How to Download and Run R Scripts from this Site
55This post outlines how to download and run R scripts from this website. We have many Fantasy Football scripts that show how to download and calculate fantasy projections, determine the riskiness of a player, identify sleepers, and much more!
What Software You Need
- Git and GitHub (for downloading the scripts)
- R (for running the scripts)
- RStudio (best text editor for viewing, editing, and running R scripts)
Downloading the R Scripts
First, install and setup Git and GitHub. Git is version control software, and GitHub is a service for storing Git projects called repositories. We store data and R scripts on our GitHub repository (repo) located here:
https://github.com/FantasyFootballAnalytics/FantasyFootballAnalyticsR
After setting up Git and GitHub, you can download our data and scripts by forking our GitHub repo. Go to our repo (above) and, in the top-right corner of the page, click “Fork”. Once you have forked the repo, you can download the repo files to your local computer by running the following line of code in the command line (Windows) or Terminal (Mac):
git clone https://github.com/USERNAME/FantasyFootballAnalyticsR
(editing “USERNAME” to be your GitHub username). For more info on how to fork a repository (so you can download our data and scripts), including how to keep your local copy in sync with changes to the repo, see here.
Running the R Scripts
First, install R and RStudio. After installing R and RStudio, open RStudio. In RStudio, open an R script (.R extension). You can run individual lines of code by highlighting the lines you want to run and clicking “Run” in the top-right corner. You can run the whole R script by clicking “Source” in the top-right corner. Many of our R scripts depend on first running two scripts called Functions.R and League Settings.R. You can run them using the source() function:
source(paste(getwd(),"/R Scripts/Functions/Functions.R", sep="")) source(paste(getwd(),"/R Scripts/Functions/League Settings.R", sep=""))
Before running the scripts, you will have to download any packages used by the scripts you are trying to run. We load the packages needed at the beginning of each script with the library() function. For instance, in the example below, we need the reshape, MASS, and psych packages:
#Library library("reshape") library("MASS") library("psych")
Before you can load these packages the first time, however, you will have to install the packages. Here’s how to install a package:
install.packages("reshape")
For more info on how to install R packages, see here. You may also have to change the working directory to be the root directory of the repository so the script knows where to look for files. Here’s how to set the working directory:
setwd("C:/ENTER/YOUR/DIRECTORY/HERE")
For more info on how to set the working directory, see here. Finally, the script may depend on other scripts. For the order the scripts are run (which can be helpful for understanding which scripts it depends on), see here. Note that some of the scripts may be out of date and may need to be adapted. Most importantly, the scripts show proof-of-concept of the topics discussed on the site.
Sharing Your R Scripts with the Community
If you create new scripts or make changes to the R scripts, you are encouraged to share them with the Fantasy Football Analytics community. This is why we make the scripts available to you for free—so the community can collectively use and improve them. This benefits everyone. Please consider sharing your new or improved files by creating a pull request back to our repo. For info on how to create a pull request, see here.
Troubleshooting
You may run into errors when running the R scripts. Here are some common issues and how to troubleshoot.
Error in library(“XML”) : there is no package called ‘XML’
- This suggests that you are trying to load the XML package without first installing it. Here’s how to install it:
- install.packages(“XML”)
- For more info on how to install packages, see here.
Error: object ‘projections’ not found
- This suggests that you did not load the data file at the top of the script—the line of code should look something like: load(filename.RData)
- Make sure you downloaded the data file to the correct location, you set your working directory, and run this line of code to the load the data.
Learning R and GitHub
For info on how to learn R, see here. For a beginner’s guide to GitHub, see here.
Really cool stuff here. Great job! While I haven’t gone through all of your posts, it looks like you are only comparing projections of each site on an annual basis. Do you have weekly projections scrapped in the repository anywhere?
Hey Justin,
Good question. We’re in the process of creating tools to download weekly projections for the upcoming season. Stay tuned!
-Isaac
That’s interesting. I’m not sure if you share the code for the scraping tools but if you do I would be interested in helping out with any spare time I have
You can download the scraping scripts from the GitHub repo. They’ll need to be updated for the current season (we’re working on it), but feel free to adapt them on your own and to share them with the community by creating a pull request to the repo—the fixes should be relatively minor.
I cannot get Calculate League Projections.R to run. I am getting a file not found error on line #22. It seems to coming from the sourcesofProjections array. How do I fix this?
Hi Norm,
You can specify the sourcesOfProjections in the LeagueSettings.R file (https://github.com/isaactpetersen/FantasyFootballAnalyticsR/blob/master/R%20Scripts/Functions/League%20Settings.R) that gets loaded at the top of the script. Note that we’re changing our code for scraping (for the purposes of the apps), so some of the scripts may be a bit out of date, but it should be manageable to get them working.
Hope that helps,
Isaac
Is there a list of known sources that work currently? I am getting a lot of ‘undefined column selected’ errors trying to get the scripts up and running. I’d like to trim my sources down to the ones that work and go from there.
Hi Austin,
We’ve been working to finish our draft tools, so we haven’t been able to keep the repo up-to-date unfortunately. I haven’t had a chance to check which ones work, but most are from last season and may require some minor edits. You have several options: 1) test which scripts work on the repo and use those, 2) use our draft tools instead of the scripts on the repo, 3) wait for us to update the repo after we finish the draft tools, or 4) make the minor edits necessary to update the scripts for this season. I’d welcome people to help us and the community by updating the scripts for this season. Feel free to update them and create a pull request to GitHub so you can share them with the community (see article above for info how to do that)!
Thanks,
Isaac
Hi Isaac, just tremendous work. I just started trying this out and I’m floored by all your skill and effort.
I still have to learn how to use Git Hub (ran into issues with the “git clone…” which I haven’t figured out yet) so I just copied all the lines into R.
I’ve been playing FF for 7-8 years now but it’s the first time I’ll try to use more in-depth statistical analyses.
My main purpose with this reply is purely to say thanks for showing me where to begin.
I really like your work first off. I have some R experience, but nothing on an expert level. I downloaded the git hub files, but I found the updates are lacking and it is very broken. I have fixed the projections and tailored it to what I want to see and standardized the data tables. My problems now are the calculations.
I like the app, but i am a person who likes to build his own projections too.
So far my issue at is the data is not consistant for the calculate league projections and I am unable to effectively change columns from character to numeric for example(current issue).
projectionsAvg <- projections[, lapply(.SD, mean, na.rm=TRUE), by=c("name","player","pos","team"), .SDcols=availableVars]
This gives me a Gmeans error.
Hi Randal,
Yes, the scripts are currently out-of-date because we’ve been working on getting the tools ready for draft season. We have plans to update the scripts on the repo after we finish the tools. I’d have to know how your data are structured and see the exact error to help. The statement you showed is trying to calculate means for every variable in the availableVars vector across all rows (i.e., analysts) for the same player (i.e., name/player/pos/team). I’d first make sure that all variables (name/player/pos/team/availableVars) are in the data table. I’d also make sure that all of the availableVars are numeric—otherwise you’ll want to convert them to numeric first.
Hope that helps,
Isaac
Hi Isaac,
I had a quick question regarding downloading the repo files… after using the clone command I get an error regarding access rights. I have set up GitHub, verified email etc, and forked the files. Just unsure how to properly save it to my system. Thanks for your help!
“Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.”
Hey Nate,
I don’t have much experience cloning repos, but have you tried forking the repo first as described above? If not, you might try to google how to clone a public repo. Let me know if you still have trouble and/or if you find the solution.
Thanks!
-Isaac
Nate, Isaac,
I got the same error. Cloning the direct link to the forked repo worked for me (i.e. git clone https://github.com/USERNAME/FantasyFootballAnalyticsR instead of git clone git@github.com:USERNAME/FantasyFootballAnalyticsR). Hope that helps!
Thanks Josh, I updated the post with your syntax.
-Isaac
I’m getting a fatal error: repository not found. Thoughts?
Hi Griff,
Not sure. What did you try? Did you see the comments above?
-Isaac
Hey Isaac, I love the site. I’m attempting to download weekly football stats by position from yahoo (using your scripts), but I’m unable to parse the new tables for yahoo’s stats web page (qb week 1 example below).
https://sports.yahoo.com/nfl/stats/byposition?pos=QB&conference=NFL&year=season_2015&timeframe=Week1&qualified=0&sort=49&old_category=QB
can you provide a quick 3 line response with how to scrape the table on that page?
Hi Alan,
I don’t have our weekly stats scripts available yet, but we do hope to post them to the GitHub repo soon. What error/problem are you getting when doing readHTMLTable() on that site?
https://fantasyfootballanalytics.net/2014/06/scraping-fantasy-football-projections.html
-Isaac
If i use readhtmltable:
Warning message:
XML content does not seem to be XML: ‘https://sports.yahoo.com/nfl/stats/byposition?pos=QB&conference=NFL&year=season_2015&timeframe=Week1&qualified=0&sort=49&old_category=QB’
I tried using RCurl and rvest (specifying xpath) but cannot get the data.
Hello,
I was just wondering if you are still planning on updating the scripts you have in the GitHub repos? I tried to run some of the scripts under the projections folder and they seem to be broken.
Accuscore:
source(‘G:/FantasyFootballAnalyticsR/R Scripts/Projections/Accuscore Projections.R’)
Loading required package: bitops
Error: failed to load external entity “http://accuscore.com/fantasy-sports/nfl-fantasy-sports/”
CBS:
Error in `[.data.table`(cbsList[[i]], , `:=`(sourceName, rep(cbs_source, :
Cannot use := to add columns to a null data.table (no columns), currently. You can use := to add (empty) columns to a 0-row data.table (1 or more empty columns), though.
ESPN:
source(‘G:/FantasyFootballAnalyticsR/R Scripts/Projections/ESPN Projections.R’)
Error in dev.off() : cannot shut down device 1 (the null device)
FantasyPros:
source(‘G:/FantasyFootballAnalyticsR/R Scripts/Projections/FantasyPros Projections.R’)
Show Traceback
Rerun with Debug
Error in `$<-.data.frame`(`*tmp*`, "positionRank_fp", value = NA) :
replacement has 1 row, data has 0
Greatly appreciated.
-Wayne
Yes, we’re working on it:
https://fantasyfootballanalytics.net/2014/10/download-run-r-scripts.html#comment-1050
Hi Isaac,
This is my third comment here (counting my question on the GH issues page) so it’s about time I tell you how awesome I think this work is. Very few out there have not only done this type of analysis but made it and the data available for free. Kudos to you.
By the way, I took a look at your CV and just wanted to mention that I’m nearing the end of my PhD in Psychology (perception-action). We probably know some people in common but I’m not going to start namedropping here.
Anyways…. I’m posting here to ask about automating the download of your raw statistics CSV. I’d like to set up a cron job to download that every week. I could dive into programming a headless browser (since it’s a js page) but maybe you have a better suggestion. Does that CSV have a raw URL I can pull from?
Thanks,
Henry
Good idea, not aware of a raw URL or a way to set up a cron job. Let us know if you figure it out! Best of luck in your PhD program! What school?
Hi – I just want to let you know how much I like your site. You have created some great stuff here and I’m diving into it! I always wanted to learn about analytics and the various tools that are used. What a fun way to learn it – analytics and football – 2 things I really like.
Anyway, I started downloading everything and will get it all going in the near future. Hopefully, I can contribute as well. I have a system admin background with some coding in there.
Thanks Again!
Hey Isaac,
Nice site! Before I go about learning R, what statistics do you give access to? I looked around your site but couldn’t find anything. Do you have any limits on how much data one can grab from your site? I’m basically looking to pull in the data for all relevant players one time then update the data per week to generate my own projections. I would appreciate a response / point me in the right direction where you answer this question on your site.
Thanks,
Dave
You can download the data using the “Download” button in the top right of the Projections tool:
http://apps.fantasyfootballanalytics.net/projections
Sorry, Isaac, I think I wasn’t clear. That is great that you include the projections, which is clearly a focus on your site, but I am looking for the raw data. For example, when I search week 1, I get a list of projections for every player, but nothing about their actual performance for that week. I am looking to get their actual stats for the week as well (rushing yards, TDs, etc.). Do you have any plans to include this data?
We have historical performance statistics for seasonal data (for examining historical accuracy of projections). Historical weekly data are on our to-do list. Your best bet for downloading historical performance data would either be Yahoo or ProFootballReference.
Hi Isaac,
Great site! I’m just a beginner but I believe the confusion about downloading your github repository from Griff is the path. The correct command is
get clone https://github.com/isaactpetersen/FantasyFootballAnalyticsR
I’m not sure, but I think once you’ve forked it, it’s part of your GitHub account, so you should be able to git clone it to your HD using (replacing USERNAME with your username):
git clone https://github.com/USERNAME/FantasyFootballAnalyticsR
It’d be helpful if someone can confirm what works. Thanks.
Hi Isaac,
The git clone command didn’t work as described above and when clicking the hotlink itself in FireFox I receive a 404 not found message.
Is there a known work around for this? Thank you and thank you for this site.
You have to change “USERNAME” in the filepath to your GitHub username. Hope that helps!
Looks like I’m good. Maybe coffee hadn’t kicked in this morning but when I did as instructed, got the message. This time it worked.
Thanks!
Is there a way to verify whether the installation and package download was done correctly or better detailed documentation? I have everything dowloaded and installed (i think).
Only way I can think of is to try it out! Documentation is here:
https://fantasyfootballanalytics.net/2016/06/ffanalytics-r-package-fantasy-football-data-analysis.html
Thanks, I’ll take a look at it and see if it helps!!
Or maybe a how to on using the software seems simple enough but I’m trying to figure out what script t run first and if I need to update the season in the script itself Thanks!
Hey Isaac,
I’ve been researching weekly fantasy football leagues like DraftKings and FanDuels. I’ve been hoping to use the R script “optimize team” to generate the best team on a week to week basis rather than for a full season. Part of what I have been trying to do is to constrain the team by salary rather than by risk. Also I would be using my own data set rather than what comes in the package. My attempts to adjust the code for my purposes have been unsuccessful thus far. Any suggestions or help in overcoming this problem would be much appreciated.
Regards,
Austin Springsteen
Our optimizer uses salary as a constraint, as well. Feel free to check out our optimization R code attached to the following article:
https://fantasyfootballanalytics.net/2013/08/optimization-simulation.html
Hey Isaac,
I ran into this issue when trying to run the calculate projections addin. Any idea why?
> ffanalytics:::Run_Projection()
Listening on http://127.0.0.1:7513
Note: the specification for S3 class “AsIs” in package ‘jsonlite’ seems equivalent to one from package ‘RJSONIO’: not turning on duplicate class definitions for this class.
> myProjections <- getProjections(scrapeData=runScrape(week = 0, season = 2016, analysts = c(-1, 3, 4, 5, 6, 7, 9, 17, 18, 19, 20, 28), positions = c("QB")), avgMethod = "average", leagueScoring = userScoring, vorBaseline, vorType, teams = 10, format = "ppr", mflMocks = -1, mflLeagues = -1, adpSources = c("CBS", "ESPN", "FFC", "MFL", "NFL", "Yahoo"))
Retrieving player data
Error in structure(.External(.C_dotTclObjv, objv), class = "tclObj") :
[tcl] invalid command name "toplevel".
https://fantasyfootballanalytics.net/2016/06/ffanalytics-r-package-fantasy-football-data-analysis.html#comment-17634
Good afternoon,
I tried to use your FF Apps online for the Auction Lineup Optimization and Projection Rankings, however it was not working properly and it said to refresh and try again. After I did that, the error said it could not be forked correctly and the resource was not available. Also a server time out error occurred while refreshing. This is while using most recent versions of Firefox, Chrome, and mobile browsers.
Additionally, I tried to use the FFanalytics R package and it said it was not a valid download and unable to be used in 3.2.x versions of R. I have used github, downloaded the tar.gz, and tried all options available. I have R Studio as well, but it does not work.
Please advise the best way to continue to use this great service.
Thank you for your effort in creating all of this.
NC.
Hey Isaac!
For those of use drafters that are doing it today September 7th. Is there a way to still get access to the Snake Draft Optimizer to kill it in our drafts today? Its looks likes the app
http://apps.fantasyfootballanalytics.net/projections
Has already switched over to the weekly lineup optimizer
Click “Change Data Settings” and change “Week” to “Full Season”
Issac, can you get in touch with me about a partnership offering. You can reach me @ jade1atcableone.net and the name is Brian.
TY
Hey Isaac! I am trying to run the script and I am having some trouble with the rglpk package. When I try to download it I get the following error:
ERROR: loading failed
* removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rglpk’
Warning in install.packages :
installation of package ‘Rglpk’ had non-zero exit status
Have you seen this before? Any ideas how to fix it?
Thanks!!
You might try installing Rglpk from 1) a local version from source (https://cran.r-project.org/web/packages/Rglpk/index.html), 2) a local zip version of the binaries (https://cran.r-project.org/web/packages/Rglpk/index.html), or 3) try this: http://mailman13.u.washington.edu/pipermail/statnet_help/2015/001984.html
Dear Isaac and the rest of the FFA team. First of all, thanks for a great resource. As a researcher and teacher in statistics this is a great resource for both my students and myself. And as a huge NFL fan living in Europe I do use it all the time.
I have cloned your repository and trying to update projections for the upcoming season. However, I very soon run into a problem with the readHTMLTable function in the Rankings scripts. It seems it is unable to read the content of the URL provided in the script (e.g. http://www.fantasypros.com/nfl/rankings/idp-cheatsheets.php). I have tried to run with through an RCurl function (getURL) without any luck. And as I am not able to see how links have changed at the FantasyPros page, I do not know if its the link that has changed, or if it is something with the function that has changed. Any ideas?
I would be happy to send you a pull request as soon as I have found the solution and updated the scripts.
Best
Lars
Hi Guys new member. Great stuff. What about projections of ownership? Is that data being scrapped from anywhere?
We do not currently have access to this data
Is this R package still being supported? I’ve been trying to get it up and running this, year, but have been hampered by a lot of outdated/broken URLs.
Same question as Jacquizz! Broken URLs–seems like the websites may have changed HTML formatting or something so it hasn’t been an easy fix either. Any support would be very appreciated! thanks!
Mr. Petersen,
My name is Jarby Brown. I am an aspiring sports data analyst professional, and I play fantasy football in a PPR league. I am planning on using your optimization tool myself using R to not only crush my inferiors in fantasy football, but more importantly to expand my knowledge and skill set of R and analytics in general. For a PPR league, I assume the only difference in the process is to scrape the PPR projections off of the different projection sites used in the optimization rather than the regular projections. Is there a PPR optimization already done on your website? If you have any tips on other things when it comes to running your analysis using PPR projections I would be very appreciative to hear them.
Thanks ahead of time,
Jarby Brown
Yes, we account for PPR in our webapps (https://fantasyfootballanalytics.net/2013/09/win-your-fantasy-football-snake-draft.html) and in the ffanalytics (https://fantasyfootballanalytics.net/2016/06/ffanalytics-r-package-fantasy-football-data-analysis.html) package. Just change your league settings in the webapps or package for PPR scoring.