Scraping Fantasy Football Projections from the Web

77

In this post, I show how to download fantasy football projections from the web using R.  In prior posts, I showed how to scrape projections from ESPN, CBS, NFL.com, and FantasyPros.  In this post, I compile the R scripts for scraping projections from these sites, in addition to the following sites: Accuscore, Fantasy Football Nerd, FantasySharks, FFtoday, Footballguys, FOX Sports, WalterFootball, and Yahoo.

Why Scrape Projections?

Scraping projections from multiple sources on the web allows us to automate importing the projections with a simple script.  Automation makes importing more efficient so we don’t have to manually download the projections whenever they’re updated.  Once we import all of the projections, there’s a lot we can do with them, like:

The R Scripts

To scrape the projections from the websites, I use the readHTMLTable function from the XML package in R.  Here’s an example of how to scrape projections from FantasyPros:

The R Scripts for scraping the different sources are located below (note that updated scripts are available via our ffanalytics R package):

Density Plot

Below is a density plot of the projections from the different sources:Calculate projections

Conclusion

Scraping projections from the web is fast, easy, and automated with R.  Once you’ve downloaded the projections, there’s so much you can do with the data to help you win your league!  Let me know in the comments if there are other sources you want included (please provide a link).

Share

— Isaac Petersen

My name is Isaac and I'm an assistant professor with a Ph.D. in Clinical Psychology. Why am I writing about fantasy football and data analysis? Because fantasy football involves the intersection of two things I love: sports and statistics. With this site, I hope to demonstrate the relevance of statistics for choosing the best team in fantasy football.

77 Comments

  1. Mike Filicicchia says:
    July 12, 2014 at 7:50 pm

    Could you include Footballguys in the projections?

    Reply
  2. Brian says:
    July 17, 2014 at 8:51 pm

    Hey, Thanks a lot for posting your scripts. I’m working on building a draft tool in R and am stuck on what to do for QB combination logic. For example, the combination of QB12 and QB14 if played correctly could potentially give you top 5 production. I’m in data analytics but I’m no stats expert, so I’d be very curious to hear how you would approach that.

    Reply
  3. Leo says:
    August 12, 2014 at 2:53 am

    Walters football add into this please he was #2 2013 fantasy pros

    Reply
  4. JP says:
    August 13, 2014 at 8:17 pm

    Where can I get the Function.R file?

    Reply
  5. leo says:
    August 14, 2014 at 12:34 am

    JP and I are getting into this R Script mostly JP he is the Albert Einstein of this generation! How accurate are your projections we both were wondering? You should get with fantasy pros and track it. Thanks for all the help!

    Reply
  6. Mike says:
    August 14, 2014 at 9:29 am

    I was trying to play with the Yahoo projections script to pull the projections from my customized league instead of messing with multipliers. However, I can’t seem to get it to work. Is there an issue because the custom league can only be seen once I login? The error is with all the http links saying that it cannot load the data from that entity. I even tried logging into my custom league in a browser first but that didn’t help. Is this even possible?

    Reply
  7. JP says:
    August 14, 2014 at 12:37 pm

    Thanks Isaac!

    Reply
  8. Z says:
    August 16, 2014 at 9:45 am

    With the risk of sounding like a moron…

    Can someone spend a few minutes (or post a link) to a place that can tell a lay person how to use the script. I have a basic understanding of coding (and I have downloaded the Github app to my computer) and have grabbed the scripts from Github, but I don’t know what to do with them now that I have them.

    I’ve been painstakingly copying and pasting from different sites and using excel to average them all together to get a basic stat projection (and then applying our league point structure – which is an odd one – and no site allows me to choose it – .25 per rush attempt, 8 yards for 1 pt receiving…)

    This sounds like it would save me SO much time and would love to learn how to do this.

    Reply
  9. Z says:
    August 16, 2014 at 11:18 am

    Wait a sec… I just looked in the data folder and noticed a .csv file from each site…

    I didn’t realize it would do all of this truly automatically!

    Does it update the stats (and if so, how often)? If not, how do I get it to run again?

    Mind. Blown.

    Sorry for cluttering up your comments with my naivety, but wow. Just wow.

    Also, what is used to figure out the InflatedCost in the AvgCost.csv?

    Reply
  10. Jon says:
    August 17, 2014 at 11:49 pm

    Isaac,

    Great stuff. Do you know if FantasyPros, ESPN, etc., have archived projections available? Or only the current year? Would be nice to work with a five-year data set….

    Thanks,

    Jon

    Reply
  11. Jon says:
    August 18, 2014 at 6:48 pm

    The only multi-year archive I’ve found is fftoday.com, which runs back to 2008. (e.g., http://www.fftoday.com/rankings/playerproj.php?Season=2008&PosID=20&LeagueID=1)

    I don’t know their methodology, but at first glace it doesn’t appear to be consensus sourcing (see: Toby Gerhart, projected 5th among RBs in 2014).

    Would be interesting to construct a joint PDF for each position using a dataset like this. The advantage of using projections over ADP would be recouping the information lost in, say, Jimmy Graham’s #1 TE ranking in 2014 (40 pts higher than #2). And the advantage of a joint distribution would be preserving the variance in the data, so that one could quickly compute the probability of any player scoring between n and (n + m) points based on projection and position.

    Jon

    Reply
  12. Scott says:
    August 24, 2014 at 8:58 pm

    Hi Isaac – thank you for all your work and information. I’m a stat and data nerd and love what you’ve done. I’m curious if there’s a way to integrate defense and kicker projections and enable them as part of the shiny app for snake draft.

    Reply
    • Isaac Petersen says:
      August 25, 2014 at 6:04 pm

      Hey Scott,

      Yes, this is on our to-do list for next season. There aren’t as many sources of projections for kickers and IDP/DST. Let me know if you find any.

      Thanks!
      -Isaac

      Reply
  13. pbwhite says:
    August 25, 2014 at 2:49 pm

    Hi Isaac, First off –Great site! Are there plans to scrape the sites for weekly projections?

    Reply
  14. John says:
    August 28, 2014 at 4:13 pm

    This is amazing stuff. I wish I would have found this before now. My draft starts in less than 4 hours…

    I am running the code — got through projections but in the Calculate League Projections.R I got this error:
    > #Duplicate last names
    > projections$lastName source(paste(getwd(),”/R Scripts/Calculations/Wisdom of the Crowd.R”, sep=””), echo=TRUE)

    I also had to comment out two of the name corrections from the footbally guys script as well — getting an error there.

    Reply
    • Isaac Petersen says:
      August 28, 2014 at 11:52 pm

      Hey John,

      Hope you didn’t try to run all that right before your draft. We have apps that do it for you!
      https://fantasyfootballanalytics.net/2014/07/fantasy-football-draft-optimizer-shiny-app-2014-update.html

      Regarding your R question. First, that doesn’t appear to be an error. It appears to be combining two different lines:
      projections$lastName —– this line shows the last names of players
      source(paste(getwd(),”/R Scripts/Calculations/Wisdom of the Crowd.R”, sep=””), echo=TRUE) —– this line runs the “Wisdom of the Crowd” script.

      It would give an error if they were on the same line, but they aren’t on the same line in my scripts on GitHub:
      https://github.com/isaactpetersen/FantasyFootballAnalyticsR/blob/master/R%20Scripts/Calculations/Calculate%20League%20Projections.R

      I’d have to see the actual errors in the football guys script to troubleshoot. FootballGuys is password-protected, so you’ll have to enter your own username and password.

      Hope that helps!

      Reply
      • John says:
        August 29, 2014 at 8:41 am

        Sorry – I didn’t paste the error — just the code that produced it. I think something about and NA through the error (think it was an index out of bounds but can’t remember). My fix below. I’ll check out your app. I did use your scripts from github.

        Here’s the code I rewrote –it’s not concise — but I think it functions the way you intend.

        #Duplicate last names
        projections$lastName <- gsub("Sr", "", gsub("Jr", "", gsub("III", "", gsub("[[:punct:]]", "",projections$player))))
        projections$lastName <- gsub(" $","",projections$lastName,perl=T)
        projections$lastName <- sapply(strsplit(projections$lastName, "\s+"),tail,n=1)

        Reply
  15. Evan Herman says:
    September 1, 2014 at 9:04 pm

    They’re seems to be stronger projection sites out there like: 4for4, Bloomberg Sports, PickingPros, and Profootballfocus, Yahoo partnered with Profootballfocus this year so Yahoo’s projections will be more accurate, Numberfire seems to be gaining ground, Yahoo used to get theirs from Accuscore, but they are not too good, ESPN is average as well, I just looked at FantasyFootballNerd, and they got some horrible projections for week1. They got Brady at 31 completions for 402 yards, Rivers @ 28 completions, Locker @ 49 attempts for 27 completions, Charles 23 rush attempts and Ellington @ 22, these are all too high

    Reply
    • Isaac Petersen says:
      September 1, 2014 at 9:30 pm

      Hey Evan,

      These scripts are for scraping the season projections (not weekly projections). We have goals to include weekly projections in the future, though. Most of those sites you listed (except PickingPros) appear to subscription-only, so we wouldn’t be able to include them (via scraping). I will look into including PickingPros next season.

      Thanks!
      -Isaac

      Reply
  16. Axel Fant-Eldh says:
    September 12, 2014 at 8:59 am

    The Yahoo script is malfunctioning for me. The columns have the wrong designations, so I’m worried that the data may not be handled properly overall (passing yards are registered as return TDs, rushing yards as reception yards, rushing attempts as passing yards, reception yards as interceptions etc.).

    Maybe it’s just for me, but you might want to look into that.

    Reply
  17. Brian Adams says:
    November 12, 2014 at 11:04 pm

    Isaac,

    Is it possible to get the code for the density plot?

    Thanks,

    Reply
  18. Brittany Pugh says:
    November 13, 2014 at 5:11 pm

    Hi Issac. I used your web scrapping code for a personal project I was working on in R and your code worked perfectly. Very helpful. I just want to say thank you for posting your script for web scraping.

    Best,

    Brittany Pugh

    Reply
  19. Isaac Petersen says:
    November 27, 2014 at 9:57 pm

    Happy to send you info. Feel free to shoot me an email.

    Reply
  20. george says:
    August 20, 2015 at 7:45 pm

    Hey Isaac – thanks for the great website. question on the yahoo scrape. Do i need to adjust my proxy settings somehow in Rstudio? noticed that the yahoo URL needs to be logged in to yahoo to direct to the pages it is scraping.

    Thanks again.

    George

    Reply
  21. JAP says:
    August 23, 2015 at 11:29 pm

    Hi Isaac. I am trying to scrape my Yahoo private league projections and I am running into the problem of having to login to Yahoo. How do I solve this in R (I am using this to learn R also) Thanks

    Reply
  22. ryan says:
    August 30, 2015 at 5:13 pm

    I’m a newb with R, like to the extent of I know what it can be used for, but I’ve always installed packages and just ran other peoples scripts that I’ve acquired and manipulated to make it work for me.

    I’m trying to scrape projections / stats from Yahoo using the script linked.

    Are their any instructions for dummies like me to make this work? I’m getting all kinds of error messages.

    Thanks!

    Reply
    • Isaac Petersen says:
      August 30, 2015 at 6:33 pm

      Hi Ryan,

      We haven’t had a chance to update the scripts on the repo because we’ve been busy working on the tools. That’s on our to-do list. Feel free to update them and create a pull request to GitHub so you can share them with the community!

      Thanks,
      Isaac

      Reply
  23. Vik K says:
    September 6, 2015 at 11:04 am

    Isaac,

    Is there a problem with the download projections App? Everytime I have tried to download projections today it has tried to save the file as an HTML or text file. I even re-booted my computer and same problem.

    I have downloaded several times in the past and no problems. Please help – my draft is tomorrow and got to prepare.

    Thanks and did you guys get the weekly projections ready for this season? I know you were planning to.

    Thanks again and the website is amazing !!

    Vik

    Reply
  24. Kyle says:
    September 16, 2015 at 11:30 am

    Love your site! Very well done!

    Can you pull weekly DFS price data from Draft Kings, Fan Duel, and Yahoo? Having the cost for each player, along with all of these other stats would be amazing to optimize line-ups. Users would be able to identify players who have the highest ratio of projected points to price, develop high floor teams (low risk), identify players with high ceilings, and countless other things.

    Once again, awesome work!

    Reply
  25. Adam says:
    October 27, 2015 at 12:06 pm

    Just yesterday, I came upon your scrapers for the fantasy football projections. I was hoping to utilize them for DFS on DraftKings. I updated the LeagueSettings.R file to select the 2015 season, and week 7 last night (as a test to compare the week 7 projections to actuals), but when I ran the ESPN scraper, it still appeared that I was getting full season worth of projections, rather than the weekly.

    Have you encountered this before? Is there a step [or ten 🙂 ] that I missed?

    Thank you for taking the time to share all of this with the community. It is really helping me build my R skills.

    Reply
    • Isaac Petersen says:
      October 27, 2015 at 6:09 pm

      We haven’t updated the scripts on the GitHub repo for the current season (it’s on our to-do list). You might have to update the URL in the script to get ESPN’s weekly projections.

      Reply
  26. Andrew says:
    October 27, 2015 at 4:28 pm

    Hey Isaac,

    Your site is amazing! I’ll probably spend whatever free time I have for the next couple of days reading further into the material on this site.
    Meanwhile, I was wondering if you could answer a few questions that I had.

    First of all, it seems like the approach you take for projecting players’ performances is almost entirely based around other experts’ projections. I was wondering if you’ve attempted coming up with your own projections purely based on players’ and teams’ statistics: for instance, by training a model based on players’ stats vs. opponent’s strength in defense, etc. For instance, maybe for a WR’s projection you would consider the WR’s QB’s stats and his O-line’s stats. Maybe even different factors like days of rest, home/away game, weather can come into play?

    If you had your own projections based on raw data, maybe you could even corporate that with what you already have and weigh it accordingly, based on how confident you are of your own algorithm :p

    Secondly, if you have already tried this, or plan on trying this, do you know where I can find these additional data (such as schedules, difficulty of opponent, etc)? The only data that I’m seeing from your site are stats revolving around others’ projections. I was hoping to try to come up an algorithm for my own projections and was wondering if you could help me find some more extensive data.

    Again, thanks for your awesome site and your contributions!

    Reply
    • Isaac Petersen says:
      October 27, 2015 at 6:16 pm

      Hi Andrew,

      I love the idea to generate our own projections in a transparent way based on how various variables (e.g., age, strength of surrounding offense, prior performance) predict future performance, especially because other websites’ projections are basically a black box. Creating accurate projections that consider many factors is major undertaking, though (e.g., figuring out what variables to consider, collecting big data, organizing big data, finding a model that predicts well historically, and continual cross-validation), so it’s one of our future directions to consider. In the meantime, we’ve shown that aggregated projections are more accurate than any individual source of projections, so it might provide a useful starting point. Let us know if you find good sources of these data!

      -Isaac

      Reply
  27. Jared says:
    November 2, 2015 at 4:30 pm

    Can you show us how to scrape the salary for each player from FanDuel.com???

    Reply
  28. Dean says:
    April 18, 2016 at 2:41 pm

    When will your new 2016 projections be out?

    Reply
    • Isaac Petersen says:
      April 18, 2016 at 8:55 pm

      Hey Dean,

      I think most sites release projections after the NFL draft, so hopefully soon after that.

      -Isaac

      Reply
  29. Garrett says:
    May 20, 2016 at 4:29 pm

    Hey Isaac,

    Any chance you have historical data from your fantasypros scrapes? I’m looking to do study on the variability of consensus accuracy by examining several offseasons worth of predictions.

    Best,
    Garrett

    Reply
  30. Corban says:
    August 3, 2016 at 10:50 pm

    FFChamps.com. Let me know if you want my login

    Reply
  31. Ryan says:
    August 23, 2016 at 9:15 am

    Just found your site today, and I’ll be going through it over the next few days. Do your scrapes separate the individual Experts or Analysts out for each site or does it only grab the average by all the analysts?

    Reply
  32. Nick says:
    September 1, 2016 at 10:24 pm

    Hi, I know this is an old post but I was wondering if you could help

    I keep getting the following error when trying to scrape the yahoo projections:
    Error: failed to load external entity “http://football.fantasysports.yahoo.com/f1/34124/players?status=ALL&cut_type=9&myteam=0&sort=PTS&sdir=1&count=0&pos=QB&stat1=S_PS_2016”

    when i run(and everything up to it in that script(I edited the league ID that is default to my ID)):
    yahoo <- lapply(yahoo_urls, function(x) {data.table(readHTMLTable(x, stringsAsFactors = FALSE)[2]$'NULL')})

    everything seems to be working up to that point but seem to always have a problem there.

    Reply
  33. Anthony says:
    September 7, 2017 at 12:53 pm

    Do you have any python web scraping sources that would be a good resource?

    Reply
  34. Adam says:
    October 2, 2019 at 9:15 pm

    Where do you scrape the Salary column from in your weekly projections? It looks as if you use FanDuel projections. I just wanted confirmation on whether or not that was true.
    Thanks

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *