Examining the Accuracy of Fantasy Football Projections with an Interactive Scatterplot in R
58In prior posts, we presented the accuracy of different analysts in projecting football players’ performance, finding that the average was more accurate than any individual analyst. In this post, we present a Shiny app to examine the accuracy of historical fantasy football projections. The app allows you to examine the accuracy of historical projections using different analysts, positions, seasons, league scoring settings, and types of averaging. The app also includes an interactive scatterplot. In order to examine the accuracy of historical projections, you’ll need an FFA Insider subscription.
The App
The app is located here:
http://apps.fantasyfootballanalytics.net/
How to Examine Historical Accuracy of Fantasy Football Projections
- Click “Settings”, then “General Settings”.
- Select a previous season or week (so we know how projections compared to actual performance).
- Change the league settings to tailor the projected/actual points to your league settings.
- Choose the type calculation type: average (mean), weighted average, or robust average. For more info on these calculation types, see here.
- Choose the analysts to include and, if you selected a weighted average, how much to weight each analyst in the average projections.
- Click “Save Settings”.
- Click the “Accuracy Tab”.
Note: there are other settings you can modify, as well. For a description of these settings, see here.
Interactive Scatterplot
The page displays two scatterplots. The top scatterplot of projected versus actual points is from ggplot2 and displays a LOESS smoother and confidence interval, along with an estimate of the R-squared value for the linear (not LOESS) fit.
The bottom scatterplot of projected versus actual points is an interactive scatterplot. You can select which positions to display in the legend. Hovering over the dots, you will see how many points each player was projected to score and actually scored. For instance, in 2014, we can see that Robert Griffin greatly under-performed expectations, whereas Demarco Murray exceeded expectations and Tom Brady fell close to expectations.
Accuracy Table
The table examines the accuracy of historical projections by position with several accuracy metrics:
- mean error (ME): closer to zero is better (positive values mean the projections are under-estimates, negative values mean the projections are over-estimates)
- root-mean squared error (RMSE): lower is better
- mean absolute error (MAE): lower is better
- mean percentage error (MPE): closer to zero is better (positive values mean the projections are under-estimates, negative values mean the projections are over-estimates)
- mean absolute percentage error (MAPE): lower is better
- mean absolute scaled error (MASE): lower is better
- R-squared (RSQ): higher is better
R-squared is measure of relative fit, whereas the others are measures of absolute fit. Note: the high percentage estimates of error (MPE and MAPE) reflect that a number of players scored very few points, which skews percentage estimates of error.
Interesting Observations
- The average of analysts was more accurate than the individual analysts, consistent with the principle of the wisdom of the crowd. For more info, see here.
- The weighted average was slightly more accurate than the mean or robust average. For more info, see here. Note, however, that the default weights were calculated based on historical accuracy, so it remains to be seen whether these weights will apply to future projections. If the best analysts are consistently more accurate than other analysts, the weighted average will likely continue to outperform the mean. If, on the other hand, analysts don’t reliably outperform each other, the mean might be more accurate.
- The weighted average explained about 60% of the variation in players’ actual performance. That means that the projections are somewhat accurate but have much room for improvement. Nevertheless, the projections are likely more accurate than pre-season rankings.
- Projections were more accurate for some positions than others. Projections were most accurate for QBs and WRs. Projections were least accurate for Team Defenses (DST) and individual defensive players (IDP). For more info, see here.
- Projections over-estimated players’ performance by about 5–6 points on average across most positions (based on mean error). It will be interesting to see if this pattern holds in future seasons. If it does, we could account for this over-expectation in players’ projections. In a future post, I hope to explore the types of players for whom this over-expectation occurs.
But don’t take my word for it. Test it out yourself and see what you find. And let me know if you find something interesting!
Uh oh– in 2011 the graph is a bit skewed because of a mistaken stat that Mike Williams had 698 projected points for the season. That chart also shows Brandon Lloyd with an estimated 437 projected points for the season.
Hi Andy,
Good catch! Just fixed it.
Thanks,
Isaac
This article also mentions wisdom of the crowds, but uses Average Draft Position instead.
http://lifehacker.com/use-the-wisdom-of-crowds-to-draft-the-best-fantasy-foot-1617837803
It’s interesting because ADP more purely shows the wisdom of the “crowds”, as long as you’re using data from real drafts and not mocks. ADP is also similar to projections in that it can show the relative differences between players better than pure rankings (i.e. Player X has an ADP of 10.4, Player Y: 10.9, Player Z: 12.3). I’d be interested in seeing how ADP compares to the average of “expert” analysts in terms of accuracy. Your thoughts?
Hi Shrinidhi,
We have plans to examine the accuracy of ADP/rankings vs projections. ADP and rankings assume general league settings, whereas projections can be made for custom leagues. Also projections show *how much* better players are than each other, unlike ADP/rankings. For these reasons, I’d argue that projections are much more useful than ADP/rankings:
https://fantasyfootballanalytics.net/2014/08/use-projections-not-rankings.html
-Isaac
Well, ADP values typically aren’t consecutive integers, so they can show how much players are better than each other. Lets say the first three players have ADPs of 1.95, 2.05, and 2.93. This would indicate that the first two players are much closer in value than the third.
Here’s ESPN’s ADP for reference: http://games.espn.go.com/ffl/livedraftresults
As long as you standardized ADP and projected fantasy points, they would both show the same thing, just with different granularity (which is why, as you said, projections are better for custom leagues).
Even if ADP is continuous, it doesn’t tell you how much better a player is. A player with an ADP of 2.2 might be 5 points better than a player with an ADP of 5.2. But a player with an ADP of 7.4 might be 20 points better than a player with an ADP of 7.5. In other words, differences in ADP don’t correspond linearly to differences in points. For that and other reasons, points are more meaningful than ADP, even when ADP is continuous.
ADP is pre-draft and projections are season long AND weekly.
ADP also reflects a wide variety of draft dates (which skews heavily towards off-season baseline rankings) and a wide variety of league types (e.g., std, 2 QB leagues, PPR, dynasty, keeper, 3 WR, flex, no flex, etc.
I would not trust ADP for the type of analysis that Isaac is doing.
Isaac,
I am a big fan of the site. This app is very useful for me, thank you. Can you provide some clarity on how you set VOR baselines, and how one might set these for a custom league?
Thanks,
Paul Larios
Hi Paul,
There are a number of ways to set VOR baselines. See below for a discussion of some of these ways:
https://fantasyfootballanalytics.net/2013/04/win-your-snake-draft-calculating-value.html
Hope that helps!
-Isaac
Thanks Isaac.
I see there is a section in this VOR script that allows one to enter players being picked during a draft. I understand this script will automatically alter VOR values for players based on the remaining player pool. I am wondering- will this script modify VOR values to account for positions that I need to fill my starting lineup? For example, if I draft WR in round 1, will the script show that I should draft a RB over a WR (of equal projected points) to optimize my draft? Please let me know, this type of iteration would be an awesome tool during drafts.
Hi Paul,
The script currently doesn’t alter VOR values, just alters who’s available to be displayed. We have plans to calculate a “Dynamic VOR” next season based on who you’ve drafted and roster needs. As of now, you’ll need to take into account your positional needs when drafting.
-Isaac
Isaac,
First of all…..WOW. This website is amazing and seems to be getting better the more I read. I am loving the numbers and the ability to customize the projection list based on my specific league format. However, most of the “coding” things fly in one ear and out the other. So a couple of things:
#1- I keep reading in comments about being able to designate players as “drafted”. I have been throughout the app and can not find this specific tool. I am sure it is just my lack of knowledge or is that something specifically in the coding portion? (I know layman here haha)
#2- I run a “dynasty” or “keeper” league format that seems to be super rare but every time I explain everyone loves the formatting. Is there any plans to integrate “keeper” projections in the future? My layman thinking process would think it would play off of the “sleeper” portion of the app. Or could be as simple as scraping “dynasty” or “Keeper” projections? Just throwing ideas out there.
Again awesome website and I am stoked to increase my knowledge in the coding aspect and have loved breaking out my stats book from a couple years ago to make sure I was thinking down the right path.
Thanks.
-Matt
Annnnnnd Isaac feel free to disregard my first question. Wow. The “exclude players” type box would be the obvious answer to my question. Yikes. Bed time.
Hopefully I did not waste your time there. Thanks! So excited about this website.
Hi Matt,
We’ve received some requests for dynasty/keeper leagues. Because we prefer projections over rankings (https://fantasyfootballanalytics.net/2014/08/use-projections-not-rankings.html), we calculate our rankings based on projections. I’ve seen sites that provide keeper rankings, but I haven’t seen keeper projections. Are you familiar with any sources of keeper projections? We’d need keeper projections to be able to do this, and unfortunately it’s not as simple as inflating current projections over time based on the player’s age. Even among rookies, some are expected to start, whereas others are projected to increase in playing time as they gain experience. I’m interested in the idea, though, if you have ideas.
Thanks,
Isaac
Hey guys,
It’s great to finally find a fantasy football site that shares a similar viewpoint on statistical analysis of projections in order to utilize a value based drafting strategy. Over the last 5 years I have been compiling a custom spreadsheet which uses projections from any source and grants the user the ability to add in league settings, roster positions, and even a “personal preference factor” to create an overall value based draft list.
This sheet also factors in consistency ratings over the past 3 years, loss over replacement, and the ability, if chosen, to adjust anomalous projections (ex. Injured players / excessive increase in projection from a previous year, and rookie adjustment) for any given player.
When I started this sheet it was only meant to be a draft aid, however even in its infant stages it yielded better results than the site from which the projections were imported, and over the last three years it has outperformed fantasy pros as well as most other sites (I have not compared to this site.) This VBD projection based draft strategy sheet is not overwhelmingly people friendly, it is all calculations, no macros, and for max effectiveness requires hand entry from the user. If anyone is interested in checking this out, including this site, I would love to send a copy of it. In exchange all I would ask is for additional input on future amendments, and opinions on its overall functionality,
Thanks- Harrison
Disclaimer – The above author has no affiliation with any sports, or fantasy sports, companies or websites. The use of this information is done so completely at the risk of user, no guarantees are provided for the above claims and information.
I’ve loved all of the tools I’ve found on this website, as they speak the same language as me: objective analysis by using proven techniques. One question I had about your preference to projections vs. rankings:
I’ve heard a number of pundits claim that even though projections for a player are accurate, they do not accurately reflect the desirability of a player for a fantasy team which scores week to week, head to head. Many players are on poor offenses for example and may score “garbage time points” after the team is dreadfully behind. The proposed problem with this is of course the unpredictability of such weeks when players “hit.” On the other hand, if you were able to accurately predict high scoring weeks, there would be a strong case for strategies such as “quarterback by comittee” where rather than spending an early draft pick on a highly projected player, you choose players whose schedule complements each other and start the one projected to do the best, ending with [QB] #1 values over the season.
Have you conducted (or considered) any research into whether the accuracy of weekly projections for any positions or (or any subcategory, i.e. players in poor offenses/playing poor defenses) support this, or what the impact of a player with high week to week volatility is on a fantasy team?
Hi Daniel,
Good question. We haven’t delved into weekly projections yet, but plan to release weekly projections this coming season, and examine their accuracy throughout the season. Including a weekly volatility index for each player is on our to-do list.
Thanks,
Isaac
I know some apps use weekly projections/strength of schedule to determine complementary players, the question to me is whether the reliability of weekly projections at the beginning of the year is a viable strategy. It seems to me that the place to start on this is analysis of historical data. So although projections for week to week performance is something I would love to see from this site, I am actually posing a separate question: does historical projection vs. performance data support considering complementing strength of schedules in draft strategy, and if so, under what conditions and to what degree?
Hi Daniel,
Not sure I follow the question. What do you mean by complementary players. Strength of schedule is different from (though may be factored into) players’ weekly projections. In any case, we haven’t done analyses on weekly projections yet, so this season we hope to be able to look at them more closely.
Thanks,
Isaac
I see how the question may have been confusing now. I am asking about strategies such as those recommended here: http://www.fantasysharks.com/quarterback-committee-strategy/
The idea is to draft multiple low tier players (in this case QBs outside the top seven or so) with the intent of things like: “pairing these two quarterbacks together [so] the toughest defense your starting quarterback would see would be when you have to play Rivers at Minnesota in Week 3.” Also, I believe there is a paid app produced by footballguys which allows you to assign weights to particular weeks if you feel those weeks are more important (i.e. playoffs) and suggest running back partners as you draft based on the aforementioned weekly projections and how they “fit” together.
Like I said before, this question is not really about generating weekly projections so much as a recommendation for a viability study. As you’ve said before in one of your articles, the seasonal aggregate is much easier to predict than individual weeks. Is this extreme enough to where all these efforts to compare strength of schedule are for naught when trying to draft a team for the season? Are players on “good” teams inherently more predictable than players on “bad” teams (i.e. both will produce the same totals throughout the year, but one will do it in a predictable fashion, and the other will do it on your bench)?
Sure can you post it online for DL ?
Hi M,
Yes, we’re working on getting weekly projections for Week 1 now. Stay tuned!
-Isaac
Does this mean the “week” option in the “Lineup Optimizer” app will start working soon? Also, will you post something with instructions for how to use it for weekly leagues?
Hi Daniel,
Yes, we’re working on it now.
Thanks,
Isaac
Can you tell where you got the data from?
Hi Cody,
All of the sources are listed in our apps:
http://apps.fantasyfootballanalytics.net/projections
Click “Change Data Settings” to see which sites we get the data from.
-Isaac
I love everything you guys are doing. Is there any chance that you can look into pulling weekly projections from all the different sources to help set lineups? I see you have a lineup optimizer, but it seems to be set up for the “draft kings” type of leagues. I’d love it if I could look up an average weekly ranking to choose between two individual players.
Thanks,
Eric
Hi Eric,
Yes, use the Projections Tool (http://apps.fantasyfootballanalytics.net/projections), and use the “Compare Players” option in the right sidebar.
Hope that helps,
Isaac
Just came across this website and this is a great resource. I just started using R for fantasy mainly for webscrapping. Would like to get into using it for line ups too. Saw your optimizer, is there any way to build covariances into the model. For example Tom Brade and Gronks scores are going to be related to one another with a positive correlation, or a defense going against an offensive player are going to have negative correlation. How can you build this into your simulation model? Also is there any way to produce multiple line ups? Thanks
We discuss this a bit here: https://fantasyfootballanalytics.net/2015/03/fantasy-football-is-like-stock-picking.html. Here’s a good website with a correlation matrix for various positions on same/different teams: http://rotoviz.com/2014/10/correlation-matrix-answers-every-question-youve-ever-dfs-stacking/. We might be able to build this into our model in the future. I’ll add to our to-do list.
Hello!
I have been reading through your articles, and they interest me. I am also interested in fantasy football and statistics; I created my own spreadsheet that ranked the players according to one projection and created a draft order. However, I am curious about how I could make my own projections or analyze statistics in order to create rankings. What methods did you use to get started, and is there any information you could tell me to help me learn it on my own?
Thanks!
We apply the wisdom of the crowd—instead of generating our own projections from scratch, we aggregate all of the publicly available projections to reduce the biases of any individual source. To generate your own projections, you would ideally want to know the factors that influence individuals’ performance (e.g., age, injury history, strength of surrounding offense, forecasted touches), the direction of each influence, and the magnitude of each effect. You might look into multiple regression for estimating the direction and magnitude of each association, noting that associations could be spurious (not reflecting a causal influence), so it’s good to carefully select variables that are likely to have a causal influence (and hence, predictive utility). Once you have a predictive model for various statistical categories, you can translate the players’ values on the predictors into predictions in terms or rushing yards, passing yards, receptions, rushing TDs, etc.
Regarding Daniel’s initial question about using committee strategies…
I would think the analysis would be complex but the primary question is whether you can predict combinations of QBs ranked outside of say the top 10 for the season that perform inside the top 10 on a weekly basis based on matchups/SOS. The assumption is that one can reliably predict weekly performance (which is less reliable than season projections). Here is my initial thought:
1) Take all QBs projected 11-20 (outside of top 10 but not too low) for the season
2) Using SOS, predict which of these should score in top 10 across weeks 1-15 (some sights provide weekly numbers but only for specific QBs?). Getting projections for each QB 11-20 each week may be hard.
3) Compare “hits” and “misses” at the end of the season or even quantify and compare the weekly performances predicted to be top 10 to a group of QBs (e.g., average of the season projected top 10)
Another interesting question is to look at SOS accuracy as some sights break this down by position. How accurate are these projections at the end of the year. For example, if BAL is projected to rank 4th in points surrendered to QBs, where do they end up in actual rank at the end of the year? These data, if accurate, would provide insight into individual player projections as well.
Any way you guys can add a column and accuracy chart for Fantasy Points per Game? If not, maybe add the average replacement points on to the the player’s actual value. Tony Romo’s data point and Andrew Luck’s data point for 2015 were way off projected value, but those two were injured players. In fantasy football, those weeks do not give us zeroes in our lineup. All (well most) of us will pick up a replacement and fill those positions and points.
For example, Keenan Allen was a beast before his injury. I think the fantasy experts who recommended him should have a higher accuracy rating than those who said he was going to do poorly. Fantasy owners who drafted Keenan Allen surely benefited for all the games he played. After his injury, players could have picked up other viable options via free agency.
Julian Edelman was a MUCH better receiver than Pierre Garcon. He won weeks for many owners whereas Pierre Garcon did not. Yet because of Edelman’s injury, he finished with less points than Garcon. Does it really make sense if Yahoo predicts a higher projection for Edelman and gets penalized for it?
I think adding a FP/G column & chart will fix this problem. It’s a very simple fix. Adding replacement points is a better solution but will require more work. There is a box in settings that mentions replacement points but it did not change any of the actual numbers. Other than that, I love this site & very impressed with all the work that you guys have done.
Hi Moore,
Good suggestion, we’ll add this to our to-do list. One challenge is that some sources of projections may already take into account injury likelihood and how many games a player is likely to play, as a result. We’ll think about how to handle this.
Thanks!
-Isaac
whenever I try and use your accuracy graph, in never is able to load. Should use a different browser?
Try using the Projections Tool (instead of the Lineup Optimizer).
Why is the Accuracy tab not available?
Did you select past years’ data? You can only examine the accuracy of past data.
Thanks. That makes sense. It wasn’t clear how to select prior years, but I figured it out.
One other question, why do all of the optimize options take you to what appears to be the same place. I see no difference between the snake draft optimizer, auction draft optimizer, rankings, etc.
I think the site is terrific, I’m just having a hard time understanding the various options.
I am looking at the results and something is not quite right. The points for some of the defenses are higher than then ceiling. Can you help with this.
We’d need more info to reproduce, see here: https://fantasyfootballanalytics.net/about-the-site/faq#incorrectProjection
Why do all of the optimize options take you to what appears to be the same place. I see no difference between the snake draft optimizer, auction draft optimizer, rankings, etc
We have two tools: Projections Tool (also for Snake Drafts) and Lineup Optimizer (for DFS leagues and Auction Drafts). They have different interfaces. You can access the two different apps here:
https://fantasyfootballanalytics.net/about-the-site/faq#apps
Hi. The “How to Examine Historical Accuracy of Fantasy Football Projections” instructions don’t seem to work anymore. Is there any way to download predicted vs actual stats for some season? Thanks!
What problem are you having with these instructions? Please provide the info requested below for us to reproduce:
https://fantasyfootballanalytics.net/about-the-site/faq#errors
When I select “2016” and “Season” it shows “Projections for 2016 – Week 20”.
I’m not able to reproduce. Please provide all of the info requested at the link above (after first trying the troubleshooting steps).
Thanks, this is Chrome (63.0.3239.84) in incognito mode on OS X (10.12.6). To repro, go to Settings > General Settings, and select any week other than the last week. The data still shows the last week in the season. I tried in Safari too but the app doesn’t load at all. Let me know if I can do anything else to help debug.
Can you verify that you are logged in? What’s your account?
I wonder if this might have something to do with adblocker extensions? I disabled Adblocker Plus in Chrome just now, and the `General Settings` modal seems to be working normally. Might be worth checking whether you’re using an adblocker, and if disabling it on our app helps fix the problem(s).
Another idea if that still doesn’t do the trick: there’s an extension for Chrome called “iFrame Allow” which allows all websites to be displayed in iFrames within Chrome: https://chrome.google.com/webstore/detail/iframe-allow/gifgpciglhhpmeefjdmlpboipkibhbjg?hl=en
Maybe worth a shot if disabling the adblocker doesn’t work.
Hi! Having trouble with any of the apps loading. I just get a white screen. I’ve got everything whitelisted and it was working a few weeks ago. I didn’t check in January. Is everything shutdown for the season?
Hi,
The accuracy tab is grayed out for me and not clickable even after logging in and selecting previous weeks/seasons. Tried different browsers as well, any solution?
How can I use the R package to download historical performance data? I’ve found how to download the projections in R, and I see how to download historical through the app, but if I want to compile weekly performance data for several years without manually using the app to download performance 64 times, how do I do that?
Hello;
I am having trouble loading accuracy for previous years. I open the app and select 2018 for instance but the accuracy tab remains locked(greyed out). What am I missing I followed the directions on this page and it is still happening. I logged out, logged in, opened incognito window and still nothing.
Hi! Please fill out an error log as described here (after trying all the steps): https://fantasyfootballanalytics.net/about-the-site/faq#errors
Are the accuracy data downloads still available? I am subscribed and have access to download historical data but don’t see the accuracy tab any longer.
We will be adding it soon. Thanks for your interest!
The accuracy tab is live!:
https://fantasyfootballanalytics.net/2023/08/2023-accuracy-tab-update.html