Which Site Has the Best Fantasy Football Projections? 2014 Update
25In prior posts, I demonstrated how to download, calculate, and compare fantasy football projections from ESPN, CBS, NFL.com, and FantasyPros, which aggregates projections from many different sources to increase prediction accuracy. Last year, I compared fantasy projections from ESPN, CBS, NFL.com, and FantasyPros, including our average and latent projections to determine who had the best fantasy football projections. In this post, I compare fantasy football projections over the last two years to see if the best projections in 2012 are still the best in 2013.
The R Script
The R Script for comparing the projections from different sources is located at:
https://github.com/FantasyFootballAnalytics/FantasyFootballAnalyticsR/blob/master/R%20Scripts/Posts/Evaluate%20Projections.R
To compare the accuracy of the projections, I use the following metrics:
- R-squared (R2) – higher is better
- Mean absolute scaled error (MASE) – lower is better
Whose Predictions Were the Best?
| Source | 2012 | 2013 | Average | |||
|---|---|---|---|---|---|---|
| R-squared | MASE | R-squared | MASE | R-squared | MASE | |
| ESPN | .528 | .577 | .393 | .684 | .460 | .630 |
| CBS | .619 | .501 | .388 | .676 | .504 | .589 |
| NFL.com | .510 | .642 | .419 | .595 | .465 | .618 |
| FantasyPros | .674 | .411 | .500 | .520 | .587 | .466 |
| Average | .671 | .422 | .503 | .520 | .587 | .471 |
| Latent | .654 | .427 | .485 | .531 | .569 | .479 |
- FantasyPros
- Average
- Latent
- CBS
- NFL.com
- ESPN
Here is a scatterplot of the FantasyPros projections in relation to players’ actual fantasy points scored in 2013:
Interesting Observations
- Projections that combined multiple sources of projections (FantasyPros, Average, Latent) were more accurate than single projections (CBS, NFL.com, ESPN).
- The R-squared of the most accurate projection source was .67 in 2012 and .50 in 2013. This suggests that players are more predictable in some years than others. It also indicates that 1/3 to 1/2 of the variance in actual points is unexplained by projections, so there is much room for improvement in terms of prediction accuracy.
- There was little consistency in performance across time among sites that used single projections (CBS, NFL.com, ESPN). In 2012, CBS was the most accurate single source of projection but they were the least accurate in 2013. Moreover, the least accurate in 2012 was NFL.com, but they were the most accurate in 2013. This suggests that no single source reliably outperforms the others. While some sites may do better than others in any given year (because of fairly random variability–i.e., chance), it is unlikely that they will continue to outperform the other sites. This may be similar to the finding that there is little consistency in the performance of mutual fund managers over time. The following charts are from Leonard Mlodinow’s book, “The Drunkard’s Walk: How Randomness Rules Our Lives“. The chart below depicts the performance of the top mutual funds from 1991 to 1995.
The next chart depicts the performance of the same funds in the same order over the subsequent 5 years (1996 to 2000).
In other words, the best funds from 1991-1995 weren’t necessarily the best funds from 1996-2000. That’s likely why a cat beat investors in a stock market challenge. This suggests that mutual fund managers differ in great part because of luck or chance rather than reliable skill. Although our sample size is much smaller with fantasy football projections, there appears to be little consistency in fantasy football sites’ rank-order in accuracy over time.
Conclusion
In conclusion, the most accurate projections were from FantasyPros.com, which combines many sources of projections. FantasyPros had the most accurate projections each of the last two years. No single projection (CBS, NFL.com, ESPN) reliably outperformed the others, suggesting that differences between them are likely due in large part to chance. I plan to calculate projections from more sites (Accuscore, Yahoo, FantasySharks) so we can compare even more projections. In sum, crowd projections are more accurate than individuals’ judgments. People often like to “go with their gut” when picking players. That’s fine—fantasy football is a game. Do what is fun for you. But, crowd projections are the most reliably accurate of any source. Do with that what you will!
Well, that would seem to suggest that a consensus of player rankings (basically an ADP) is better than any one source. It might be my bias — but that seems counter intuitive, if not boring! 🙂
Hey Lenny,
Yep, in general an ADP is more accurate than most individual sources. A few individual sources may beat the ADP in any given year, but it will likely be because of chance, and they most likely won’t reliably outperform the ADP over time. I’m curious why you think this is counter intuitive, There are many domains where “wisdom of the crowd” outperforms so-called experts (e.g., stock market). Every individual source has biases, and averaging together the crowd’s picks removes a lot of those biases (increasing the ratio of true signal to noise).
Hey Issac — I live in a country where the ADP selects Democrats and Republicans to govern us. Therefore, I don’t typically put much faith in the “wisdom of the crowd.” 🙂
Haha touché! Politics aside, there certainly are problems with wisdom of the crowd (https://en.wikipedia.org/wiki/Wisdom_of_the_crowd#Problems). Wisdom of the crowd tends to work best when there is a correct answer to the question posed. I’m not sure this is the case with politics. In contrast, there is a correct answer to how many passing yards Peyton Manning will throw next year. Ultimately, the question as to which is better (experts or the crowd) is a testable question, and evidence suggests that the crowd is more accurate than experts in predicting fantasy performance. Do with that what you will. If you think you can outperform the crowd, you should send me your projections and we can compare them at the end of the season. We could have a contest with users’ entries. Sounds fun to me!
Any chance of updating these numbers for all the sources available in the draft tools? It would be a big help for setting up my weightings…
Thanks for all the work on the site, this is pretty much exactly what I was looking for.
Hey Ehren,
I don’t have historical projections for most of the sites, so I wouldn’t be able to examine their accuracy from last year. Next year, on the other hand, I will be able to examine the accuracy of all of these sources and will be able to give you better weightings. In the meantime, I personally will use an equal weighting, because then a robust average is calculated to remove the outliers.
-Isaac
[…] Who has the Best Projections? […]
First of all, I think what you’re doing here is great. I’m a math guy myself so it’s refreshing to see this kind of analysis done when it comes to something like fantasy sports. As far as your projections go, what made you decide between the sources you did (CBS, ESPN etc)? There are numerous fantasy sites I look at (FantasyPros being one of them) that also have projections such as: footballguys.com, fftoolbox.com, fantasyfootballcalculator to name a few. Do you think you’re going to add more sources to get data from to come up with a definite “best” one? Yes I agree that FantasyPros is the best out of the ones you mentioned, but the ones I mentioned possibly could be more accurate. I’m not saying they are, maybe they’re crap, but you never know until you test them out.
Also, I used the snake draft optimizer in my draft the other day and it was great! Thanks again for all your hard work.
Hey Joe,
We were only able to compare projections to actual performance for sources for which we had last year’s projections. These were the only sites for which I have last year’s projections. We have many more sources this year, though, (17 sources now), so after the season, we will be able to compare all of them. I’m looking forward to it!
Glad you found the Snake Draft Optimizer helpful!
-Isaac
Oh I see. Can’t wait to see what you come up with. I assume you’re extremely busy but during the season will you be updating the site such as the WR Goldmine, Sleepers, etc?
Yes, one of our writers has plans to do goldmining and identify sleepers again this season. Stay tuned!
Perhaps I did not find it but have you discussed what variables are included in their projections? Are they listed someplace publicly or is it all proprietary? I ask because sometimes I want to deviate from a weekly projection but am not sure if the factor I am considering is already in their model.
Hey Peter,
When you ask “what variables are included in their projections” do you mean which variables do they project (running yards, passing TDs, etc.), or do you mean what predictor variables did they use to derive the projections (strength of schedule, strength of surrounding offense, etc.)? If the former, You can see and download their projections by using our app (http://apps.fantasyfootballanalytics.net/projections). If the latter, the unfortunate answer is no. As you point out, their projections are proprietary so we don’t know how they are calculated by the black box. We have considered calculating our own projections, but this would be challenging and time-consuming because doing it well would require lots of data. Plus, our evidence suggests that the wisdom of the crowd is better than individual projections, so you’re probably better off going with the average than with a single source anyway.
Hope that helps!
-Isaac
Yes, I was hoping for the latter but suspected the worst. Thanks!
Man that is insane that you have came up with this formula to see who really is the most accurate. I know fantasy football has a lot to do with luck but can you break it down by position.
For example Like ESPN is correct when it comes to QBs but Fantasy Pros is more accurate with WRs and TEs.
I will def bookmark this page and tell a Couple of my fantasy friends. Keep doing what yo do..look forward to see who is the most accurate a lil over halfway through season.
Steel City
Yes, we could break it down by position. We’re finding that individual sources of projections aren’t reliably better than other sources, so I suspect that we’d find a similar pattern when looking at projections at individual positions. Will definitely consider it for next season. Thanks!
Hello Isaac-
Have you done any NBA analysis? If so, what have you found are the most accurate projections sources? I see that you shy away from daily/weekly projections, but have you found any daily NBA projections sites that are somewhat accurate? Looking to do some analysis of my own.
Thanks for the help!
Cheers,
Tyler
Hey Tyler,
Haven’t done any NBA analysis. I’m an avid basketball player/fan, though, so I’d be open to expanding our coverage if people help contribute. I suspect many of the same principles would still apply (e.g., wisdom of the crowd is reliably more accurate than projections by individual sources). Also, psychological factors (https://en.wikipedia.org/wiki/Clustering_illusion) explain the shooting “streaks” we perceive (https://en.wikipedia.org/wiki/Hot-hand_fallacy). Here are some potentially good sources of NBA data to get you started:
http://www.basketball-reference.com/
http://www.basketballgeek.com/data/
http://downloads.nbastuffer.com/nba-play-by-play-data-sets
http://stats.nba.com/tracking/#!/player/
Also, we are starting to do weekly fantasy football projections (see here: https://fantasyfootballanalytics.net/2014/11/wide-receiver-gold-mining-week-13.html).
Hope that helps!
-Isaac
Hi Isaac,
Great site and article. I am researching which site to use for the upcoming season. I have used Yahoo in the past – any idea how they compare?
Thanks!
Hi Caroline,
Here’s the analysis from this year that includes Yahoo:
https://fantasyfootballanalytics.net/2015/02/best-fantasy-football-projections-2015.html
Hope that helps!
-Isaac
Very nice Isaac! I am going to use your stats this year. Last year I used football guys app and it wad great for adp in my opinion. Wasn’t free I think it was $5 but years before I spent $8-$11 on a magazine.
Cheers,
Dan
Excellent work and presentation, Isaac! Thank you for your time & effort, sir.
When evaluating predictions in this manner, wouldn’t it be advantageous to drop the constant from the model, or at least evaluate that as a negative metric in and of itself? Otherwise a source can consistently overvalues or undervalues players gets credit for evaluating those players accurately, despite the fact that their assessment was actually fairly inaccurate.
Hi Tyler,
Not sure what you mean by the constant in the model. MASE doesn’t include a constant in the model.
-Isaac
This is great as always. Thanks for sharing such wonderful insights. I notice that the projections so far draw from ESPN and FantasySharks. Can you provide some features of FD-guru website as well? I found their pricing comparable to my budget and thus, your insights would be greatly helpful.