Are Subscription Sources More Accurate?
21Introduction
In this article, we examine whether there are advantages to paying for subscription fantasy football projections. We tested whether projections from subscription sources have higher accuracy than projections from free, publicly available sources. There are arguments that subscription sources would possibly be more accurate as you may expect to get better accuracy as part of what you are paying for. We examined projections from the 2015 season for QB, RB, WR and TE positions.
Overall Accuracy
We calculated the projected seasonal points based on standard scoring settings as used in our Projections tool and compared with the actual points. For the aggregation of sources we used the regular mean. We have 10 free sources and 6 subscription sources. The free sources were: CBS, Yahoo, ESPN, FOX, NFL, FFToday, NumberFire, EDS Football, WalterFootball and RTSports. Because the subscription sources are not publicly available we chose not to disclose the names of the sources. For each of the groups we calculated R2 (higher is better) and MASE (lower is better) values as well as values for both groups combined. The results are below.
| Source Type | R-Squared | MASE |
|---|---|---|
| Free | .635 | .548 |
| Subscription | .618 | .568 |
| All | .641 | .541 |
Based on the results, the subscription sources were less accurate than the free sources but add to the overall accuracy. In light of that, one possible reason the subscription sources were less accurate could be because there were more free sources than subscription sources. To investigate that possibility, we examined the accuracy of all possible combinations of 6 sources among the free sources. The results below show the mean R2 and MASE values for all the combinations. Reducing the number of free sources did reduce the projections’ accuracy, but the free sources were still more accurate than the subscription sources. In other words, even after accounting for how many sources of projections were included, free projections were still more accurate than subscription projections.
| R-Squared | MASE | |
|---|---|---|
| Combinations of free sources | .631 | .556 |
Accuracy of Individual Subscription Sources
We then examined whether any of the individual subscription sources was more accurate than the crowd.
| Source Type | R-Squared | MASE |
|---|---|---|
| Subscription 1 | .567 | .616 |
| Subscription 2 | .576 | .608 |
| Subscription 3 | .588 | .612 |
| Subscription 4 | .587 | .615 |
| Subscription 5 | .599 | .586 |
| Subscription 6 | .552 | .641 |
None of the subscription sources was more accurate than the crowd of free projections (R2 = .64, MASE = .55) or all projections (R2 = .64, MASE = .54).
Position Accuracy
Let’s examine whether the results are different when we look at individual positions:
| Source Type | R-Squared | MASE |
|---|---|---|
| Free | .707 | .416 |
| Subscription | .677 | .434 |
| All | .705 | .414 |
| Source Type | R-Squared | MASE |
|---|---|---|
| Free | .488 | .673 |
| Subscription | .492 | .686 |
| All | .503 | .657 |
| Source Type | R-Squared | MASE |
|---|---|---|
| Free | .613 | .575 |
| Subscription | .560 | .633 |
| All | .616 | .571 |
| Source Type | R-Squared | MASE |
|---|---|---|
| Free | .550 | .583 |
| Subscription | .541 | .592 |
| All | .559 | .572 |
For the QB, WR, and TE positions, the free sources were more accurate than the subscription sources, while the subscription sources were slightly more accurate for the RB position measured by R2 but slightly less accurate as measured by MASE. For every position except the QB position, combining the free and subscription sources also increased the overall accuracy.
We also calculated all possible combinations of 6 sources among the free sources by position. As the results below show, the accuracy for the free sources did decrease. However, as was the case with overall accuracy, the free sources were still more accurate than the subscription sources even after accounting for how many sources of projections were included.
| Position | R-Squared | MASE |
|---|---|---|
| QB | .702 | .422 |
| RB | .482 | .688 |
| WR | .605 | .585 |
| TE | .548 | .589 |
Conclusion
We have seen that subscription sources are not more accurate than the free sources. In general, free projections were actually more accurate than subscription projections. However, including subscription projections did improve the accuracy of projections both overall and for each position (except quarterbacks). As we have demonstrated before, individual analysts do not reliably beat the “Wisdom of the Crowd” and this analysis further supports that—none of the subscription analysts beat the crowd. So while free sources seem to be more accurate, on average, than subscription sources, it is combining them that adds to the accuracy of the overall projections. The most accurate projections combined free and subscriptions sources. So if you are asking whether you should use free or subscription sources for your projections, the answer is: use both!
You can find data and script for the analysis here.
Great read! Keep these articles coming
Dennis, when I look at the results I begin to wonder if the results reflect a positional bias? Are subscriptions services focused on RB based on historical performance? And are free services concentrated on overall value of the player and not the position? The great thing about this information is it opens up the discussion of what information is being valued among all the sources. Is it analytics, beat writers report, is it the analyst that serve on staff? What is so different? I value the opinions of the analyst just as much as the analytics. Analytics in football often has too many variables as compared to baseball driven analytics,
Hi Cris, I don’t think there is a positional bias. I think there is a lot of information available out there for all analysts if you go look for it. I think the differences reflects the individual analysts perception and interpretation of the information available. All the analysts have the information about previous performance, so I believe the difference come into play when you take the off-season information into account and try and apply that when projecting the performance for the next season. So the differences in projections probably reflect more of a difference in opinion about the players than difference in available information.
This is a fantastic article Dennis. I really appreciate you guys putting this stuff out. This is something I have wondered for a few years. I assumed pay would be more accurate and if I was running an optimizer I wouldn’t want to contiminate the pay projection by adding them. Turns out I was very wrong.
Thank you
*adding free ones
Interesting article. I really like the work you guys do on this site.
It’s not obvious to me from what you’ve shown that there is not any single subscription source that outperforms the crowd. Can you show the table of individual performance compared to the performance of the crowd? (similar to the ones here: https://fantasyfootballanalytics.net/2016/03/best-fantasy-football-projections-2016-update.html )
The reason I think this matters is because subscription sources are usually contrasting their performance with that of other subscription sites. So in terms of their marketing claims it doesn’t matter how a group of sites perform, it only matters how each one individually performs.
Thanks again for taking the time to share your analysis with us.
Hi Timothy,
Great question. We updated the post with the accuracy estimates of the individual subscription sources. None was more accurate than the crowd.
Hope that helps,
Isaac
Thanks, Isaac! It’s nice to see more confirmation that the individual sources do not reliably beat the crowd. Keep up the good work!
These accuracy results can be misleading, since it very much depends on which list of players your comparing to. I could do the same study but my group of players may vary and get different results. Case and point, I have done a similar study and used 15 different sources for 2015 (based on 200 players). FFA was the most accurate site for preseason results with a .409 r2 – the average for accuracy among all the models was .347 and the low was .174.
I applied this to positions to see if any position was easier to predict than others. QBs were by far the most accurate in the 70% range. All the other positions fell in the 60% range. However, despite FFA’s strong overall accuracy, this is how they scored: QB – 7th, RB – 5th, WR – 9th, TE – 15th, K – 5th, D – 6th. I haven’t really figured out if being accurate at position is better than being accurate overall.
I also applied this accuracy study to a week by week time frame. FFA once again was the top model, placing in the top 3 every single week. They also had the highest weekly accuracy of .572 over the 16 week season.
This site is outstanding. I have done my study every year for 4 years, FFA is always a top producer in accuracy. Very impressive work.
Hi Jason,
Good point—the accuracy depends on which positions and players are examined. We examine the accuracy of all available players we can find projections for among the most common positions across people’s fantasy football leagues (QBs, RBs, WRs, and TEs). Nevertheless, you can examine our (and others’) accuracy for other positions using our tools:
https://fantasyfootballanalytics.net/2015/07/accuracy-of-fantasy-football-projections-interactive-scatterplot-in-r.html
In general, we (and others) are more accurate for seasonal than weekly projections, which makes sense because there are fewer plays (and hence, more randomness) inherent in weekly than seasonal projections. If you have more accurate sources of projections that we might be able to include, please send them to us, and we’ll do our best to include them!
Thanks!
-Isaac
Three more to consider are: FBG (Football Guys – Draft Dominator is free till before the season starts – you can export those projections), FS (Fantasy Sharks) and FP (Fantasy Pros). CBS does IDP projections, I could send those to you for both the season and the week.
How would you like those, via email, what format (xlsx, csv). I’m still struggling to learn R, so I’m still an excel guy, until I finally make the plunge.
If you’re interested, I built a model that is a stand alone, it quantifies a projection solely off of historical data. Been using it for 2 seasons. It does a pretty good job and rivals your weekly results. It’s a lot of work to maintain because it’s based on last 3, 5 and YTD (scored and allowed) data for every position, including IDP. But in 2015, it received a .408 r2, which was only .001 less than FFA on my study. It also did a better job when judging accuracy on individual positions. Regardless, it’s still in beta. If you would like more models for the aggregate. I’m open to helping.
We have plans to include many of those projections. To the best of my knowledge, we already include all sources of projections included in the FantasyPros projections. What’s the link to the CBS IDP projections? Feel free to send me an email if you want to help out!
Thanks!
-Isaac
My league hosts through CBS, but unless you are in a IDP setup, not sure CBS publishes them for the public. But since there are so few sites that create IDP projections, every one counts right? I’ll email them to you once they become available.
Hey Jason,
Thanks for your help! The challenge is that we’d need a way to update the projections when CBS updates them. Do they not have them publicly available?
-Isaac
Hey Isaac, have you considered adjusting for the number of “analysts” involved in developing each single site’s projections? For instance, if CBS’ projections are basically the average of 2 individual’s projections and ESPN’s projections are basically the average of 5 individual’s projections, would you want to look into weighting those projections differently in your final average projections?
Hi Michael,
We weight based on historical accuracy. Presumably, the more sources are included, the higher the historical accuracy, so it should be accounting for the number of sources (to the extent that it matters).
Hope that helps,
Isaac
Why did you exclude Defense?
Another free source that you might want to consider adding to your weekly rankings database is Justin Boone from The Score. His overall accuracy has consistently been rated in the top 10 over the last few years according to Fantasy Pros.
Thank you for the suggestion but we need projections, not rankings, for our tools.
Is there a way this could be broken down into perhaps finding WHERE projections are most/least accurate? Like perhaps certain sources are most accurate for the top 10 or so QBs, but the next 22 are just all over. Or conversely very accurate on game to game basis, but just fail to capture the monster games and duds. I’m thinking even one major early season ending injury could drastically skew someone’s accuracy if they had projected that player for a career season.
Hello Victor,
We try to pull data from as many sources to give as accurate as possible projections, thus using a wisdom of the crowds approach.