2018 Update of ffanalytics R Package
66Introduction
As part of our effort to provide users ways to replicate our analyses and improve their performance in fantasy football, we are continuously looking at ways we can improve our tools. Since the ffanalytics package was introduced back in 2016, we have had a tremendous response and feedback on the package and have worked through some of the kinks and are now introducing a new version of the ffanalytics package that should be more robust and provide more consistent results.
Improvements
One of the major concerns with data scrapes is that websites change and so does the way data are rendered on individual pages. Where the old version of the ffanalytics package was highly dependent on how columns were ordered on the different sites, the new version of the package does not depend on column order. Instead the package looks for specific components of the page and, based on predefined mapping, can determine which columns are which.
The other major concern is that different sites lists the same player with a slightly different name (i.e. Rob Kelly or Robert Kelly) or at different positions, so matching up the data can be tricky. The new version of the package maps player IDs from the different sites back to the player ID used by MyFantasyLeague.com, so it now doesn’t matter what name or position a specific player is listed as on a specific site because the data for a specific player will match up correctly.
Finally, the new version of the package relies heavily on the vocabulary and terminology used in the tidyverse package. We highly recommend familiarizing yourself with that package.
It is our intent that the new version of the package will provide more consistent scrapes for users.
Player information
The player information used in the package is based on player information from MyFantasyLeague.com (MFL). When the package is loaded, it pulls the most recent player data from MFL into the player_table object. When the scrape runs, it determines the player ID for the player on the site that is being scraped and maps that ID back to the MFL player ID. The scraped data contains an id column which holds the player ID for MFL, and a src_id column that has the ID for the site. However, when aggregating the stats and generating the projections the player ID used is the MFL player ID.
Custom calculations
The customization of calculations is done as input to the projections_table() function. You can specify your own scoring rules, analyst weights, tier thresholds and VOR baselines to be used in the calculations. See ?projections_table for specifics, and check out the scoring_settings vignette on how to customize scoring rules
Package Manual
In addition to the documentation that is included within the package and can be explored directly in R/RStudio, we have also made that documentation available as a pkgdown site here.
Installation and getting started
We have updated the instructions on installing and getting the package here so you can check that page out, and feel free to leave comments with feedback and suggestions.
Awesome! but i can’t seem to punch through the MFL login. even with a registered account, for every type of analyst i get:
“Error: 1: Operation in progress2: failed to load external entity “http://football.myfantasyleague.com/2018/export?TYPE=players&L=&W=0&JSON=0&DETAILS=1″”
Please make sure you have installed the new version of the package. Instructions have been updated here: https://fantasyfootballanalytics.net/2016/06/ffanalytics-r-package-fantasy-football-data-analysis.html
Thanks so much! I did re-run the install from GitHub- and it was successful– can you say which is the correct version so i have it right? Here’s my current environment:
sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.6
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] ffanalytics_0.1.91 miniUI_0.1.1 shiny_1.0.5
loaded via a namespace (and not attached):
[1] Rcpp_0.12.14 cellranger_1.1.0 RColorBrewer_1.1-2
[4] plyr_1.8.4 base64enc_0.1-3 bitops_1.0-6
[7] tools_3.3.2 rpart_4.1-10 digest_0.6.13
[10] tibble_1.3.4 gtable_0.2.0 checkmate_1.8.5
[13] htmlTable_1.11.1 lattice_0.20-34 rlang_0.1.6
[16] Matrix_1.2-7.1 rstudioapi_0.7 binman_0.1.0
[19] gridExtra_2.3 httr_1.3.1 stringr_1.2.0
[22] cluster_2.0.5 knitr_1.18 caTools_1.17.1
[25] htmlwidgets_0.9 grid_3.3.2 nnet_7.3-12
[28] data.table_1.10.4-3 R6_2.2.2 tcltk_3.3.2
[31] readxl_1.0.0 XML_3.98-1.9 survival_2.41-2
[34] RSelenium_1.7.1 foreign_0.8-67 latticeExtra_0.6-28
[37] Formula_1.2-2 semver_0.2.0 ggplot2_2.2.1
[40] magrittr_1.5 Hmisc_4.1-1 backports_1.1.2
[43] scales_0.4.1 htmltools_0.3.6 splines_3.3.2
[46] assertthat_0.2.0 xtable_1.8-2 mime_0.5
[49] colorspace_1.3-2 httpuv_1.3.5 stringi_1.1.5
[52] acepack_1.4.1 openssl_0.9.9 RCurl_1.95-4.10
[55] lazyeval_0.2.0 munsell_0.4.3 wdman_0.2.2
It appears you are still using the incorrect version of the package. The version should be 2.0.0. Make sure you install the package from https://github.com/FantasyFootballAnalytics/ffanalytics. The version at https://github.com/isaactpetersen/ffanalytics is now outdated.
Sincere apologies for not realizing the repo had changed (i reinstalled from the same). Successful install but now i get an error (below).
This is reproduced with any src combination as well. perhaps it is user error in what i’m feeding to projections_table()? is there a step between scrape_data() and projections_table()?
##############
> my_scrape ffanalytics::projections_table(my_scrape)
Error: is.data.frame(df) is not TRUE
FYI that i solved my problem below (but couldn’t thread off of it for some reason)– it was an older version of tibble (1.3.4) that cause the bug. updating tibble to 1.4.2 solved the problem.
All dependencies installed (listed on this site as well as the github readme), but I’m still hung up — using scrape_data produces the error “could not find function make_df_colnames”, and scrape_source says “Error: $ operator is invalid for atomic vectors”.
More info:
> chk % transpose() %>%
map(~imap(.x, ~scrape_source(.x, season, week, .y))) %>%
transpose() %>% map(discard, is.null) %>% map(bind_rows,
.id = “data_src”)
1.
scrape_data(src = c(“CBS”, “ESPN”, “FFToday”, “NumberFire”),
pos = c(“QB”, “RB”, “WR”, “TE”, “DST”, “K”), season = 2018)
This should be fixed now. Please reinstall the package.
Single-source scraper is still borked (same error about the vector), but scrape_data is working on a full-season level now, thanks! And sorry my traceback got truncated above, that probably removed any sort of helpfulness it might have provided.
For
scrape_source, thesrcargument is one of the elements of theprojection_sourceslist. So if you want to scrape ESPN usingscrape_source, setsrc = projection_sources$ESPNLooks like many sites might not have their weekly data available yet — just as a trial I tried to grab various sites’ Wk1 data and most are still returning the full-season figures. URL appears well-formed, though, so I’m guessing it’s a problem on their end, not yours (esp. since at least one, CBS, *does* work weekly already). Still, might be worth putting on your “look into this” list.
trying to customize scoring settings for .5ppr but the “scoring_settings” vignette is missing. Any instructions I can reference?
Try installing the package with devtools::install_github(“FantasyFootballAnalytics/ffanalytics”, build_vignettes = TRUE) to generate the vignette
The function get_aav doesn’t return any columns, ive tired:
my_projections % add_ecr() %>% add_risk() %>%
add_adp() %>% add_aav()
and also
my_projections % add_aav()
what i am doing wrong?
how did you generate the my_projections object?
Hi Dennis,
I used the below:
my_scrape <- scrape_data(src = c("CBS",
"ESPN",
"FantasyPros",
"FantasySharks",
"FFToday",
"FleaFlicker",
"NumberFire",
"Yahoo",
"NFL",
"RTSports"
,"FantasyData"
,"FantasyFootballNerd"
"Walterfootball"
),
pos = c("QB", "RB", "WR", "TE", "DST"),
season = 2018, week = 0)
i’m having the same problem
Please provide full code and list of errors you are getting.
i ran everything as noted above in my original post, here are the warnings i am getting. (i reinstalled again today to the newest version)
In evalq(as.numeric(gsub(“^\\$”, “”, aav)), ) :
NAs introduced by coercion
When I run projections_table() on a data_result object, I get the following error:
Error in right_join_impl(x, y, by$x, by$y, suffix$x, suffix$y, check_na_matches(na_matches)) :
Can’t join on ‘id’ x ‘id’ because of incompatible types (logical / character)
In addition: Warning messages:
1: Length of logical index must be 1 or 331, not 0
2: Length of logical index must be 1 or 331, not 0
Any suggestions?
Please provide the code that generate the data_result object
getting this error when running projections_table() on scrape_data object. Thoughts?
Error in right_join_impl(x, y, by$x, by$y, suffix$x, suffix$y, check_na_matches(na_matches)) :
Can’t join on ‘id’ x ‘id’ because of incompatible types (logical / character)
In addition: Warning messages:
1: Length of logical index must be 1 or 331, not 0
2: Length of logical index must be 1 or 331, not 0
Please provide the code that generate the scrape_data object
scrape_data(src = c(“CBS”, “ESPN”, “FantasyPros”,
“FantasySharks”, “FFToday”, “FleaFlicker”, “NumberFire”, “Yahoo”,
“NFL”, “RTSports”, “Walterfootball”), pos = “K”, season = 2018, week = 0)
Script ran perfectly thank you! Although I did get a number of #N/A responses when I matched ‘id’ from the predictions with ‘id’ in the ‘player_table’. Is there a solution to this?
Could you provide some examples?
add_aav isn’t working
> proj = proj %>% add_aav()
Error in if (week != 0) { : argument is of length zero
looked at the projections_table object and the problem seems to be that, at least, the “week” attribute isn’t being passed through to the position tibbles (resulting in the argument of length 0 error since it’s comparing to a null). in fact, it seems like none of the arguments it needs are being passed through (lg_type, season, and week), so for the time being, this function appears to be flat busted.
Could you provide the code that generated the “proj” object?
proj = projections_table(datascrape,
scoring_rules = scorelist,
src_weights = NULL,
vor_baseline = VORbase,
tier_thresholds = NULL)
where datascrape is the output of
datascrape = scrape_data(src = c(“CBS”,
“ESPN”,
“FantasySharks”,
“FantasyPros”,
“FFToday”,
“FleaFlicker”,
“NumberFire”,
“Yahoo”,
“FantasyFootballNerd”,
“NFL”,
“RTSports”,
“Walterfootball”),
pos = c(“QB”,
“RB”,
“WR”,
“TE”,
“K”,
“DST”),
season = 2018,
week = 0)
there’s also a small sequence of additions prior to add_aav
proj = proj %>% add_ecr()
proj = proj %>% add_player_info()
proj = proj %>% add_risk()
and then
proj = proj %>% add_aav()
If you run add_player_info last it should work. The package has been updated to handle running add_player_info at any point in the sequence
just re-tried it with add_player_info at the end, but still didn’t see any auction values being added (compared this CSV with the one generated before it and had exactly the same number of columns).
so if I wanted to calculate predictions excluding my leagues keepers, how is the best way to do that? should I subset the player list based on keepers and go from there?
nevermind, I sorted it out.
I subsetted the my_scrape position data like so:
my_scrape$QB <- subset(my_scrape$QB, !(id %in% c('1234','1235',…)))
for each position.
if you create a vector with ids for the keepers (say keeper_ids), then you can use the map function filter for all positions using one line of code:
my_scrape <- my_scrape %>% map(filter, !(id %in% keeper_ids))
I may be going crazy, but didn’t there used to be a Shiny GUI for the R package? It doesn’t seem to be there anymore.
We have not created a Shiny GUI for the updated version of the R package, as it would have too much resemble to our apps at https://apps.fantasyfootballanalytics.net
First – thanks for all the work you put into the package – a fun way to learn R.
I know it used to be possible to get a df of the weighted statistics by using Projections$avgStats… looks like this isn’t possible anymore. How would I go about getting a tibble or dataframe of the average stats?
you can use the aggregate_stats function on the result from scrape_data. This will create a tibble with pos, id , data_col, avg_type, and stat_value columns. To create a table with a column for each stat you can then use the spread function. For example if my_data is the result of scrape_data then you can do this to crate a tibble with aggregated stats
aggregated <- my_data %>% aggregate_stats() %>% spread(data_col, stat_value)
Note that the resulting table will contain aggretaged stats for all three average types so you will need to subset on avg_type to get a specific average
Can I rely on Auction Values? is it up to date?
The add_aav function will scrape the most recent auction values from the different sites
Hi, Appreciate all of the work on this. I am just getting into R and this will be a great way to learn… Wondering if you could help with the error below:
Code:
my_scrape <- scrape_data(src = c("CBS", "ESPN", "Yahoo"),
pos = c("QB", "RB", "WR", "TE", "DST"),
season = 2018, week = 0)
Error:
Error in get(dbname) : object 'QBMapEnv' not found
Thanks!
I am not able to reproduce. The `QBMapEnv` object is not one that is used in the package, so I am not sure where that would be referencing. Did you remember to load the package via `library(ffanalytics)`?
I’ve been playing around with this, and don’t see the functionality offhand. is there a way to scrape my roster, then scrape all available players, and then the projections?
There is not functionality offhand to separate scrapes by roster and available players. Available players is league specific so that information is not built into the package, but you can split the projections by groups of players. You can use the player_table object to create a roster object and an available players object if you have the ids or names of players in each group. Then use each of the groups to break out the projections.
yeah, I thought about that but more work than I would like to put in (mainly the maintenance).
Basically I’d like to be able to run a script every week that tells me what available players there are that are better than my current roster, and projections for start/sit.
I was looking at the WR gold mining example script, and it looks like I might be able to make part of that work, because I’m on Yahoo. However I haven’t tried.
This would require that package to be able to retrieve 1) your roster, and 2) available players in your league.
right, and it looks like you might be able to just pull the full player list, or just available players in yahoo (status=ALL vs status=A in the URL). Similarly an individual team in the league (i.e. my team).
Once I have that pulled I’ll also need to be able to match that with the player_table produced by your package.
once I have the player ids for the players on my and everyone else’s team in my league, then it’s just a matter of matching it with the projections that are already there.
Hello! The 2018 update is works very smoothly, thanks so much for writing.
I have a question related to data scrapes. If I scrape data for week 0, everything works great and I can subsequently calculate projections, create figures, etc.
If I change from week = 0 to week=1 in the scrape_data() function, I get an error when calling projections_table():
Error in rowSums(., na.rm=TRUE) : ‘x’ must be numeric
I’m guessing this is some change in the input data table. Any suggestions for what I can do to fix? Literally the only difference in the code I’m running is week = 0 versus week = 1.
Thank you!
did you try enclosing the 1 in single quotes?
Using quotes around the week (single or double) gives a debug error.
Error in week + ifelse(week > 17, 3, 0) :
non-numeric argument to binary operator
Called from: self$get_query(season, week, p_id, …)
?scrape_data() indicates that the syntax is without quotes also.
If it’s helpful, here are the commands I’m executing:
my_scrape2 <- scrape_data(src = c("ESPN", "Yahoo", "FantasyPros","FFToday","NumberFire","NFL","FleaFlicker","FantasyFootballNerd"), pos = c("QB", "RB", "WR", "TE", "DST"),season = 2018, week = 1)
my_projections<-projections_table(my_scrape2, scoring_rules = my_scoring_rules)
Again, week=0 seems to do OK. Very weird.
Thank you. : )
I had the same issue crop up today as well. The way that I “fixed” it, is that I used try() to determine if I could calculate projected points for each individual analyst and then only used the sources that allowed me to calculate points.
Obviously, this is just a band-aid, but it should get you through week1.
all.sources <- c("CBS", "ESPN", "FantasyData", "FantasyPros",
"FantasySharks", "FFToday", "FleaFlicker", "NumberFire", "Yahoo",
"FantasyFootballNerd", "NFL", "RTSports", "Walterfootball")
bad.sources <- c()
# function to create projections for a given week
create_project <- function(use_week,use_analyst) {
hold_scrape <- scrape_data(src = use_analyst
, pos = c("QB","RB", "WR", "TE","DST"), season = 2018, week = use_week)
hike22 <- projections_table(hold_scrape)
1
}
bad_sources <- c()
for(i in 1:length(all.sources)) {
bad_sources <- c(bad_sources,try(create_project(use_week = 1,use_analyst = all.sources[i])))
}
use.sources <- all.sources[which(bad_sources == 1)]
The issues with individual sources should be fixed now. Try reinstalling the package.
This issue has been fixed. Please reinstall the package.
Thank you! I reinstalled, and it looks like I might be getting a new error now when I execute a scrape (still week 1):
Error in week + ifelse(week > 17, 3, 0) :
non-numeric argument to binary operator
I’ll try Peter’s suggestion above, too, since it sounds like it’s only with certain data sources?
I believe that the fantasyfootballnerd.com projections are pulling from past seasons. I noticed this when I had Colin Kaepernick projected to play in Week 2! That sent me straight to google…
They are pulling from this link, which confirms that it is from past seasons:
http://www.fantasyfootballnerd.com/service/weekly-projections/json/test/QB/2
For now, I am excluding fantasyfootballnerd as a source!
Thanks again for all of your work on this – it is really an amazing piece of software!
Thanks for the update Peter
Everytime I use the sleepers tab in the table it says no data available. This has been going on since last week.
Love this package! Currently sitting at 1st place in my 12 team league and I think this is a major reason why!
I have run the data scrape (week = 0) a few times since the beginning of the season and it does appear to be updating projected totals. And I’ve tried running it for specific week projections and it appears to work for that as well. I was wondering if there was any way to run a scrape/projections for “Rest of Season”? I think this would be very useful when looking at potential adds/drops and trades.
Any plan to update the package to resolve the fantasy football nerds issue.
I get old data of Colin Kaepernick and RG3.
I tried to install the package from CRAN on my new 3.5.1 instance and got the below error. Do you plan on releasing a new version anytime soon?
> install.packages(“ffanalytics”)
Warning in install.packages :
package ‘ffanalytics’ is not available (for R version 3.5.1)
it appears that there’s a problem with projection_source for some sources.
I’m trying to scrape weekly projections with this code, and it’s not giving me all the sources:
my_scrape <- scrape_data(src = c("CBS", "ESPN", "FantasyData", "FantasySharks", "FFToday", "NFL", "Yahoo"),pos = c("QB","RB", "WR", "TE", "K", "DST"), season = 2019, week = 11)
I'm not getting projections from ESPN or FantasyData.
it looks like the base urls contained in projection_source may be wrong, but I don't have a current espn fantasy league so I can't fully test it.
Is there an easy fix?
I'm on r version 3.6.1, and I'm in v2.0.2.0001 for the ffanalytics package. I forced a fresh update earlier and the error is persisting.
nm, I see it’s on github.
I am getting this error when i follow the instructions above. I’ve got all the packages installed and libraries in my data. In R Markdown….all green.
https://www.cbssports.com/fantasy/football/stats/QB/2018/season/projections/nonppr
Error in make_df_colnames(data_table) : could not find function “make_df_colnames”
Hi guys! With the R package, I can only scrape data from FFToday. Does anyone have success with other sites? thx