DSCI 325: Handout 27 Web scraping with R using APIs Spring 2017

Transcription

1 DSCI 325: Handout 27 Web scraping with R using APIs Spring 2017 In this handout, we will explore scraping data from the web (in R) using APIs. The information in this handout is largely based on the following tutorial prepared by Bradley Boehmke: What is an API? Many websites and companies allow programmatic access to their data via web APIs (API stands for application programming interface). Essentially, this gives a web-based database a way to communicate with another program (such as R). Each organization s API is unique, but you typically need to know the following for your specific data set of interest: URL for the organization and data you are pulling Data set you are pulling from Data content (so that you can specify variables you want the API to retrieve, you need to be familiar with the data library) In addition, you may also need the following: API key (i.e., API token). This is typically obtained by supplying your name/ to the organization. OAuth. This is an authorization framework that provides credentials as proof for certain information. The remainder of this handout will introduce these topics with the use of examples. Example: Pulling U.S. Bureau of Labor Statistics data No API key or OAuth is required, but a key is recommended. 1

2 To register for a key, visit this website: All available data sets from the Bureau of Labor Statistics and their series code information are available here: In this example, we will consider pulling Mass Layoff Statistics: Key information is contained in the series identifier. For the Mass Layoff data, the series ID code is MLUMS00NN The BLS provides this breakdown: Questions: 1. The S00 component represents the division/state from which data will be pulled. What happens if you leave this specified as S00? If you change it to D30? How would you pull for only Minnesota? 2. What if you want to pull data from only the Health care and social assistance industry? 2

3 The following link contains some answers to FAQs: To read this data into R from the web, you should make use of the following package. blsapi {blsapi} R Documentation Request Data from the U.S. Bureau Of Labor Statistics API Description Allows users to request data for one or multiple series through the U.S. Bureau of Labor Statistics API. Users provide parameters as specified in < and the function returns a JSON string or data frame. Usage blsapi(payload = NA, api.version = 1, return.data.frame = FALSE) Arguments payload a string or a list containing data to be sent to the API. api.version an integer for which api version you want to use (i.e. 1 for v1, 2 for v2) return.data.frame a boolean if you want to the function to return JSON (default) or a data frame. If the data frame option is used, the series id will be added as a column. This is helpful if multiple series are selected. install.packages("blsapi") library(blsapi) # Supply series identifier to pull data payload=list('seriesid'='mlums00nn ', 'registrationkey'='yourkey') layoffs = blsapi(payload,api.version=2,return.data.frame=true) > str(layoffs) 'data.frame': 29 obs. of 5 variables: $ year : chr "2013" "2013" "2013" "2013"... $ period : chr "M05" "M04" "M03" "M02"... $ periodname: chr "May" "April" "March" "February"... $ value : chr "1383" "1174" "1132" "960"... $ seriesid : chr "MLUMS00NN " "MLUMS00NN " "MLUMS00NN " "MLUMS00NN "... Task: Use the blsapi package to pull data from the National Employment, Hours, and Earnings data set. Pull the average weekly hours of all employees in the construction sector. 3

4 Data for a three-year period is returned by default. To pull data from a specified set of years, you can specify additional arguments: payload2=list('seriesid'='mlums00nn ', 'startyear'=2000, 'endyear'=2010, 'registrationkey'='yourkey') layoffs2 = blsapi(payload2,api.version=2,return.data.frame=true) Finally, note that you can also pull multiple series at a time: payload3=list('seriesid'=c('mlums00nn ','mlums27nn '), 'startyear'=2000, 'endyear'=2010, 'registrationkey'='yourkey') layoffs3 = blsapi(payload3,api.version=2,return.data.frame=true) 4

5 Example: Pulling NOAA Data The rnoaa package can be used to request data from the National Climatic Data Center (now known as the National Centers for Environmental Information) API. This package requires you to have an API key. To request a key, click here and provide your address. NOAA data set descriptions are available here: Description rnoaa is an R interface to NOAA climate data. Data Sources Many functions in this package interact with the National Climatic Data Center application programming interface (API) at all of which functions start with ncdc_. An access token, or API key, is required to use all the ncdc_ functions. The key is required by NOAA, not us. Go to the link given above to get an API key. More NOAA data sources are being added through time. Data sources and their function prefixes are: buoy_* - NOAA Buoy data from the National Buoy Data Center gefs_* - GEFS forecast ensemble data ghcnd_* - GHCND daily data from NOAA isd_* - ISD/ISH data from NOAA homr_* - Historical Observing Metadata Repository (HOMR) vignette ncdc_* - NOAA National Climatic Data Center (NCDC) vignette (examples) seaice - Sea ice vignette storm_ - Storms (IBTrACS) vignette swdi - Severe Weather Data Inventory (SWDI) vignette tornadoes - From the NOAA Storm Prediction Center argo_* - Argo buoys coops_search - NOAA CO-OPS - tides and currents data For example, let s start by pulling all weather stations in Winona County, MN, using Winona County s FIPS code. We will focus on the GHCND data set, which contains records on daily measurements such as maximum and minimum temperature, total daily precipitation, etc. install.packages("rnoaa") library(rnoaa) stations <- ncdc_stations(datasetid= GHCND, locationid= FIPS:27169, token ='yourkey') 5

6 To pull data from one of these stations, we need the station ID. Suppose we want to pull all available data from the Winona, MN US station. The following commands supply the data to pull, the start and end dates (you are restricted to a one-year limit), station ID, and your key. climate = ncdc(datasetid='ghcnd', startdate = ' ', enddate = ' ', stationid='ghcnd: USC , token = 'yourkey') > climate$data date datatype station value fl_m fl_q fl_so fl_t T00:00:00 PRCP GHCND:USC T00:00:00 SNOW GHCND:USC T00:00:00 SNWD GHCND:USC T00:00:00 TMAX GHCND:USC T00:00:00 TMIN GHCND:USC T00:00:00 TOBS GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 SNOW GHCND:USC T00:00:00 SNWD GHCND:USC T00:00:00 TMAX GHCND:USC T00:00:00 TMIN GHCND:USC T00:00:00 TOBS GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 SNOW GHCND:USC T00:00:00 SNWD GHCND:USC T00:00:00 TMAX GHCND:USC T00:00:00 TMIN GHCND:USC T00:00:00 TOBS GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 SNOW GHCND:USC T00:00:00 SNWD GHCND:USC T00:00:00 TMAX GHCND:USC T00:00:00 TMIN GHCND:USC T00:00:00 TOBS GHCND:USC T00:00:00 PRCP GHCND:USC

7 Next, let s pull data on precipitation for 2007 (note the use of the datatypeid argument). By default, ncdc limits the results to 25, but we can adjust the limit argument as shown below. precip = ncdc(datasetid='ghcnd', startdate = ' ', enddate = ' ', limit=365, stationid='ghcnd:usc ', datatypeid = 'PRCP', token = 'yourkey') Finally, we can sort the observations to see which days in 2007 experienced the greatest rainfall. precip.data=precip$data precip.data %>% arrange(desc(value)) date datatype station value fl_m fl_q fl_so fl_t T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC T00:00:00 PRCP GHCND:USC

8 Example Leveraging an Organization s API without an R Package In some situations, an R package may not exist to communicate with an organization s API interface. Hadley Wickham has developed the httr package to easily work with web APIs. It offers multiple functions, but in this example, we will focus on the use of the GET() function to access an API, provide some request parameters, and get output. Suppose we wanted to pull College Scoreboard data from the Department of Education. Though an R package does in fact exist to facilitate such a data pull, we will illustrate the use of the httr package, instead. Start by requesting a key: Data library: Query explanation: 8

9 Note that to run this query, we need only enter the appropriate URL in a web browser: If you wanted to obtain a csv file instead, change the URL as follows: name,2013.student.size&api_key=xejzxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx The above example retrieves data for a very specific query. What if we want all information collected by the U.S. Department of Education on Winona State University? We will use R to access this data set. install.packages("httr") library(httr) library(dplyr) URL <- " # import all available data for Winona State University wsu_req <- GET(URL, query = list(api_key = 'yourkey', school.name = "Winona State University")) This request will provide us with all information collected on Winona State University. To retrieve the contents, use the contents() function. This returns an R object (specifically a list). wsu_data = content(wsu_req) names(wsu_data) > names(wsu_data) [1] "metadata" "results" Note that the data is segment into metadata and results; we are interested in the results. names(wsu_data$results[[1]]) [1] "2008" "2009" "2006" "ope6_id" "2007" "2004" "2013" "2005" "location" 9

10 [10] "2014" "2002" "2003" "id" "1996" "1997" "schoo l" "1998" "2012" [19] "2011" "2010" "ope8_id" "1999" "2001" "2000" To see what data are available, we can look at a single year: names(wsu_data$results[[1]]$'2013') [1] "earnings" "academics" "student" "admissions" "repayment" "aid" "cost" [8] "completion" With such a large data set containing many embedded lists, we can explore the data by examining names at different levels: > names(wsu_data$results[[1]]$'2013'$cost) [1] "title_iv" "avg_net_price" "attendance" "tuition" "net_price" > names(wsu_data$results[[1]]$'2013'$cost$attendance) [1] "academic_year" "program_year" > names(wsu_data$results[[1]]$'2013'$cost$attendance$academic_year) NULL > wsu_data$results[[1]]$'2013'$cost$attendance$academic_year [1] > names(wsu_data$results[[1]]$'2013'$admissions) [1] "sat_scores" "admission_rate" "act_scores" > names(wsu_data$results[[1]]$'2013'$admissions$act_scores) [1] "midpoint" "25th_percentile" "75th_percentile" > names(wsu_data$results[[1]]$'2013'$admissions$act_scores$midpoint) [1] "math" "cumulative" "writing" "english" > names(wsu_data$results[[1]]$'2013'$admissions$act_scores$midpoint$"math") NULL > wsu_data$results[[1]]$'2013'$admissions$act_scores$midpoint$"cumulative" [1] 23 10

11 We can pull data collected on a certain variable over many years as follows: # subset list for annual student data only wsu_yr <- wsu_data$results[[1]][c(as.character(1996:2014))] # extract median cumulative ACT score data for each year wsu_yr %>% sapply(function(x) x$admissions$act_scores$midpoint$"cumulative") %>% unlist() #extract net price for each year wsu_yr %>% sapply(function(x) x$cost$avg_net_price$overall) %>% unlist() # extract median debt data for each year wsu_yr %>% sapply(function(x) x$aid$median_debt$completers$overall) %>% unlist() > # extract median cumulative ACT score data for each year > wsu_yr %>% + sapply(function(x) x$admissions$act_scores$midpoint$"cumulative") %>% + unlist() > #extract net price for each year > wsu_yr %>% + sapply(function(x) x$cost$avg_net_price$overall) %>% + unlist() > # extract median debt data for each year > wsu_yr %>% + sapply(function(x) x$aid$median_debt$completers$overall) %>% + unlist()