STAT 408. Data Scraping and SQL STAT 408. Data Scraping SQL. March 8, 2018

Size: px

Start display at page:

Download "STAT 408. Data Scraping and SQL STAT 408. Data Scraping SQL. March 8, 2018"

Rosemary Rice
6 years ago
Views:

1 and and March 8, 2018

2 and

3 and scraping is defined as using a computer to extract information, typically from human readable websites. We could spend multiple weeks on this, so this will be a basic introduction that will allow you to: extract text and numbers from webpages and extract tables from webpages.

4 A bit about HTML and HTML elements are written with a start tag, an end tag, and with the content in between: content. The tags which typically contain the textual content we wish to scrape. Some tags include: < h1 >, < h2 >,...,: for headings < p >: Paragraph elements < ul >: Unordered bulleted list < ol >: Ordered list < li >: Individual List item < div >: Division or section < table >: Table

5 HTML Example and Figure 1: MSU website

6 with rvest and library(rvest) library(stringr) msu.math <- read_html(" msu.math ## {xml_document} ## <html lang="en-us"> ## [1] <head>\n<meta http-equiv="content-type" content ## [2] <body class="responsive">\n <header class=

7 with rvest and msu.math %>% html_nodes('h1') ## {xml_nodeset (1)} ## [1] <h1>\n \t\tdepartment of Mathematic msu.math %>% html_nodes('h1') %>% html_text() ## [1] "\n \t\tdepartment of Mathematical

8 Tidying Up and msu.math %>% html_nodes('h1') %>% html_text() %>% str_replace_all("\\s+", " ") %>% str_replace_all(pattern = "\n", replacement = "") %>% str_replace_all(pattern = "\t", replacement = "") ## [1] " Department of Mathematical Sciences "

9 h3 and msu.math %>% html_nodes('h3') %>% html_text() %>% str_replace_all("\\s+", " ") %>% str_replace_all(pattern = "\n", replacement = "") %>% str_replace_all(pattern = "\t", replacement = "") ## [1] "Faculty" "Undergraduate" "News" ## [4] "Events" "More Information" "Resources" ## [7] "Follow Us"

10 A River Runs Through It and Figure 2: IMDB: A River Runs Through It

11 Get Story line and river <- read_html(" story.line <- river %>% html_nodes('#titlestoryline') %>% html_nodes('p') %>% html_text() %>% str_replace_all(pattern = "\n", replacement = "") The storyline is : The Maclean brothers, Paul and Norman, live a relatively idyllic life in rural Montana, spending much of their time fly fishing. The sons of a minister, the boys eventually part company when Norman moves east to attend college, leaving his rebellious brother to find trouble back home. When Norman finally returns, the siblings resume their fishing outings, and assess both where they ve been and where they re going. Written byjwelch5742.

12 SelectorGadget for accessing actors and Figure 3: Using SelectorGadget

13 Actors and # Get Actors river %>% html_nodes('#titlecast') %>% html_nodes(".itemprop span") %>% html_text() ## [1] "Craig Sheffer" "Brad Pitt" "Tom Skerritt" ## [4] "Brenda Blethyn" "Emily Lloyd" "Edie McClurg" ## [7] "Stephen Shellen" "Vann Gravage" "Nicole Burdette" ## [10] "Susan Traylor" "Michael Cudlitz" "Rob Cox" ## [13] "Buck Simmonds" "Fred Oakland" "David Creamer"

14 Selecting Tables: baseball data and Figure 4: HTML Table

15 Tables and batting <- read_html(" batting.list <- batting %>% html_nodes('table') %>% html_table() batting.df <- tbl_df(batting.list[[1]]) kable(batting.df) Tm #Bat BatAge R/G G PA AB R H 2B 3B HR ARI ATL BAL BOS CHC CHW CIN CLE COL DET HOU KCR LAA LAD MIA MIL MIN NYM NYY OAK PHI PIT SDP

16 Exercise: Get Team Info and Visit the baseball reference website for the Colorado Rockies https: // and scrape a table or text.

17 Solution: Get Team Info and batting.co <- read_html(" tables.co <- batting.co %>% html_nodes('table') %>% html_table() tbl_df(tables.co[[1]]) ## # A tibble: 48 x 28 ## Rk Pos Name Age G PA AB R H `2B` ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> ## 1 1 C Tony Wolters* ## 2 2 1B Mark Reynolds ## 3 3 2B DJ LeMahieu ## 4 4 SS Trevor Story ## 5 5 3B Nolan Arenado ## 6 6 LF Gerardo Parra* ## 7 7 CF Charlie Blackmon* ## 8 8 RF Carlos Gonzalez* ## 9 Rk Pos Name Age G PA AB R H 2B ## 10 9 UT Ian Desmond ## #... with 38 more rows, and 18 more variables: `3B` <chr>, HR <chr>, ## # RBI <chr>, SB <chr>, CS <chr>, BB <chr>, SO <chr>, BA <chr>, ## # OBP <chr>, SLG <chr>, OPS <chr>, `OPS+` <chr>, TB <chr>, GDP <chr>, ## # HBP <chr>, SH <chr>, SF <chr>, IBB <chr>

18 and

19 ite and For this class we will use ite which enables users to store database files locally, but the principles are the same for querying a server-based database. We will use a European soccer database available at which can be downloaded with the following link: com/hugomathien/soccer/downloads/database.sqlite

20 Accessing base and library(dbi) library(rite) ## connect to a database WHICH IS STORED LOCALLY my.database <- dbconnect(ite(), dbname="~/google Drive/teaching/STAT408/data/database.sqlite") dblisttables(my.database) ## [1] "Country" "League" "Match" ## [4] "Player" "Player_Attributes" "Team" ## [7] "Team_Attributes" "sqlite_sequence" dbdisconnect(my.database)

21 Identifying fields in table and my.database <- dbconnect(ite(), dbname="~/google Drive/teaching/STAT408/data/database.sqlite") dblistfields(my.database, "Player") ## [1] "id" "player_api_id" "player_name" ## [4] "player_fifa_api_id" "birthday" "height" ## [7] "weight"

22 commands and The most basic queries have the following structure: SELECT var1name, var2name (filter columns) FROM tablename (identify table) WHERE condition1 (filter rows) GROUP_BY var3name (aggregate data) HAVING condition2 (filter aggregated data) ORDER_BY var (arrange ordering)

23 Query 1 and Select all columns for player and view first 5 rows. kable((dbgetquery(my.database,"select * FROM Player"))[1:5,]) id player_api_id player_name player_fifa_api_id birthday height weigh Aaron Appindangoye :00: Aaron Cresswell :00: Aaron Doran :00: Aaron Galindo :00: Aaron Hughes :00:

24 Query 2 and Retain player name, weight, height for players over 200 cm kable(dbgetquery(my.database,"select player_name, height, weight FROM Player WHERE height > 200" player_name height weight Abdoul Ba Asmir Begovic Bogdan Milic Costel Pantilimon Daniel Burn Danny Wintjens Fejsal Mulic Fraser Forster Jurgen Wevers Kevin Vink Konrad Jalocha Kristof van Hout Lacina Traore Nikola Zigic Paolo Acerbis Peter Crouch Pietro Marino Robert Jones Stefan Maierhofer Vanja Milinkovic-Savic Wojciech Kaczmarek Zeljko Kalac

25 Query 3 and Compute average weight for players of 200 cm dbgetquery(my.database,"select AVG(weight) as mean_weight FROM Player WHERE height > 200") ## mean_weight ##

26 Create a database and new.db <- dbconnect(rite::ite(), ":memory:") dblisttables(new.db) ## character(0) player <- tbl_df(dbgetquery(my.database,"select * FROM Player")) dbwritetable(new.db, "player", player) dblisttables(new.db) ## [1] "player" dbdisconnect(new.db) dbdisconnect(my.database)

27 Additional and also has functionality for merging and updating tables. See the cheat sheet for more details.

28 Exercise and Select the average goals scored in matches in different countries from the match table

29 Solution and Select the average goals scored in matches in different countries from the match table my.database <- dbconnect(ite(), dbname="~/google Drive/teaching/STAT408/data/database.sqlite") st <- "SELECT AVG(home_team_goal + away_team_goal) as total_goals, country_id FROM match GROUP by country_id" dbgetquery(my.database,st) ## total_goals country_id ## ## ## ## ## ## ## ## ## ## ## dblisttables(my.database) ## [1] "Country" "League" "Match" ## [4] "Player" "Player_Attributes" "Team" ## [7] "Team_Attributes" "sqlite_sequence"

30 Solution and Select the average goals scored in matches in different countries from the match table st2 <- "Create Table goals as SELECT AVG(home_team_goal + away_team_goal) as total_goals, country_id FROM match GROUP by country_id " dbgetquery(my.database,st2) ## Warning in rsqlite_fetch(res@ptr, n = n): Don't need to call dbfetch() for ## statements, only for queries ## data frame with 0 columns and 0 rows dbgetquery(my.database, "SELECT * from goals INNER JOIN country on goals.country_id = country.id ## total_goals country_id id name ## Belgium ## England ## France ## Germany ## Italy ## Netherlands ## Poland ## Portugal ## Scotland ## Spain ## Switzerland

HW 6+7 Advanced: CS 110X C 2013

HW 6+7 Advanced: CS 110X C 2013 Note: This homework (and all remaining homework assignments) is a partner homework and must be completed by each partner pair. When you complete this assignment, you must