Patrick Downes Rutgers University-New Brunswick School of Management and Labor Relations WEB SCRAPING FOR RESEARCH

Size: px

Start display at page:

Download "Patrick Downes Rutgers University-New Brunswick School of Management and Labor Relations WEB SCRAPING FOR RESEARCH"

Piers Harvey
5 years ago
Views:

1 Patrick Downes Rutgers University-New Brunswick School of Management and Labor Relations WEB SCRAPING FOR RESEARCH

2 Gauging our pace How would you rate your experience (1=a little, 3=a lot) with R? with web languages (e.g., HTML, CSS, PHP)? with computer programming more broadly (e.g., C++, Java, C#)? with syntax-driven statistics software (e.g., STATA, SAS)?

3 A (topical) 90-minute flurry A framework for web scraping Extract, Parse, Store A primer on R/Rstudio A worked example Strengths/Weaknesses of Scraping Ethics of web scraping With great power, comes great responsibility

4 Scraping Web Data The internet houses massive amounts of data in somewhat structured manner This could be human-coded, but is hard Some websites allow software to retrieve their data (using an API) Otherwise, software retrieving websites as a human would are called web scrapers

5 But first, a bit on how the internet works A user makes a request to a server (e.g., a URL) Sometimes this request is sent with data (e.g., GET protocol) The server issues some response (e.g., HTML) This response may also direct the user s machine to download other files (e.g., javascript, CSS, images) These files are processed by the user s browser into something the web developer intended

6 De-constructing a URL action=query&list=backlinks&bltitle=star%20 Wars&bllimit=10 REQUEST: = GET protocol Action=query List = backlinks Bltitle=Star%20Wars Bllimit=10

7 More Introduction to the Internet <HTML></HTML> markup The actual code displaying page Includes the text, images, etc to be presented Cascading Style Sheets (CSS) Used for formatting the text, images, etc Typically uses selectors to identify different elements and format them appropriately

8 An example

9 An example

10 An example

11 Application Programming Interfaces (API s) API s allow programmers to directly access the data and functions in a server Websites often require developers to register Dev s are constrained to using the functions provided in the API If there s an API, use it

12 Sometimes an API won t do When there is no API or it does not provide what we need, we can build software to emulate a human user This is highly idiosyncratic, as every website is different And sites vary in how far they go to prevent scrapers In part because there are less than ethical scrapers out there

13 Web Scraping Steps Steps for scraping the web Extract HTML Parse HTML Structure and Store

14 Extract HTML Get URLs Start with a URL/list of URLs (Mturk?) Crawl for URLs Visit page and download HTML text Using rvest, read_html(url)

15 Parse HTML Search HTML to get text, links, images, attributes you care about Rely primarily on CSS Selectors (i.e., selecting elements by their class or id) Can also use regular expressions (i.e., stringmatching) SelectorGadget is a helpful tool in this regard

16 Parse HTML <div id= personal_info class= field-name > <li class= contact-info > </li> <li class= contact-info > </li> </div> <div class= field-content > <p> Patrick Downes is a professor of HR </p> </div>

17 Structure and Store Establish some storage object In R, this is usually a data frame, vector, or list Can also write this out to a file for external analysis Organize information from each page into the object Uses loops, generally In R, the apply() family

18 R Resources Rob Kabacoff s Quick-R Datacamp R courses Rstats subreddit Rtutorial R-bloggers Resources

19 R Packages Collections of methods/functions Important functions (e.g., rvest) install.packages( rvest ) Installs the package library( rvest ) Loads the package into the workspace?read_html Loads the help file for the function

20 Talking through the Faculty Scrape Script

21 Ethics and Etiquette Whether scraping or using an API, Don t overload the server Store data rather than re-requesting it Pause your script to avoid DDOS Only take what you need Don t steal private data Create value with data Don t just duplicate it Scrape during off hours

22 When to scrape When the site permits it When there s no API When you need a little data from many pages Sports data (e.g., attendance, starters, weather) When pages take a similar format/same website When a login is not required (often greatly increases complexity)

23 Scraping Weaknesses Every website is different More advanced techniques are needed for less structured data Websites change over time Any change can break your script Prefer one-shot collections Can take time Both to develop the script And to run the script Savvy web developers will write code that makes scraping difficult (to block competitors)

ECPR Methods Summer School: Automated Collection of Web and Social Data. github.com/pablobarbera/ecpr-sc103

ECPR Methods Summer School: Automated Collection of Web and Social Data Pablo Barberá School of International Relations University of Southern California pablobarbera.com Networked Democracy Lab www.netdem.org