Package rvest. R topics documented: August 29, Version Title Easily Harvest (Scrape) Web Pages

Size: px

Start display at page:

Download "Package rvest. R topics documented: August 29, Version Title Easily Harvest (Scrape) Web Pages"

Jack Bailey
6 years ago
Views:

1 Version Title Easily Harvest (Scrape) Web Pages Package rvest August 29, 2016 Wrappers around the 'ml2' and 'httr' packages to make it easy to download, then manipulate, HTML and XML. Depends R (>= 3.0.1), ml2 Imports httr (>= 0.5), selectr, magrittr Suggests testthat, knitr, png, stringi (>= 0.3.1), rmarkdown, covr Encoding UTF-8 License GPL-3 LazyData true VignetteBuilder knitr RoygenNote URL BugReports NeedsCompilation no Author Hadley Wickham [aut, cre], RStudio [cph] Maintainer Hadley Wickham <hadley@rstudio.com> Repository CRAN Date/Publication :57:12 R topics documented: encoding google_form html html_form html_nodes html_session

2 2 encoding html_table html_tet jump_to pluck session_history set_values submit_form Inde 13 encoding Guess and repair faulty character encoding. These functions help you respond to web pages that declare incorrect encodings. You can use guess_encoding to figure out what the real encoding is (and then supply that to the encoding argument of html), or use repair_encoding to fi character vectors after the fact. guess_encoding() repair_encoding(, from = NULL) from A character vector. The encoding that the string is actually in. If NULL, stringi These function are wrappers around tools from the fantastic stringi package, so you ll need to make sure to have that installed. # A file with bad encoding included in the package path <- system.file("html-e", "bad-encoding.html", package = "rvest") <- read_html(path) %>% html_nodes("p") %>% html_tet() guess_encoding() # Two valid encodings, only one of which is correct read_html(path, encoding = "ISO ") %>% html_nodes("p") %>% html_tet() read_html(path, encoding = "ISO ") %>% html_nodes("p") %>% html_tet()

3 google_form 3 google_form Make link to google form given id Make link to google form given id google_form() Unique identifier for form google_form("1m9b8dsynfydjpwsk6ur_bzf8rv_04ma3rmaabiveoui") html Parse an HTML page. html is deprecated: please use read_html() instead. html(,..., encoding = "") ## S3 method for class 'response' read_ml(,..., encoding = "", as_html = FALSE) ## S3 method for class 'session' read_ml(,..., as_html = FALSE) A url, a local path, a string containing html, or a response from an httr request.... If is a URL, additional arguments are passed on to GET(). encoding as_html Specify encoding of document. See iconvlist() for complete list. If you have problems determining the correct encoding, try stri_enc_detect Optionally parse an ml file as if it s html.

4 4 html_form # From a url: google <- read_html(" encoding = "ISO ") google %>% ml_structure() google %>% html_nodes("div") # From a string: (minimal html 5 document) # minimal <- read_html("<!doctype html> <meta charset=utf-8> <title>blah</title> <p>i'm the content") minimal minimal %>% ml_structure() # From an httr request google2 <- read_html(httr::get(" html_form Parse forms in a page. Parse forms in a page. html_form() A node, node set or document. See Also HTML 4.01 form specification: html_form(read_html(" html_form(read_html(" bo_office <- read_html(" bo_office %>% html_node("form") %>% html_form()

5 html_nodes 5 html_nodes Select nodes from an HTML document More easily etract pieces out of HTML documents using XPath and css selectors. CSS selectors are particularly useful in conjunction with it makes it easy to find eactly which selector you should be using. If you have t used css selectors before, work your way through the fun tutorial at html_nodes(, css, path) html_node(, css, path) css, path Either a document, a node set or a single node. Nodes to select. Supply one of css or path depending on whether you want to use a css or path 1.0 selector. html_node vs html_nodes html_node is like [[ it always etracts eactly one element. When given a list of nodes, html_node will always return a list of the same length, the length of html_nodes might be longer or shorter. CSS selector support CSS selectors are translated to XPath selectors by the selectr package, which is a port of the python cssselect library, It implements the majority of CSS3 selectors, as described in REC-css3-selectors /. The eceptions are listed below: Pseudo selectors that require interactivity are ignored: :hover, :active, :focus, :target, :visited The following pseudo classes don t work with the wild card element, *: *:first-of-type, *:last-of-type, *:nth-of-type, *:nth-last-of-type, *:only-of-type It supports :contains(tet) You can use!=, [foo!=bar] is the same as :not([foo=bar]) :not() accepts a sequence of simple selectors, not just single simple selector.

6 6 html_session # CSS selectors ateam <- read_html(" html_nodes(ateam, "center") html_nodes(ateam, "center font") html_nodes(ateam, "center font b") # But html_node is best used in conjunction with %>% from magrittr # You can chain subsetting: ateam %>% html_nodes("center") %>% html_nodes("td") ateam %>% html_nodes("center") %>% html_nodes("font") td <- ateam %>% html_nodes("center") %>% html_nodes("td") td # When applied to a list of nodes, html_nodes() returns all nodes, # collapsing results into a new nodelist. td %>% html_nodes("font") # html_node() returns the first matching node. If there are no matching # nodes, it returns a "missing" node if (utils::packageversion("ml2") > "0.1.2") { td %>% html_node("font") } # To pick out an element at specified position, use magrittr::etract2 # which is an alias for [[ library(magrittr) ateam %>% html_nodes("table") %>% etract2(1) %>% html_nodes("img") ateam %>% html_nodes("table") %>% [[ (1) %>% html_nodes("img") # Find all images contained in the first two tables ateam %>% html_nodes("table") %>% [ (1:2) %>% html_nodes("img") ateam %>% html_nodes("table") %>% etract(1:2) %>% html_nodes("img") # XPath selectors # chaining with XPath is a little trickier - you may need to vary # the prefi you're using - // always selects from the root noot # regardless of where you currently are in the doc ateam %>% html_nodes(path = "//center//font//b") %>% html_nodes(path = "//b") html_session Simulate a session in an html browser. Simulate a session in an html browser.

7 html_table 7 html_session(url,...) is.session() url Location to start session... Any additional httr config to use throughout session. Methods An object to test to see if it s a session. A session object responds to a combination of httr and html methods: use cookies(), headers(), and status_code() to access properties of the request; and html_nodes to access the html. # s <- html_session(" s %>% jump_to("hadley-wickham.jpg") %>% jump_to("/") %>% session_history() s %>% jump_to("hadley-wickham.jpg") %>% back() %>% session_history() s %>% follow_link(css = "p a") html_table Parse an html table into a data frame. Parse an html table into a data frame. html_table(, header = NA, trim = TRUE, fill = FALSE, dec = ".") header trim fill dec A node, node set or document. Use first row as header? If NA, will use first row if it consists of <th> tags. Remove leading and trailing whitespace within each cell? If TRUE, automatically fill rows with fewer than the maimum number of columns with NAs. The character used as decimal mark.

8 8 html_tet Assumptions html_table currently makes a few assumptions: No cells span multiple rows Headers are in the first row tdist <- read_html(" tdist %>% html_node("table.infobo") %>% html_table(header = FALSE) births <- read_html(" html_table(html_nodes(births, "table")[[2]]) # If the table is badly formed, and has different number of rows in # each column use fill = TRUE. Here's it's due to incorrect colspan # specification. skiing <- read_html(" skiing %>% html_table(fill = TRUE) html_tet Etract attributes, tet and tag name from html. Etract attributes, tet and tag name from html. html_tet(, trim = FALSE) html_name() html_children() html_attrs() html_attr(, name, default = NA_character_) trim name default A document, node, or node set. If TRUE will trim leading and trailing spaces. Name of attribute to retrieve. A string used as a default value when the attribute does not eist in every node.

9 jump_to 9 Value html_attr, html_tag and html_tet, a character vector; html_attrs, a list. movie <- read_html(" cast <- html_nodes(movie, "#titlecast span.itemprop") html_tet(cast) html_name(cast) html_attrs(cast) html_attr(cast, "class") jump_to Navigate to a new url. jump_to() takes a url (either relative or absolute); follow_link takes an epression that refers to a link (an <a> tag) on the current page. jump_to(, url,...) follow_link(, i, css, path,...) url A session. A URL, either relative or absolute, to navigate to.... Any additional httr configs to apply to this request. i css path You can select with: an integer selects the ith link a string first link containing that tet (case sensitive) Nodes to select. Supply one of css or path depending on whether you want to use a css or path 1.0 selector. Nodes to select. Supply one of css or path depending on whether you want to use a css or path 1.0 selector. s <- html_session(" s <- s %>% follow_link("github") s <- s %>% back() s %>% follow_link("readr")

10 10 session_history pluck Etract elements of a list by position. Etract elements of a list by position. pluck(, i, type) i type A list A string or integer. Type of output, if known session_history History navigation tools History navigation tools session_history() back() A session.

11 set_values 11 set_values Set values in a form. Set values in a form. set_values(form,...) Value form Form to modify... Name-value pairs giving fields to modify An updated form object search <- html_form(read_html(" set_values(search, q = "My little pony") set_values(search, hl = "fr") ## Not run: set_values(search, btni = "blah") submit_form Submit a form back to the server. Submit a form back to the server. submit_form(session, form, submit = NULL,...) session form submit Session to submit form to. Form to submit Name of submit button to use. If not supplied, defaults to first submission button on the form (with a message).... Additional arguments passed on to GET() or POST()

12 12 submit_form Value If successful, the parsed html response. Throws an error if http request fails. To access other elements of response, construct it yourself using the elements returned by submit_request. test <- google_form("1m9b8dsynfydjpwsk6ur_bzf8rv_04ma3rmaabiveoui") f0 <- html_form(test)[[1]] f1 <- set_values(f0, entry = "abc")

13 Inde Topic deprecated html, 3 back (session_history), 10 status_code, 7 stri_enc_detect, 3 submit_form, 11 cookies, 7 encoding, 2 follow_link (jump_to), 9 GET, 3, 11 google_form, 3 guess_encoding (encoding), 2 headers, 7 html, 3 html_attr (html_tet), 8 html_attrs (html_tet), 8 html_children (html_tet), 8 html_form, 4 html_name (html_tet), 8 html_node (html_nodes), 5 html_nodes, 5, 7 html_session, 6 html_table, 7 html_tet, 8 iconvlist, 3 is.session (html_session), 6 jump_to, 9 pluck, 10 POST, 11 read_ml.response (html), 3 read_ml.session (html), 3 repair_encoding (encoding), 2 session_history, 10 set_values, 11 13

Package rvest. R topics documented: February 20, Version Title Easily Harvest (Scrape) Web Pages

Package rvest. R topics documented: February 20, Version Title Easily Harvest (Scrape) Web Pages Version 0.2.0 Title Easily Harvest (Scrape) Web Pages Package rvest February 20, 2015 Wrappers around the XML and httr packages to make it easy to download, then manipulate, both html and ml. Depends R