Web scraping tools, a real life application

Size: px

Start display at page:

Download "Web scraping tools, a real life application"

Robert Copeland
5 years ago
Views:

Web scraping tools, a real life application ESTP course on Automated collection of online proces: sources, tools and methodological aspects Guido van

1 Web scraping tools, a real life application ESTP course on Automated collection of online proces: sources, tools and methodological aspects Guido van den Heuvel, Dick Windmeijer, Olav ten Bosch, Statistics Netherlands THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2 Aim of this afternoon Learn about building web scrapers with the CBS Robot Framework Learn about web technology (HTML, CSS, XPath) Learn about the Robot Framework Introduce some useful tools for inspecting web sites Learn to configure and run the Robot Framework 2

3 Overview Introducing the Robot Framework Introducing web technologies Data extraction Coffee break Site navigation 3

4 The CBS Robot Framework Used for automated site navigation and data extraction Rule based configuration Does not require programming But: allows programming for advanced use Uses a full-blown browser (phantomjs) Works with rendered pages, not page source Includes a JavaScript engine Generates CSV data files and extensive logs. 4

5 Running the Framework Main directory: Documents/ESTP Webscraping Course Config directory: RobotConfig Commands: (in windows command prompt) newrobot <robotname> runrobot <robotname> Output directories: RobotOutput\<robotname>\data : inspect with Excel RobotOutput\<robotname>\log : inspect with notepad 5

6 Demo 1: "Hello, world!" Use newrobot to initialise a new robot Use runrobot to run the robot Inspect the output Data: no data yet, this robot does nothing Log: it starts, initialises and ends immediately I will use boohoo.com for my demos Boohoo: large online UK clothing retailer 6

7 Framework config Text file Edit with Brackets (or other code / text editor) Format: JSON (actually, a Node.js JavaScript module) Different sections starturls extractionrules navigationrules & backrules (and some others, which are for advanced use) 7

8 JSON quick reference name: value assign value to named property "string" number character string number { } object (group of properties) [ ] array of values See also: 8

9 Robotconfig: StartUrls One or more start URLs Start URL: web page that is used as a starting point for scraping and crawling Each start URL is a separate object Must have a unique name Must contain url property May contain extractioncontext and/or navigationcontext properties 9

10 Demo 2: start URL Choose a single page with data that we want to scrape as start URL Add this URL to starturls in the config Run the robot Inspect output Still no data Logging: we see that the robot visited the starting page 10

11 Extracting data 1. Decide which parts of the page to extract Items (products, news articles, forum posts,...) Properties of these items (description, price, author,...) 2. Inspect HTML code Tool: Firebug (Firefox plugin) 3. Formulate CSS selectors that select these items Tool: Firepath (Firebug add-on) 4. Incorporate these selectors in the robot config 11

12 One slide HTML introduction "Boxes within boxes" Tags Important tags: <a>, <p>, <h?>, <div>, <span>, <ul> / <li>, <table> / <tr> / <td>, <body>, <html> Text content Attributes id class 12

13 HTML Tags quick reference <a> Hyperlink <p> Paragraph <h?> Header.? is a single digit between 1 and 6 <div> Section; Generic rectangular block of content <span> (Part of a) line of text <ul> / <li> Unordered List / List item <table> Table <tr> / <td> Table row / Table cell <body> Document body: visible part of the page <html> The entire HTML document See also: 13

14 CSS selectors Originally used in Cascading Style Sheets to denote which tags have specific layout By means of the HTML structure, and/or in conjunction with the class attribute Use in scrapers to select items / properties HTML structure has meaning Example: item brand as descendant tag of item description Layout often has meaning Example: specific layout for page titles, author names, prices,... Class name often reflects this meaning 14

15 CSS Selectors quick reference tag #id.class [attr=value] tag.class Select tags with indicated tag name Select tag with indicated id Select tags with indicated class Select tags for which attribute equals value select tags with indicated tag name and class selector1 selector2 selector1>selector2 selector1,selector2 select tags obeying selector2 within tags obeying selector1 as previous, but children only select tags obeying selector1 or selector2 See also: 15

16 Demo 3: CSS selectors Using firebug / firepath: Formulate a CSS selector to select all items on the Zalando page chosen as the start URL Formulate a CSS selector to select the names of these items Exercise: formulate CSS selectors for item prices and item IDs. 16

17 Hidden features HTML code may contain data not shown to users But which is of interest to us (item ID, SKU, EAN,...) Inspect HTML code for this Since it is in the HTML, this data can be selected with CSS selectors, just like other data Discussion: should we use hidden data? 17

18 Robot config: extractionrules Contains one or more extraction rules blocks Each extraction rules block: Contains a CSS selector to select items Contains one or more properties to select, each with their own CSS selector Has a unique name, the extraction context Extraction context is used in starturls to denote which extraction rule block to use 18

19 Demo 4: Items of interest Add extraction context to robot config with the item selector devised in the previous demo Run robot Inspect output Data file with correct number of records, no data yet Log now shows number of items selected 19

20 Demo 5: Extracting Data Add Name property to extraction context, with CSS selector devised earlier Run robot Inspect output Data now contains a Name column with item names Log the same as before Repeat above steps for Price and Id 20

21 COFFEE BREAK "A programmer is a person who converts caffeine into code" 21

22 Site navigation overview Menus Top / Side menu: often hyperlinks Pulldown / mouseover menu: combination of CSS and JavaScript Multi-level menus Next page button Often implemented in JavaScript: AJAX Filters, facets Almost always implemented in JavaScript Links to individual items 22

23 Robot config: navigationrules Contains one or more navigation rules blocks navigation context: name of a navigation rules block Each navigation rules block contains one or more navigation rules Each navigation rule contains an "action" and a selector action: OPEN (also CLICK, HOVER, SCROLL) selector: XPATH selector (or CSS selector) Browser performs indicated action for each item selected 23

24 XPath selectors XPath: language to select tags in [X/HT]ML code Similar to CSS selectors, but much more powerful Syntax somewhat comparable to directory names HTML can be seen as a hierarchy, just like a file system Example: html/body/div/h1/a 24

25 XPath syntax overview /tag find tags as children of the current tag //tag find tags as descendants of current tag [n] select the nth tag of the indicated type [condition] select tags which obey the given select the indicated attribute of the current tag text() select the text contents of the current tag =,!= comparison operators: equal to / not equal to id('<id>') select the tag with the indicated id See:

26 XPath examples Select all hyperlinks with link text "Politics" inside a <ul> tag with class "nav2" //div[contains(@class, 'next')] Selects all <div> tags for which the class attribute contains the word "next" (id('main-menu')//ul/li)[3] First, select all <li> tags which are children of <ul> tags inside a tag with id "main-menu", then select the 3rd of these. 26

27 Demo 6: Next! Add navigation rule for next page button Add navigation context to start URL Run robot Inspect output Log: Now contains additional pages visited Data: Now has data from all pages in the result set Note: filter out tablet / mobile versions 27

28 Demo 7: Menu items Change start URL to go to menu page first Add navigation context for this page Add navigation rule to run through menu Run robot Inspect output Log: now visits multiple categories Data: now we have data from all these categories 28

29 Navigation variables Menu item clicked gives extra info on items Would be nice if this info stored in data Navigation variables store this menu info Very useful for classifying items 29

30 Demo 8: Tracking navigation Add navigation variable to config Run robot Inspect output Log: now logs presence of navigation variable Data: now contains extra column with menu info 30

Understanding this structure is pretty straightforward, but nonetheless crucial to working with HTML, CSS, and JavaScript.

Understanding this structure is pretty straightforward, but nonetheless crucial to working with HTML, CSS, and JavaScript. Extra notes - Markup Languages Dr Nick Hayward HTML - DOM Intro A brief introduction to HTML's document object model, or DOM. Contents Intro What is DOM? Some useful elements DOM basics - an example References