Web Scraping. Juan Riaza.!

Size: px

Start display at page:

Download "Web Scraping. Juan Riaza.!"

Shona Grant
5 years ago
Views:

1 Web Scraping Juan Riaza!

2 Who am I? So5ware Developer OSS enthusiast Pythonista & Djangonaut Now trying to tame Gophers Reverse engineering apps Hobbies: cooking and reading

3 CompeIng in a data-driven world

4 Data-driven world Web Scraping: Turn web content into useful data BeOer data leads to beoer decisions All decisions and processes should be dictated by data

5 Data sources # $ % Websites & RSS! ( ) * Documents + Open data (Datasets/APIs), Third party APIs

6 Let s understand the web Web pages are built using text-based markup languages: HTML Designed for human end-users to be accessed via a web browser not for the ease automated use Human-friendly design makes it difficult to access this data because it is unstructured

7 What is Web Scraping The main goal in scraping is to extract structured data from unstructured sources, typically, web pages.

8 What is an API AlternaIve user interface that so5ware uses to interact with other so5ware Difficult to build and maintain (+cost/effort)

9 Re APIs Most of the world hasn t embraced API-centric development Most of the world s interesing data isn t API accessible If you want to use this data, you need to use unconvenional tacics...

10 Re APIs

11 We can build a user facing API that works the way we want it to

12 Re websites SemanIc web Microdata (RFD)

13 Re websites SemanIc web Microdata (RFD) Just broken HTML

14 Some stats How many are rendered in quirk mode? ~ 85% What s more popular? TITLE or BODY? TITLE What percent validate in general? ~ 4.13% hgp://validator.w3.org

15 What for? - ". / ; < = >?@ Lead generaion Track online reviews Map users acivity Price monitoring Research data Financial data Data aggregaion

16 Lead generalon Clearbit Fullcontact

17 Lead generalon

18 Consumer Products Average Selling Prices, Market Share, and Sales Ranking for the Bestselling Products/Brands in a category Price-matching InflaIon Tracking

19 Pricing analylcs Brandview

20 Retail

21 Real Estate EsImate house prices, rental values, average house prices, and housing stock movements Provide macro indicators

22 Business intelligence (and Data viz)

23 Data viz

24 Data viz

26 Data journalism 'mariachis' Cómo encontramos a los de las Sicav

27 Data journalism hgp://populate.tools

28 Data journalism hgp://datahippo.org

29 Data viz pudding.cool fivethirtyeight.com

30 Scary Other stuff Memex

31 Scary Other stuff Might affect credit score

32 Scary Other stuff If you are not paying for it, you're not the customer; you're the product being sold

33 Your imaginaion is the limit

34 teller.io

37 Legal OperaIng a web crawler is legal Obey robots.txt: The Robots Exclusion Protocol Affects performance Site Terms of Use (ToS) ~ Intellectual Property / Copyright infringement

38 Legal

39 Visual tools import.io PorIa dexi.io but they are limited

40 Visual tools

41 Automated (Machine Learning)

42 pdfdata.io

43 OK, but How the internet works?

45 Do you speak HTTP?

46 RTFM Hypertext Transfer Protocol -- HTTP/1.1 RFC

47 How to make a HTTP request from scratch $ curl -v

48 Let s dissect a request Action Path GET / HTTP/1.1 Host: User-Agent : Curl Key Header Value

49 HTTP Methods GET POST PUT DELETE Create Read Update Delete

50 HTTP Status Codes 1 InformaIonal 2 Success 3 RedirecIon 4 Client Error 5 Server Error 999 Useful: hopstatuses.com

51 HTTP Headers Accept-Language User-Agent (again, RTFM) Cookies, persistence

52 Browser developer tools Firefox Developer Tools well it s all about Google Chrome developer tools hgps://developer.chrome.com/devtools

53 Useful extensions Quick Javascript Switcher Hola.org AB Tons of XPath helpers

54 HTTP for humans

55 Show me the code! import requests url = ' headers = {'User-Agent': 'riaza'} params = {'name': 'Juan Riaza', 'location': 'Vitoria-Gasteiz'} response = requests.get(url, headers=headers, params=params) html = response.text

56 Now we have a big chunk of html ideas?

57 HTML is not a regular language

58 How does the browser process this page? <html> <head> <meta name="viewport" content="width=device-width,initialscale=1"> <link href="style.css" rel="stylesheet"> <title>critical Path</title> </head> <body> <p>hello <span>web performance</span> students!</p> <div><img src="awesome-photo.jpg"></div> </body> </html>

60 CSS Selectors hgps://

61 XPath XPath is a language for addressing parts of an XML document A MUST have skill for accurate web data extracion More powerful than CSS Selectors: fine-grained look at the text content complex condiioning axes hgps://

62 Node types in a XPath tree Element node: represents an HTML element/tag <p> </p> AOribute node: represents an aoribute from an element node href="page.html" Comment node <! a comment > Text node: represents the text enclosed in an element node "Some title"

63 <html> <head> <title>my page</title> </head> <body> <h2>welcome to my <a href="#">page</a></h2> <p>this is the first paragraph.</p>  </body> </html> XPath overview

64 XPath overview

65 How I could parse HTML with Python?

66 HTML parsers lxml pythonic binding for the C libraries libxml2 and libxslt beaulfulsoup html.parser, lxml, html5lib

67 import requests from lxml.html import fromstring from urlparse import urljoin A complete example sess = requests.session() sess.headers.update({'user-agent': 'Mozilla/5.0...'}) products = [] def parse_products(tree): beers = tree.xpath('//ul[contains(@class, "itemslist")]/li') for beer in beers: title = beer.xpath('.//h3/a/text()')[0].strip() url = beer.xpath('.//h3/a/@href')[0] product_id = beer.xpath('.//input[@name="id"]/@value')[0] price = beer.xpath('.//span[@class="currencyprice"]/text()')[0] product = {'title': title, 'url': url, 'product_id': product_id, 'price': price} products.append(product) def parse_page(tree): parse_products(tree) next_page = tree.xpath('//ul[@class="pagination"]/li[@class="next"]/a/@href') if next_page: next_page = urljoin(' next_page[0]) response = sess.get(next_page) tree = fromstring(response.text) parse_page(tree) response = sess.get(' tree = fromstring(response.text) parse_page(tree) print(products, len(products))

68 An open source and collabora/ve framework for extrac/ng the data you need from websites. In a fast, simple, yet extensible way.

69 How does it looks like? import scrapy class BrewDogSpider(scrapy.Spider): name = 'brewdog_spider' start_urls = [' def parse(self, response): for product in self.parse_products(response): yield product next_page = response.xpath( '//ul[@class="pagination"]/li[@class="next"]/a/@href' ).extract_first() if next_page: url = response.urljoin(next_page) request = scrapy.request(url) yield request def parse_products(self, response): beers = response.xpath('//ul[contains(@class, "itemslist")]/li') for beer in beers: title = beer.xpath('.//h3/a/text()').extract_first().strip() url = beer.xpath('.//h3/a/@href').extract_first() product_id = beer.xpath('.//input[@name="id"]/@value').extract_first() price = beer.xpath('.//span[@class="currencyprice"]/text()').extract_first() product = {'title': title, 'url': url, 'product_id': product_id, 'price': price} yield product

70 BaGeries included Validating scraped data Checking for duplicates Storing on database Third parties integrations (google translate!) Built-in support for generating feed exports in multiple formats (JSON, CSV, XML) and storing them in multiple backends Backends FTP, S3, local filesystem

71 Deployment Scrapy Cloud

72 Avoid gebng banned Rotate your User Agent Disable cookies (if not needed) Randomize download delays Use a pool of rotaing IPs (scrapoxy.io) Pretend to be more human-like Use a commercial soluion: Crawlera luminai.io

73 Avoid gebng banned

75 How to protect against web scraping In-house implementaion DisIl networks Incapsula Fake data Unreachable data Easy communicaion (robots.txt)

76 I want to outsource it Specific scope In-house development Professional services Datasets on demand On-going costs (fix spiders, proxies, etc.)

77 La punta del iceberg

Index. Autothrottling,

Index. Autothrottling, A Autothrottling, 165 166 B Beautiful Soup, 4, 12 with scrapy, 161 Selenium, 191 192 Splash, 190 191 Beautiful Soup scrapers, 214 216 converting Soup to HTML text, 53 to CSV (see CSV module) developing