Web scraping and social media scraping handling JS

Transcription

1 Web scraping and social media scraping handling JS Jacek Lewkowicz, Dorota Celińska University of Warsaw March 28, 2018

2 JavaScript A typical problem What will we be working on today? Most of modern websites use JavaScript (JS) With JS the content of the website is generated dynamically Which may make scraping content impossible or significantly more difficult: 1 A part of website may not be rendered correctly 2 Access to some areas may be granted upon clicking a button

3 Convention JavaScript A typical problem In snippets, we will highlight in violet the areas where you may put your own content In commands, the areas in [] are optional UNIX-like systems use / as the path separator and DOS uses \. In this presentation the paths will be written in UNIX-like convention if not stated otherwise

4 JavaScript JavaScript A typical problem High-level dynamic, untyped interpreted run-time language One of the three core languages related to web development (the most popular language of GitHub!) Used to make dynamic webpages interactive and provide online programs, including video games

5 JavaScript A typical problem Problem getting blog content Let us assume that we want to collect titles of the blog s articles Looks easy! They are stored in a table. We have already done similar scrapers two weeks ago

6 Very basic spider JavaScript A typical problem import scrapy from scrapy import Request class exitem(scrapy.item): title = scrapy.field() class exspider(scrapy.spider): name = ex start_urls = [ ] def parse(self,response): for i in range(0,13): item = exitem() item[ title ] = response.xpath( //your/xpath/here/text() ).extract()[i] yield item

7 Output of scraper JavaScript A typical problem We do not extract anything... at all Let us debug the code, e.g. with scrapy shell! a blank page... but... it worked in the browser...

8 Solution 1 naive Open the website in a browser Save the source code of the website after the page is loaded Work on a local copy of the source code Pros: easy and sometimes may be a good workaround Cons: tedious with limited possibilities, also slow

9 Solution 2 PhantomJS headless browser has no graphical interface, that is where the name originated (user looks like a ghost) Pros: usually a good workaround Cons: limited possibilities, sometimes does not render correctly, suspended development

10 Solution 3 Selenium A web driver you may work with various browsers via your code Pros: a mature project, does not require human activity Cons: nearly none, but for some reasons not covered during the course

11 Scrapy + Selenium # example from import scrapy from selenium import webdriver class ProductSpider(scrapy.Spider): name = "product_spider" allowed_domains = [ ebay.com ] start_urls = [ def init (self): self.driver = webdriver.firefox() def parse(self, response): self.driver.get(response.url) while True: next = self.driver.find_element_by_xpath( //td[@class="pagn-next"]/a ) try: next.click() # get the data and write it to scrapy items except: break self.driver.close()

12 Solution 4 A monster child of Scrapy guys... Pros: relatively easy to use, does not require human activity, great Scrapy integration Cons: probably a lot

13 Installation pip install scrapy-splash Typically one works with an instance of in a docker docker run -p 8050:8050 scrapinghub/splash usually is enough

14 Configuration in settings.py 1 Add server address: SPLASH_URL = 2 Enable middleware DOWNLOADER_MIDDLEWARES = { scrapy_splash.cookiesmiddleware : 723, scrapy_splash.middleware : 725, scrapy.downloadermiddlewares.httpcompression.httpcompressionmiddleware : 810, } 3 Enable Spider middlewares SPIDER_MIDDLEWARES = { scrapy_splash.deduplicateargsmiddleware : 100, } 4 Set a custom Dupefilter Class DUPEFILTER_CLASS = scrapy_splash.awaredupefilter 5 for more options see

15 Adding to Scrapy code import scrapy from scrapy import Request class exitem(scrapy.item): title = scrapy.field() class exspider(scrapy.spider): name = ex start_urls = [ ] # A convenient way is to parse information about splash to start_requests metadata # this setup can be used in any project (always looks the same) def start_requests(self): for url in self.start_urls: yield scrapy.request(url, self.parse, meta={ splash :{ endpoint : render.html, args :{ wait :0.5,}}}) def parse(self,response): for i in range(0,13): item = exitem() item[ title ] = response.xpath( //your-xpath-here/text() ).extract()[i] yield item

16 Output with Scrapy-

17 Output of Scrapy shell scrapy shell