Index. Autothrottling,

Transcription

1 A Autothrottling, B Beautiful Soup, 4, 12 with scrapy, 161 Selenium, Splash, Beautiful Soup scrapers, converting Soup to HTML text, 53 to CSV (see CSV module) developing long run cache intermediate step results, 90 database cache, 92 file-based cache, 92 saving space, 93 updating cache, 94 exporting data JSON files, NoSQL database, relational database, saving class, saving dictionary, extracting all images, 46 extracting all links, extracting required information, 53 navigating product pages, target URLs, using classes, 62 using dictionaries, find and find_all, 45 finding comments, 52 finding tags on property, 48 finding tags through attributes, installing, 41 nutrition table, parsing file, 45 parsing HTML text, parsing remote HTML, 44 performance improvements changing parser, 86 parse only needed, saving while working, source code, 95 tags and attributes adding, changing, deleting, 51 unforeseen changes, Breadth First Search (BFS), 56 builtwith library, 7 8 Gábor László Hajba 2018 G. L. Hajba, Website Scraping with Python, 219

2 C Caching, scrapy DBM storage, 155 default, dummy policy, 156 file system storage, 155 HTTP options, LevelDB storage, 156 RFC2616 policy, 157 Chrome Developer Tools, see DevTools Cookies, CSV file contents, feed exporter file format, 150 mycsv, 153 truncate() method, 152 item pipeline, 147, 149 CSV module headers, 68 line endings, 68 quick glance, D, E DBM storage, 155 Depth First Search (DFS), 56 DevTools definition, 8 9 website scrapers, 9 11 Digital transformation, 2 Dummy policy, 156 F, G, H Feed exporter file format, 150 mycsv, 153 truncate() method, 152 File system storage, 155 I Image extraction, J JSON file, K Kayak.com, L LevelDB storage, 156 Link extractor, M, N, O Meat & fish department, 23 Middlewares, MongoDB, 83 database, installing, 83 writing to,

3 P, Q Parse method, Parsing robots.txt, Pipelines, 103 Portia tools, 12 Protopage.com, PythonAnywhere, 203 configuration, 204 running the script, script, 203 script manually, storing data in database, uploading script, R Requests library, 36 Reverse engineering kayak.com, search expressions, 172 RFC2616 policy, 157 S, T, U, V Sainsbury scraper allowed_domains, 107 checklist, 108 CSV file (see CSV file) database MongoDB, SQLite, 140 downloading images, 158, 160 duplicate filter, extensions, 104 extracting information, 118, 120 genspider command, 106 items dictionary-like objects, 127 dropping, flat class, 124 parse_product_detail method, 123, 125 static imports, 124 JSON file, middlewares, navigation category pages, product listing pages, 116 parse method, pipelines, 103 project structure, 99 robots.txt file, 100 ROBOTSTXT_OBEY property, 100 selectors, settings.py file, 101 spider, 127 start_urls variable, 107 USER_AGENT property, 100 using shell, Sainsbury s Halloween 2017 Beef category, country of origin, 30 detailed product page, image s HTML code, 29 landing page,

4 Sainsbury s Halloween 2017 (cont.) Meat & fish department, navigation websites BFS and DFS code, 33, graph, 32 HTML content, 37 installation, 36 link extraction, Requests library, 36 search algorithms, 35 nutrition details, 20 nutrition information, 30 unordered list class pages, productlister class, 27 productnameandpromotions class, 27 Roast dinner option, 25 robots.txt file, Sainsbury s scraper to Splash, , 182 ScrapingHub, 194, 203 Scrapy autothrottling feature, caching (see Caching, scrapy) concurrent requests, 164 cookies, download delay, 164 framework, 4 logging, 162 log level, 163 scrapy-selenium, with Selenium, with Splash, tool, installing, 98 using Beautiful Soup, 161 Scrapy Cloud accessing data, API, 200, 202 creating project, deploying spider, limitations, 202 start and wait, Selectors, Selenium Beautiful Soup, installation, 184 integration with scrapy, Sainsbury s website, 185 scrapy-selenium, Selenium tools, 12 Splash Beautiful Soup, converting Sainsbury s scraper, , 182 drawback, 183 error message, 183 install Docker, 173 integration with scrapy, protopage.com, Sainsbury s, 174 welcome screen, 174 with source code, SQLite database,

5 W, X, Y, Z Web drivers, 184 Website scraping Beautiful Soup scrapers, layout, 3 preparation steps robots.txt, 6 terms and conditions, 5 website technologies, 7 8 PythonAnywhere, 203 configuration, 204 running the script, script, 203 script manually, storing data in database, uploading script, Requests library, 4 Scrapy Cloud, 193 accessing data, API, 200, 202 creating project, deploying spider, limitations, 202 start and wait, WordPress, 2 223