Using Internet as a Data Source for Official Statistics: a Comparative Analysis of Web Scraping Technologies

Transcription

1 NTTS 2015 Session 6A - Big data sources: web scraping and smart meters Using Internet as a Data Source for Official Statistics: a Comparative Analysis of Web Scraping Technologies Giulio Barcaroli(*) (barcarol@istat.it), Monica Scannapieco (*) (scannapi@istat.it), Marco Scarnò (*) (m.scarnò@cineca.it), Donato Summa (*) (donato.summa@istat.it) (*) Istituto Nazionale di Statistica (Istat) (**) Consorzio Interuniversitario per il Calcolo Automatico (CINECA)

2 Web scraping definition and types Web scraping is the process of automatically collecting information from the World Wide Web, based on tools (called scrapers, internet robots, crawlers, spiders etc.) that navigate, extract the content of websites and store scraped data in local data bases for subsequent elaboration purposes. We can distinguish two different kinds of web scraping: 1. specific web scraping, when both structure and content of websites to be scraped are perfectly known, and crawlers just have to replicate the behaviour of a human being visiting the website and collecting the information of interest. Typical areas of application: data collection for price consumer indices (ONS, CBS, Istat); 2. generic web scraping, when no a priori knowledge on the content is available, and the whole website is scraped and subsequently processed in order to infer information of interest.

3 An application on «ICT in enterprises» survey

4 Web scraping different techniques and tools Different solutions for the web scraping are being investigated, based on the use of (i) the Apache suite Nutch/Solr ( for crawling, content extraction, indexing and searching results is a highly extensible and scalable open source web crawler; it facilitates parsing, indexing, creating a search engine, customizing search according to needs, scalability, robustness, and scoring filter for custom implementations; (ii) HTTrack ( a free and open source software tool that permits to mirror locally a web site, by downloading each page that composes its structure. In technical terms it is a web crawler and an offline browser; (iii) JSOUP ( permits to parse and extract the structure of a HTML document. It has been integrated in a specific step of the ADaMSoft system ( this latter selected as already including facilities that allow to handle huge data sets and textual information.

5 Web scraping solutions evaluation These techniques are evaluated by taking into account: 1. efficiency: number of websites actually scraped on the total and execution performance; 2. effectiveness: completeness and richness of collected text that can influence the quality levels of prediction.

6 Web scraping techniques evaluation: efficiency Solution # websites reached Average number of webpages per site Time spent Nutch 7020 / 8550=82,1% 15,2 32,5 hours Type of Storage Binary files on HDFS Storage dimensions 2,3 GB (data) 5,6 GB (index) HTTrack 7710 / 8550=90,2% 43,5 6,7 days HTML files on file system 16, 1 GB JSOUP 7835/8550=91,6% hours HTML ADaMSoft compressed binary files 500MB

7 Web scraping techniques evaluation: effectiveness The evaluation of the effectiveness of the different solutions is being based on the application of the steps of text and data mining to collected data in order to predict a subset of the target information of the survey. The developed application is available on the Adamsoft website: appscripts.html

8 Prediction of survey information by text and data mining Application of Naïve Bayes to predict all questions in section B8 Question B8:"indicate if the Website have any of the following facilities" a) Online ordering or reservation or booking (web sales functionality) Precision Performance of Naive Bayes Sensitivity Specificity Observed proportion Predicted proportion b) Tracking or status of orders placed c) Description of goods or services, price lists d) Personalized content in the website for regular/repeated visitors e) Possibility for visitors to customize or design online goods or services f) A privacy policy statement, a privacy seal or a website safety certificate g) Advertisement of open job positions or online job application

9 Web scraping: from sample to whole population So far, the three different solutions for web scraping have been applied to a limited number of websites (related to the subset of enterprises respondents in the sampling survey and declaring to have a website: 8,600). Next step is the scraping of all the websites owned by the enterprises included to the population of interest (212,000). Two problems: 1. URLs retrieval: how to individuate all the websites owned by the 212,000 (between 90,000 and 100,000 are expected to own one website); 2. massive scraping: how to increase efficiency when scaling a factor 10: O(10^4) O(10^5)

10 General idea: for each enterprise: Web scraping: URLs retrieval 1. Querying search engines with the enterprise denomination 2. Processing the first ten URLs retrieved in order to choose the right one for the given enterprise Processing: a) matching of the enterprises information (denomination, fiscal code, etc. available from administrative data) and the content of the first ten URLs retrieved; b) use of the subset of enterprises (from survey data) for which the correct URL is known, as a training set in order to maximise the precision of the choice function; c) application of the choice function to the whole set.

11 Web scraping: mass scraping Use of Nutch on top of MapReduce / Hadoop to harness parallelism Completed tasks: enhancement of Nutch by using the following plugins: HTML-Plugin (Nutch custom search) to retrieve HTML tags Metatag plugin (urlmeta) to add custom metatag information integration of Nutch with analysis activities in order to execute the whole process Future task: deployment and execution of Adamsoft/JSOUP and Nutch (HTTrack is abandoned due to its scalability problems) on CINECA PICO platform (1,080 cores, 54 nodes, 6.9 TB RAM)

12 Conclusions 1. A first remark is that a scraping task can be carried out for different purposes in an Official Statistics production environment, and the choice of a unique tool for all the purposes may not always be possible. 2. As for this specific case, the final evaluation of the different solutions will depend on the evaluation of the results of their execution for massive scraping on an adequate platform (PICO). 3. Finally, we highlight that the scraping application here presented is a sort of generalized scraping task, as it does not require any specific assumption on the structure of the websites. In this sense it goes a step further with respect to previous experiences.