Web scraping job vacancies

Transcription

1 Web job vacancies (ESSnet on Big Data - Work package 1) Frantisek (Fero) Hajnovic frantisek.hajnovic@ons.gov.uk Big data team

2 Outline Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

3 Outline Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

4 Outline Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

5 Outline Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

6 Outline Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

7 Outline Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

8 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

9 Proof of concept P.O.C. on random sample of 50 (large) companies Survey vs. Company websites vs. Job portals Survey CW Indeed... Tesco HSBC

10 Useful quick insights Which portal is better? Boots problem Gap: survey - online

11 pros and cons + Quick and simple scrapers + Entries already linked (matched) + Lightweight (less risk) - at least for small sample - Sample bias - Effort to increase the sample

12 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

13 Full-size Directory Proper spiders Careerjet CV-library Universal job match T&Cs / robots.txt + Lot of data + Not influenced by sample - More risky Need to match

14 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

15 Matching company names company company name JV count Careerjet Milton Keynes Borough Council 25 MILTON KEYNES COUNCIL INCL EDUCATION EXCL SCHOOLS WITH EXTERNAL PAYROLL PROVIDERS 34 Milton Keynes council Survey Casing, stop-words, (TF-)IDF scores, INCL/EXCL 434 entries matched (3.7%)

16 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

17 Scraping company websites One website, one spider - no problem 50 websites

18 Specific spider code Name and rep. unit URL Extraction XPath Regex pattern

19 Scraping company websites Type of access to the relevant HTML Simple HTTP. E.g. Caring homes Selenium. E.g. Care UK Obtaining count Direct count. E.g. Caring homes Counting vacancies. E.g. University of Portsmouth Pagination Not necessary. E.g. Caring homes Necessary. E.g. Somerset county

20 Scraping company websites

21 Scraping company websites

22 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

23 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

24 Project architecture Python project Spiders (scrapy, xpath, regex, beautiful soup) Full-size 7... Bric k Car eer ed jet PSB (portal sample-based) Ind e UJM rary -lib CV Car eer jet PFS (portal full-size) Sample-based Tests, scripts, notebooks... (nose, bash, jupyter,...) fhajnovic.ons@gmail.com CW (comp. websites) (selenium) (sample of 50 companies) er (mailjet)

25 s from

26 Deploying project Python project Google cloud Spiders (scrapy, xpath, regex, beautiful soup) Managing instance Full-size Sample-based 7... Bric k Car eer ed jet PSB (portal sample-based) Ind e UJM rary -lib CV Car eer jet PFS (portal full-size) Deploy (bash) CW (comp. websites) (selenium) (sample of 50 companies) Run (Cron-job) 24 h Turn on/off, Store data Mongo DB instance Tests, scripts, notebooks... (nose, bash, jupyter,...) fhajnovic.ons@gmail.com er (mailjet)

27 Technologies used

28 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

29 Visualise the data Python project Google cloud Spiders (scrapy, xpath, regex, beautiful soup) Managing instance Full-size Sample-based 7... Bric k Car eer ed jet PSB (portal sample-based) Ind e UJM rary -lib CV Car eer jet PFS (portal full-size) Deploy (bash) CW (comp. websites) (selenium) (sample of 50 companies) Run (Cron-job) 24 h Turn on/off, Store data Mongo DB instance Tests, scripts, notebooks... (nose, bash, jupyter,...) fhajnovic.ons@gmail.com er (mailjet) Visualise Dashboard (flask, bokeh, js)

30 Dashboard

31 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

32 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

33 Scatter plot with best fit

34 Bland-Altman plot with KDE

35 BA-plots side by side

36 Krippendorff s alpha inter-rater agreement <-1, 1> 1 = perfect agreement 0 = absence of reliability -1 = systematic disagreement K.A. = 0.755

37 Comparing portals

38 Sample based Full-size Company names matching Automated framework Scraping company websites Dashboard To-do Comparisons

39 Nowcasting 1 month Survey data train 1 day train... In total Per industry Per company predict... train train... Scraped data

40 Nowcasting survey entry Possible model inputs Scraped values Previous survey values Company parameters Industry (dummy 0-1 coding) Outlying factor Possibly lots of training data 6k entries in survey monthly Portal 1 Portal 2 Portal n For company X at time t Comp. website Survey(t-1) Survey(t-2) Survey(t-k) Regression (neural network?) Employee size Industry 1 Industry 2 Industry m Outlying factor Survey(t)

41 Scale up and expand! Why not? New FS spider 1-3 days New SB spider 1-3 days + sample New CW spider 10 minutes Sample 100 Improve matching Data from partners

42 The deadly triangle

43 thanks! Questions?