Making spreadsheets machine-readable. With Dave from ScraperWiki

Transcription

1 Making spreadsheets machine-readable With Dave frm ScraperWiki

2 Abut Me / Databaker David McKee n Twitter

3 Backgrund Databaker takes cmplicated published spreadsheets, and cnverts them int tabular CSV. By way f example we will feature a prject we have delivered fr the UK s Office f Natinal Statistics (ONS).

4 The Prblem in Jan-Mar 2012, there were 1.09 millin unemplyed wmen aged between 16 and 64 in the UK (figures nt seasnally adjusted) Hw d yu use this data?

5 Hw d we use this data? T difficult! Keep the cmplicated shape Labrius, manual prcess t cnvert Cmplex, errr-prne manual wrkflw Maybe it shuldn t have been in a spreadsheet t start with but t late nw!

6 What ges wrng? Ad-hc dwnstream mangling f data vs. apprved data in readily usable frmat Analysis based n what s easy, rather than what s imprtant Nt easy t imprt int existing tls t infrm decisins - slws decisin making

7 Aim: cnvert human-readable spreadsheets int smething truly machine readable

8 Beautiful, dangerus spreadsheets Fcussed n the printed page, nt data reuse Need t migrate away t smething else - genuinely tabular CSV is prtable Specifically, prtable t

9 Break the prblem dwn Identify cells which act as headers Want t be rbust against minr changes especially inserted rws Select each dimensin s headers in turn Identify cells which act as values May want t d this frm the intersectin f headers Determine crrect set f header cells fr each value Needs a simple but rbust rule

10 Identifying cells Bags f cells nn-cntiguus limited t a single sheet Hw t describe where they are? Gender: all the cells which are All Persns, Male r Female Date: all the cells belw All Persns, except blank, italic cells r the Gender cells we already described gender = tab.filter(ne_f(['male', 'Female', 'All Persns']) date = tab.filter( All Persns ).fill(down).nt_blank().nt_italic().difference(gender)

11 What abut the values? Nn-cntiguus Cntains unwanted numbers Belw indicatrs, right f dates values = indicatrs.waffle(dates)

12 Matching values t dimensin cells Gd practice - abve r t the left! Header either directly up/left f value r the clsest date.dimensin( date, DIRECTLY, LEFT) indicatr.dimensin( indicatr, DIRECTLY, ABOVE) gender.dimensin( gender, CLOSEST, ABOVE)

13 Selectrs Yu can nly select what yu can describe Select by text, frmatting, brders, fnt, fnt size, cell ref. Cmbine grups f cells set peratrs (unin, difference, intersectin) get cells belw this bag and right f anther bag Get ther cells nearby relative mvement (the cells ne beneath these) fill (all the cells t the right) All these transfrm a bag f cells int anther bag f cells

14 Previewing Check yur wrk!

15 Cmplicatins Headers in the wrng place - fake spans 66 needs t lk left, 77 right Real spans are nt a prblem can just use the tp-left cell can als get the whle span as a bag f cells

16 Recipes Scripts t cnvert spreadsheets Technical users wh dn t cnsider themselves prgrammers can write them All the pwer f Pythn

17 Bigger Picture Critical task f crrectly allcating headers dne Training critical - they need t write their wn recipes Lts f pst prcessing t get identifiers cnsistent and crrect Currently ONS-specific utput frmat, intend t change

18 Mre infrmatin Databaker is pen surce Databaker dcumentatin / dwnlads n Twitter

19 Thank yu! Wuldn t have happened withut: UN OCHA & ONS - spreadsheet prblems, Open Surce OKFN - libraries, rganising csv,cnf and yu! fr listening