TOOLS & TECHNIQUES FOR WORKING WITH DATA

Transcription

1

2 TOOLS & TECHNIQUES FOR WORKING WITH DATA

3 TL;DR 1. There are many tools & techniques to work with data. 2. Know about alternatives and try them. 3. Not every data management task is the same. 4. You might need different tools for separate parts of larger tasks. 3

4

5 1. Tools we will use working with our example data csvkit a. csvcut b. csvstat c. csvsql d. csvsort e. in2csv Unix/Mac commands a. wc b. head c. pbcopy d. piping DB Browser for SQLite 5

6

7

8 A brief ode to CSV: Possibly the most widely supported structured data format in the world. One of the simplest possible structured formats for data. Strikes a delicate balance, remaining readable by both machines & humans. Source: 8

9 csvkit csvkit is a suite of command-line tools for converting to and working with CSV, the king of tabular file formats. It is inspired by pdftk, gdal and the original csvcut tool. 9

10 Row count ~ fulcrum-live $ wc -l Fire_Inspections.csv Fire_Inspections.csv wc (short for word count) is a Unix-like command. wc -l prints the line count csvstat --count outputs total row count. $ csvstat --count Fire_Inspections.csv Row count:

11 csvstat -n, --names Display column names and indices. ~ fulcrum-live $ csvstat -n Fire_Inspections.csv 1: Inspection Number 2: Inspection Type 3: Inspection Type Description 4: Address 5: Inspection Address Zipcode 6: Battalion 7: Station Area 8: Fire Prevention District 9: Billable Inspection 10: Inspection Start Date 11: Inspection End Date 12: Inspection Status 13: Return Date 14: Corrective Action Date... 11

12 ~ fulcrum-live $ head -n 101 Fire_Inspections.csv > subset.csv Preview a subset $ csvstat --count subset.csv Row count: 100 head is a program on Unix-like systems used to display the beginning of a text file. 12

13 Preview the data Can also be done in command line with csvlook. 13

14 Heads up! We ll need to split this column into separate column for latitude and longitude. 14

15 ~ fulcrum-live csvstat Prints descriptive statistics for all columns in a CSV file. Will intelligently determine the type of each column and then print analysis. $ csvstat Fire_Inspections.csv 1. "Inspection Number" Type of data: Contains null values: Unique values: Smallest value: Largest value: Sum: Mean: Median: StDev: Most common values: Number False ,298 29,621,444, , ,958 89, ,121 (1x) 234,119 (1x) 15

16 ~ fulcrum-live 9. "Billable Inspection" csvstat csvstat returning Boolean and Date types. Type of data: Contains null values: Unique values: Most common values: Boolean False 2 False (202888x) True (10321x) 10. "Inspection Start Date" Type of data: Contains null values: Unique values: Smallest value: Largest value: Most common values: Date False (661x) 16

17 Gotcha! Upon import of our data subset, we discovered that the date format was not correct for import into Fulcrum. Don t worry, csvkit has got your back! 17

18 ~ fulcrum-live 14. "Neighborhood District" csvstat csvstat returning a Text type and listing the most common values. Type of data: Contains null values: Unique values: Longest value: Most common values: Text True characters Financial District (2034x) Tenderloin (727x) South of Market (644x) Mission (622x) Nob Hill (525x) 18

19 ~ fulcrum-live $ csvsql --query 'SELECT DISTINCT "Neighborhood District" FROM "Fire_Inspections";' Fire_Inspections.csv csvsql Run SQL queries directly on your CSV!! Mission Lone Mountain/USF Noe Valley Haight Ashbury Nob Hill Lakeshore Tenderloin Russian Hill Chinatown Mission Bay Financial District/South Beach South of Market 19

20 ~ fulcrum-live Putting it all together $ csvcut -C 2,13-24,26-32 Fire_Inspections.csv > fire-inspections-subset.csv $ csvsort -r -c 9 fire-inspections-subset.csv head -n in2csv -f csv > fire-inspections-subset-sorted.csv 20

21 ~ fulcrum-live Putting it all together $ csvcut -C 2,13-24,26-32 Fire_Inspections.csv > fire-inspections-subset.csv $ csvsort -r -c 9 fire-inspections-subset.csv head -n in2csv -f csv > fire-inspections-subset-sorted.csv 21

22 ~ fulcrum-live Putting it all together $ csvcut -C 2,13-24,26-32 Fire_Inspections.csv > fire-inspections-subset.csv $ csvsort -r -c 9 fire-inspections-subset.csv head -n in2csv -f csv > fire-inspections-subset-sorted.csv 22

23 ~ fulcrum-live Putting it all together $ csvcut -C 2,13-24,26-32 Fire_Inspections.csv > fire-inspections-subset.csv $ csvsort -r -c 9 fire-inspections-subset.csv head -n in2csv -f csv > fire-inspections-subset-sorted.csv 23

24

25 DB Browser for SQLite csvkit could not do it all (for me), so we turn to the SQLite Browser tool for our last task. 25

26 DB Browser for SQLite Steps in SQLite Browser: Create new database Import CSV Create new columns for latitude & longitude Execute UPDATE statement UPDATE "fire-inspections-subset-sorted" SET lat = replace(substr(location, 1, instr(location, ', ') - 1), '(', ''), lon = replace(substr(location, instr(location, ', ') + 1), ')', ''); Export CSV containing our lat & lon columns Import into Fulcrum Profit! 26

27 Location column, sample value UPDATE "fire-inspections-subset-sorted" SET lat = replace(substr(location, 1, instr(location, ', ') - 1), '(', ''), lon = replace(substr(location, instr(location, ', ') + 1), ')', ''); ( , ) 27

28 Location column, sample value UPDATE "fire-inspections-subset-sorted" SET lat = replace(substr(location, 1, instr(location, ', ') - 1), '(', ''), lon = replace(substr(location, instr(location, ', ') + 1), ')', ''); ( , ) 28

29 Location column, sample value UPDATE "fire-inspections-subset-sorted" SET lat = replace(substr(location, 1, instr(location, ', ') - 1), '(', ''), lon = replace(substr(location, instr(location, ', ') + 1), ')', ''); ( , ) 29

30 Location column, sample value UPDATE "fire-inspections-subset-sorted" SET lat = replace(substr(location, 1, instr(location, ', ') - 1), '(', ''), lon = replace(substr(location, instr(location, ', ') + 1), ')', ''); ( , ) 30

31 Location column, sample value UPDATE "fire-inspections-subset-sorted" SET lat = replace(substr(location, 1, instr(location, ', ') - 1), '(', ''), lon = replace(substr(location, instr(location, ', ') + 1), ')', ''); ( , ) 31

32 The new lat & lon columns look good! 32

33

34 Some things to note: csvstat and csvsql can be SLOW! csvkit is written in Python, so keep that in mind. If it s too slow for a specific task - you re probably better off pulling the data into SQLite for querying. pbcopy is a great little tool that allows you to direct your command line output to your clipboard. - Instead of copying/pasting the results of a csvsql --query 'SELECT DISTINCT ' command from the command line, you can pipe it to pbcopy and paste it directly into your Choice field list in the Fulcrum app builder. - For example: csvsql --query 'SELECT DISTINCT "Neighborhood District" FROM "Fire_Inspections";' Fire_Inspections.csv pbcopy 34

35 Thank you very much for your time But before we go...let s enjoy the fruits of our labor and IMPORT THIS DATA! 35

36