Data. Notes. are required reading for the week. textbook reading and a few slides on data formats and data cleaning

Transcription

1 CS 725/825 Information Visualization Spring 2018 Data Dr. Michele C. Weigle Notes } We will not cover these slides in class, but they are required reading for the week. } There are a few supplemental images for the textbook reading and a few slides on data formats and data cleaning 2

2 Data Ward, Grinstein, Keim, Fig Tables m attributes n items value Ward, Grinstein, Keim, Fig

3 Networks and Trees Field Dataset 6

4 Geometry 7 Multidimensional Table Item is determined by both row header and column header. Columns are not attributes, they are additional keys. 8

5 Temporal Semantics _time_series_of_seasonal_(red_dots)_and_annual_average_(black_line)_of_global_upper_ocean_heat_content_for_the_0-700m_layer_between_ gif 9 Data Formats } Delimited Text } tabbed delimited } comma delimited (CSV) } Extensible Markup Language (XML) } looks a bit like HTML } user-defined tags to identify data } JavaScript Object Notation (JSON) } collection of name/value pairs } smaller than XML } easier to parse 10

6 Example of Data Formats CSV JSON , , ,27 {"observations": [ {"date":" ", "max_temp":26}, {"date":" ", "max_temp":34}, {"date":" ", "max_temp":27} ] } XML <weather_data> <observation> <date> </date> <max_temp>26</max_temp> </observation> <observation> <date> </date> <max_temp>34</max_temp> </observation> <observation> <date> </date> <max_temp>27</max_temp> </observation> </weather_data> Yau, Visualize This, Ch 2 11 How to convert between data formats? } Write a program to convert from one format to another } awk (my favorite, but I'm old school), Python, Perl, PHP,... } Other tools } search Google for "csv to json", "csv to xml", "xml to json" } Mr. Data Converter } } developed by a graphics editor at The New York Times } input: CSV or tab-delimited data } output: HTML table, JSON, MySQL, Python, PHP, Ruby, XML,... 12

7 Tabula - PDF to CSV } Upload a PDF file containing a data table. } Browse to the page you want, then select the table by clicking and dragging to draw a box around the table. } Click "Preview & Export Extracted Data". } Tabula will try to extract the data and display a preview. Inspect the data to make sure it looks correct. If data is missing, you can go back to adjust your selection. } Click the "Export" button. } Now you can work with your data as text file or a spreadsheet rather than a PDF! 13 Beautiful Soup 14

8 Data in the Real World } Data can be missing, have typos, be inconsistent, spread over multiple tables, etc. } Two big issues: } format } accuracy 15 World Disasters - Inconsistent 16

9 World Disasters - Missing 17 What to do with dirty data? 18

10 Data Cleaning Tools Quick Tools Full Apps }Data Science Toolkit } } lots of quick conversion tools } Mr. People } } formats lists of names } Mr. Data Converter } } Trifacta Wrangler } /wrangler/ } video: } Open Refine (was Google Refine) } } video: 70J_H_zAWM } more info: /using-google-refine-for-datacleaning 19 What about accuracy? } Nathan Yau (Visualize This, Data Points) was intern at The New York Times } One day, his entire goal was to verify 3 numbers in a dataset } Must have accurate data before can trust the visualization Yau, Visualize This, Ch 1 20

11 Recall the marriage rate chart 21 Graphing the raw data New Hampshire?!? 22

12 Let's look at the Excel file 23 Now, let's look at the PDF 24

13 But wait, there's more funny stuff PDF Excel 25 Bottom Line } If you see something weird in your graph that you can't explain, go back and double-check your data } Even if you didn't make an error, maybe someone else did 26