Data Foundations. Topic Objectives. and list subcategories of each. its properties. before producing a visualization. subsetting

Transcription

1 CS 725/825 Information Visualization Fall 2013 Data Foundations Dr. Michele C. Weigle Topic Objectives! Distinguish between ordinal and nominal values and list subcategories of each.! Give an example of structured data and describe its properties.! Explain the importance of data preprocessing before producing a visualization! Contrast interactive subsetting with query-based subsetting! Explain the purpose of dimensional reduction. 2

2 Outline Reading Assignment (due today): Ch 2 - Data Foundations! Data! Types of Data! Structure in and between Records! Data Cleaning! Data Preprocessing! Data Sources Homework Assignment: Due next Tuesday 3 The first step is data m variables (or, attributes) n records (or, values) WGK, Fig

3 Variables: Independent vs. Dependent 5 Variables: Domain and Range! f(domain) = range!!! 6 domain - independent variables range - dependent variables f(weight) = City MPG

4 Outline! Data! Types of Data! Structure in and between Records! Data Cleaning! Data Preprocessing! Data Sources 7 Ordinal vs. Nominal Data! Binary! 0, 1! Discrete! integers: 1, 2, 3, 4, 5! Continuous! real numbers: [0, 5] or 0.1, 0.25, , 0.456! Categorical! ODU-CS faculty: Nelson, Weigle, Olariu, Maly, Nadeem,! Ranked! undergrad classes: Freshman, Sophomore, Junior, Senior! Arbitrary! last names: Helser, Bokinsky, Clark, Tarr, Kum, Van Leeuwen, Weigle,... 8

5 Scale! Ordering relation! 1, 2, 3, 4, 5! Freshman, Sophomore, Junior, Senior! Distance metric! distance between 5 and 1 is 4! Existence of absolute zero! absolute minimum weight is 0 9 Outline! Data! Types of Data! Structure within and between Records! Data Cleaning! Data Preprocessing! Data Sources 10

6 Structured Data! Data has structure! syntax! semantics! Time is special! Data has topology 11 Example of Structured Data! Vehicle Name - nominal, arbitrary! Next 8 variables - ordinal, binary! Price, cost - ordinal, discrete, absolute zero! etc. 12

7 Data Formats! Delimited Text! tabbed delimited! comma delimited (CSV)! Extensible Markup Language (XML)! looks a bit like HTML! user-defined tags to identify data! JavaScript Object Notation (JSON)! collection of name/value pairs! smaller than XML! easier to parse 13 Example of Data Formats CSV JSON , , ,27 {"observations": [ {"date":" ", "max_temp":26}, {"date":" ", "max_temp":34}, {"date":" ", "max_temp":27} ] } XML <weather_data> <observation> <date> </date> <max_temp>26</max_temp> </observation> <observation> <date> </date> <max_temp>34</max_temp> </observation> <observation> <date> </date> <max_temp>27</max_temp> </observation> </weather_data> Yau, Visualize This, Ch 2 14

8 How to convert between data formats?! Write a program to convert from one format to another! awk (my favorite, but I'm old school), Python, Perl, PHP,...! Other tools! search Google for "csv to json", "csv to xml", "xml to json"! Mr. Data Converter! developed by a graphics editor at The New York Times! input: CSV or tab-delimited data! output: HTML table, JSON, MySQL, Python, PHP, Ruby, XML, Outline! Data! Types of Data! Structure within and between Records! Data Cleaning! Data Preprocessing! Data Sources 16

9 Data in the Real World! Data can be missing, have typos, be inconsistent, spread over multiple tables, etc.! Two big issues:! format! accuracy 17 World Disasters - Inconsistent 18

10 World Disasters - Missing 19 What to do with dirty data? 20

11 Data Cleaning Tools Quick Tools Full Apps! Data Science Toolkit! lots of quick conversion tools! Mr. People! formats lists of names! Mr. Data Converter! data_converter/! Data Wrangler! app/! video: Open Refine (was Google Refine)! video: watch?v=b70j_h_zawm! more info: using-google-refine-for-datacleaning 21 What about accuracy?! Nathan Yau (Visualize This, Data Points) was intern at The New York Times! One day, his entire goal was to verify 3 numbers in a dataset! Must have accurate data before can trust the visualization Yau, Visualize This, Ch 1 22

12 Yes, this is actually a problem. 23 Graphing the raw data 24

13 Let's look at the Excel file 25 Now, let's look at the PDF 26

14 But wait, there's more funny stuff PDF Excel 27 Bottom Line! If you see something weird in your graph that you can't explain, go back and double-check your data! Even if you didn't make an error, maybe someone else did 28

15 Outline! Data! Types of Data! Structure within and between Records! Data Cleaning! Data Preprocessing! Data Sources 29 Raw vs. Processed Data 30

16 What statistics could we compute?! Mean! Standard deviation! Distribution Mean = 440 Median = Remember Anscombe's Quartet?! N = 11! Mean of the x values = 9.0! Mean of the y values = 7.5! Equation of the least-squared regression line: y = x! Sums of squared errors (about the mean) = 110.0! Regression sums of squared errors (variance accounted for by x) = 27.5! Residual sums of squared errors (about the regression line) = 13.75! Correlation coefficient = 0.82! Coefficient of determination = 0.67 F.J. Anscombe, "Graphs in Statistical Analysis", American Statistician, Feb 1973, graphs from 32

17 The four data sets are not the same 33 Other Analysis Techniques Outlier detection Cluster analysis

18 Other Analysis Techniques Correlation analysis 35 Visual Analytics Mantra "Analyze First, Show the Important, Zoom, Filter and Analyze Further, Details-on-Demand" - Daniel Keim Keim et al., "Visual Analytics: Combining Automated Discovery with Interactive Visualizations",

19 Normalization! Example: Transform range of values so that all values fall between ! Why?! compare seemingly unrelated variables! How?! d normal = (d orig - d min ) / (d max - d min ) 37 Subsetting! Set of constraints to retrieve only data that meets desired conditions 38

20 Query-Based Subsetting "Find all employees who have not made a sale in the past year" SELECT * FROM EMP AS e WHERE NOT EXISTS (SELECT * FROM SALE AS s WHERE s.eid = e.eid) 39 Interactive Subsetting 40

21 Dimension Reduction! What about when there are lots of dimensions (attributes)?! allow user to select more important dimensions! or reduce automatically through computational methods! principal component analysis (PCA)! multidimensional scaling (MDS)! Kohonen self-organizing maps (SOMs) 41 PCA Example WGK, Fig

22 How to visualize nominal values? 43 Label nominal values? 44

23 Aggregation and Summarization! What to do with too much data?! Two components to aggregation!!! method of grouping points method of displaying the groups Key to visualizing aggregated data! provide sufficient info for user to decide whether they want to perform a drill-down on data (explore contents of one or more clusters) 45 Original Data WGK, Fig 2.5a 46

24 Aggregated Data WGK, Fig 2.5b 47 Outline! Data! Types of Data! Structure within and between Records! Data Cleaning! Data Preprocessing! Data Sources 48

25 Data Sources! Giant list of data sources! Some notable ones! Data.gov! Google Public Data Explorer! Census Bureau! Census data visualization gallery - Federal Reserve! FRASER - FRED Data for Maps! TIGER! detailed data on roads, railroads, rivers, ZIP codes (from Census Bureau)! OpenStreetMap! free wiki world map! Geocommons! contains data and ability to make maps 50 Yau, Visualize This, Ch 2

26 Outline! Data! Types of Data! Structure within and between Records! Data Cleaning! Data Preprocessing! Data Sources 51