STA490HY1Y. Initial Examination of Data

Transcription

1 STA490Y1Y Initial Examination of Data Alison L. Department of Statistical Sciences University of Toronto

2 Course mantra It s OK not to know. Expressing ignorance is encouraged. It s not OK to not have a willingness to learn.

3 Cleveland s Visualization of the Barley Data Trebi Wisconsin No. 38 No. 457 Glabron Peatland Velvet No. 475 Manchuria No. 462 Svansota Trebi Wisconsin No. 38 No. 457 Glabron Peatland Velvet No. 475 Manchuria No. 462 Svansota Trebi Wisconsin No. 38 No. 457 Glabron Peatland Velvet No. 475 Manchuria No. 462 Svansota Crookston Waseca University Farm Morris Grand Rapids Duluth Barley Yield (bushels/acre)

4 Initial Examination of Data Purpose: Understand the structure of the data.

5 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert items or binned quantitative), binary

6 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert items or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations?

7 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert items or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems.

8 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert items or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems. Get ideas for more sophisticated analyses.

9 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert items or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems. Get ideas for more sophisticated analyses. Check on whether or not assumptions of more sophisticated analyses seem reasonable.

10 IDA Should be motivated by original research questions.

11 IDA Should be motivated by original research questions. Avoid data dredging. (Look long enough and you ll find some meaningless pattern.)

12 IDA Should be motivated by original research questions. Avoid data dredging. (Look long enough and you ll find some meaningless pattern.) Trivial? Requires judgment and common sense.

13 IDA Should be motivated by original research questions. Avoid data dredging. (Look long enough and you ll find some meaningless pattern.) Trivial? Requires judgment and common sense. May be all that is needed.

14 Types of Missing Data 1. Missing Completely At Random (MCAR) The probability that a data value is missing does not depend on the missing value, nor on the values of all other variables.

15 Types of Missing Data 1. Missing Completely At Random (MCAR) The probability that a data value is missing does not depend on the missing value, nor on the values of all other variables. 2. Missing At Random (MAR) The probability that a data value is missing, conditional on the values of the other variables for the observation, is not related to the missing value.

16 Types of Missing Data 1. Missing Completely At Random (MCAR) The probability that a data value is missing does not depend on the missing value, nor on the values of all other variables. 2. Missing At Random (MAR) The probability that a data value is missing, conditional on the values of the other variables for the observation, is not related to the missing value. 3. Informative / Non-ignorable (NMAR) Difficult to deal with.

17 Tools for IDA 5 number summary (for all data and for subsets).

18 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d.

19 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms (kernel density estimates), boxplots, dotplots.

20 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms (kernel density estimates), boxplots, dotplots. Frequency tables (1- and 2-way) for categorical variables

21 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms (kernel density estimates), boxplots, dotplots. Frequency tables (1- and 2-way) for categorical variables Scatterplots.

22 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms (kernel density estimates), boxplots, dotplots. Frequency tables (1- and 2-way) for categorical variables Scatterplots. Correlations.

23 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms (kernel density estimates), boxplots, dotplots. Frequency tables (1- and 2-way) for categorical variables Scatterplots. Correlations. What else?

24 AMore Sophisticated Tools for IDA Cluster Analysis Dimension reduction LOESS: Locally Weighted Scatterplot Smoothing Idea: fit a simple polynomial using regression on small ranges of the independent variable, and smoothly join up the pieces. Amount of smoothing controlled by a smoothing parameter.

25 Course mantra It s OK not to know. Expressing ignorance is encouraged. It s not OK to not have a willingness to learn.

26 For next week 1. Initial examination of student-performance-and-time-of-day data input the data into RStudio carry out IDA (no confidence intervals or P-values... yet...) document what you did and observed using R Markdown send me an html or pdf of your R Markdown document before class 2. Read: Lefevre et al (2010) Beer comsumption increases human attractiveness to malaria mosquitoes. PLoS ONE, vol. 5, issue 3. Make sure you understand the figures.