Lecture 1: Exploratory data analysis

Transcription

1 Lecture 1: Exploratory data analysis Statistics 101 Mine Çetinkaya-Rundel January 17, 2012

2 Announcements Announcements Any questions about the syllabus? If you sent me your gmail address your RStudio account should be working. If you haven t, please do so ASAP. Did you take the survey? If not, it s under Tests & Quizzes on Sakai. Did you get a clicker? If you ve made arrangements to get a used clicker using the list on Google Docs please remove yourself from the list. Once you have your clicker register it at support/ registeryourclicker. Enter your NetID in the Student ID field. If you missed lab last Wednesday, review it on your own. It s posted on the course webpage at stat.duke.edu/ courses/ Spring12/ sta Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

3 Announcements Introduction to data Process of scientific inquiry: 1 Identify a question or problem 2 Collect relevant data on the topic 3 Analyze the data 4 Form a conclusion Statistics focuses on making stages (2)-(4) objective, rigorous, and efficient. Statistics is the study of how best to collect, analyze, and draw conclusions from data. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

4 Case study 1 Case study 2 Data basics 3 Examining numerical data Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, 2012

5 Case study (1) Identify a question or problem Is whether or not students identify with their parents political beliefs associated with their class year? How would we go about answering this question? Can you help design an appropriate study? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

6 Case study (2) Collect relevant data on the topic 86 students took a survey where they were asked about their class year and whether or not they identify with their parents political beliefs. Student class parent politic 1 junior no 2 first-year yes 3 first-year NA sophomore yes Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

7 Case study (3) Analyze the data class parent politic no yes Total first-year sophomore junior senior Total Why is the total number of responses only 76 when 86 students took the survey? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

8 Case study (3) Analyze the data (cont.) How can we answer the research question using these data? class parent politic no yes Total first-year sophomore junior senior Total Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

9 Case study (4) Form a conclusion Clicker question What is an appropriate conclusion for this study? (a) Proportion of graduate students who identify with their parents political beliefs will be less than 33%. (b) Proportion of high school students who identify with their parents political beliefs will be higher than 97%. (c) Staying in school longer causes students to not identify with their parents political beliefs. (d) While there is a difference in the proportion of students who identify with their parents political beliefs among the different class years, we cannot determine from this analysis alone if being in school for longer is the cause of having different political beliefs. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

10 Data basics 1 Case study 2 Data basics Observations and variables Types of variables Relationships among variables Associated and independent variables 3 Examining numerical data Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, 2012

11 Data basics Observations and variables Observations and variables data matrix variable Stu. gender intro extra dread 1 male extravert 3 2 female extravert 2 3 female introvert 4 4 female extravert 2 observation male extravert 3 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

12 Data basics Types of variables Types of variables all variables numerical categorical continuous discrete regular categorical ordinal measured counted unordered categories ordered categories Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

13 Data basics Types of variables Types of variables (cont.) gender: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

14 Data basics Types of variables Types of variables (cont.) gender: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

15 Data basics Types of variables Types of variables (cont.) gender: sleep: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

16 Data basics Types of variables Types of variables (cont.) gender: sleep: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

17 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

18 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

19 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: countries: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

20 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: countries: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

21 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: countries: dread: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

22 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: countries: dread: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

23 Data basics Types of variables Types of variables (cont.) gender: sleep: bedtime: countries: dread: gender sleep bedtime countries dread 1 male female female female female female Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

24 Data basics Types of variables Clicker question What type of variable is a zip code? (a) numerical, continuous (b) numerical, discrete (c) categorical (d) categorical, ordinal Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

25 Data basics Relationships among variables Relationships among variables 14 # of alcoholic drinks / week age at first alcohol consumption Does there appear to be a relationship between number of alcoholic drinks consumed per week and age at first alcohol consumption? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

26 Data basics Associated and independent variables Associated and independent variables Clicker question Based on the scatterplot on the right, which of the following statements is correct about the head and skull lengths of possums? skull width (mm) head length (mm) (a) There is no relationship between head length and skull width, i.e. the variables are independent. (b) Head length and skull width are positively associated. (c) Skull width and head length are negatively associated. (d) A longer head causes the skull to be wider. (e) A wider skull causes the head to be longer. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

27 Data basics Associated and independent variables Associated vs. independent When two variables show some connection with one another, they are called associated variables. Associated variables can also be called dependent variables and vice-versa. If two variables are not associated, i.e. there is no evident connection between the two, then they are said to be independent. It is also possible for observations to be independent as well. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

28 1 Case study 2 Data basics 3 Examining numerical data Scatterplots for paired data Dot plots and the mean Histograms and shape Variance and standard deviation Box plots, quartiles, and the median Robust statistics Transforming data Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, 2012

29 Population to sample It is usually not feasible to collect information on the entire population due to high costs of data collection so statisticians instead work with samples that are (hopefully) representative of the populations they come from. population sample We try to understand certain features of the population as a whole using summary statistics and graphs based on these samples. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

30 Scatterplots for paired data Cars:... vs. weight From the cars data presented in the textbook: 60 miles per gallon (city rating) price ($1000s) weight (pounds) weight (pounds) What do these scatterplots reveal about the data? How might they be useful? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

31 Scatterplots for paired data Life expectancy vs. income Do life expectancy and income appear to be associated or independent? Was the relationship the same throughout the years, or did it change? world Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

32 Dot plots and the mean Dot plots Useful for visualizing one numerical variable gpa Do you see anything out of the ordinary? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

33 Dot plots and the mean Dot plots (cont.) Useful for visualizing one numerical variable. Darker colors represent areas where there are more observations gpa How would you describe the distribution of GPAs in this data set? Make sure to say something about the center, shape, and spread of the distribution. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

34 Dot plots and the mean Dot plots & mean gpa The mean, also called the average (marked with a triangle in the above plot), is one way to measure the center of a distribution of data. The mean GPA is Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

35 Dot plots and the mean Mean The sample mean, denoted as x, can be calculated as x = x 1 + x x n, n where x 1, x 2,, x n represent the n observed values. The population mean is also computed the same way but is denoted as µ. It is often not possible to calculate µ since population data is rarely available. The sample mean is a sample statistics, or a point estimate of the population mean. This estimate may not be perfect, but if the sample is good (representative of the population) it is usually a good guess. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

36 Dot plots and the mean Stacked dot plot Higher bars represent areas where there are more observations, makes it a little easier to judge the center and the shape of the distribution gpa Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

37 Histograms and shape Histograms - GPA Higher bars represent areas where there are more observations, preferable when sample size is large but hides finer details like individual observations frequency gpa Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

38 Histograms and shape Anatomy of a histogram Order the data in ascending order: sort(d$gpa) [1] [16] [31] [46] [61] [76] Make a frequency table where the number of observations that fall in a certain bin are recorded by counting how many observations fall in each bin. Let s use a bin width of 0.1: GPA 2.9 to 3 3 to to to to to 4 Count Note: Histogram is shown on the previous slide. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

39 Histograms and shape Histograms - Extracurricular hours Histograms provide a view of the data density. Higher bars represent where the data are relatively more common. Histograms are especially convenient for describing the shape of the data distribution. The chosen bin width can alter the story the histogram is telling. frequency extracurricular hrs / week Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

40 Histograms and shape Bin width Which one(s) of these histograms are useful? Which reveal too much about the data? Which hide too much? frequency frequency extracurricular hrs / week extracurricular hrs / week frequency frequency extracurricular hrs / week extracurricular hrs / week Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

41 Histograms and shape Shape of a distribution: modality The mode is defined as the most frequent observation in the data set. Does the histogram have a single prominent peak (unimodal), several prominent peaks (bimodal/multimodal), or no apparent peaks (uniform)? Note: In order to determine modality, it s best to step back and imagine a smooth curve over the histogram. Use the limp spaghetti method. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

42 Histograms and shape Shape of a distribution: skewness Is the histogram right skewed, left skewed, or symmetric? Note: Histograms are said to be skewed to the side of the long tail. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

43 Histograms and shape Shape of a distribution: unusual observations Are there any unusual observations or potential outliers? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

44 Histograms and shape Clicker question Which of these variables do you expect to be uniformly distributed? (a) weights of adult females (b) salaries of a random sample of people from North Carolina (c) exam scores (d) birthdays of classmates (day of the month) Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

45 Histograms and shape Ages of my FB friends What would you guess my age is? frienddata Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

46 Histograms and shape Are you typical? watch?v=4b2xovkffz4 How useful are centers alone for conveying the true characteristics of a distribution? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, 2012

47 Variance and standard deviation Variability in data How would you describe the amount of variability in the number of hours of sleep students get per night? 40 frequency sleep (hrs / night) Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

48 Variance and standard deviation Deviation The distance of an observation from the mean is its deviation: x i x. sort(d$sleep) [1] [30] [59] mean(d$sleep) [1] 4.6 x 1 x = = 3.6 x 2 x = = 3.6 x 3 x = = 2.6. x 86 x = = 4.4 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

49 Variance and standard deviation Variance Sample variance, s 2, is roughly the average squared deviation from the mean. s 2 = ni=1 (x i x) 2 n 1 Note: When calculating the sample variance we divide by n 1 instead of n. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

50 Variance and standard deviation Variance Sample variance, s 2, is roughly the average squared deviation from the mean. s 2 = ni=1 (x i x) 2 n 1 Note: When calculating the sample variance we divide by n 1 instead of n. The variance of amount of sleep students get per night can be calculated as: s 2 = ( 3.6)2 + ( 3.6) (4.4) = = 2.76 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

51 Variance and standard deviation Variance (cont.) Calculating variance by hand is rather tedious, especially when the data set is large. var(d$sleep) [1] 2.76 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

52 Variance and standard deviation Variance (cont.) Why do we use the squared deviation in the calculation of variance? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

53 Variance and standard deviation Variance (cont.) Why do we use the squared deviation in the calculation of variance? To get rid of negatives so that observations equally distant from the mean are weighed equally. To weigh larger deviations more heavily Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

54 Variance and standard deviation Standard deviation The sample standard deviation, s, is the square root of the variance. s = n i=1 (x i x) 2 n 1 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

55 Variance and standard deviation Standard deviation The sample standard deviation, s, is the square root of the variance. s = n i=1 (x i x) 2 n 1 The standard deviation of car prices can be calculated as: s = 2.76 = 1.66 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

56 Variance and standard deviation Standard deviation The sample standard deviation, s, is the square root of the variance. s = n i=1 (x i x) 2 n 1 The standard deviation of car prices can be calculated as: s = 2.76 = 1.66 sd(d$sleep) [1] 1.66 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

57 Variance and standard deviation Variability in car prices sleep, x = 4.6, s x = out of 86 students (80%) are within 1 SD of the mean. 80 out of 86 students (93%) are within 2 SDs of the mean. 86 out of 86 students (100%) are within 3 SDs of the mean. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

58 Variance and standard deviation Describing distributions When describing distributions make sure to talk about the shape, center, spread, and if any, unusual observations Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

59 Variance and standard deviation Notation recap mean variance SD sample x s 2 s population µ σ 2 σ Do you see a trend in what types of letters are used for sample statistics vs. population parameters? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

60 Box plots, quartiles, and the median Median The median is the value that splits the data in half when ordered in ascending order. 0, 1, 2, 3, 4 If there are an even number of observations, then the median is the average of the two values in the middle. 0, 1, 2, 3, 4, = Since the median is the midpoint of the data, 50% of the values are below it. Hence, it is also the 50 th percentile. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

61 Box plots, quartiles, and the median Q1, Q3, and IQR The 25 th percentile is also called the first quartile, Q1. The 50 th percentile is also called the median. The 75 th percentile is also called the third quartile, Q3. summary(d$study_hours) Min. 1st Qu. Median Mean 3rd Qu. Max. NAs Between Q1 and Q3 is the middle 50% of the data. The range these data span is called the interquartile range, or the IQR. IQR = = 10 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

62 Box plots, quartiles, and the median Box plot The box in a box plot represents the middle 50% of the data, and the thick line in the box is the median # of study hours / week Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

63 Box plots, quartiles, and the median Anatomy of a box plot 40 suspected outliers max whisker reach # of study hours / week upper whisker Q 3 (third quartile) median Q 1 (first quartile) 0 lower whisker Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

64 Box plots, quartiles, and the median Whiskers and outliers Whiskers of a box plot can extend up to 1.5 * IQR away from the quartiles. max upper whisker reach : Q IQR = = 35 max lower whisker reach : Q1 1.5 IQR = = 5 An outlier is defined as an observation beyond the maximum reach of the whiskers. It is an observation that appears extreme relative to the rest of the data. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

65 Box plots, quartiles, and the median Outliers (cont.) Why is it important to look for outliers? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

66 Box plots, quartiles, and the median Household income Clicker question Which of the below is the most reasonable estimate for the median household income? (n = 48) household income ($ thousands) (a) $50K (b) $150K (c) $300K (d) $400K (e) $500K Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

67 Robust statistics Extreme observations How would sample statistics such as mean, median, SD, and IQR of household income be affected if the largest value was replaced with $10 million? What if the smallest value was replaced with $10 million? household income ($ thousands) Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

68 Robust statistics Robust statistics household income ($ thousands) robust not robust scenario median IQR x s original data 165K 150K 211K 180K move largest to $10 million 165K 150K 398K 1,422K move smallest to $10 million 190K 163K 4,186K 1,424K Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

69 Robust statistics Robust statistics Median and IQR are more robust to skewness and outliers than mean and SD. Therefore, for skewed distributions it is more appropriate to use median and IQR to describe the center and spread for symmetric distributions it is more appropriate to use the mean and SD to describe the center and spread If you would like to estimate the typical household income for a Duke student, would you be more interested in the mean or median income? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

70 Robust statistics Recap: finding the mean and the median Below are the final exam scores of 5 Stats 101 students: 79, 82, 94, 83, 92 Median: Put the values in increasing order: 79, 82, 83, 92, 94 Median is the value in the middle: 79, 82, 83, 92, 94 Mean is the arithmetic average: = 86 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

71 Robust statistics Recap: finding the mean and the median (cont.) Let s add an extremely low score to the data set: 3, 79, 82, 83, 92, 94 Clicker question What is the new median? (a) 72.2 (b) 82 (c) 82.5 (d) 83 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

72 Robust statistics Recap: finding the mean and the median (cont.) Let s add an extremely low score to the data set: 3, 79, 82, 83, 92, 94 Clicker question What is the new median? (a) 72.2 (b) 82 (c) 83 Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

73 Robust statistics Recap: finding the mean and the median (cont.) Let s add an extremely low score to the data set: 3, 79, 82, 83, 92, 94 Clicker question What is the new median? (a) 72.2 (b) 82 (c) 83 What is the new mean? Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

74 Robust statistics Mean vs. median If the distribution is symmetric, center is the mean Symmetric: mean median If the distribution is skewed or has outliers center is the median Right-skewed: mean > median Left-skewed: mean < median Right skewed Left skewed mean median mean median Symmetric mean median Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

75 Robust statistics Clicker question Which is true for the distribution of percentage of time actually spent taking notes in class versus on Facebook, Twitter, etc.? 20 frequency % of time spent taking notes in class (a) mean is larger than median (b) mean is smaller than median (c) mean is roughly equal to the median (d) impossible to tell Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

76 Robust statistics Clicker question Which is true for the distribution of number of schools accepted? 15 frequency # of schools accepted (a) mean is much larger than median (b) mean is much smaller than median (c) mean is roughly equal to the median (d) impossible to tell Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

77 Transforming data Duke basketball games attended frequency # of Duke basketball games attended Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

78 Transforming data Extremely skewed data When data are extremely skewed, transforming them might make modeling easier. A common transformation is the log transformation. frequency frequency # of Duke basketball games attended log(# of Duke basketball games attended) Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58

79 Transforming data Pros and cons of transformations Transformed data are easier to work with when applying statistical models because they are much less skewed and the outliers are much less extreme. # of games log(# of games) However results of an analysis might be difficult to interpret because the log of a measured variable is usually meaningless what does log of number of games mean after all? What other variables would you expect to be extremely skewed? Note: We ll talk more about transformations when we get to inference. Statistics 101 (Mine Çetinkaya-Rundel) L1: Exploratory data analysis January 17, / 58