Exploratory Data Analysis! 1D!

Transcription

1 Exploratory Data Analysis! 1D!!

2 philosophical view

3 What is EDA? Paraphrasing John Tukey: It is a mindset (a willingness to look for what can be seen, whether or not it is an>cipated), a flexibility (let the data speak for themselves, explore lots of avenues) A way to make pictures (the picture-examining eye is the best finder we have of the wholly unan>cipated)

4 more concrete view

5 What is EDA? Verify expected rela>onships actually exist in the data Find unexpected structure in the data that must be accounted for Ensure the right ques>ons are being asked Generate addi>onal ques>ons to be considered Provide a basis for further data collec>on

6

7 There is a large emphasis on graphical exploration because it is the best way to reveal unanticipated structure. We want to plot the raw data plot simple statistics position objects and plots to maximize pattern recognition

8 one categorical variable

9 diamonds We examine a categorical variable by looking at counts

10 diamonds %>% count(cut) count() is a data manipulation verb table() is a vector function table(diamonds$cut)

11 da>ng_profiles %>% count(educa>on, sort = TRUE) %>% mutate(pct = 100 * n/sum(n))

12 da>ng_profiles %>% count(educa>on, sort = TRUE) %>% mutate(pct = 100 * n/sum(n)) Things to consider: 1. Group sizes 2. Vital few vs. trivial many 3. Missing data 4. Is there a natural ordering? 5. Recoding 6. Collapsing 7. Data type: chr vs. fct vs. ord

13 da>ng_profiles <- da>ng_profiles %>% mutate(educa>on = fct_explicit_na(educa>on)) levels(fct_infreq(da>ng_profiles$educa>on)) Things to consider: 1. Group sizes 2. Vital few vs. trivial many 3. Missing data 4. Is there a natural ordering? 5. Recoding 6. Collapsing 7. Data type: chr vs. fct vs. ord

14 da>ng_profiles %>% ggplot(aes(fct_rev(fct_infreq(educa>on)))) + geom_bar() + coord_flip() Use a bar plot to examine the distribu>on of a categorical variable

15 da>ng_profiles %>% ggplot(aes(fct_rev(fct_infreq(educa>on)))) + geom_bar() + coord_flip() Use a bar plot to examine the distribu>on of a categorical variable by default geom_bar() places the missingdata group last

16 Your Turn 1 Consider the pets variable in the dating_profiles dataset. Make a summary of the group counts and make a bar plot. Then, go through the checklist below and think about what you would consider doing. 1. Do group sizes vary a lot? 2. Are there a vital few and/or trivially many? 3. Any missing data? 4. Is there a natural ordering to the groups? 5. Should we consider recoding/collapsing?

17 da>ng_profiles %>% count(pets, sort = TRUE) %>% mutate(pct = 100 * n/sum(n)) da>ng_profiles %>% mutate(pets = fct_rev(fct_infreq(fct_explicit_na(pets)))) %>% ggplot(aes(pets)) + geom_bar() + coord_flip()

18

19 what about pie charts?

20 Pie charts Try placing the wedges in order from largest to smallest

21 Pie charts

22 Pie charts What is revealed about the data from this pie chart?

23 Pie charts

24 Pie charts Pie charts are awful because they encode information in angles and areas that are very difficult for humans to judge Graphics that make comparisons via position on a common scale are best

25 one numerical variable

26 diamonds We examine a numerical variable by looking at statistical summaries and binned counts (histograms)

27 diamonds %>% summarise( min = min(price), q1 = quan>le(price, 0.25), median = median(price), mean = mean(price), q3 = quan>le(price, 0.75), max = max(price) ) summarise() is a data manipulation verb summary(diamonds$price) summary() is a generic function

28 diamonds %>% summarise( min = min(price), q1 = quan>le(price, 0.25), median = median(price), mean = mean(price), q3 = quan>le(price, 0.75), max = max(price) ) summary(diamonds$price) Things to consider: 1. What values are most common? 2. Which values are rare/extreme? 3. Can you see unusual paferns? 4. Measures of center (mean, median, etc.) 5. Measures of spread (std. dev., IQR, etc.) 6. Skewness 7. Missing data

29 diamonds %>% ggplot(aes(price)) + geom_histogram(binwidth = 100) Use a histogram to examine the distribu>on of a numerical variable

30 diamonds %>% ggplot(aes(price)) + geom_histogram(binwidth = 100) + geom_vline(xintercept = mean(diamonds$price), color = "red") + geom_vline(xintercept = median(diamonds$price), color = "blue") sta>s>cs and other values can be overlaid

31 histogram examples

32 diamonds %>% filter(carat < 3) %>% ggplot(aes(carat)) + geom_histogram(binwidth = 0.01)

33 faithful

34 summary(faithful$erup>ons)

35 faithful %>% ggplot(aes(erup>ons)) + geom_histogram(binwidth = 0.25)

36 histograms on the density scale

37 diamonds %>% ggplot(aes(fct_rev(cut), y = stat(prop), group = 1)) + geom_bar() bar plots can be on a propor>on scale the height of each bar is the propor>on of values in that bar

38 diamonds %>% ggplot(aes(price, y = stat(density))) + geom_histogram(binwidth = 3000, color = "white") histograms can be on a density scale

39 diamonds %>% ggplot(aes(price, y = stat(density))) + geom_histogram(binwidth = 3000, color = "white") histograms can be on a density scale the area of each bin is the propor>on of values in that bin

40 top_movies %>% ggplot(aes(gross_adj, y = stat(density))) + geom_histogram(binwidth = 100, boundary = 300, color = "white")

41 top_movies %>% ggplot(aes(gross_adj, y = stat(density))) + geom_histogram(breaks = c(300, 400, 600, 1500), color = "white")

42 top_movies %>% ggplot(aes(gross_adj, y = stat(density))) + geom_histogram(breaks = c(300, 400, 1500), color = "white") no>ce that the density scale reasonably displays the distribu>on

43 top_movies %>% ggplot(aes(gross_adj)) + geom_histogram(binwidth = 100, boundary = 300, color = "white")

44 top_movies %>% ggplot(aes(gross_adj)) + geom_histogram(breaks = c(300, 400, 600, 1500), color = "white")

45 top_movies %>% ggplot(aes(gross_adj)) + geom_histogram(breaks = c(300, 400, 1500), color = "white") a count (or propor>on) scale loses the shape of the distribu>on

46 top_movies %>% ggplot(aes(gross_adj)) + geom_histogram(breaks = c(300, 400, 1500), color = "white") a count (or propor>on) scale loses the shape of the distribu>on this is not a histogram

47 top_movies %>% ggplot(aes(gross_adj, y = stat(density))) + geom_histogram(breaks = c(300, 400, 600, 1500), color = "white") On the density scale, the height of a bin is the density: the propor>on per unit on the horizontal axis

48 top_movies %>% ggplot(aes(gross_adj, y = stat(density))) + geom_histogram(breaks = c(seq(300, 400, by = 10), 600, 1500), color = "white") On the density scale, the height of a bin is the density: the propor>on per unit on the horizontal axis

49 top_movies %>% ggplot(aes(gross_adj, y = stat(density))) + geom_histogram(breaks = c(300, 350, 400, 450, 1500), color = "white") Recap: there are 25 movies represented in the [400,450) bin, and 92 in the [450, 1500) bin. But the last bin is much wider, so it is less crowded (less dense)