Acquisition Description Exploration Examination Understanding what data is collected. Characterizing properties of data.

Transcription

1 Summary Statistics

2 Acquisition Description Exploration Examination what data is collected Characterizing properties of data. Exploring the data distribution(s). Identifying data quality problems. Selecting Collection Methods Types Dimensionality Sparsity Resolution Summary stats. Visualizations Multidimensional analysis Missing data Noise and artifacts Outliers Inconsistencies

3 Basic Statistical Descriptions of Motivation: to better understand the data Freshman Sophomore Juniors Seniors Bedtime Times Sick Bedtime Times Sick Bedtime Times Sick Bedtime Times Sick

4 Basic Statistical Descriptions of Motivation: to better understand the data Freshman Sophomore Juniors Seniors Bedtime Times Sick Bedtime Times Sick Bedtime Times Sick Bedtime Times Sick Mean SD

5 Basic Statistical Descriptions of characteristics Central Tendency: Mean, median, mode Spread : Variance, standard deviation, max, min, Z-score 5

6 Types of Attributes Quantitative Discrete/ Nominal Continuous Binary Ordinal Ratio Interval Qualitative 6

7 Types of Attributes Quantitative Discrete/ Nominal Continuous Binary Ordinal Ratio Interval Qualitative 7

8 Frequency Conditions Pre(n=197) Post(n=195) Total(n=392) Hypertension 94 (47.7) 96 (49.2) 190 (48.5) Diabetes 44 (22.3) 47 (24.1) 91 (23.2) Coronary Artery Disease Frequency 59 (30.0) 35 (18.0) 94 (24.0) Asthma 22 (11.2) 17 (08.7) 39 (10.0) None 54 (27.4) 64 (32.8) 118 (30.1) Valid for all attribute types 8

9 Central Tendency: Mode Mode: Value that occurs most frequently in the data Multi-modal Bimodal Trimodal etc. Also valid for all attribute types Conditions Pre(n=197) Post(n=195) Total(n=392) Hypertension 94 (47.7) 96 (49.2) 190 (48.5) Diabetes 44 (22.3) 47 (24.1) 91 (23.2) Coronary Artery Disease 59 (30.0) 35 (18.0) 94 (24.0) Asthma 22 (11.2) 17 (08.7) 39 (10.0) None 54 (27.4) 64 (32.8) 118 (30.1) 9

10 Percentiles Given an ordinal or continuous feature x and a number p between 0 and 100, the p th percentile is a value x p of x such that p% of the observed values of x are less than x p. 10

11 Central Tendency: Mean and Median Mean Highly sensitive to outliers Sorting not needed Many variations Trimmed mean: Chopping off extreme values Often a percentage Geometric and Harmonic Means Only defined for interval and ratio data Exception of binary data 11 n x ҧ = 1 n i=1 x i Median Less sensitive to outliers (x) is sorted, then median can be identified as the as follows Odd value of X at position (n+1)/2 Even - average of values for X at position n/2 and (n+1)/2 n 1 x[ ] ( N Odd) 2 Median n n 1 x[ ] x[ ] 2 2 ( N Even) 2 Median is defined for ordinal data Odd computed the same way Even commonly the smaller value (n/2) is selected

12 Central Tendency: A Deeper Consideration of Means Geometric Mean Used when the values are multiplicative rather than additive When data are skewed and transformed with a logarithm Pros Scale invariant Cons Careful about zeros! With multiple scales: units are lost n n i=1 x i 12

13 Central Tendency: A Deeper Consideration of Means Patient 1 Patient 2 4.5/5 3/5 68/100 75/100 Arithmetic Mean Patient 1 Patient 2 ( ) 2 = vs (3 + 75) 2 = 39 13

14 Central Tendency: A Deeper Consideration of Means Patient 1 Patient 2 4.5/5 3/5 68/100 75/100 Arithmetic Mean Patient 1 Patient 2 ( ) 2 = vs (3 + 75) 2 = 39 Arithmetic Mean (Scaled) Patient 1 Patient * 20 = 90 3 * 20 = 60 ( ) 2 = 79 vs ( ) 2 =

15 Central Tendency: A Deeper Consideration of Means Patient 1 Patient 2 4.5/5 3/5 68/100 75/100 Geometric Mean Patient 1 Patient 2 sqrt(4.5 * 68) = 17.5 vs sqrt (3 * 75) = 15 15

16 Central Tendency: A Deeper Consideration of Means Harmonic Means Commonly associated with rates (of the same quantity/distance) Less commonly used Relations between means: ( σ i=1 n n 1 x i ) 1 arithmetic mean > geometric mean > harmonic mean 16

17 Central Tendency: A Deeper Consideration of Means Suppose an ambulance is 10 miles from a hospital Driving to the hospital they drove 60mph, Returning they drove 30mph What is the average trip speed? = 45 (no) 1 2 ( = ) Other examples: What is the average cost per cell? $480 for cell type A, 30$ per cell $480 for cell type B, 40$ per cell $480 for cell type C, 32$ per cell ( = ) 17

18 Spread: Range / IQR Range: is the difference between the maximum and minimum values. IQR: difference between 75 th and 25 th percentile values Describes central 50% of a distribution regardless of shape 18

19 Spread: Variance / Standard Deviation The variance or standard deviation is the most common measure of the spread of a set of points: Standard deviation s is square root of variance s 2 s 2 = 1 n 1 (x i x) ҧ 2 i=1 n n x ҧ = 1 n i=1 x i Like mean: highly susceptible to outliers: 19 Alternatives include Median Absolute Deviation (MAD) MAD = Median(x i median(x))

20 Coefficient of Variation (CV) Measure of relative spread, in relation to mean of the population As the CV utilizes mean, not valid for ordinal or nominal data types, also typically not valid for interval data Coefficient has no units 20 CV = SD തX (100%) Example: Comparing the variability in shock level to that of blood pressure can be difficult. Shock μ =.69, σ =.20 BP: μ = 138, σ = 26 CV Shock = CV BP = % = % = 18.8

21 Extra: Higher-Order Descriptive Statistics Kurtosis Measures the tail-heaviness of the distribution. Very loosely concerns the likelihood of seeing a value farther from the mean. Extremely dependent on the variance Skewness Measure of the asymmetry of the distribution of a variable. The skew value of a normal distribution is zero, Positive skew value indicates a tail on the right side Negative skew value indicates a tail on the left side 21

22

23 Exploring through Visualization

24 Many Ways To Visualize 24

25 Scatter plot Provides a first look at data to see clusters of points, outliers Each pair of values is a pair of coordinates and plotted as points in the plane Often additional attributes can be displayed by using the size, shape, and color of the markers that represent the objects. 25

26 Scatterplot Matrix Influence of Collection Site and Methods on Postmortem Morphine Concentrations in a Porcine Model 26

27 Strip Plot 27

28 Jitter / Swam Plot horizontal spread is artificially added All Points Are Shown 28

29 Measuring the Dispersion of : Five number summary: min, Q 1, median, Q 3, max Quartiles: Q 1 (25 th percentile), Q 3 (75 th percentile) Inter-quartile range: IQR = Q 3 Q 1 Boxplot: is represented as: 29 i.e., the height Q 1, Q 3, IQR: The ends of the box are at the first and third quartiles of the box is IQR Median (Q 2 ) is marked by a line within the box Whiskers: Two lines outside the box extended to Minimum and Maximum Quartiles & Boxplots

30 Box Plots Address Things Like Is a feature significant? Does the location differ between subgroups? Does the variation differ between subgroups? Are there outliers in the data? More on this next week 30

31 Combining Figure Types 31

32 Graph display of tabulated frequencies, shown as bars Usually shows the percentage for nominal attributes Can be broken down to compare multiple same attributes across multiple classes Bar Chart 32

33 Bar Chart Variations 33

34 What Can Bar Charts Address? What is the highest percentage categories? How are the category values spread out? What is the modality of the data? How do categories compare between groups? 34

35 Histograms Graph display of tabulated frequencies, shown as bars Usually shows the distribution of values of a single variable of objects in each bin. The height of each bar indicates the number of objects. The shape depends on the number of bins. 35

36 What Can Histograms Address? What kind of population distribution do the data come from? Where are the data located? How spread out are the data? Are the data symmetric or skewed? Are there outliers in the data? 36

37 Histograms Often Tell More than Boxplots The two histograms may have the same boxplot The same values for: min, Q1, median, Q3, max But they have rather different data distributions 37

38 Histogram vs Bar Chart 38

39 Histogram vs Bar Chart Between histograms and bar charts Histograms are used to show distributions of variables while bar charts are used to compare variables Histograms plot binned quantitative data while bar charts plot categorical data Bars can be reordered in bar charts but not in histograms 39

40 Odds ratio Odds Plot? The ratio of one class to another as a function of feature values. OddRatio( x, y) i j k p( x y 1) ij p( x y 0) ij Odds of a patient having diabetes given their plasma glucose concentration Plasma glucoseconcentration 1 = patient has diabetes 0 = patient does not have diabetes 40

41 What Can Odds Plots Address? How do feature values affect the probability of occurrence? Is there a threshold for the effect? 41

42 Quantile Plot Displays all of the data (allowing the user to assess both the overall behavior and unusual occurrences) Plots quantile information For a data x i data sorted in increasing order, f i indicates that approximately 100 f i % of the data are below or equal to the value x i 42

43 Quantile-Quantile (Q-Q) Plot Graphs the quantiles of one univariate distribution against the corresponding quantiles of another Example shows unit price of items sold at Branch 1 vs. Branch 2 for each quantile. Unit prices of items sold at Branch 1 tend to be lower than those at Branch 2 43

44 Heatmap Heatmaps visualize data through variations in color. They are useful for: Cross-examining multivariate data Showing variance across multiple variables, Revealing patterns, Displaying similar variables Typically, all the rows are one category (labels displayed on the left or right side) and all the columns are another category (labels displayed on the top or bottom). Cells either contain color categorical data or numerical data, based on a color scale. 44

45 Heatmap 45

46 Bonus: KDE Estimates A major problem with histograms, is that the choice of binning can have a disproportionate effect on the resulting visualization which may lead to different interpretations of the data. 46

47 KDE Estimates Instead we can attempt to estimate the parameters of the underlying distribution with respect to a number of known distribution types Kernel is a type of probability density function (PDF) Essentially, at every point (+), a kernel function is created with the point at its center The PDF is then estimated by adding all of these kernel functions and dividing by the number of data Intuitively, a kernel density estimate is a sum of bumps. A bump is assigned to every point, and the size of the bump represents the probability assigned at the neighborhood of values around that point; thus, if the data set contains 2 points at x = points at x = 0.5 the bump at x = 1.5 is twice as big as the bump at x = 0.5. Not only a visualization, but we can sample from this as well to get new data 47

48 Lots of Kernel Options 48

49 Cleaning 49