Exploratory/Visual Data Analysis

Transcription

1 Exploratory/Visual Data Analysis Intelligent Data Analysis 9/14/2018 Budapest University of Technology and Economics Fault Tolerant Systems Research Group Budapesti Műszaki és Gazdaságtudományi Egyetem Méréstechnika 1 és Információs Rendszerek Tanszék 1

2 Data mining brickstones Clustering Classification Association rules Regression 2 2

3 Clustering 3 3

4 Association rules 4 4

5 Classification 5 5

6 Regression 6 6

7 Model Data analysis New knowledge Data THE SYSTEMATIC WAY: EXPLORATORY DATA ANALYSIS Look and see 7 7

8 Dr. Snow and 1854 London cholera epidemics About half of our sensory neurons are dedicated to vision, endowing us with a remarkable pattern-recognition ability. Prof. Alfred Inselberg 8 8

9 Exploratory data analysis (EDA) Summary of the main characteristics of a data set o Identification of outliers, trends, other patterns o Often with visual methods. o A statistical model can be used or not, For seeing what the data can tell o beyond formal modeling or hypothesis testing o hypotheses new data collection and experiments 9 9

10 Exploratory data analysis (EDA) Approach to analyzing data sets o Summary of their main characteristics in Easy-to-understand, often: visual graphs without using statistical model or having formulated a hypothesis. Prepares Confirmatory Data Analysis o Checking statistical model or a hypothesis. Dynamic visualization capabilities o Identification of outliers, trends and patterns Reduction of the sensitivity of inferences to errors : o Robust statistics - low sensitivity to outliers o Nonparametric statistics no a priori assumptions 10 10

11 Approach-visual exploratory analytics Resources sensors processors Process based on interactivity 1. Graphical presentation of the data multiple diagrams 2. Visual evaluation exploiting human overview 3. Visual selection, manipulation multiple diagrams 4. Interpretation, correlation with other models, evaluation (like architecture etc.)

12 Visualisation in Everyday Life Trend Analysis and Forcast Time Series Analysis Correlation Analysis Analysis of Spatial Data 12 12

13 Additional knowledge People buying coffee often buy milk There is a significant difference in salaries depending on gender The memory consumption of a software grows exponentially wrt. number of requests in queue. The population follows a N(100, 15) disribution BME students fall into 3 main different groups (according to their grades) 13 13

14 Cleaning Data analysis Modell Data Data analysis Exploratory Confirmatory Additional knowledge 14 14

15 Data analysis Exploratory analysis Goal: formulate hypotheses Know the data/domain Highly ad-hoc Mainly descriptive statistics+data mining+visualization Confirmatory analízis Goal: test hypotheses Validate Mainly statistical tests + statistical inference 15 15

16 Data analysis E.g. distribution analysis Exploratory Hypothesis: variable x follows normal distribution Confirmatory Variable x follows N(12, 4) distribution 16 16

17 Data analysis E.g., linear regression Exploratory Hypothesis: variables x and y are linearly correlated Confirmatory Y = 3.2 x 1.2 R 2 =

18 Cleaning Data analysis Modell Data Data analysis Exploratory Confirmatory Additional knowledge 18 18

19 Data cleaning Reasons for inconsistencies o Input/measurement errors, wrong join, partial observation, fraud, Outliers o =/= gross error o Can be an analysis goal to identify o Hard to detect in case of multiple dimensions o Visualization with (low) dimensions number may help 19 19

20 Boxplot: five-number summary Q1 Q3 Min. Median Max. Q1 1.5 IQR Normal Extr. small Small Large Extr. large 20 20

21 Example: Robust and non-paramteric measures Sample set o 1000 points o U(1, 5) unif. Distribution mean = median = 3 ms 3ms ± 2 ms 1 sample of 20 sec Response time New median: sort(resp. times)[501] = 3.02 ms Resp. t. median Resp. t. mean Robust Non-robust New mean: (2 * 10^4 + 3 * 10^3 )/ 1001 = 25 ms! 21 21

22 Parallel coordinates: multi(many)factor analysis You can do much, much more o Redundancy reduction o Correlation analysis o Clustering o Data mining o Approximation o Optimization Inselberg, A: Parallel Coordinates, Visual Multidimensional Geometry and Its Applications, Springer

23 Example: host CPU usage Host CPU usage Ratio of bad vcpu ready Different failure domains, as outlier clusters Enough slack Better workload management Scheduling helps 23 23

24 Example: Strange observation Computing power use = CPU use ratio CPU clk rate (const.) Should be pure proportional Correlation coefficient:

25 Repetition: Basic statistics Min: Max: Mean: Variance: smallest value gratest value x ҧ = x 1+x 2 + +x n n σ 2 = σ n i=1 (xi x ҧ ) 2 n Think Stats: Probability and Statistics for Programmers 25 25

26 Reviewing the Calculations Avoiding erroneous assumptions Intuition! Overly primitive model Single outlier! Lack of robustness For all cases: Means: M x = 9 M y ~7.5 Variance: σ x = 11 σ y ~4.12 Correlation: C x, y ~0.816 Regression: y ~ x

27 Example: Visualisation of State Spaces

28 Example: State Space of the CAN Bus

29 Example: System Model Performance Model Modelling System model Design of the Analysis Expectations Data Collection Measurements Benchmarks Qualitative model Performance model Parameters to measure Design of the experiments Simulation Analysis Exploratory Analysis Hypothesis testing 29 29

30 Example: performance analysis Outliers/background operations Deviations?? Linearity Unpredictable behavior

31 Reminder: Tabular Representation Rows of the table = Model elements Columns of the table = Properties Name Type Size (kb) Last modified Documents directory Contracts.pdf file Pictures directory Logo.png file Groundplot.jpg file Data analysis languages (e.g. R, Python): dataframe o One row: one measurement/observation o Columns have their own Types 31 31

32 Numerical and Categorical Variables Numerical o Arithmetic operations are interpreted meaningful (average, sum, inc, dec, ) Variables Categorical o No operation between the values Numerical Categorical Age Phone number Average temperature Sex 32 32

33 Numerical Variables Continuous o Measured in a specific range with a specific precision o e.g. temperature of a the server room Numerical Variable Categorical Integer Continuous Discrete o Counted finite number of values in a specific range o e.g. number of disks 33 33

34 Categorical variables Ordered (ordinal) o Excelent, good, fair, poor o Value range is ordered o Fully ordered? Un-ordered (nominal) o Types o City names Numerical Ordered Variables Categorical Un-ordered 34 34

35 Single variable Variables Változók Numerikus Numerical Kategorikus Categorical Test results: [28, 28, 30, ] Training groups: [G01, G02, G03, ] 35 35

36 Column Charts / Bar Charts Input variable: Course codes Question: How many students have subscribed? Are there popular time slots / trainers? Absolute Frequency! Height of bar: Frequency of the given value Design decision: Splitting of the value range e.g.: Tuesday-Thursday-Friday? 36 36

37 Single variable Variables Változók Numerikus Numerical Kategorikus Categorical Test results: [28, 28, 30, ] Training groups: [G01, G02, G03, ] 37 37

38 Histogram Input variable: Test results Question: What results were born? Height of bar : Frequency of the given interval of values Absolute Frequency! Design decision : Choosing length of the intervals e.g.: 1-point-resolution vs. 0,5-point-resolution? 38 38

39 Histogram Input variable: Test results Question: What results were born? Many have the entry test almost made. The most of those who made the entry test, made the whole test also. Those not appeared at all. The average/ median was at 20 points

40 Simple Statistical Description Where is the middle of the values? 40 40

41 Simple Statistical Description How far are the values scattered? 41 41

42 Simple Statistical Description Are there outliers? 42 42

43 Box plots Input variable: Test results Question: What results were born approximately? An art of abstraction: just take some intervals, the exact values are not that important 43 43

44 Description of (Continuous) Observations Description of the middle 1. Average arithmetic mean 2. Median the element separating the upper half from the lower half (ordered data sets!) 3. Mode the most frequent element o Example: {3, 4, 4, 5, 5, 6, 10, 20} Mean: ~ Median: 5 Mode: 4 and 5 (often as 4.5) Description of the spread o Percentiles (frequency for categorical types) 44 44

45 Describing (cont.) Observations If the elements of a data set are ordered, the middle element is the median of the data set. In the case if there is no middle element (an even number of elements), the median is the average of the two middle elements. The mode is the most frequent element (the most frequent elements) of the data set. If there is no unique most frequent element, the data set has multiple Modes

46 Percentiles Percentile o n% of the values are weaker than the n th percentile o {3, 4, 4, 5, 5, 6, 10, 20} 50. percentile: percentile: percentile: 10 Quartiles o Q1: 25. percentile o Q3: 75. percentile o Q2: Median Frequency: n% of the values lie in the given categorie(s) 46 46

47 Example: Representation of Percentiles 47 47

48 Box and whisker plot 48 48

49 Box and whisker plot Limit of the upper outliers Q3 Median Q1 Limit of the lower outliers How to do it in Excel? (

50 Box and whisker plot Task2 has produced (in average) better results than Task1. 50% of the results of Task1 lie between 4.5 and

51 Box and whisker plot How were the results per training groups? Very good entry tests in G06, G11 and G17 G08? 51 51

52 Box and whisker plot Abstraction: With box plot we can miss important information

53 Boxplot (Box and whisker plot) Interquartile range ±1.5 IQR is nearly 3σ 53 53

54 Boxplots vs qualitative domains Min. Median Max. Qualitative Extr. small Small Normal Large Extr. large Q1 1.5 IQR Quantitative Extr. small Small Normal Large Extr. large 54 54

55 Median instead of Mean Why? Data set: 1000 Points in (1, 5) with uniform distribution Mean = Median = 3 ms 3ms ± 2 ms 1 element: 20 s Response time Median Mean New Median: sort(resp. times)[501] = ms Robust Not robust New Mean: (2 * 10^4 + 3 * 10^3 )/ 1001 = 23 ms! 55 55

56 Relation between two Variables Variable Numerical Categorical 2 numerical 2 categorical 1 numerical, 1 categorical 56 56

57 Numerical, per Category 57 57

58 Relation between two Variables Variable Variable Numerical Categorical Numerical Categorical 2 numerical 1 numerical, 1 categorical 2 categorical 58 58

59 Scatter Plot Input variable: Results of the two main test tasks Question: Is there any correlation? Pairs of results are displayed (visualised). Where one of them is missing, the pair cannot be shown

60 Scatter Plot Input variable: Results of the two main test tasks Question: Is there any correlation? Who had a good result for Task1, had not necessarily a good one for Task2. How should overlapping be handled? 60 60

61 Overplotting 61 61

62 Overplotting Solution 1: Jitter 62 62

63 Overplotting Solution 2: Transparency 63 63

64 Overplotting Solution 3: Size 64 64

65 Relation between two Variables Variable Variable Numerical Categorical Numerical Categorical 1 numerical, 2 numerical 2 categorical 1 categorical 65 65

66 Mosaic Plot Relation between 2 or more categorical variables Guardian's list of "1000 songs to hear before you die"

67 MULTIPLE VARIABLES 67 67

68 More than Two Variables Changing the properties of the graphical elements o Color o Size o Texture o Place (non-trivial way, but look at tree maps, there the place has a direct meaning) E.g. bubble chart, heatmap, treemap 68 68

69 3D Plot 69 69

70 Bubble Chart: Average Age by Regions 70 70

71 Scatterplot matrix 71 71

72 Heatmap: Operations Statistics We mostly write simple texts. In Start we only start, but we never return

73 Pairwise Correlation of Multiple Values variable names independent values blue: positive, red: inverse darker colours: stronger correlation strong correlation Drawn with the corrgram package of the statistics software R. Correlation (see Probability Theory): Strength and direction of the linear dependency between two variables Over the diagonal: scatterplot matrix Goal: Filtering out the related variables, identification of the outliers. Which variables are important for the prediction of the load? 73 73

74 Tree Map: e.g. File System 74 74

75 Parallel Coordinates Multi-dimensional visualization Compact, scalable Axis order? [2, 4, 4, 5, 6] [3, 6, 6, 4, 5] [1, 2, 2, 3, 3] [3, 4, 6, 5, 6] [2, 5, 4, 6, 4] [1, 3, 2, 3, 2] A B C D E A D C E B 75 75

76 Parallel Coordinates: Analysis of the Test Cases 1 test case: 1 broken line The variables appear on the x-axis 76 76

77 Parallel Coordinates: Analysis of the Test Cases Timeout? The ones detecting an error did not even came to the actual computation. Run time and memory usage seem to be in a positive relation (if the test is successful) 77 77

78 Parallel Coordinates: the Alternatives 78 78

79 Radar Chart: An Extension of Parallel Coord

80 References Tukey, John W. "Exploratory data analysis." (1977): 2. Inselberg, Alfred, and Bernard Dimsdale. "Parallel coordinates for visualizing multi-dimensional geometry." Computer Graphics Springer, Tokyo, Kocsis Imre. Vizuális analízis. In: Antal Péter, Antos András, Horváth Gábor, Hullám Gábor, Kocsis Imre, Marx Péter, Millinghoffer András, Pataricza András, Salánki Ágnes. Intelligens adatelemzés. 141 p. Budapest: Typotex Kiadó, pp (ISBN: ) 80 80