Data Analysis. Concepts and Techniques. Chapter 2. Chapter 2: Getting to Know Your Data. Data Objects and Attribute Types

Transcription

1 Data Aalysis Cocepts ad Techiques Chapter 2 1 Chapter 2: Gettig to Kow Your Data Data Objects ad Attribute Types Basic Statistical Descriptios of Data Data Visualizatio Measurig Data Similarity ad Dissimilarity Summary 2 1

2 Types of Data Sets Record Relatioal records Data matrix, e.g., umerical matrix, crosstabs Documet data: text documets: termfrequecy vector Trasactio data Graph ad etwork World Wide Web Social or iformatio etworks Molecular Structures Ordered Video data: sequece of images Temporal data: time-series Sequetial Data: trasactio sequeces Geetic sequece data Spatial, image ad multimedia: Spatial data: maps Image data: Video data: Documet 1 Documet 2 Documet 3 team coach pla y ball score game wi lost timeout TID Items 1 Bread, Coke, Milk 2 Beer, Bread 3 Beer, Coke, Diaper, Milk 4 Beer, Bread, Diaper, Milk 5 Coke, Diaper, Milk seaso 3 Importat Characteristics of Structured Data Dimesioality Curse of dimesioality Sparsity Oly presece couts Resolutio Patters deped o the scale Distributio Cetrality ad dispersio 4 2

3 Data Objects Data sets are made up of data objects. A data object represets a etity. Examples: sales database: customers, store items, sales medical database: patiets, treatmets uiversity database: studets, professors, courses Also called samples, examples, istaces, data poits, objects, tuples. Data objects are described by attributes. Database rows -> data objects; colums ->attributes. 5 Attributes Attribute (or dimesios, features, variables): a data field, represetig a characteristic or feature of a data object. E.g., customer _ID, ame, address Types: Nomial Biary Numeric: quatitative Iterval-scaled Ratio-scaled 6 3

4 Attribute Types Nomial: categories, states, or ames of thigs Hair_color = {aubur, black, blod, brow, grey, red, white} marital status, occupatio, ID umbers, zip codes Biary Nomial attribute with oly 2 states (0 ad 1) Symmetric biary: both outcomes equally importat e.g., geder Asymmetric biary: outcomes ot equally importat. e.g., medical test (positive vs. egative) Covetio: assig 1 to most importat outcome (e.g., HIV positive) Ordial Values have a meaigful order (rakig) but magitude betwee successive values is ot kow. Size = {small, medium, large}, grades, army rakigs 7 Numeric Attribute Types Quatity (iteger or real-valued) Iterval Measured o a scale of equal-sized uits Values have order E.g., temperature i C or F, caledar dates No true zero-poit Ratio Iheret zero-poit We ca speak of values as beig a order of magitude larger tha the uit of measuremet (10 K is twice as high as 5 K ). e.g., temperature i Kelvi, legth, couts, moetary quatities 8 4

5 Discrete vs. Cotiuous Attributes Discrete Attribute Has oly a fiite or coutably ifiite set of values E.g., zip codes, professio, or the set of words i a collectio of documets Sometimes, represeted as iteger variables Note: Biary attributes are a special case of discrete attributes Cotiuous Attribute Has real umbers as attribute values E.g., temperature, height, or weight Practically, real values ca oly be measured ad represeted usig a fiite umber of digits Cotiuous attributes are typically represeted as floatigpoit variables 9 Chapter 2: Gettig to Kow Your Data Data Objects ad Attribute Types Basic Statistical Descriptios of Data Data Visualizatio Measurig Data Similarity ad Dissimilarity Summary 10 5

6 Basic Statistical Descriptios of Data Motivatio To better uderstad the data: cetral tedecy, variatio ad spread Data dispersio characteristics media, max, mi, quatiles, outliers, variace, etc. Numerical dimesios correspod to sorted itervals Data dispersio: aalyzed with multiple graularities of precisio Boxplot or quatile aalysis o sorted itervals Dispersio aalysis o computed measures Foldig measures ito umerical dimesios Boxplot or quatile aalysis o the trasformed cube 11 Measurig the Cetral Tedecy Mea (algebraic measure) (sample vs. populatio): Note: is sample size ad N is populatio size. Weighted arithmetic mea: Trimmed mea: choppig extreme values Media: Mode Middle value if odd umber of values, or average of the middle two values otherwise Estimated by iterpolatio (for grouped data): / 2 ( media L1 ( freq Value that occurs most frequetly i the data Uimodal, bimodal, trimodal Empirical formula: freq) l ) width media x 1 x i 1 i 1 Media iterval w x x i i 1 mea mode 3 ( mea media ) i w i i x N 12 6

7 Symmetric vs. Skewed Data Media, mea ad mode of symmetric, positively ad egatively skewed data symmetric positively skewed egatively skewed August 24, 2015 Data Miig: Cocepts ad Techiques 13 Measurig the Dispersio of Data Quartiles, outliers ad boxplots Quartiles: Q 1 (25 th percetile), Q 3 (75 th percetile) Iter-quartile rage: IQR = Q 3 Q 1 Five umber summary: mi, Q 1, media, Q 3, max Boxplot: eds of the box are the quartiles; media is marked; add whiskers, ad plot outliers idividually Outlier: usually, a value higher/lower tha 1.5 x IQR Variace ad stadard deviatio (sample: s, populatio: σ) Variace: (algebraic, scalable computatio) 2 1 s 1 i 1 ( x x) i 2 1 [ xi ( xi ) i 1 i 1 Stadard deviatio s (or σ) is the square root of variace s 2 ( or σ 2) ] ( xi ) xi N i 1 N i

8 Boxplot Aalysis Five-umber summary of a distributio Miimum, Q1, Media, Q3, Maximum Boxplot Data is represeted with a box The eds of the box are at the first ad third quartiles, i.e., the height of the box is IQR The media is marked by a lie withi the box Whiskers: two lies outside the box exteded to Miimum ad Maximum Outliers: poits beyod a specified outlier threshold, plotted idividually 15 Visualizatio of Data Dispersio: 3-D Boxplots August 24, 2015 Data Miig: Cocepts ad Techiques 16 8

9 Properties of Normal Distributio Curve The ormal (distributio) curve From μ σ to μ+σ: cotais about 68% of the measuremets (μ: mea, σ: stadard deviatio) From μ 2σ to μ+2σ: cotais about 95% of it From μ 3σ to μ+3σ: cotais about 99.7% of it 17 Graphic Displays of Basic Statistical Descriptios Boxplot: graphic display of five-umber summary Histogram: x-axis are values, y-axis repres. frequecies Quatile plot: each value x i is paired with f i idicatig that approximately 100 f i % of data are x i Quatile-quatile (q-q) plot: graphs the quatiles of oe uivariat distributio agaist the correspodig quatiles of aother Scatter plot: each pair of values is a pair of coordiates ad plotted as poits i the plae 18 9

10 Histogram Aalysis Histogram: Graph display of tabulated 40 frequecies, show as bars It shows what proportio of cases fall ito each of several categories Differs from a bar chart i that it is the area of the bar that deotes the value, ot the height as i bar charts, a crucial distictio whe the categories are ot of uiform width The categories are usually specified as 0 o-overlappig itervals of some variable. The categories (bars) must be adjacet Histograms Ofte Tell More tha Boxplots The two histograms show i the left may have the same boxplot represetatio The same values for: mi, Q1, media, Q3, max But they have rather differet data distributios 20 10

11 Quatile Plot Displays all of the data (allowig the user to assess both the overall behavior ad uusual occurreces) Plots quatile iformatio For a data x i data sorted i icreasig order, f i idicates that approximately 100 f i % of the data are below or equal to the value x i Data Miig: Cocepts ad Techiques 21 Quatile-Quatile (Q-Q) Plot Graphs the quatiles of oe uivariate distributio agaist the correspodig quatiles of aother View: Is there is a shift i goig from oe distributio to aother? Example shows uit price of items sold at Brach 1 vs. Brach 2 for each quatile. Uit prices of items sold at Brach 1 ted to be lower tha those at Brach

12 Scatter plot Provides a first look at bivariate data to see clusters of poits, outliers, etc Each pair of values is treated as a pair of coordiates ad plotted as poits i the plae 23 Positively ad Negatively Correlated Data The left half fragmet is positively correlated The right half is egative correlated 24 12

13 Ucorrelated Data 25 13