Data Preprocessing. Motivation

Transcription

1 Data Preprocessig Mirek Riedewald Some slides based o presetatio by Jiawei Ha ad Michelie Kamber Motivatio Garbage-i, garbage-out Caot get good miig results from bad data Need to uderstad data properties to select the right techique ad parameter values Data cleaig Data formattig to match techique Data maipulatio to eable discovery of desired patters 1

2 Data Records Data sets are made up of data records A data record represets a etity Examples: Sales database: customers, store items, sales Medical database: patiets, treatmets Uiversity database: studets, professors, courses Also called samples, examples, tuples, istaces, data poits, objects Data records are described by attributes Database row = data record; colum = attribute 3 Attributes Attribute (or dimesio, feature, variable): a data field, represetig a characteristic or feature of a data record E.g., customerid, ame, address Types: Nomial (also called categorical) No orderig or meaigful distace measure Ordial Ordered domai, but o meaigful distace measure Numeric Ordered domai, meaigful distace measure Cotiuous versus discrete 4

3 Attribute Type Examples Nomial: category, status, or ame of thig Hair_color = {black, brow, blod, red, aubur, grey, white} marital status, occupatio, ID umbers, zip codes Biary: omial attribute with oly states (0 ad 1) Symmetric biary: both outcomes equally importat e.g., geder Asymmetric biary: outcomes ot equally importat. e.g., medical test (positive vs. egative) Ordial Values have a meaigful order (rakig) but magitude betwee successive values is ot kow Size = {small, medium, large}, grades, army rakigs 5 Numeric Attribute Types Quatity (iteger or real-valued) Iterval Measured o a scale of equal-sized uits Values have order E.g., temperature i C or F, caledar dates No true zero-poit Ratio Iheret zero-poit We ca speak of values as beig a order of magitude larger tha the uit of measuremet (10m is twice as high as 5m). E.g., temperature i Kelvi, legth, couts, moetary quatities 6 3

4 Discrete vs. Cotiuous Attributes Discrete Attribute Has oly a fiite or coutably ifiite set of values Nomial, biary, ordial attributes are usually discrete Iteger umeric attributes Cotiuous Attribute Has real umbers as attribute values E.g., temperature, height, or weight Practically, real values ca oly be measured ad represeted usig a fiite umber of digits Typically represeted as floatig-poit variables 7 Data Preprocessig Overview Descriptive data summarizatio Data cleaig Data itegratio Data trasformatio Summary 8 4

5 Measurig the Cetral Tedecy Sample mea: 1 x x i i1 Weighted arithmetic mea: x Trimmed mea: set weights of extreme values to zero Media Middle value if odd umber of values; average of the middle two values otherwise Mode Value that occurs most frequetly i the data Uimodal, bimodal, trimodal distributio i1 i1 w x i w i i 9 Measurig Data Dispersio: Boxplot Quartiles: Q 1 (5th percetile), Q 3 (75th percetile) Iter-quartile rage: IQR = Q 3 Q 1 Various defiitios for determiig percetiles, e.g., for N records, the p-th percetile is the record at positio (p/100)n+0.5 i icreasig order If ot iteger, roud to earest iteger or compute weighted average E.g., for N=30, p=5 (to get Q1): 5/100* = 8, i.e., Q1 is 8-th largest of the 30 values E.g., for N=3, p=5: 5/100*3+0.5 = 8.5, i.e., Q1 is average of 8-th ad 9-th largest values Boxplot: eds of the box are the quartiles, media is marked, whiskers exted to mi/max Ofte plots outliers idividually Outlier: usually, a value higher (or lower) tha 1.5 x IQR from Q3 (or Q1) 10 5

6 Measurig Data Dispersio: Variace Sample variace (aka secod cetral momet): m s 1 i1 ( x x) 1 Stadard deviatio = square root of variace Estimator of true populatio variace from a sample: 1 s 1 ( xi x) 1 i1 i i1 x i x 11 Graph display of tabulated frequecies, show as bars Shows what proportio of cases fall ito each category Area of the bar deotes the value, ot the height Crucial distictio whe the categories are ot of uiform width! Histogram 1 6

7 Scatter plot Visualizes relatioship betwee two attributes, eve a third (if categorical) For each data record, plot selected attribute pair i the plae 13 Correlated Data 14 7

8 Not Correlated Data 15 Data Preprocessig Overview Descriptive data summarizatio Data cleaig Data itegratio Data trasformatio Summary 16 8

9 Why Data Cleaig? Data i the real world is dirty Icomplete: lackig attribute values, lackig certai attributes of iterest, or cotaiig oly aggregate data E.g., occupatio= Noisy: cotaiig errors or outliers E.g., Salary= -10 Icosistet: cotaiig discrepacies i codes or ames E.g., Age= 4 ad Birthday= 03/07/1967 E.g., was ratig 1,, 3, ow ratig A, B, C E.g., discrepacy betwee duplicate records 17 Example: Bird Observatio Data Chage of rage boudaries over time, e.g., for temperature Differet uits, e.g., meters versus feet for elevatio Additio or removal of attributes over the years Missig etries, especially for habitat ad weather People wat to watch birds, ot fill out log forms GIS data based o 30m cells or 1km cells Locatio accuracy ZIP code versus GPS coordiates Walk alog trasect but report oly sigle locatio Icosistet ecodig of missig etries Hairy vs. Dowy Woodpecker 0, -9999, -3.4E+38 eed cotext to decide Varyig observer experiece ad capabilities Cofusio of species Missed species that was preset Cofusio about reportig protocol Report max versus sum see Report oly iterestig species, ot all 18 9

10 How to Hadle Missig Data? Igore the record Usually doe whe class label is missig (for classificatio tasks) Fill i maually Tedious ad ofte ot clear what value to fill i Fill i automatically with oe of the followig: Global costat, e.g., ukow Ukow could be mistake as ew cocept by data miig algorithm Attribute mea Attribute mea for all records belogig to the same class Most probable value: iferece-based such as Bayesia formula or decisio tree Some methods, e.g., trees, ca do this implicitly 19 How to Hadle Noisy Data? Noise = radom error or variace i a measured variable Typical approach: smoothig Adjust values of a record by takig values of other earby records ito accout Dozes of approaches Biig, average over eighborhood Regressio: replace origial records with records draw from regressio fuctio Idetify ad remove outliers, possibly ivolvig huma ispectio For this class: do t do it uless you uderstad the ature of the oise A good data miig techique should be able to deal with oise i the data 0 10

11 Data Preprocessig Overview Descriptive data summarizatio Data cleaig Data itegratio Data trasformatio Summary 3 Data Itegratio Combies data from multiple sources ito a coheret store Etity idetificatio problem Idetify real world etities from multiple data sources, e.g., Bill Clito = William Clito Detectig ad resolvig data value coflicts For the same real world etity, attribute values from differet sources might be differet Possible reasos: differet represetatios, differet scales, e.g., metric vs. US uits Schema itegratio: e.g., A.cust-id B.cust-# Itegrate metadata from differet sources Ca idetify idetical or similar attributes through correlatio aalysis 4 11

12 Covariace (Numerical Data) Covariace computed for data samples (A 1, A,..., A ) ad (B 1, B,..., B ): 1 Cov( A, B) i1 ( A A)( B i 1 B) If A ad B are idepedet, the Cov(A, B) = 0, but the coverse is ot true Two radom variables may have covariace of 0, but are ot idepedet If Cov(A, B) > 0, the A ad B ted to rise ad fall together The greater, the more so If covariace is egative, the A teds to rise as B falls ad vice versa i i1 A B A B i i 5 Covariace Example Suppose two stocks A ad B have the followig values i oe week: A: (, 3, 5, 4, 6) B: (5, 8, 10, 11, 14) AVG(A) = ( )/ 5 = 0/5 = 4 AVG(B) = ( ) /5 = 48/5 = 9.6 Cov(A,B) = ( )/ = 4 Cov(A,B) > 0, therefore A ad B ted to rise ad fall together 6 1

13 Correlatio Aalysis (Numerical Data) Pearso s product-momet correlatio coefficiet of radom variables A ad B: Cov( A, B) A, B Computed for two attributes A ad B from data samples (A 1, A,..., A ) ad (B 1, B,..., B ): 1 A i A Bi B r A, B 1 i1 sa sb Where A ad B are the sample meas, ad s A ad s B are the sample stadard deviatios of A ad B (usig the variace formula for s ). Note: -1 r A,B 1 r A,B > 0: A ad B positively correlated The higher, the stroger the correlatio r A,B < 0: egatively correlated A B 7 Correlatio Aalysis (Categorical Data) (chi-square) test (Observed Expected) Expected The larger the value, the more likely the variables are related The cells that cotribute the most to the value are those whose actual cout is very differet from the expected cout Correlatio does ot imply causality # of hospitals ad # of car-thefts i a city are correlated Both are causally liked to the third variable: populatio 8 13

14 Chi-Square Example Play chess Not play chess Sum (row) Like sciece fictio 50 (90) 00 (360) 450 Not like sciece fictio 50 (10) 1000 (840) 1050 Sum(col.) Numbers i parethesis are expected couts calculated based o the data distributio i the two categories (50 90) 90 (50 10) 10 (00 360) 360 ( ) It shows that like_sciece_fictio ad play_chess are correlated i the group 9 Data Preprocessig Overview Descriptive data summarizatio Data cleaig Data itegratio Data trasformatio Summary 30 14

15 Why Data Trasformatio? Make data more mieable E.g., some patters visible whe usig sigle time attribute (etire date-time combiatio), others oly whe makig hour, day, moth, year separate attributes Some patters oly visible at right graularity of represetatio Some methods require ormalized data E.g., all attributes i rage [0.0, 1.0] Reduce data size, both #attributes ad #records 31 Normalizatio Mi-max ormalizatio to [ew_mi A, ew_max A ]: v mi A v' (ew_max A ew_mi max mi E.g., ormalize icome rage [$1,000, $98,000] to [0.0, 1.0]. The $73,000 is mapped to Normalizatio by decimal scalig: A where j is the smallest iteger such that Max( ν ) < 1 A 73,600 1,000 (1.0 0) ,000 1,000 Z-score ormalizatio (μ: mea, σ: stadard deviatio): A ) ew_mi A A v A v' 73,600 54,000 E.g., for μ = 54,000 ad σ = 16,000, $73,000 is mapped to ,000 v v' 10 j 3 15

16 Data Reductio Why data reductio? Miig cost ofte icreases rapidly with data size ad umber of attributes Goal: reduce data size, but produce (almost) the same results Data reductio strategies Dimesioality reductio Data Compressio Numerosity reductio Discretizatio 33 Dimesioality Reductio: Attribute Subset Selectio Feature selectio (i.e., attribute subset selectio): Select a miimum set of attributes such that the miig result is still as good as (or eve better tha) whe usig all attributes Heuristic methods (due to expoetial umber of choices): Select idepedetly based o some test Step-wise forward selectio Step-wise backward elimiatio Combiig forward selectio ad backward elimiatio Elimiate attributes that some trusted method did ot use, e.g., a decisio tree 34 16

17 Pricipal Compoet Aalysis Fid projectio that captures largest amout of variatio i the data Space defied by eigevectors of the covariace matrix Compressio: use oly first k eigevectors x e 1 e x 1 39 Data Reductio Method: Samplig Select a small subset of a give data set Reduces miig cost Miig cost usually is super-liear i data size Ofte makes differece betwee i-memory processig ad eed for expesive I/O Choose a represetative subset of the data Simple radom samplig may have poor performace i the presece of skew Develop adaptive samplig methods Stratified samplig Approximate the percetage of each class (or sub-populatio of iterest) i the overall database Used i cojuctio with skewed data 41 17

18 Samplig with or without Replacemet Raw Data 4 Samplig: Cluster or Stratified Samplig Raw Data Cluster/Stratified Sample 43 18

19 Data Reductio: Discretizatio Applied to cotiuous attributes Reduces domai size Makes the attribute discrete ad hece eables use of techiques that oly accept categorical attributes Approach: Divide the rage of the attribute ito itervals Iterval labels replace the origial data 44 Data Preprocessig Overview Descriptive data summarizatio Data cleaig Data itegratio Data trasformatio Summary 45 19

20 Summary Data preparatio is a big issue for data miig Descriptive data summarizatio is used to uderstad data properties Data preparatio icludes Data cleaig ad itegratio Data reductio ad feature selectio Discretizatio May techiques ad commercial tools, but still major challege ad active research area 46 0