Chapter 2 and 3, Data Pre-processing

Transcription

1 CSI 4352, Itroductio to Data Miig Chapter 2 ad 3, Data Pre-processig Youg-Rae Cho Associate Professor Departmet of Computer Sciece Baylor Uiversity Why Need Data Pre-processig? Icomplete Data Missig values, or Lack of attributes of iterest Noisy Data Errors, or Outliers Redudat Data Duplicate data, or Duplicate attributes e.g., Age = 47, Birthday = 01/07/1968 Icosistet Data Cotaiig discrepacies i format or ame e.g., Ratig by 1, 2, 3, Ratig by A, B, C Huge Volume of Data 1

2 Importace Lower Quality Data, Lower Quality Miig Results!! Miig quality depeds o data quality as well as miig techiques. Majority of Data Miig Data pre-processig comprises the majority of the works for data warehousig ad data miig Major Tasks Data Cleaig Fill i missig values, smooth oisy data, remove outliers, remove redudacy, ad resolve icosistecy Data Itegratio Itegratio of multiple databases or files Data Trasformatio Normalizatio ad aggregatio Data Reductio Reducig represetatio i volume with similar aalytical results Discretizatio of cotiuous data 2

3 CSI 4352, Itroductio to Data Miig Chapter 2 ad 3, Data Pre-processig Geeral Data Characteristics Descriptive Data Summarizatio Data Cleaig Data Itegratio Data Trasformatio Data Reductio Data Types Record Relatioal records Data matrix, e.g., umerical matrix, crosstabs Documet data, e.g., text documets Trasactio data Ordered Data Sequetial data, e.g., trasactio sequeces, biological sequeces Temporal data, e.g., time-series data Spatial data, e.g., maps Graph WWW, iteret Social or iformatio etworks Biological etworks 3

4 Attribute Types Nomial e.g., ID umber, professio, zip code Ordial e.g., rakig, grades, sizes Biary e.g., medical test (positive or egative) Iterval e.g., caledar dates, temperature, height Ratio e.g., populatio, sales Discrete vs. Cotiuous Attributes Discrete Attribute Fiite set of values Sometimes, represeted as iteger values Biary attributes are a special case of discrete attributes Cotiuous Attribute Real umbers as values Typically, represeted as floatig-poit variables I practice, show as a fiite umber of digits 4

5 Characteristics of Data Dimesioality Curse of dimesioality Sparsity Lack of iformatio Resolutio Patters depedig o the scale Similarity Similarity measures for complex types of data CSI 4352, Itroductio to Data Miig Chapter 2 ad 3, Data Pre-processig Geeral Data Characteristics Descriptive Data Summarizatio Data Cleaig Data Itegratio Data Trasformatio Data Reductio 5

6 Descriptive Data Miig Motivatio To better uderstad the properties of data distributios, e.g., cetral tedecy, spread ad variatio Measuremets media, max, mi, quatiles, outliers, etc. Aalysis Process Foldig the measures ito umeric dimesios Graphic aalysis o the trasformed dimesio space Cetral Tedecy Measures Mea Weighted arithmetic mea: Trimmed mea: choppig extreme values Media Middle value if odd umber of values Average of two middle values otherwise Estimatio by iterpolatio for grouped data: N media L ( Mode The value that occurs the most frequetly i the data Uimodal, bimodal, trimodal distributio 1 x / 2 i 1 i 1 ( w freq i w x i med i freq ) low ) width 6

7 Cetral Tedecy i Skewed Data Symmetric Data Mea, Media, Mode Skewed Data Data Dispersio Measures Quartiles ad Outliers Quartiles: Q 1 (25 th percetile), Q 3 (75 th percetile) Iter-quartile rage: IQR = Q 3 Q 1 Outliers: data with extreme low ad high values usually, values lower/higher tha Q IQR / Q IQR Variace ad Stadard Deviatio 2, i populatio: ( xi ) N i 1 N i 1 x 2 i 2 s 2, s by samplig: 2 1 s 1 i 1 ( x x) i 2 1 [ xi ( xi ) i 1 i 1 ] Degree of Freedom: # idepedet pieces of iformatio (= # idepedet measuremet # parameters) 7

8 Graphic Aalysis Boxplot Display of five-umber summary Histogram Display of tabulated frequecies Quatile-Quatile (Q-Q) Plot Descriptio of the relatioship betwee two uivariate distributios Scatter Plot Descriptio of the relatioship betwee two attributes of a bivariate distributio Boxplot Aalysis Five-umber summary of a Distributio Miimum / Q 1 / Media / Q 3 / Maximum Boxplot Represeted as a box The bottom of the box is Q1 The top of the box is Q3 The media is marked by a lie Whiskers: two lies outside of the box exted to miimum ad maximum 8

9 Histogram Aalysis Histogram Uivariate graphic method Represeted as a set of bars reflectig the frequecies of the discrete values Groupig data values ito classes if they are cotiuous Boxplot vs. Histogram Ofte, histogram gives more iformatio tha boxplot Frequecy Frequecy ~59 60~79 80~99 100~ ~139 40~59 60~79 80~99 100~ ~139 Uit Price Uit Price Quatile Plot Aalysis Quatile Plot Plots quatile iformatio of the data (sorted i a ascedig order) Displays all the data Q-Q (Quatile-Quatile) Plot Plots the quatiles of oe uivariate distributio agaist the quatiles of the other Describes the relatioship betwee two distributios 9

10 Scatter Plot Aalysis Scatter Plot Displays the poits of bivariate data Describes the relatioship betwee two attributes (variables) positively correlated data egatively correlated data clusters patters outliers CSI 4352, Itroductio to Data Miig Chapter 2 ad 3, Data Pre-processig Geeral Data Characteristics Descriptive Data Summarizatio Data Cleaig Data Itegratio Data Trasformatio Data Reductio 10

11 Missig Data Data is ot always available e.g., may tuples have o record value for several attributes Missig data may be due to equipmet malfuctio icosistet with other recorded data ad thus deleted data ot etered due to misuderstadig certai data may ot be cosidered importat at the time of etry ot register history or chages of the data Missig data may eed to be iferred How to Hadle Missig Data Igore the missig values Not effective Fill i the missig values maually Tedious, ifeasible? Fill i the missig values automatically with ukow : ot effective The attribute mea The attribute mea of all samples belogig to the same class The most probable value by iferece or classificatio techiques 11

12 Noisy Data Noise Radom error or variace i a measured variable Icorrect data may be due to faulty data collectio istrumets data trasmissio problem techology limitatio icosistecy i data coversio Other Data Problems Duplicate records Icomplete data Icosistet data How to Hadle Noisy Data Biig Sort data ad partitio ito bis Smooth by bi meas, smooth by bi media, smooth by bi boudaries Regressio Smooth by fittig the data ito regressio fuctios Clusterig Detect ad remove outliers Ispectio Semi-automatically Detect suspicious values ad check by huma 12

13 Partitioig for Biig Equal-Width (Distace) Partitioig Divides the rage ito N itervals of equal distace (uiform grid) If A ad B are the lowest ad highest values of the attribute, the the width of itervals will be (B-A)/N. Straightforward Problem: (1) Outliers may domiate the partitios. (2) Skewed data is ot hadled well. Equal-Depth (Frequecy) Partitioig Divides the rage ito N itervals of equal frequecy, i.e., each cotaiig approximately same umber of samples. Problem: Not possible for categorical attributes Data Smoothig for Biig Example Sorted data of price (i dollars): 4,8,9,15,21,21,24,25,26,28,29,34 Partitio ito three equal-frequecy bis Bi 1: 4, 8, 9, 15 Bi 2: 21, 21, 24, 25 Bi 3: 26, 28, 29, 34 Bi 1: 9, 9, 9, 9 Bi 2: 23, 23, 23, 23 Bi 3: 29, 29, 29, 29 smoothig by bi meas Bi 1: 4, 4, 4, 15 Bi 2: 21, 21, 25, 25 Bi 3: 26, 26, 26, 34 smoothig by bi boudaries 13

14 Regressio Liear Regressio Modeled as a liear fuctio of oe variable, Y = w X + b Ofte, uses a least-square method. y 1 Y y 1 y = x + 1 x 1 X Multiple Regressio Modeled as a liear fuctio of a multi-dimesioal feature vector, Y = b 0 + b 1 X 1 + b 2 X 2 May o-liear fuctios ca be trasformed. Log-Liear Model Approximates discrete multi-dimesioal probability distributios. Clusterig Outlier Detectio outliers 14

15 CSI 4352, Itroductio to Data Miig Chapter 2 ad 3, Data Pre-processig Geeral Data Characteristics Descriptive Data Summarizatio Data Cleaig Data Itegratio Data Trasformatio Data Reductio Data Itegratio Defiitio Process to combie multiple data sources ito coheret storage Process to provide uiform iterface to multiple data sources Process Data Modelig Schema Matchig Data Extractio Data Modelig Creatig global schema (mediated schema) Schema Matchig Matchig betwee two attributes of differet sources The most critical step of data itegratio Schema-level matchig / Istace-level matchig 15

16 Istace-Level Matchig Defiitio Detectig ad resolvig data value coflicts Etity Idetificatio For the same real world etity, values from differet sources might be differet Possible reasos: (1) differet represetatios, e.g., Greg Hamerly = Gregory Hamerly (2) differet format, e.g., Sep 16, 2009 = 09/16/09 (3) differet scale, e.g., meters iches Schema-Level Matchig Defiitio Detectig ad resolvig attribute coflicts ad redudat attributes Object Idetificatio The same attribute (or object) might have differet ames i differet sources. e.g, trasactio id = TID Oe attribute might be a derived attribute i aother table. e.g., Age = Birthday Attribute Redudacy Aalysis Ca be aalyzed by correlatio / variatio measures e.g., 2 test, Pearso coefficiet, t-test, F-test 16

17 Pearso Coefficiet Pearso Coefficiet Evaluates correlatio betwee two samples. Give two samples X={x 1,x 2,,x } ad Y={y 1,y 2,, y }, r i 1 ( x x)( y y) i ( 1) x i y co-variace betwee X ad Y idividual variace (stadard deviatio) of X ad Y If r > 0, X ad Y are positively correlated. If r = 0, X ad Y are idepedet. If r < 0, X ad Y are egatively correlated. t-test ad F-Test t-test (t-statistics) Idepedet two-sample t-test: t 2 1 s x x / s / 2 Evaluates statistical variace betwee two samples. p-value ANOVA (Aalysis of Variace) / F-test (F-statistics) Evaluates statistical variace amog three or more samples 17

18 Chi-Square Test 2 Test ( 2 Statistic) Evaluates whether a observed distributio i a sample differs from a theoretical distributio (i.e., hypothesis). Where E i is a expected frequecy ad O i is a observed frequecy, The larger 2, the more likely the variables are related (positively or egatively). i 1 ( Oi Ei E 2 2 ) i Example Play chess Not play chess Sum (row) Like sciece fictio 250 (90) 200 (360) 450 Not like sciece fictio 50 (210) 1000 (840) 1050 Sum (col.) CSI 4352, Itroductio to Data Miig Chapter 2 ad 3, Data Pre-processig Geeral Data Characteristics Descriptive Data Summarizatio Data Cleaig Data Itegratio Data Trasformatio Data Reductio 18

19 Data Trasformatio Defiitio Process that maps a etire set of values of a give attribute ito a ew set of values Purpose To remove oise from data To chage scales Methods Smoothig (icludig biig ad regressio) Normalizatio Geeral Normalizatio Methods Mi-Max Normalizatio Maps the values i the rage [mi, max] ito a ew rage [mi, max ]. v' mi' v mi max' mi' max mi z-score Normalizatio Trasforms the values of a attribute A based o its mea ad stadard deviatio. v ' v A A Decimal Scalig v v' 10 Moves decimal poit of values where j is the maximal digit j 19

20 Quatile Normalizatio (1) Motivatio I a Q-Q plot, if two distributios are the same, the the plot should be a straight lie. Ca be exteded to dimesios Descriptio q k = (q k1,, q k ): a vector of the kth quatile for all dimesios proj d q k 1 ( i 1 1 qki,..., i 1 q ki Algorithm Sort each colum (dimesio) of X to give X Assig the meas across rows of X ito each elemet of the row Rearrage each colum of X to the same order of X ) Quatile Normalizatio (2) Advatages Efficiet i high dimesioal data ( popularly used for biological data pre-processig ) Disadvatages I practice, each dimesio may have differet distributio Refereces Bolstad, B.M., et al., A compariso of ormalizatio methods for high desity oligoucleotide array data based o variace ad bias, Bioiformatics, Vol.19 (2003) 20

21 CSI 4352, Priciples of Data Miig Chapter 2 ad 3, Data Pre-processig Geeral Data Characteristics Descriptive Data Summarizatio Data Cleaig Data Itegratio Data Trasformatio Data Reductio Data Reductio Defiitio Process to obtai a reduced represetatio of a data set, which is much smaller i volume but produces almost the same aalytical results Problems Data miig algorithms take a very log time to ru o the complete data sets Data aalysis methods are complex, iaccurate i the high dimesioal data Methods Dimesioality reductio Numerosity reductio 21

22 Dimesioality Reductio Curse of Dimesioality Whe dimesioality icreases, data becomes icreasigly sparse Possible combiatios of subspaces will grow expoetially Desity ad similarity betwee data values becomes less meaigful Purpose To avoid the curse of dimesioality To elimiate irrelevat features ad reduce oise To reduce time ad space required i data miig To allow easier visualizatio Methods Feature extractio Feature selectio Feature Extractio Process (1) Combiig a multitude of correlated features (2) Creatig a ew dimesioal feature space for the combied features Example Pricipal compoet aalysis (PCA) Fid the eigevectors of the covariace matrix Defie a ew space with the eigevectors Wavelet trasformatio Problem New dimesioal spaces might ot be meaigful i the domai of data sets 22

23 Feature Selectio Methods Elimiatig redudat features or irrelevat features Selectig sigificat (iformative) features Example Redudat features: e.g., purchase price of a product ad the amout of sales tax paid Irrelevat features e.g., paret s ame is irrelevat for selectig studet scholarship cadidates Iformative features e.g., studet s ame, studet s GPA, paret s icome are iformative for selectig studet scholarship cadidates Heuristic Search for Feature Selectio Problem of Feature Selectio If d features, how may possible combiatios of the features? 2 d Typical Heuristic Methods Step-wise feature selectio: Repeatedly pick the best feature Step-wise feature elimiatio: Repeatedly remove the worst feature Best combied feature selectio ad elimiatio Optimal brach ad boud 23

24 Numerosity Reductio Purpose To reduce data volume by choosig alterative, smaller forms of data represetatio Parametric Methods Assume the data fits some model, estimate model parameters, store oly the parameters, ad discard the data. e.g., Regressio No-parametric Methods Do ot assume models, ad use data values. e.g., Discretizatio, Clusterig, Coceptual Hierarchy Geeratio Discretizatio Methods Dividig the rage of cotiuous data ito itervals Selectig sigificat (frequet) data Strategy Supervised vs. Usupervised Splitig (top-dow) vs. Mergig (bottom-up) Examples Biig: top-dow, usupervised Samplig: top-dow, supervised Etropy-based Discretizatio: top-dow, supervised 24

25 Coceptual Hierarchy Geeratio Orderig Attributes Partial/total orderig of attributes at the schema level e.g., street < city < state < coutry Hierarchy Geeratio A hierarchy for a set of values by explicit data groupig e.g., {Dallas, Waco, Austi} < Texas Automatic Method Based o the umber of distict values per attribute coutry state city street 15 distict values 365 distict values 3567 distict values 674,339 distict values Questios? Lecture Slides are foud o the Course Website, 25