DATA PREPROCESSING. Tzompanaki Katerina

Transcription

1 DATA PREPROCESSING Tzompanaki Katerina

2 Background: Data storage formats Data in DBMS ODBC, JDBC protocols Data in flat files Fixed-width format (each column has a specific number of characters, filled in with special characters if needed) Delimited format: tab, comma,, other Attention: Convert field delimiters inside strings Verify the number of attributes before and after convertion 9/3/18 2

3 Background: Data and attributes Some frequently encountered terminology: Data objects are also called data points, samples, examples, vectors, instances, or data tuples. They are entities in a given context in a given dataset, eg patients, products etc. Attributes are also called features, variables, dimensions. Attribute vector is a set of attributes used to describe a given data object. Eg., the attribute vector <Name, Disease, Prescription> describes patient data objects. (Observed) Values for attributes are called observations. Eg, cancer, high blood pressure, flu may be the observations for the disease attribute in a given dataset. 9/3/18 3

4 Background: Attribute types Nominal (or categorical) attributes refer to names of things, or categories that normally have no order. E.g., marital status (single, married, divorced), color (blue, green, etc) or userid (323, 235,etc). Binary attribute is a nominal attribute with two possible values: 0 or 1 stating absence or precense. Eg, for a patient we could have the following binary attributes: smoker (yes, no), sex (male,female), test (positive, negative). Ordinal attribute is an attribute whose values have an ordering or ranking. Eg., grades (A>B>C), sizes (large>medium>small) Qualitative attributes: they describe a feature of an object without giving an actual size or quantity. 9/3/18 4

5 Background: Attribute types Numeric attributes are used to describe measurable quanities and are represented using numbers (integers or reals). They provide a ranking and allow for mathematical operations. Eg, temperature (20 C-15 C), age (44 years old is 2 times older than 22 years old) etc. Quantitative attributes: they describe measurable quantities. u Another categorisation: u Discrete attributes have a finite or countably infinite set of values, which may or may not be represented as integers. Eg, hair color, smoker, size, age, etc. u Continuous attributes are attributes that are not discrete, thus can be represented as numbers with floating points. Eg, length, income, price, etc. 9/3/18 5

6 Background: Basic Statistical Description of Data Measures of central tendency The mean of an attribute x in a multi-set of N observations, is the central value. x = N i=1 N x i = x 1 +!+ x N N The median is the middle value in an order set of values. If the number of values is even the median is not unique. The median better represents skewed data (aka not symmetric) and is less sensitive to outliers. The mode is the most frequent value. If several values have the highest frequency then we talk about multimodal datasets. Can also be used for nomimal attributes. 9/3/18 6

7 Background: Basic Statistical Description of Data Measures of data dispersion The range of a numeric attribute is the difference of the maximum and the minimum observation (max()-min()). The quantiles separate an ordered numerical set into equal size (containing the same fraction of data) sub-sets. The k th q-quantile for a given data distribution is the value v such that at most k/q of the data values are less than v and at most (q-k)/q of the data values are more than v, where k is an integer such that 0 <k <q. The 100-th quantile is called percentile. 9/3/18 7

8 Background: Basic Statistical Description of Data Measures of data dispersion The variance (σ 2 ) and standard deviation (σ) indicate how spread out the distribution of an attribute x is. A low standard deviation means that the observations tend to be very close to the mean, while a high standard deviation indicates that the observations are spread out over a large range of values. σ 2 (x) = 1 N (x i x) 2 N 1 The covariance cov(x,y) of two attributes shows how correlated the attributes are. A positive covariance cov(x,y)>0 shows that y raises as x increases while a negative one cov(x,y)<0 indicates that y decreases while x increases. Finally we define the covariance matrix for x,y (can be extended to cover all data variables/attributes): x y Note that x σ 2 (x) cov(x,y) cov(x,y)=cov(y,x) (symmetric matrix) y cov(y,x) σ 2 (y) i=1 cov(x, y) = 1 N (x i x)(y i y) N 1 i=1 9/3/18 8

9 Background: Displaying Data Histograms are used to summarize the distribution of observations. Each bar represents the frequency of the observation. For ordered numeric values, we split the range into equally sized buckets. The range of a bucket is called width. Scatter plots are used to observe correlations between pairs of numeric attributes. Positive (left) and negative (right) correlation. 9/3/18 9

10 Why Preprocessing? Data in the real world is dirty incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data noisy: containing errors or outliers inconsistent: containing discrepancies in codes or names No quality data, no quality mining results! Quality decisions are based on quality data Data warehouses need consistent integration of quality data 9/3/18 10

11 Data understanding: Relevance What data are available for the task? Are these data relevant? Are additional relevant data available? How much historical data are available (provenance)? Who is the data expert? 9/3/18 11

12 Data understanding: Quantity Number of instances (records, objects) Rule of thumb: 5,000 or more desired If less, results are less reliable; use special methods (like boostingnot covered in this course) Number of attributes Rule of thumb: for each attribute, 10 or more instances If many attributes, use feature reduction and selection Number of targets Rule of thumb: more than 100 instances for each class If very unbalanced, use stratified sampling 9/3/18 12

13 Forms of data preprocessing 9/3/18 13

14 Major Tasks in Data Preprocessing Data cleaning Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies Data integration Integration of multiple databases, data cubes, or files Data transformation Normalization and aggregation Data reduction Reduced data volume but still the same or similar analytical results Data discretization Part of data reduction but with particular importance, especially for numerical data 9/3/18 14

15 Major Tasks in Data Preprocessing Data cleaning Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies Data integration Integration of multiple databases, data cubes, or files Data transformation Normalization and aggregation Data reduction Reduced data volume but still the same or similar analytical results Data discretization Part of data reduction but with particular importance, especially for numerical data 9/3/18 15

16 Data Cleaning Reformat/Convert data Fill in missing values Handle dates Identify outliers and smooth out noisy data Correct inconsistent data 9/3/18 16

17 Reformatting Data Convert data to a standard format Missing values Unified date format Binning of numeric data Fix errors and outliers Convert nominal fields, whose values have order, to numeric. Why? to be able to use comparison operators ( > and < ) on these fields) 9/3/18 17

18 Missing Data Data is not always available E.g., many tuples have no recorded value for several attributes (eg customer income in sales data) Missing data may be due to equipment malfunction inconsistency with other recorded data data not entered due to misunderstanding that certain data may not be considered important at the time of entry not registered history or changes of the data v Missing data may need to be inferred! 9/3/18 18

19 Handling Missing Data Ignore the tuple: usually done in classification tasks when the tuple s class label (target value) is missing Fill in the missing value manually Use a global constant to fill in the missing value Measure of central tendency: use the attribute mean/median to fill in the missing value, or the attribute mean for all samples belonging to the same class Use the most probable value to fill in the missing value: in a supervised manner, find the most possible value using inference-based mechanisms such as a Bayesian formula or decision tree 9/3/18 19

20 Handling Missing Data Ignore the tuple: usually done in classification tasks Ineffective when the tuple s class label (target value) is missing Fill in the missing value manually Inefficient and tedius Use a global constant to fill in the missing value Measure of central tendency: use the attribute mean/median to fill in the missing value, or the attribute mean for all samples belonging to the same class Use the most probable value to fill in the missing value: in a supervised manner, find the most possible value using inference-based mechanisms such as a Bayesian formula or decision tree Not foolproof Smarter Best choice 9/3/18 20

21 Unified Date Format We want to transform all dates to the same format internally Some systems accept dates in many formats e.g. Sep 24, 2003, 9/24/03, , etc dates are transformed internally to a standard value Frequently, just the year (YYYY) is sufficient For more details, we may need the month, the day, the hour, etc Representing date as YYYYMM or YYYYMMDD can be OK, but has problems What are the problems with YYYYMMDD dates? YYYYMMDD does not preserve intervals: /3/18 21

22 Unified Date Format Options To preserve intervals, we can use Unix system date: Number of seconds since Jan 1, 1970 Number of days since Jan 1, 1960 (SAS) Problem: values are non-obvious don t help intuition and knowledge discovery harder to verify, easier to make an error 9/3/18 22

23 KSP Date Format KSP _ Date = YYYY + days _ starting _1_ Jan _if _leap _ year Preserves intervals between days The year is obvious Sep 24, 2003 is ( )/365= (round to 4 digits) Can be extended to include time 9/3/18 23

24 Conversion: Nominal to Numeric Some methods can deal with nominal values internally. Other methods (regression, nearest neighbor, neural networks) require only numeric inputs. To use nominal fields in such methods we need to convert them to a numeric value. Different strategies for binary, ordered, multi-valued nominal fields. 9/3/18 24

25 Conversion: Binary to Numeric Binary fields E.g. Gender=M, F Convert to Field_0_1 with 0, 1 values e.g. Gender = M à Gender_0_1 = 0 Gender = F à Gender_0_1 = 1 9/3/18 25

26 Conversion: Ordered to Numeric Ordered attributes (e.g. Grade) can be converted to numbers preserving natural order, e.g. A à 4.0 A- à 3.7 B+ à 3.3 B à 3.0 Why is it important to preserve natural order? To allow meaningful comparisons, e.g. Grade > 3.5 9/3/18 26

27 Conversion: Nominal, Few Values Multi-valued, unordered attributes with small (rule of thumb < 20) no. of values e.g. Color=Red, Orange, Yellow,, Violet for each value v create a binary flag variable C_v, which is 1 if Color=v, 0 otherwise Also called one-hot-encoding or dummy variable method. ID color ID C_red C_orange C_yellow 100 red 101 yellow /3/18 27

28 Conversion: Nominal, Many Values Examples: US State Code (50 values) Profession Code (7,000 values, but only few frequent) How to deal with such fields? Ignore ID-like fields whose values are unique for each record. For other fields, group values naturally : e.g. 50 US States à 3 or 5 regions Profession à select most frequent ones, group the rest Create binary flag-fields (one-hot-encoding) for selected values. 9/3/18 28

29 Noisy Data Noise: random error or variance in a measured variable Incorrect attribute values may be due to faulty data collection instruments data entry problems data transmission problems technology limitation inconsistency in naming convention Other data problems, which require data cleaning duplicate records incomplete data inconsistent data 9/3/18 29

30 How to Handle Noisy Data? Binning method first sort data and partition into bins then smooth by bin means, smooth by bin median, smooth by bin boundaries, etc. Clustering detect and remove outliers Combined computer and human inspection detect suspicious values and check by putting a human in the loop Regression smooth by fitting the data into (linear) regression functions 9/3/18 30

31 Simple Discretization Methods: Binning Equi-width (distance) partitioning: It divides the range into N intervals of equal size if A and B are the lowest and highest values of the attribute, the interval width will be: W = (B-A)/N. The most straightforward method Equi-depth (frequency) partitioning: It divides the range into N intervals, each containing approximately the same number of samples 9/3/18 31

32 Binning Methods for Data Smoothing Sorted data for product prices: 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34 Partition into (equi-depth) bins: Bin 1: 4, 8, 9, 15 Bin 2: 21, 21, 24, 25 Bin 3: 26, 28, 29, 34 Smoothing by bin means: Bin 1: 9, 9, 9, 9 Bin 2: 23, 23, 23, 23 Bin 3: 29, 29, 29, 29 Smoothing by bin boundaries: Bin 1: 4, 4, 4, 15 Bin 2: 21, 21, 25, 25 Bin 3: 26, 26, 26, 34 9/3/18 32

33 Major Tasks in Data Preprocessing Data cleaning Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies Data integration Integration of multiple databases, data cubes, or files Data transformation Normalization and aggregation Data reduction Reduced data volume but still the same or similar analytical results Data discretization Part of data reduction but with particular importance, especially for numerical data 9/3/18 33

34 Data Integration Data integration combines data from multiple sources into a coherent store Schema integration integrate metadata from different sources Entity identification problem identify same real world entities in multiple data sources, e.g., A.cust-id B.cust-# Detecting and resolving data value conflicts for the same real world entity, attribute values from different sources maybe different possible reasons: different representations, different scales, e.g., meter vs. foot 9/3/18 34

35 Handling Redundant Data in Data Integration Redundant data occur often when integrating multiple databases The same attribute may have different names in different databases One attribute may be a derived attribute in another table, e.g., monthly vs annual revenue Redundant data may be able to be detected by correlation analysis Careful integration of multiple sources may help reduce/ avoid redundancies and inconsistencies and improve mining speed and quality 9/3/18 35

36 Major Tasks in Data Preprocessing Data cleaning Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies Data integration Integration of multiple databases, data cubes, or files Data transformation Normalization and aggregation Data reduction Reduced data volume but still the same or similar analytical results Data discretization Part of data reduction but with particular importance, especially for numerical data 9/3/18 36

37 Data Reduction Dimensionality reduction: reduce the number of considered attributes Principal component analysis Wavelet transformation Numerosity reduction: reduce the volume of data to smaller but representative data representations Sampling: pick some of the data Clustering: create clusters of similar items, use clusters instead of members. Histograms: binning method Data compression: compress data in lossless (if original data can be reconstructed) or lossy (otherwise) manner 9/3/18 37

38 Dimensionality Reduction Purpose Avoid curse of dimensionality Reduce amount of time and memory required by data mining algorithms Allow data to be more easily visualized May help to eliminate irrelevant features or reduce noise Feature selection Select the most important features Feature extraction/engineering Find representative combinations of features to use instead. 9/3/18 38

39 Dimensionality reduction: Feature selection Feature selection Select a minimum set of features such that the distribution of different classes is as close as possible to the original distribution. 2 n possible subsets! Expert knowledge can be utilized to keep the most important features. Automatic feature selection Model-based selection The most important features are selected using a supervised ML algorithm (eg decision tree). Iterative selection Iteratively, the least important features are discarded (backward elimination) or the most important are added (forward selection) until the desired number is reached. 9/3/18 39

40 Dimensionality reduction: Feature selection 9/3/18 40

41 Dimensionality reduction: Feature extraction Principal Component Analysis (PCA) Given N data vectors from k-dimensions, find c <= k orthogonal vectors that can be best used to represent the data The original data set is reduced to a new one consisting of N data vectors on c principal components (reduced dimensions) Each data vector is a linear combination of the c principal component vectors Works for numeric data only We will see PCA in detail, when we will study unsupervised learning methods. 9/3/18 41

42 Principal Component Analysis (PCA) The perpendicular (orthogonal) arrows show the principal components of the data. The blue is the first principal component, the pink is the second one. * 9/3/18 42

43 Numerosity Reduction: Sampling Simple random sample without replacement (SRSWOR) of size s: randomly pick s samples, all with equal probability Simple random sample with replacement (SRSWR) of size s: the same item can be picked more than once 9/3/18 43

44 Numerosity Reduction: Sampling Simple random sample without replacement (SRSWOR) of size s: randomly pick s samples, all with equal probability Simple random sample with replacement (SRSWR) of size s: the same item can be picked more than once 9/3/18 44

45 Numerosity Reduction: Sampling Cluster sample: when data are clustered, pick randomly s number of them. Eg. data retrieved in memory pages 9/3/18 45

46 Numerosity Reduction: Sampling Cluster sample: when data are clustered, pick randomly s number of them. Eg. data retrieved in memory pages Stratified sample: create strata (levels) in the data to represent different categories. Then, pick a number of samples from each strata accordingly. In this way, all strata will be guaranteed to exist in the samples. 9/3/18 46

47 Numerosity Reduction: Clustering Create partitions of data objects (clusters), so that objects within a cluster are similar to one another and dissimilar to objects in other clusters. Then use clusters instead of elements in the clusters. 9/3/18 47

48 Numerosity Reduction: Histograms Use histogram representations instead of full data. As we saw before, histograms (binning method) partition the data distribution of an attribute A into disjoint buckets that are of Equal-width: In an equal-width histogram, the width of each bucket range is uniform. Equal-frequency (or equal-depth): In an equal-frequency histogram, the buckets are created so that, roughly, the frequency of each bucket is constant (i.e., each bucket contains roughly the same number of contiguous data samples). 9/3/18 48

49 Major Tasks in Data Preprocessing Data cleaning Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies Data integration Integration of multiple databases, data cubes, or files Data transformation Normalization and aggregation Data reduction Reduced data volume but still the same or similar analytical results Data discretization Part of data reduction but with particular importance, especially for numerical data 9/3/18 49

50 Data Transformation Smoothing: remove noise from data Discretization: binning, histograms, clusters Normalization: scale to fall within a small, specified range min-max normalization z-score normalization normalization by decimal scaling Concept hierarchy generalization: replace a value with a higher class Aggregation: summarization, data cube construction Common tasks with data cleaning 9/3/18 50

51 Normalization min-max normalization ν ' = ν min(ν) max(ν) min(ν) z-score normalization (standardization) ν ' = ν ν σ v has zero-mean and unit variance è gaussian distribution scaling to unit length ν ' = v v 9/3/18 51

52 Concept Hierarchies For numerical data it can be regarded as discretization method. Eg salaries fall into different ranges. For nominal data, hierarchies can be implicitly or explicitly defined in schemas or by the data Specification of a partial ordering of attributes explicitly at the schema level by users or experts. Eg street < city < province or state < country Specification of a set of attributes for the hierarchy, but not of their partial ordering. To find the ordering use the distict attribute values cardinality. country province city street 15 distinct values 365 distinct values 3567 distinct values 9/3/ ,339 distinct values

53 Sources Han and Kamber: Data Mining, Concepts and Techniques Nguyen Hung Son: Data cleaning and data preprocessing Prof. Pier Luca Lanzi: Data Exploration and Preparation Muller and Guido: Introduction to Machine Learning with Python 9/3/18 53