Data Preprocessing. Supervised Learning

Transcription

1 Supervised Learning Regression Given the value of an input X, the output Y belongs to the set of real values R. The goal is to predict output accurately for a new input. The predictions or outputs y are categorical, while x can take any set of values (real or categorical). The goal is to select the correct class for a new instance. 1

2 The task of assigning objects to one of several predefined categories. 2

3 Feature set (X) Model Class label (y) 3

4 Definition of The input data for a classification task is a collection of records, each known as an instance, composed of a feature set and a class label., then, is the task of learning a target function f that maps each feature set x to one of the predefined class labels y. The target function is also known informally as the classification model. A classification model is useful 4 for several purposes.

5 Types of Models Two purposes of classification models: 1 2 Descriptive Modeling A classification model can serve as an explanatory tool to distinguish between objects of different classes. Predictive Modeling A classification model can also be used to predict the class label of unknown records. 5

6 Models: An Example Input Features: X Class: Y Make Cylinders Length Weight Style Honda Four Hatchback Instances Toyota Four Wagon BMW Six Sedan 6 Descriptive modeling: Explain what features define a hatchback, wagon, or sedan. Predictive modeling: Given another car, determine the style to which it belongs.

7 7 Building a Model A classification technique (or classifier) is a systematic approach to building classification models from an input dataset. Each technique employs a learning algorithm to identify a model that best fits the relationship between the features and class of the input data. A key objective of the learning algorithm is to build models with good generalization capability.

8 Building a Model Data Training Set Test Set Learning Algorithm Learn Model Apply Model Model 8

9 9 Types of Learners Eager learners learn a model that maps the input features to the class label a soon as the training data becomes available. Lazy learners delay modeling the training data until it is needed to classify the test examples. An example of a lazy leaner is a rote classifier, which memorizes the entire training data and performs classification only if a test instance matches one of the training examples exactly.

10 Rote Learning Make Cylinders Length Weight Style Honda Four Hatchback Toyota Four Wagon BMW Six Sedan BMW Six Sedan 10

11 Nearest Neighbor Classifier The Idea: Find which training data is closest to the test instance, and classify the test instance as that class. Problems: Computationally expensive. Not interpretable. Very sensitive to noise/outliers. 11

12 Nearest Neighbor Learning Make Cylinders Length Weight Style Honda Four Hatchback Toyota Four Wagon BMW Six Sedan BMW Six Sedan 12

13 k-nearest Neighbor (knn) Classifier The Idea: Find the k training instances that are closest to the test instance, and classify the test instance as the majority class of the k nearest training instances. If it walks like a duck, quacks like a duck, and looks like a duck, then it s probably a duck. 13

14 k-nearest Neighbor (knn) Classifier Training New Instance? 14

15 The knn Algorithm The k-nearest neighbor classification algorithm. 1: Let k be the number of nearest neighbors and D be the set of training examples. 2: for each test example z = x, y do 3: Compute d x, x, the distance between z and every example, x, y D. 4: Select D z D, the set of k closest training examples to z. 5: y = argmax x i,y i D z I v = y i v 6: end for 15

16 Nearest Neighbors 1-nearest neighbor 2-nearest neighbor 3-nearest neighbor The k nearest neighbors of an example x are the data points that have the k smallest distances to x. 16

17 Choosing the k for knn If k is too small, then the nearest-neighbor classifier may be susceptible to overfitting because of noise in the training data. If k is too large, the nearest-neighbor classifier may misclassify the test instance because its list of nearest neighbors may include data points that are located far away from its neighborhood. 17

18 Choosing the k for knn k-nearest neighbor classification with small k. k-nearest neighbor classification with large k. 18

19 knn Prediction: : Take the majority vote of class labels among the k-nearest neighbors: Majority voting: y = argmax v I v = y i x i,y i D z where v is a class label, y i is the class label for one of the nearest neighbors, and I is an indicator function that returns the value 1 if its argument is true and 0 otherwise. 19

20 knn Prediction: Regression Regression: Predict the average value of the class value of the k-nearest neighbors. Average value: y = 1 x i, y i D z y i x i,y i D z where v is a class label and y i is the class label for one of the nearest neighbors. 20

21 knn Prediction: Distance-Weighted In the majority voting approach, every neighbor has the same impact on the classification. Instead, the influence of each nearest neighbor x i can be weighted according to distance: w i = 1 d x, x i 2. Then, the class label can be determined as follows: y = argmax v w i I v = y i. x i,y i D z 21

22 knn Efficiency Very efficient in model induction (training) Only store the training data. Not particularly efficient in testing Computation of distance measure to every training instance Note that 1NN and knn are equally efficient Retrieving the k nearest neighbors is (almost) no more expensive than retrieving a single nearest neighbor k nearest neighbors can be maintained in a queue. 22

23 knn Summary Nearest-neighbor classification is part of a more general technique known as instance-based learning. Lazy learners such as nearest-neighbor classifiers do not require model building. Generate their predictions based on local information. Can produce arbitrarily shaped decision boundaries. Can easily produce wrong predictions without appropriate data preprocessing. 23

24 Back to Building a Model 24 Data Training Set Test Set Learning Algorithm Learn Model Apply Model For Nearest Neighbors, the model is the training set itself. Model

25 Let s see how a model that actually learns works. 25

26 Components of Learning Suppose that a bank wants to automate the process of evaluating credit card applications. Input x (customer information that is used to make a credit application). 26 Target function f: X Y (ideal formula for credit approval), where X and Y are the input and output space, respectively. Dataset D of input-output examples x 1, y 1,, x n, y n. Hypothesis (skill) with hopefully good performance: g: X Y ( learned formula to be used)

27 Components of Learning 27 Unknown Target Function f: X Y (ideal credit approval formula) Training Examples x 1, y 1,, x n, y n (historical records of credit customers) Hypothesis Set H (set of candidate formulas) Learning Algorithm A Final Hypothesis g f (learned credit approval formula) Use data to compute hypothesis g that approximates target f

28 Simple Learning Model: The Perceptron For x = x 1,, x d ( features of the customer ), compute a weighted score and: Approve credit if w i x i > threshold, d i=1 Deny credit if d i=1 w i x i < threshold. 28

29 Simple Learning Model: The Perceptron This formula can be written more compactly as d x = sign w i x i threshold, where x = +1 means approve credit and x = 1 means deny credit ; sign s = +1 if s > 0 and sign s = 1 if s < 0. This model is called a perceptron. i=1 29

30 Simple Learning Model: The Perceptron x 1 x 2 x 3 y Input Nodes x 1 x 2 x Output Node Σ t = 0.4 y 30

31 Perceptrons (in R 2 ) x = sign w 0 + w 1 x 1 + w 2 x 2 31 The perceptron is a linear (binary) classifier: Customer features x: points on the plane Labels y: (+1), (-1) Hypothesis : line (divide positive and negative)

32 And now Let s see some classifying! 32