Machine Learning Topic 5: Linear Discriminants Bryan Pardo, EECS 349 Machine Learning, 2013 Thanks to Mark Cartwright for his extensive contributions to these slides Thanks to Alpaydin, Bishop, and Duda/Hart/Stork for images and ideas 1
General Classification Learning Task There is a set of possible examples Each example is an k-tuple of attribute values 1 1 There is a target function that maps X onto some finite set Y The DATA is a set of tuples <example, target function values> D Find a hypothesis h such that... X!! = { x,... x } 1 n! x =< a,..., ak f : X Y!!!! x, f ( x ) >,... < x, f ( x ) > } 1 m = { < 1 m!!! x, h( x) f ( x) >
Where would you draw the line? a 2 0 0 6 8 a 1 3
Linear Discriminants A linear combination of the attributes. g x w, w 0 ( ) = w 0 + w T x = w0 + w i Easily interpretable k a i i=1 Are optimal when classes are Gaussian and share a covariance matrix 4
g( x) = 0 Two-Class Classification defines a decision boundary that splits the space in two a 2 g( x) = w 1 a 1 + w 2 a 2 + w 0 = 0 If a line exists that does this without error, the classes are linearly separable g( x) < 0 h( x ) = " $ # %$ 1 if g( x ) > 0 1 otherwise g( x) > 0 a 1 5
Any hyperplane can be written as the set of points satisfying the equation below, where w and x are vectors in R d Defining a Hyperplane w T x + w0 = 0 The vector w is a normal vector: it is perpendicular to the hyperplane. It is often referred to as the weight vector. The parameter w 0 determines the offset of the hyperplane from the origin along the normal vector. It is often referred to as the threshold. dist to origin = d 0 = w 0 w Euclidean norm
Estimating the model parameters There are many ways to estimate the model parameters: Minimize least squares criteria (classification via regression) Maximize Fisher linear discriminant criteria Minimize perceptron criteria (using numerical optimization techniques, e.g. gradient descent) Many other methods for solving for the inequalities using constrained optimization 7
Classification via regression Label each class by a number Call that the response variable Analytically derive a regression line Round the regression output to the nearest label number Fall 2014 8
An example 1 r Decision boundary 1 1 1 1.5 0 O 0 0 0 x Mark Cartwright and Fall 2012 9
1 r What happens now? Decision boundary 1 1 1 1.5 0 O 0 0 0 x Mark Cartwright and Fall 2012 10
Estimating the model parameters There are many ways to estimate the model parameters: Minimize least squares criteria (i.e. classification via regression as shown earlier) Maximize Fisher linear discriminant criteria Minimize perceptron criteria (using numerical optimization techniques, e.g. gradient descent) Many other methods for solving for the inequalities using constrained optimization 11
Hill-climbing (aka Gradient Descent) Start somewhere and head up (or down) hill. Objective function J(w) w: the value of some parameter
Hill-climbing Easy to get stuck in local maxima (minima) Objective function J(w) w: the value of some parameter
What s our objective function? 1 0.5 0 0.5 Minimise number of misclassifications? Maximize margin between classes? Personal satisfaction? 1 1 0.5 0 0.5 1 14
Gradient Descent Simple 1 st order numerical optimization method Idea: follow the gradient to a minimum Finds global minimum when objective function is convex, otherwise local minimum Objective function (the function you are minimizing) must be differentiable Used when there is no analytical solution to finding minimum 15
SSE: Our objective function We re going to use the sum of squared errors (SSE) as our classifier error function. As a reminder, we used SSE in linear regression (next slide) Note, in linear regression, the line we learn embodies our hypothesis function. Therefore, any change in the line will change our SSE, by some amount. Mark Cartwright and Fall 2012 16
Simple Linear Regression Typically estimate parameters by minimizing sum of squared residuals (a.k.a. least squares): number of training examples Response variable residual Mark Cartwright and Fall 2012 17
SSE: Our objective function In classification (next slide), the line we learn is INPUT to our hypothesis function. Therefore, a change of the line may not change the SSE objective function, if moving it does not change the classification of any training point. Note also, in the next slide, both the horizontal and vertical dimensions are INPUTS to the hypothesis function while in the previous slide, the vertical dimension was the OUTPUT. Therefore, the dimension a1 and a2 are the elements of the vector X that gets handed to h(x). Mark Cartwright and Fall 2012 18
Simple Linear Classification Typically estimate parameters by minimizing sum of squared residuals (a.k.a. least squares): a 1 E(! w) 1 2 m ( f ( x! m ) h( x! m )) 2 1 g ( x! ) > 0 g ( x! ) = 0 g ( x! ) < 0 h ( x! ) = 1 if g ( x! ) > 0 1 otherwise Mark Cartwright and Fall 2012 19 a 2
Gradient Descent Gradient descent is a useful, simple optimization method, but it can be very slow on difficult problems. There are many many optimization methods out there for different types of problems. Take the optimization courses offered in EECS and IEMS to learn more. 20
Gradient Descent are parameters is the convergence threshold is the step size (in general, important to choose well) is the gradient (vector of partial derivatives with respect to parameters) of the objective function Algorithm: begin initialize do end return until 21
Gradient Descent In batch gradient descent, J is a function of both the parameters and ALL training samples, summing the total error, e.g. In stochastic gradient descent, J is a function of the parameters and a different single random training sample at each iteration. this is a common choice in machine learning when there is a lot of training data, and computing the sum over all samples is expensive. 22
Perceptron Criteria Perceptron criteria: NOTE: augmented to include the threshold w 0 where and so that we want is the set of misclassified training examples. We can minimize this criteria to solve for our weight vector w Restricted to 2-class discriminant We can use stochastic gradient descent to solve for this Only converges when data is linearly separable 23
Perceptron Algorithm are parameters is set to 1 (this is fine in this case, since multiplying w by a scalar doesn t affect the decision) Algorithm: begin initialize end return do if then until all examples are properly classified 24
Example: Perceptron Algorithm 1 0.5 Red is the positive class Blue is the negative class 0 0.5 1 1 0.5 0 0.5 1 25
Perceptron Algorithm Example (cont d): 1 0.5 Red is the positive class Blue is the negative class 0 0.5 1 1 0.5 0 0.5 1 26
Perceptron Algorithm Example (cont d): 1 0.5 Red is the positive class Blue is the negative class 0 0.5 1 1 0.5 0 0.5 1 27
Perceptron Algorithm Example (cont d): 1 0.5 Red is the positive class Blue is the negative class 0 0.5 1 1 0.5 0 0.5 1 28
Multi-class Classification a 2 When there are N>2 classes: you can classify using N discriminant functions. Choose the class with the maximum output Geometrically divides feature space into N convex decision regions a 1 Choose C i if g i x N ( ) = max j=1 g j x ( ) 29
Pairwise Multi-class Classification If they are not linearly separable (singly connected convex regions), may still be pair-wise separable, using N(N-1)/2 linear discriminants. a 2 g ij x wij, w ij0 K ( ) = w ij0 + w ijl l=1 x l choose C i if j i, g ij ( x) > 0 a 1 30
Appendix (stuff I didn t have time to discuss in class) 31
2 Fisher Linear Discriminant Criteria Can think of as dimensionality reduction from K-dimensions to 1 Objective: w T x Maximize the difference between class means Minimize the variance within the classes 4 J( w) = (m 2 m 1 ) 2 2 0 2 2 6 s 1 2 + s 2 2 where s i and m i are the sample variance and mean for class i in the projected dimension. We want to maximize J. Mark Cartwright and Fall 2012 32
Fisher Linear Discriminant Criteria Solution: where However, while this finds finds the direction ( ) of decision boundary. Must still solve for to find the threshold. Can be expanded to multiple classes Mark Cartwright and Fall 2012 33
Logistic Regression (Discrimination) a 2 4 3.5 3 2.5 2 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 3 3.5 4 Mark Cartwright and Fall 2012 34 a 1
Logistic Regression (Discrimination) Discriminant model but well-grounded in probability Flexible assumptions (exponential family classconditional densities) Differentiable error function ( cross entropy ) Works very well when classes are linearly separable Mark Cartwright and Fall 2012 35
Logistic Regression (Discrimination) Probabilistic discriminative model Models posterior probability To see this, let s start with the 2-class formulation: p(c 1 x) = p(! x C 1 )p(c 1 ) p(! x C 1 )p(c 1 )+p(! x C 2 )p(c 2 ) 1 = 1+exp log p(! x C 1 )p(c 1 ) p(! x C 2 )p(c 2 ) = 1 1+exp( ) = ( ) where = log p(! x C 1 )p(c 1 ) p(! x C 2 )p(c 2 ) logistic sigmoid function Mark Cartwright and Fall 2012 36
Logistic Regression (Discrimination) logistic sigmoid function 1 0 0 Squashing function that maps Mark Cartwright and Fall 2012 37
Logistic Regression (Discrimination) For exponential family ere of densities, is a linear function of x. = log p(! x C 1 )p(c 1 ) p(! x C 2 )p(c 2 ) Therefore we can model the posterior probability as a logistic sigmoid acting on a linear function of the attribute vector, and simply solve for the weight vector w (e.g. treat it as a discriminant model): To classify: Mark Cartwright and Fall 2012 38