SUPPORT VECTOR MACHINES

Transcription

1 SUPPORT VECTOR MACHINES Today Reading AIMA 18.9 Goals (Naïve Bayes classifiers) Support vector machines 1

2 Support Vector Machines (SVMs) SVMs are probably the most popular off-the-shelf classifier! Software Packages LIBSVM (LIBLINEAR) on the Resources page SVM-Light Linearly Separable x 1 x 2 x 1 and x x 1 x 2 x 1 or x x 1 x 1 x 2 x 2 2

3 Support Vector Machines A support vector machine (SVM) is a linear classifier that finds the decision boundary btw. two classes that is maximally far from any point in the training set Support vectors The margin is the distance from the decision boundary to the closest data point The support vectors are a subset of the training examples that fully determine the decision boundary Maximum margin decision hyperplane Maximize margin What defines a hyperplane? 3

4 What defines a hyperplane? A hyperplane is defined by: A vector w Perpendicular to the hyperplane Often called the weight vector A scalar b Selects the hyperplane that is distance b from the origin from among all possible hyperplanes w b How do we classify an example? D = {(x i,y i ) i =1...N} y i 2 { 1, 1} w x + b =0 w x + b<0 w x + b>0 x on the decision boundary x below the decision boundary x above the decision boundary w g(x i )=sign(w x + b) b 4

5 The hyperplane that maximizes the margin We know how to specify a hyperplane (w and b). Given the hyperplane, we know how to predict. But how do we find the hyperplane with the maximum margin? (Derivation on board) Solving the Optimization Problem min w,b 1 2 w 2 such that y (i) w x (i) + b 1 8i Need to optimize a quadratic function subject to linear constraints Quadratic optimization problems are a well-known class of mathematical programming problem and many algorithms exist for solving them The solution involves constructing a dual problem where a Lagrange multiplier (a scalar value) is associated with every constraint in the primary problem 5

6 Solving the Optimization Problem min w,b 1 2 w 2 such that y (i) w x (i) + b 1 8i Lagrange multipliers max min w,b 1 2 w 2 NX i y (i) (w x (i) + b Xi=1 1 Dual max N X i=1 i 1 2 X X i j y (i) y (j) x (i) x (j) i subject to i j 0 and X i i y (i) =0 X X Solving the Optimization Problem The solution has the form: w = X X X Each non-zero alpha indicates corresponding x i is a support vector The classifying function has the form: Relies on an inner product between the test point x and the support vectors x i X NX i y (i) x (i) and b = y (i) w x (i) for any x (i) s.t. i 6=0 i=1 X g(x i ) = sign i y (i) x (i) x + b i 6

7 Soft-margin Classification If the training data is not linearly separable, slack variables ξ i can be added to allow misclassification of difficult or noisy examples. Still, try to minimize training set errors, and to place hyperplane far from each class (large margin) ξ i ξ j How many support vectors? Determined by alphas in optimization Typically only a small proportion of the training data The number of support vectors determines the run time for prediction 7

8 How fast are SVMs? Training Time for training is dominated by the time for solving the underlying quadratic programming problem Slower than Naïve Bayes Non-linear SVMs are worse Testing (Prediction) Fast - as long as we don t have too many support vectors Multi-label classification SVMs are inherently two-class classifiers Given C classes, common techniques are: One-versus-all n Train C different SVMs where each SVM learns one class versus all the other classes One-versus-one n Train C(C-1)/2 SVMs where each SVM learns to distinguish one class from another Multi-class SVMs Transductive SVMs 8

9 Linear SVMs Summary The classifier is a decision boundary (separating hyperplane) Most important training points are support vectors which define the hyperplane Quadratic optimization algorithms can identify which training points are support vectors (vectors with non-zero Lagrange multipliers) In the dual formation and in classifying an example, the training points appear only inside inner products Non-linear SVMs General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable: Φ: x φ(x) 9

10 The Kernel trick The linear classifier relies on an inner product between vectors x it x j X g(x i ) = sign i i y (i) x (i) x + b If every example is mapped into a high-dimensional space via some transformation Φ: x φ(x) then the inner product becomes: X g(x i ) = sign i i y (i) '(x (i) ) '(x)+b A kernel function is some function that corresponds to a dot product in some transformed feature space: K(x i,x j )= φ(x i ) T φ(x j ) The Kernel trick Sec The kernel K may be cheaper to compute then the transformation φ Implictly do the transformation φ(x) = x 1 x 1 x 1 x 2 x 1 x 3 x 2 x 1 x 2 x 2 x 2 x 3 x 3 x 1 x 3 x 2 x 3 x 3 K(x, z) = = = ( n )( n ) x i z i x i z i i=1 j=1 n n x i x j z i z j i=1 j=1 n (x i x j )(z i z j ) i,j=1 * 10

11 Kernels Why use kernels? n Make non-separable problem separable. n Map data into better representational space Common kernels n Linear n Polynomial K(x,z) = (1+x T z) d n Radial basis function (infinite dimensional space) Summary Support Vector Machines (SVMs) Choose hyperplane based on support vectors Support vectors are critical points close to the decision boundary Often among the best performing classifiers * 11