A Summary of Support Vector Machine

Transcription

1 A Summary of Support Vector Machine Jinlong Wu Computational Mathematics, SMS, PKU May 4,2007 Introduction The Support Vector Machine(SVM) has achieved a lot of attention since it is developed. It is widely used in many areas because of its powerful ability of classification and regression, such as textual classification, face recognition, image processing, hand-written recognition and so forth. In this article I try to give a summary of SVM. Because of its plentiful contents, This article has to mainly focus on one aspect Support Vector Classifier(SVC). All of the techniques mentioned here are realized in a new SVM software package PKSVM. Presently PKSVM can handle two-class and multi-class classification problems using C-SVC and ν-svc (omitted here). The earliest pattern recognition systems were Linear Classifiers(Nilsson,965). We will point out the differences between SVC and LC in the following sections. Suppose n training data(sometimes called examples or observations) (x i, y i ) are given, where x i R p (p features or variables) and y i R. Without loss of generality, the whole article is limited to two-class problems, that is to say, y i s only have two values, which can be + and for simplicity. 2 SVC in the linearly separable situation In this section I will review linearly separable problems because they are the simpliest for SVC. The training set {(x, y ),..., (x n, y n )} is said to be linearly separable if there exists a linear discriminant function whose sign matches the

2 0 Data from class + (red) and class (green) Figure : A linearly separable training set (n = 9,p = 2) class of all training examples. Figure gives a trivial example with n = 9, p = 2. In the linearly separable problems, it is obvious one can use the simpliest model linear model, ŷ(x) = β T x+β 0 to separate the two classes. However, as in Figure 2, there usually exist infinite separating hyperplanes which can separate the training set perfectly. So which one should be chosen is a big problem. Vapnik and Lerner(963) proposes to choose the separating hyperplane that maximizes the margin. This optimal hyperplane for the trivial example in Figure is illustrated in Figure 3. That the optimal hyperplane is selected as the best one is the first new idea which was added into SVM in comparison with linear classifier. As in Figure 4, one can easily know the signed distance from a point x to the hyperplane L is β βt (x x 0 ) = β (βt x β T x 0 ) = β (βt x + β 0 ) () 2

3 0 Three lines which can separate the training set perfectly Figure 2: Some optional hyperplanes (p = 2) Hence, our aim is to obtain the biggest positive C which makes all examples satisfy β βt x i + β 0 C, i. Obviously it is a constrained optimization problem(op) max β,β 0 subject to C β y i(β T x i + β 0 ) C, i. Since the length of β is insignicant, we can let β be any value, of course it is allowed to assume = C all the time. With this assumption β the previous OP can be rewritten as min β,β 0 2 β 2 subject to y i (β T x i + β 0 ), i. (2) However, it is difficult to solve this OP directly because of the complicated constraint conditions. So we will resort to solving its dual problem. Detail will be exhibited in Section 4. 3

4 Figure 3: The optimial hyperplan Figure 4: The distance of any point x to the hyperplane L 4

5 SVM with RBF kernel: decision boundary (black) x val = dec boun val = x Figure 5: An inseparable example 3 Two important extentions to QP (2) 3. Hypersurfaces For real-world data sets, they can not be separated idealy by simple hyperplanes usually. If hyperplanes are replaced by hypersurfaces, it could be expected that better classification results will be obtained. Therefore, the OP (2) should be generalized to min β,β 0 2 β 2 subject to y i (β T Φ(x i ) + β 0 ), i. (3) where Φ( ) is a map from a n dimensional space to a higher(maybe infinite) dimensional space. 3.2 Soft margins Maybe someone will raise another problem after careful thought: Is this generalization able to separate any real-world data sets perfectly? 5

6 My first answer is yes. If the function Φ( ) is very complicated, and it will produce very wiggly surfaces, then the given set can always be separated perfectly. However, if one really does that, another serious problem will give birth. That is overfitting. The model obtained will be worthless because of its serious overfitting. Figure 5 gives another data set which can not be separated perfectly using linear hyperplanes. At first sight the black surfaces give a much better separating result, but at the same time they are too wiggly to classify new data well. Therefore, one mustn t utilize excessively complicated function Φ( )! Hence, my second answer is no. Compared to the first answer, this one is more realistic. That is a very depressing answer. Can t people do something to improve the separating result for this example in Figure 5? Of course we can. We can take a more step to extend our model (3) again. Since data sets are from real-world problems, they almost include more or less noises. The intention of our model is to learn some useful information from the data sets, not to learn their noises. It is not wise to use excessively complicated function Φ( ) to separate the classes of data. Coretes and Vapnik(995) shows that noisy problems are best addressed by allowing some examples to violate the margin constraints in (3). We can realize this idea using some positive slack variables ξ = (ξ,..., ξ n ), that is to say we change the previous constraints y i (β T Φ(x i ) + b), i to new constraints y i (β T Φ(x i ) + b) ξ i, i. However, it is necessary avoid each ξ i getting an unnecessarily big value, such as +. So some penalizations to all ξ i s should be applied. Combining this consideration, we get a new QP problem for the inseparable case: min β,β 0 2 β 2 { yi (Φ(x i ) T β + β 0 ) ξ i i, (4) subject to ξ i 0, ξ i constant. Computationally it is convenient to re-express (4) in the equivalent form subject to min β,β 0 2 β 2 +C ξ i i= { yi (Φ(x i ) T β + β 0 ) ξ i i, ξ i 0. (5) 6

7 where C replaces the constant in (4); the separable case corresponds to C = +. We call (5) Lagrange primal problem of Support Vector Classifier(SVC) hereinafter. 4 Duality of (5) The Lagrange primal function of (5) is L P = 2 β 2 +C ξ i α i [y i (Φ(x i ) T β + β 0 ) ( ξ i )] µ i ξ i, (6) i= i= i= where α i, µ i, ξ i 0 i. We minimize L P w.r.t β, β 0 and ξ i. Setting the respective derivatives to zero, we get β = α i y i Φ(x i ), (7) i= 0 = α i y i, (8) i= α i = C µ i, i, (9) By substituting the last three equalities into (6), we obtain the Lagrangian (Wolfe) dual objective function L D = α i 2 i= α i α i y i y i Φ(x i ) T Φ(x i ). (0) i= i = We maximize L D subject to 0 α i C and n i= α iy i = 0. In addition to (7)-(9), the Karush-Kuhn-Tucker (KKT) conditions include the constraints α i [y i (Φ(x i ) T β + β 0 ) ( ξ i )] = 0, () µ i ξ i = 0, (2) y i [Φ(x i ) T β + β 0 ] ( ξ i ) 0, (3) for i =, 2,..., n. Together these equations (7)-(3) uniquely characterize the solution to the primal and dual problem. 7

8 Hence all we need to do is to solve the dual problem max α i α i α i y i y i Φ(x i ) T Φ(x i ) α 2 i= i= i = y i α i = 0 subject to i=, 0 α i C, i (4) and get ˆα i s, and from the (7) β has the form ˆβ = ˆα i y i Φ(x i ), (5) i= with nonzero coefficients ˆα i only for those observations i for which the constraints in (3) are exactly met( = ) (due to ()). These observations are called the support vectors, that is to say an observation x i is called support vector if its respective Lagrange multiplier α i > 0. Any of these margin points (0 < ˆα i, ˆξ i = 0) can be used to solve for β 0, and we typically use an average of all the solutions for numerical stability. After β and β 0 are both obtained, the discriminant function ˆf(x) = Φ(x) T ˆβ + ˆβ 0 = ˆα i y i Φ(x) T Φ(x i ) + ˆβ 0 (6) i= The decision function can be written as Ĝ(x) = sign[ ˆf(x)] = sign[ ˆα i y i Φ(x) T Φ(x i ) + ˆβ 0 ]. (7) The tuning parameter of this procedure is C. The optimal value for C can be estimated by cross-validation. A large value of C will discourage any positive ξ i, and leads to an overfit wiggly boundary in the original feature space; a small value of C will encourage a small value of β, which causes f(x) and hence the boundary to be smoother. 5 Another useful extention of QP (2) Kernel Since SVC just uses the sign of the decision function to classify the class, only the decision function is informative eventually. However, x always appears 8 i=

9 with pairwise forms Φ(x) T Φ(x ) in (7). Defining the kernel function K(x, x ) as K(x, x ) = Φ(x) T Φ(x ), (7) can be written as Ĝ(x) = sign[ ˆf(x)] = sign[ ˆα i y i K(x, x j ) + ˆβ 0 ]. (8) Instead of hand-choosing a feature function Φ(x), Boser, Guyon, and Vapnik(992) proposes to directly choose a kernel function K(x, x ) that represents a dot product Φ(x) T Φ(x ) in some unspecified high dimensional space. For instance, any continuous decision boundary can be implemented using the Radial Basis Function(RBF) kernel K(x, x ) = e γ x x 2. Although many new kernels are being proposed by researchers, the following four basic kernels are used most widely: Linear: K(x, x ) = x T x i= Polynomial: K(x, x ) = (γx T x + r) d, γ > 0, d N Radial Basis Function (RBF): K(x, x ) = e γ x x 2, γ > 0 Sigmoid: K(x, x ) = tanh(γx T x + r) where γ, r, and d are kernel parameters. 6 Shrinking and Caching Defining a new n by n matrix Q as Q ij expressed more simply as = y i y j K(x i, x j ), QP (4) can be min g(α) = α 2 αt Qα e T α { y T α = 0 subject to, 0 α i C, i (9) where e is a vector with all elements equal to. Traditional optimization methods need to store the whole matrix Q if used to solve (9). In most situation it is impossible to put the whole Q into computer memory when the number of training examples n is very large. Therefore, traditional algorithms are not suitable for SVC. 9

10 Fortunately many particularly more powerful algorithms have been invented to solve (9), such as chunking, decomposition and sequential minimal optimization(smo). Especially SMO has attracted a lot of attention since it was proposed by Platt(998). All of them employ separate and conquer strategies to split the original big QP (9) into many much smaller subproblems, and to solve them iteratively and finally obtain the solution to (9). But the iterative computation is expensively time-comsuming when n is big. Joachims(998) proposes two new techniques to reduce the cost of computation. The first one is Shrinking. 6. Shrinking For many problems the number of free Support Vectors (0 < α i < C) is small compared to all Support Vectors (0 < α i C). The shrinking technique reduces the size of the problem without considering some bounded Support Vectors(α i = C). When the iterative process approaches the end, only variables in a small set A, we call it the active set, is allowed to move according to Theorem 5 in Fan et al.(2005). After shrinking, the decomposition method works on a smaller problem: min 2 αt AQ AA α A (e A Q AN α k N ) T α A { y T subject to Aα A = y T N α k N, 0 (α A ) i C, i =,..., A α A (20) where N = {,..., n}\a is the set of shrunken variables, and is a constant determined by the (k )th iteration. Although Theorem 5 in Fan et al.(2005) tells us the active set A exists, it does not tell when A will be obtained. Hence it may fail using the previous heuristic shrinking if the optimal solution of subproblem (20) is not the corresponding part of that of (9). If a failure happens, the whold QP (9) will be reoptimized with initial values α where α A is an optimal solution of (20) and α N are bounded variables identified on the previous subproblem. Before presenting the shrinking details, it is necessary to supply some definitions. 0

11 Defining I up (α) {i α i < C, y i = or α i > 0, y i = }, I low (α) {i α i < C, y i = or α i > 0, y i = }; (2) m(α) max i I up (α) { y i g(α) i }, M(α) max { y i g(α) i }, i I low (α) (22) where g(α) is the gradient of the objective function g(α) in (9), that is, g(α) = Qα e. α is an optimal solution to (9) if and only if m(α) M(α). (23) In other words, the above equality is equivalent to the KKT conditions. So we can stop the iteration if m(α k ) M(α k ) tol, (24) where tol is a small positive value which indicates the KKT conditions are obeyed within tol. In PKSVM, tol= 0 3 by default. The following shrinking procedure is from LIBSVM, which is one of the most popular SVM softwares at present. More details are available from C.-C. Chang et al.(200).. Some bounded variables will be shrunken after every min(n, 000) iterations. Since the KKT conditions are not satisfied within tol during the iterative process, (24) will not be obeyed yet, that is, m(α) > M(α). (25) Following Theorem 5 in Fan et al.(2005), variables in the following set may be shrunken: {i y i g(α k ) i > m(α k ), i I low (α k ), α k i = C or 0} {i y i g(α k ) i < M(α k ), i I up (α k ), α k i = C or 0} = {i y i g(α k ) i > m(α k ), α k i = C, y i = or α k i = 0, y i = } {i y i g(α k ) i < M(α k ), α k i = C, y i = or α k i = 0, y i = }. (26) Hence the active set A is dynamically reduced in every min(n, 000) itetrations.

12 2. Since the previous shrinking strategy may fail, and many iterations are spent in obtaining the final digit of the required accuracy, we would not hope these iterations are wasted because they are trying to solve a wrongly shrunken subproblem (20). Thus once the iteration attains the tolerance m(α) M(α) + 0tol, (27) we reconstruct the whold gradient g(α). After reconstruction, we shrink some bounded variables based on the same rule in step, and the iterations continue. The other useful technique for saving computational time by Joachims(998) is called Caching. To illustrate caching technique is very necessary, some analyses about computational complexity will first be presented. 6.2 Computational complexity Most time in each iteration is spent on the kernel evalutions needed to compute the q rows of Q, which q relys on the decomposition method, for SMO q = 2. This step has a time complexity of O(npq). Using the stored rows of Q, updating α k is done in time O(nq). Setting up the QP subproblem requires O(nq) as well. The selection of the next working set, which includes computing the gradient, can be done in O(nq). 6.3 Caching As illustrated in the last subsection, the most expensive step in each iteration is the kernel evalutions to compute the q rows of Q. Near the end of iterations, eventual support vectors enter the working set multiple times. To avoid recomputing the rows of Q, Caching is useful for reducing computational cost. Since Q is fully dense and may not be put into the computer memory completely, usually a special storage using the idea of a cache is utilized to store recently used Q ij. Just as in SVM light and LIBSVM, a simple least-recently-used caching strategy is implemented in PKSVM. When the cache has not enough room for a new row, the row which has not been used for the greatest number of iterations will be eliminated from the cache. 2

13 Only those rows of Q which correspond to active set A are computed and cached in PKSVM. Once shrinking occurs, we simply clean up the whole cache and recache shrunken rows. 7 Conclusions This article presents a simple summary of Support Vector Machine for classification problems. However, it also includes most of the state-of-the-art techniques to make SVM more practical for large-scale problems, such as shrinking and caching. Of course some other useful methods, i.e., working set selection, have to be skipped because of the space restriction. SVM has been developed to be a big family because of thousands of excellent research papers in the last ten years. It has become one of the most powerful and popular tools in machining learning. Although the whole article is dedicated to C-SVC, yet ν-svc, ɛ-svr, ν-svr and some other generalizations of SVM share most of the techniques mentioned here. The difference between them is small expect that the dual optimization problems differ formally. Thus one can easily consult more details about them in many textbooks and papers if it is necessary. References [] Naiyang Deng, Yingjie Tian, A New Method in Data Mining Support Vector Machine, Science Press, [2] Nils J. Nilsson, Learning machines: Foundations of Trainable Pattern Classifying Systems, McGraw-Hill, 965. [3] Vladimir N. Vapnik and A. Lerner, Pattern recognition using generalized portrait method, Automation and Remote Control, 24: , 963. [4] Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik, A training algorithm for optimal margin classifiers. In D. Haussler, editor, Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, pages 44-52, Pittsburgh, PA, July 992, ACM Press. [5] Corinna Cortes and Vladimir N. Vapnik, Support vector networks,machine Learning, 20:pp -25,

14 [6] J. C. Platt. Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors, Advances in Kernel Methods - Support Vector Learning, Cambridge, MA, 998. MIT Press. [7] T. Joachims. Making large-scale SVM learning practical. In B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors, Advances in Kernel Methods - Support Vector Learning, Cambridge, MA, 998. MIT Press. [8] Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines, 200. Software available at cjlin/libsvm [9] C.-W. Hsu, C.-C. Chang, C.-J. Lin. A practical guide to support vector classification July, [0] R.-E. Fan, P.-H. Chen, and C.-J. Lin. Working set selection using second order information for training SVM. Journal of Machine Learning Research, 6:889-98, URL cjlin/papers/quadworkset.pdf. 4