Kernel SVM. Course: Machine Learning MAHDI YAZDIAN-DEHKORDI FALL 2017

Transcription

1 Kernel SVM Course: MAHDI YAZDIAN-DEHKORDI FALL

2 Outlines SVM Lagrangian Primal & Dual Problem Non-linear SVM & Kernel SVM SVM Advantages Toolboxes 2

3 SVM Lagrangian Primal/DualProblem 3

4 SVM LagrangianPrimalProblem Minimize ()= 1 2 Subject to ( +) 1 L,, = ( + 1] This unconstrained optimization problem is solved by [Haykin, 1999]: Minimizing L with respect to the primalvariables and, and MaximizingL with respect to the dualvariables >0 The optimal solution would be saddle point saddle point 4

5 SVM LagrangianPrimal/DualProblem Primal L,, = ( + 1] For minimizingl over and, we have: (A) (B) By expanding L, L,, = 1 2 b + (C) By substituting (A) and (B) into (C), we have: Dual L = 1 2 ( ) It is a function of What about the shape of L? 5

6 SVM LagrangianPrimal/DualProblem SVM Lagrangian Primal Problem SVM Lagrangian Dual Problem Minimize L,, = 1 2 ( + 1] Maximize Subject to L = 1 2 ( ) 0 =0 For convexproblems, solving either primal or dual optimization problem leads to the global solution Note that: Here, solving the dual form (L ) is much easier than solving the primal form (L,, ) Moreover, in L the training data appears as dot products ( ) This property will be used later for non-linear SVM 6

7 SVM LagrangianPrimal/DualProblem SVM (Hard-Margin SVM) Lagrangian Dual Problem Maximize Subject to L = 1 2 ( ) 0 =0 After solving the dual problem: To calculate : To calculate b, we use the support vectors which satisfy : = () ( +)=1 7

8 Soft-Margin SVM Primal/DualProblem Soft-Margin SVM In Primal form Minimize = Subject to ( +) <1 >1 Note that the only difference respect to the previous result is : 0 Soft-Margin SVM In dual form Maximize Subject to L = 1 2 ( ) =0 8

9 Non-linear SVM (Kernel SVM) 9

10 Non-linear SVM :,,, =(, 2, ) 10

11 Non-linear SVM Cover s theorem: A complex pattern-classification problem cast in a high-dimensional space nonlinearly is more likely to be linearly separable than in a low-dimensional space Based on Cover s theorem, non-linear SVM: 1- Non-linearly, map the feature vector x onto a high-dimensional space 2- Construct an optimal separating hyperplane in the high-dimensional space Non-linear SVM 11

12 Non-linear SVM PrimalProblem Non-linear SVM Primal Problem Minimize Subject to = ( ( )+) 1 0 This explicit mapping to high dimensional space has two major problems: Explicit mapping onto high-dimensional space suffers from cures of dimensionality and risk of overfitting Working in high-dimensions is computationally expensive, which poses limits on the size of the problems Projection onto a high-dimensional space is performed implicitlyusing kernel function [Cristianini and Schölkopf, 2002] 12

13 Non-linear SVM Dual Problem Recall: the linear SVM solution in dual form depends only on the dot product between examples Maximize Subject to Therefore, the non-linear SVM in Dual form would be: Maximize L = 1 2 ( ).( ) Subject to L = 1 2 ( ) If we find a function, = ( ).( ) then operations in high dimensional space ϕ(x) do not have to be performed explicitly 0 0, Linear SVM in dual form Non-linear SVM in dual form 13

14 Kernel SVM Dual Problem SVM Lagrangian Dual Problem Non-linear SVM LagrangianDual Problem Kernel SVM LagrangianDual Problem Maximize Subject to Maximize Subject to L = 1 2 ( ) 0 L = 1 2 ( ).( ) Maximize Subject to 0 L = 1 2 (, ) 0 Explicit mapping to higher dimensional space using () Implicit mapping using kernel function (.) that:, = ( ).( ) 14

15 Kernel SVM Decision Function From the previous results for linear case (in primal form), the optimal heypeplane in the original space was: For the non-linear case we would have: Then, the decision function will be: = = h = + = + = (,)+ 15

16 Kernel: example Suppose a machine learning problem in and consider, = If (.) can be written as a dot product of two vectors, i.e., = =.( ), then it can be used as a kernel function. Can we do it?, = x x =. = + = =, 2,, 2, =.( ) Therefore, using, = we are implicitly operating on a higher dimensional defined by = Note that projection into is not performed explicitly. Instead, it is performed implicitly using (.) 2 16

17 Linear kernel Polynomial kernels Kernel Functions, = x, = x +1 =, 2, = 1, 2, 2, 2,, The degree d of the polynomial is a free parameter can be determined by cross validation Radial basis function kernels, =exp The σ is a user-defined parameter can be determined using cross validation The number of radial basis functions and their centers are automatically determined by the number of support vectors and their values 2 [Burges, 1998; Kaykin, 1999] 17

18 Kernel SVM: Example1 Consider a classical XOR problem Class 1 (=1): =(1,1), = 1, 1 Class 2 (= 1): =( 1,1), = 1, 1 Suppose a polynomial of order 2:, = +1 Note that this kernel function performs an implicit mapping to a 6-dimensional space by: = 1, 2, 2, 2,, Goal: Solve this problem by the kernel SVM in dualform, i.e. and optimal Maximize Subject to L = 1 2 (, ) 0 =0 [Cherkassky and Mulier, 1998; Haykin, 1999] 18

19 =?, =?, =?, =? Kernel SVM: Example1 The decision boundary? 19

20 Kernel SVM: Example1 The decision boundary is non-linear in the original space The decision boundary is linear in the implicit space 20

21 RBF kernel Kernel SVM: Example2 [T. Fletcher 2009] 21

22 Kernel SVM: Example3 Kernel Type, = x, = x +1, = x +1 Linear Kernel Polynomial Kernel d=2 Polynomial Kernel d=5 [Figure by Ben-Hur, et. al. 2008] 22

23 Kernel SVM: Example4 The effect of in RBF Kernel, =exp RBF kernel with =20 RBF kernel with =1 RBF kernel with = [Figure by Ben-Hur, et. al. 2008] 23

24 Kernel SVM Regression () 24

25 SVM Advantages SVM maximize margin and minimize structural risk SVM has a good generalization(theoretically and empirically) SVM provides a schema to control complexity independent of feature dimensionality The problem is a Quadratic problem (Convex) It has a global optimum with no local optimum The optimal solution can be found in polynomial time SVM classifier has a few free parameters Penalty term C and the parameter(s) of the kernel function (in RBF kernel) SVM solution is stable and repeatable (no randomness), robust and efficient Kernel SVM implicitly maps data to a high dimension 25

26 SVM for: clustering feature extraction Semi-supervised classification one-class SVM for outlier detection SVM for unbalanced Samples More Issues on SVM ν-support Vector Machines: Maximize margin while bounding the number of margin errors Kernel SVM Regression SVM that take into account difference cost of misclassification for the different classes String kernel to handle text samples 26

27 There are many SVM toolbox: OsuSVM(MATLAB with C++ Mex) SVMLight(C) LibSVM(C++) SVM toolbox Some machine learning toolbox that include SVM: Spider (Matlab) Torch (C++) Weka(Java) See following page to find furtherer machine learning toolboxes 27

28 Referene C. Cortes, V. Vapnik, Support-Vector Networks,, 20, pp , Tristan Fletcher, Support Vector Machines Explained, C. M. Bishop, Pattern Recognition and (Information Science and Statistics), Springer (2006). J.C. BURGES, A Tutorial on Support Vector Machines for Pattern Recognition, Knowledge discovery and data mining, 2 (2), K.R. Müller, S.Mika, G.Rätsch, K.Tsuda, and B.Schölkopf, An introduction to kernel-based learning algorithms. IEEE Neural Networks, 12(2), pp , N. Cristianiniand B. Scholkopf, Support Vector Machines and Kernel Methods: The New Generation of Learning Machines, AI Magazine, 23(2), pp. 31, Andrew Ng, Support Vector Machines Lecture notes, Oxford University. 28

29 Appendix 29

30 Primal LagrangianProblem Suppose an primal optimization problem with following conditions: Minimize Subject to 0 =1,, h =0 =1,, Generalized Primal Lagrangian Problem is defined as Minimize It is solved by: Minimizing L with respect to the primal variables and, and Maximizing L with respect to the dual variables 0 L,, = + h + saddle point 30