Chakra Chennubhotla and David Koes

Transcription

1 MSCBIO/CMPBIO 2065: Support Vector Machines Chakra Chennubhotla and David Koes Nov 15, 2017

2 Sources mmds.org chapter 12 Bishop s book Ch. 7 Notes from Toronto, Mark Schmidt (UBC) 2

3 SVM SVMs and Logistic Regression MOST popular methods Fast training and testing With high-dimensional features and regularization, good test error 3

4 SVM Geometry -> Intuition/Insight -> Math Margins and why maximize Objective function when data is separable by a hyperplane when data is nearly separable by a hyperplane when data is NOT separable by a hyperplane Kernel trick/primal-dual form Optimization algorithm 4

5 Geometry -> Intuition/Insight Margins 5

6 2D view of classifiers 6

7 Maximum-margin classifier Consider a linearly separable dataset Are all zero-error classifiers equally good? 7

8 Maximum-margin classifier Consider a linearly separable dataset Choose the farthest from both classes 8

9 Maximum-margin classifier Consider a linearly separable dataset Choose the farthest from both classes 9

10 Maximum-margin classifier Consider a linearly separable dataset Choose the farthest from both classes 10

11 Maximum-margin classifier Consider a linearly separable dataset Choose the farthest from both classes 11

12 Maximum-margin classifier Final classifier only depends on support vectors!! 12

13 Maximum-margin classifier 13

14 Math deriving the margins 14

15 15

16 16

17 Goal 17

18 What is the margin? 18

19 Largest Margin 19

20 Support Vector Machine 20

21 Support Vector Machine 21 (for hard margin classifier)

22 Canonical Hyperplane: Problem 22

23 Canonical Hyperplane: Solution 23

24 Maximizing Margins 24

25 Data almost separable by a hyperplane 25

26 SVM design with slack variables 26

27 Introducing slack variables Slack variables are constrained to be non-negative. When they are greater than zero they allow us to cheat by putting the plane closer to the datapoint than the margin. So we need to minimize the amount of cheating. This means we have to pick a value for C w.x c + b +1 ξ c for positive cases w.x c + b 1+ξ c with ξ c 0 for all c for negative cases 27 and w 2 2 c + C ξ c as small as possible

28 Slack Penalty C 28

29 SVM: Hinge Loss Form 29

30 SVM: estimate w 30

31 SVM: training 31

32 SVM: training Parallelizable 32

33 1. Multi-class SVM 33

34 2. Multi-class SVM: learn 3 sets of weights simultaneously 34

35 Multi-class SVM 35

36 36

37 37

38 38

39 Changing the basis 39

40 40

41 41

42 42

43 43

44 44

45 45

46 46

47 47

48 48

49 49 The kernel trick For many mappings from a low-d space to a high-d space, there is a simple operation on two vectors in the low-d space that can be used to compute the scalar product of their two images in the high-d space. a K( x, x Letting the kernel do the work b ) =f( x a ). f( x b ) doing the scalar product in the obvious way High-D b x f( x b ) Low-D f a x f( x a )

50 SVM Primal Problem: dimensionality of w may be too large Radial basis function 50

51 SVM Dual Fundamental scalability issue: kernel matrix is the square of the training set size 51 Further reading (kernel approximation):

52 Dealing with the test data If we choose a mapping to a high-d space for which the kernel trick works, we do not have to pay a computational price for the high-dimensionality when we find the best hyper-plane. We cannot express the hyperplane by using its normal vector in the high-dimensional space because this vector would have a huge number of components. Luckily, we can express it in terms of the support vectors. But what about the test data. We cannot compute the scalar product because its in the high-d space. w. f( x) 52

53 Dealing with the test data We need to decide which side of the separating hyperplane a test point lies on and this requires us to compute a scalar product. We can express this scalar product as a weighted average of scalar products with the stored support vectors This could still be slow if there are a lot of support vectors. 53

54 The classification rule The final classification rule is quite simple: All the cleverness goes into selecting the support vectors that maximize the margin and computing the weight to use on each support vector. 54 We also need to choose a good kernel function and we may need to choose a lambda for dealing with nonseparable cases.

55 Some commonly used kernels Polynomial: K( x, y) = ( x. y + 1) p Gaussian radial basis function K( x, y) = e - x-y 2 / 2 s 2 Parameters that the user must choose Neural net: K( x, y) = tanh ( k x.y - d ) 55

56 Performance Support Vector Machines work very well in practice. The user must choose the kernel function and its parameters, but the rest is automatic. The test performance is very good. They can be expensive in time and space for big datasets The computation of the maximum-margin hyper-plane depends on the square of the number of training cases. We need to store all the support vectors. SVM s are very good if you have no idea about what structure to impose on the task. The kernel trick can also be used to do PCA in a much higher-dimensional space, thus giving a non-linear version of PCA in the original space. 56