Module 4. Non-linear machine learning econometrics: Support Vector Machine

Transcription

1 Module 4. Non-linear machine learning econometrics: Support Vector Machine THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2 Introduction When the assumption of linearity is relaxed Non-linear models Polinomial regression Generalized additive models Decision Trees Support Vector Machines Etc. 2

3 Introduction: hyperplanes Hyperplane: In a p-dimensional space, an hyperplane is a flat affine subspace of dimension p-1 p=2 line p=3 plane Definition: p=2 β 0 + β 1 X 1 + β 2 X 2 =0 line equation p-dimensions β 0 + β 1 X 1 + β 2 X β p X p =0 3

4 Introduction: hyperplanes Geometric interpretation: If X= (X 1, X 2,,X p ) T satisfies the above equation the hyperplane X lies on If β 0 + β 1 X 1 + β 2 X β p X p >0 or β 0 + β 1 X 1 + β 2 X β p X p <0 X lies on one side or the other of the hyperplane We can think of a hyperplane as dividing p-dimensional space into two halves 4

5 Introduction: hyperplanes X Example: 1+2X1+3X2>0 1+2X1+3X2=0 1+2X1+3X2< X 1 5

6 Introduction: hyperplanes Separating hyperplanes: Define y 1, y 2, y n 1,1 f x = β 0 + β 1 X i1 + β 2 X i2 + + β p X ip >0 y i =1 f x = β 0 + β 1 X i1 + β 2 X i2 + + β p X ip <0 y i =-1 A test observation x* will be assigned a class (either 1 or -1) depending on which side of the hyperplane is located Magnitude of f x : if f x is far from 0, then x lies far from hyperplane Reliable class assignment for x 6

7 Introduction: hyperplanes Problem: If a hyperplane exists, then there exists an infinite number of other hyperplanes that could separate the data Possible solution: select the one that is the farthest from the data maximal margin hyperplane 7

8 Maximal margin classifier X maximal margin hyperplane: separating hyperplane for which the margin is largest margin: minimal distance from the observations to the hyperplane distance Note: similarity with fitting a regression hyperplane with leastsquares X 1 8

9 Maximal margin classifier X maximal margin classifier: A test observation will be classified depending on which side of the maximal margin hyperplane it lies Support vectors X 1 9

10 Maximal margin classifier n training observations x 1, x 2,,x n p dimensions y 1, y 2,,y n 1, 1 M width of margin Optimisation problem: Maximise M for β 0, β 1, β 2,, β p subject to: p σ j=1 β 2 j = 1 y i (β 0 + β 1 X i1 + β 2 X i2 + + β p X ip ) M for each i=1,..n Once maximised M, we classify a test observation depending on the sign of f x = β 0 + β 1 x 1 + β 2 x 2+ + β p x p 10

11 Maximal margin classifier Problems: It is not robust to individual observations it cannot be applied if no separating hyperplane exists Solution: Support vector classifier 11

12 Support vector classifier Based on hyperplane that does not perfectly separate the two classes Soft margin (it can be violated by some of the training observations) Robust to individual observations Better classification of most of the training observations 12

13 Support vector classifier How it works: Optimisation problem: Maximise M for β 0, β 1, β 2,, β p, ε 1, ε n subject to: p σ j=1 β 2 j = 1 y i (β 0 + β 1 X i1 + β 2 X i2 + + β p X ip ) M(1-ε i ), for each i=1,..n ε i 0, σ n i=1 ε i C ε 1, ε n = slack variables that allow individual observations to be on the wrong side of the margin or the hyperplane C= non-negative tuning parameter 13

14 Support vector classifier ε i =0 ε i >0 ε i >1 ith observation is on correct side of the margin ith observation is on wrong side of the margin (violates the margin) ith observation is on wrong side of hyperplane C determines the number and severity of the violations to the margin (and hyperplane) that are tolerated: C=0 no accepted violations C>0 accepted no more than C observations that can be on wrong side of hyperplane 14

15 Support vector classifier About C: Tuning parameter generally chosen via cross-validation It controls the bias-variance trade-off If C is small we want narrow margins rarely violated highly fit to the data (low bias but high variance) If larger, the margin is wider and we allow more violations lower fit to data (higher bias but lower variance) 15

16 Support vector classifier C higher Lower C 16

17 Support vector classifier Property: An observation that lies on the correct side of margin does not affect the support vector classifier Only Support vectors affect the classifier 17

18 Support Vector Machines Extension of the support vector classifier Method to enlarge the feature space to accommodate non-linear boundaries They use quadratic, cubic, or even higher-order polynomial functions of the predictors: X 1, X 2 1, X 2, X 2 2,, X p, X 2 p 18

19 Support Vector Machines p y i (β 0 + σ j=1 Maximise M β 0, β 11, β 12 β p1, β p2, ε 1, ε n p σ j=1 Subject to 2 β 2 jk = 1, k=1 p β j1 x ij + σ j=1 ε i 0, σ n i=1 ε i C β j2 2x 2 ij) M(1-ε i ), 19

20 Support Vector Machines Introducing Kernels (function that quantify the similarity of two observations): Inner product p K (x i, x i ) =σ j=1 p K (x i, x i ) =1+ (σ j=1 x ij x i j x ij x i j ) d linear kernel polynomial kernel p K (x i, x i ) =exp( γ σ j=1 x ij x i j 2 ) radial kernel 20

21 Support Vector Machines It combines a non-linear (polynomial) kernel with a support vector classifier If the linear support vector classifier can be represented by: Space of the indices for which α i 0 f x = β 0 + α i x, x i i S Inner product Parameter that 0 only if the training observation is a support vector Then the SVM: f x = β 0 + α i K(x, x i ) i S Polynomial kernel 21

22 Support Vector Machines X 2 X Examples: X 1 Polynomial kernel with d= X 1 Radial kernel 22

23 References An Introduction to Statistical Learning G. James, D. Witten, T. Hastie, R. Tibshirani; Springer, The Elements of Statistical Learning: Data Mining, Inference, and Prediction T. Hastie, R. Tibshirani, J Friedman; Springer, Introducton to machine learning E. Alpaydın; The MIT Press,