Solving the SVM Problem. Christopher Sentelle, Ph.D. Candidate L-3 CyTerra Corporation

Size: px

Start display at page:

Download "Solving the SVM Problem. Christopher Sentelle, Ph.D. Candidate L-3 CyTerra Corporation"

Piers Hutchinson
5 years ago
Views:

1 Solvng the SVM Problem Chrstopher Sentelle, Ph.D. Canddate L-3 Cyerra Corporaton

2 Introducton SVM Background Kernel Methods Generalzaton and Structural Rsk Mnmzaton Solvng the SVM QP Problem Actve Set Method Research Future Work

3 Introduced by Vapnk (1995) Bnary classfer Foundaton n statstcal learnng theory (Vapnk) Excellent generalzaton performance Easy to use Avods curse of dmensonalty Avods over-tranng Sngle global optma Burges, C., A utoral on Support Vector Machnes for Pattern Recognton

4 Support Vector Machne seeks maxmal margn separatng hyperplane 1 mn w s.t. y w, x b 1 0 Non-optmal hyperplane wth small margn Support Vector he maxmum margn provdes good generalzaton performance Support vectors le on the margn and defne the margn Margn 1 w Optmal hyperplane has largest margn

5 In many cases, problem s not lnearly separable.5 Soft-margn formulaton ntroduces slack varables to allow some data ponts to cross margn 1 mn C s a tunable parameter s.t. y w, x b 1 w C Penalty term Slack varable

6 he SVM problem formulaton can be recast to dual through method of Lagrange multplers N N N 1 LP α, b, w w, w y w, x b 1 C u N L w yx 0 w 1 1 N L y 0 b L C u L N N 1 y y D j j j 1, j x, x

7 Dual-formulaton SVM Quadratc Programmng Problem y 1, 0 y 1, 0 C m 1 ax j y y j x, x j, j s. t. Convex, Quadratc objectve 0 C, y 0 Data ponts appear wthn dot product Penalty parameter, C, shows up n bound constrants Sngle equalty constrant Bound SVs, cross margn wth nonzero slack ( C ) Non-bound SVs, on margn 0 C Non SVs ( 0 ) y 1,0 C y 1,0 C

8 Kernel methods can be appled to SVM dual formulaton 1 max yyk x, x j j, j s. t. 0 C, y 0 Kernel represents dot-product formed n a non-lnearly mapped space k x, x x, x Unnecessary to know or form mappng. j j j j

9 A chosen kernel functon must satsfy Mercer s Condton (Vapnk, 1995), A kernel functon K x, y satsfes Mercer's condton f for all functons, g x, such that g x dx s fnte K x y g x g y dxdy 0 Common kernel functons: **RBF kernel most commonly employed snce t s numercally stable and can approxmate a lnear kernel. Polynomal Gaussan RBF Sgmod Lnear xy k x, y 1 d k x, y e x y k x, y x, y, tanh x, y k x y

10 A slghtly dfferent problem s solved for Support Vector Regresson (SVR): 1 C mnmze w ˆ subject to w, x y w, x b ˆ ν -SVC trades the parameter, C, n classfcaton for a parameter, ν, that pre-specfes proporton of support vectors b y mnmze subject to 1 1 y x, w b 0, 0 w l

11 ypcally a neural network mnmzes the emprcal rsk (Vapnk, 1995) ntroduces the followng rsk bound: For some R R emp l 1 R y f x, emp l 1 wth probablty less than log lh1log 4 h l Where h s the VC (Vapnk-Chervonenks) dmenson, a measure of capacty Vapnk ntroduced noton of Structural Rsk Mnmzaton Perform trade between emprcal error and hypothess space complexty

12 Capacty or VC-dmenson s related to how many ponts can be arbtrarly labeled and correctly ft by a set of functons, f In ths example, 3 data ponts are shattered by the lne functon (VC dmenson s 3) Example from Burges, C., A utoral on Support Vector Machnes for Pattern Recognton

13 LIBSVM (SMO) Commercally avalable: SVMLght (Decomposton w/ Interor Pont) SVMorch (Decomposton for Large-Scale Regresson) SVM-QP (Actve Set method)

14 SVM dual formulaton Convex, quadratc, bound-constraned, sngle equalty constrant Problem s large Q, j mn α s.t. α y j Qα 1 y y K, α 0, 0 C Q (kernel matrx) s dense, sem-defnte, ll-condtoned Equalty constrant y α 0 s consdered nusance for some algorthms α nn x x j

15 Karush-Kuhn ucker (KK) condtons for optmalty are necessary and suffcent, gven convexty, for optmalty 1 LD (,, r, ) Q 1 ( y ) r ( 1 C) Q 1 y r 0 y 0 r r C ( C) 0 Feasble regon Non-negatvty constrants on KK multplers for nequalty constrants Complementary condtons

Vapnk Introduces SVM Chunkng Geometrcal Convex Hull Actve Set Gradent Projecton Prmal

Volatng Par WSS Quadratc nfo WSS Maxmum Gan WSS DrectSVM SmpleSVM SVM-QP SVM-RSQP

Constraned GP Cuttng Plane (SVMPerf) Stochastc Sub-gradent Pegasos *hs represents only

16 Vapnk Introduces SVM Chunkng Geometrcal Convex Hull Actve Set Gradent Projecton Prmal Methods Interor Pont Gradent Projecton Interor Pont (SVMLght) Decomposton SMO Max Volatng Par WSS Quadratc nfo WSS Maxmum Gan WSS DrectSVM SmpleSVM SVM-QP SVM-RSQP Regularzaton Path Search Incremental/ Decremental ranng Varable Projecton Sngle Equalty Constraned GP Cuttng Plane (SVMPerf) Stochastc Sub-gradent Pegasos *hs represents only a samplng of optmzaton methods reported n the lterature Low-Rank Kernel Approx OOQP for Massve SVMs

17 Decomposton method breaks problem down nto sequence of fxedsze sub-problems 1 mn wqww w wqwn n 1w w w w w n n w st.. y y, 0 C Varables are optmzed whle the set of varables reman fxed w Objectve decreased at each teraton as long as KK volators added to sub-problem and non-kk volators swtched out Proof of lnear convergence (Keerth, Ln, et al.) n

18 Sub-problems can be solved wth any approprate QP-solver (Joachms, 1998) ntroduces SVMLght Decomposton method, even number of ponts Introduces LP problem for workng set selecton Introduces shrnkng Solves sub-problem usng nteror pont method (IPM), pr_loqo (Zanghrat et al., 003) (Serafn et al., 005) Uses a gradent projecton method to solve sub-problem Works wth larger sub-problem szes Parallelzable mplementaton

19 SMO (Sequental Mnmal Optmzaton, Platt 1999) takes decomposton to extreme by optmzng data ponts at a tme Sub-problem has analytcal soluton E1 E y, y1 1 y 0 y y 1 1 C C C K K K, E y K b y 11 1 N j1 j j j 0 C Optmal soluton clpped to mantan feasblty y y 1 1 0

20 (Platt, 1998) Sequental Mnmal Optmzaton (SMO) Sub-problem contans ponts -> analytcal solutons Computes hgh number of cheap teratons (Keerth, 003) ntroduces dual-threshold method Improves on effcency of Platt s orgnal mplementaton (Fan et al. 005) ntroduces an mproved workng set selecton employng nd order (quadratc) nformaton LIBSVM (Ln et al. 001) s stll a popular SMO mplementaton

21 Gven an nequalty constraned problem 1 mn Q 1 s. t. Ax b Cx d Goal s to dentfy actve constrants n soluton,.e. nequalty constrants satsfed as equalty constrants Actve set method ncrementally dentfes actve constrants and solves correspondng equalty constraned problem untl all nequalty constrants are satsfed

22 Equalty constraned problem s solved gven current set of actve constrants 1 mn Q 1 s. t. Ax b C [] x d, Actve set Volated nactve constrants dentfed and sngle nactve constrant converted to actve constrant and problem resolved he actve set method explores faces of convex polytope formed from set of nequalty constrants

23 Incremental nature allows use of effcent rank-one updates to factorzatons used to solve equalty-constraned problem Ideal for medum-szed problems wth few support vectors (sparse soluton) Memory stll ssue for large problems, can employ Precondtoned Conjugate Gradent Kernel Approxmatons he actve set method explores faces of convex polytope formed from set of nequalty constrants

24 (Cauwenberghs, et al., 000) Introduce an actve set method for ncremental/decremental tranng Method s only applcable to postve defnte kernels (Vshwanathan et al., 003) Introduce an actve set method (SmpleSVM) Intalzes the method usng the closest par, opposte label, ponts Mentons possbltes for, but does not handle the sngular Kernel matrx case (Shlton et al., 005) Solve a mn-max problem or prmal-dual hybrd problem Avods workng wth the equalty constrant

25 (Vogt et al., 005) Does not address the case of a sem-defnte or sngular Kernel matrx Suggests a gradent projecton be used to speed-up actve set method (Schenberg et al., 005) Introduces effcently mplemented dual Actve Set method Handles sem-defnte kernels Shows compettve results aganst SVM Lght (Sentelle et al., 009, n work) Introduces Revsed Smplex, avods sngulartes common to Actve Set methods

26 Based upon the penalty barrer method Replace nequalty constrants wth penalty functon x 0 ln( x ) becomes 1 mn c x x Qx x st.. Ax=b, x 0 mn x st.. Ax=b 1 c x x Qx ln x N j1 j Optmalty condtons become: 1 LD ( x,, ) c x x Qx 1 c Qx A V e Ax b 0 VSe e ( xs, ) 0 where V dag( x, x,... x ) j1 S dag( s, s,... s ), s n n N ( Ax b) ln x j V 1 e

27 Newton s method employed to fnd soluton to optmalty condtons x x n1 n f( x ) f ( x ) Whch for IPM, the Newton step becomes A 0 0 x b Ax Q A I c A s Qx S 0 Vs evse n n In IPM, complementary condtons not mantaned, met at soluton Ideal for dense Hessans and large problems Kernel approxmaton methods proposed to allow method to deal wth non-lnear kernels nk Q LL D, L, k n Polynomal convergence, theoretcal worst case s O(n)

28 (Fne & Schenberg, 001) Implements a prmal-dual nteror pont method (Mehrotra predctoncorrector) Problem solved at each teraton grows cubcally wth problem sze Employ Kernel approxmaton to reduce computatons at each teraton Employ Incomplete Cholesky factorzaton (Ferrs & Munson, 003) Dscuss applcaton of IPM to large-scale SVM problems (Gondzo & Woodsend, 009) Show how use of separablty allows scalng of IPM to large-scale problems (outperforms all but LIBLnear on large problems)

29 ry to solve the prmal problem nstead of dual 1 mnmze w s.t. y wx, b 1 0 radtonally, dual admts easer use of kernel matrx Dual soluton not always sparse Prmal converges to approxmate soluton qucker (Chapelle, 006) C

30 (Chapelle, 006) SVM prmal objectve can be casted as mn w C L y, w, x b w,b usng Reproducng Kernel Hlbert Space, mn f f H H L y, f Equatng the gradent to zero and employng reproducng property mn β β Kβ L y, Smlar to (Ratlff et al, 007) for Kernel Conjugate Gradent method K x β

31 (Joachms, 006) ntroduces SVM-Perf Cuttng planes method Shows consderable mprovement over decomposton methods for lnear kernels Convergence s (Shalev-Shwartz et al., 007) ntroduce Pegasos Employs stochastc gradent descent, sub-gradent, trust-regon Solves text classfcaton problem (Reuters Corpus Volume 1) wth 800,000 ponts < 5 sec!!!! Lnear kernels only Convergence s O O 1 1

32 Works by projectng gradent onto convex set of constrants mn t q( x(t)) where x(t) P q( x) Actvates/deactvates more than one constrant per teraton (Da & Fletcher, 006) ntroduce sngle-equalty constraned Gradent Projecton for SVM 1 x x o tg, Gx d x x 0 gradent projected gradent

33 (Wrght, 008 NIPS conference) suggests applcaton of prmal-dual gradent projecton to SVM problem Consder saddle pont (mn-max) problem where s convex for all and s concave for all Projecton steps are as follows: x v l X x V v, mn max x l, X x v, l V v 1 1 1,, k k v k k V k k k x k k X k x v l v P v x v l x P x (Zhu et al., 008) An effcent prmal-dual hybrd gradent algorthm for total varaton mage restoraton, ech Report, UCLA

34 he SVM problem s convex n but s only non-strctly concave n (or non-strctly convex) Lookng at the projecton step, we have where s the projecton onto the box constrants Step lengths must be carefully chosen α y α Qα α C b b 1 mn max α b y 1 Qα α α α y α k k k k k k k k k b P b b P C 0 k k,

35 Can we further mprove tranng algorthms for large datasets? Popular SMO algorthm s a coordnate descent method Care must be taken to ensure numercal stablty Data scalng affects convergence speed Improper settng of C value and/or kernel parameters can prevent convergence May be slow, overall, when compared to other methods Other non-decomposton methods ntroduced n SVM communty cannot handle large tranng set szes Our goal s to fnd non-decomposton technques that can handle large tranng sets and are fast

Gven the equalty-constraned problem solved at each teraton, a drecton of descent can be solved (for SVM) Postve sem-defnte sub-matrx of orgnal Kernel matrx contanng non-bound SVs Q y ss s ys h 0 g c

36 Gven the equalty-constraned problem solved at each teraton, a drecton of descent can be solved (for SVM) Postve sem-defnte sub-matrx of orgnal Kernel matrx contanng non-bound SVs Q y ss s ys h 0 g c d In general, system of equaton s ndefnte, possbly sngular Several factors cause sngularty Kernel-type (lnear kernel) Subset of chosen non-bound support vectors Sngulartes can occur for postve defnte kernels!

37 (Rusn, 1971) Revsed Smplex Introduces revsed smplex for quadratc programmng Equalty-constraned problem solved at each teraton guaranteed non-sngular Represents a form of nerta controllng method (Sentelle et al., 009) Adapt Revsed Smplex to SVM Optmze method for SVM problem Employ Cholesky factorzaton wth rank-one updates (smlar to Schenberg 006) Show method can be vewed as a fx to exstng actve set methods whch must contend wth sngulartes

38 Conventonal actve set methods typcally solve equalty constraned problem of actve constrants, drectly Q ys ss * ys α s 0 * 1 Q y Solvng for descent drecton allows deleton of exstng constrant before addng new constrant sc c αc Q * ss ys α s αs qs 0 * y ys α c Conventonal actve set method does not mantan complementary condtons durng descent Qss ys hs 0 g ys 1s Q α Q α y 0 sc c ss s s Mantanng complementary condtons are key to guaranteeng non-sngular equatons Q ys ss y h s 0 s g e 0

39 Indefnte, non-sngular matrx admts use of NULL-space method for soluton Gven the system Qss ys h u ys 0 g v Defne NULL space as s z y Zy 0, h Zh Yh System of equatons becomes Q Zh Q Yh y g u s y Yh ss z ss y s y v Solvng for Z Q h z ss z y Zh Z ( u QYh ) A Cholesky factorzaton s mantaned Z Q Z ss Null space has an analytcal soluton Z y y Effcent rank-one updates made to Cholesky factorzaton wthout typcal QR factorzatons R R... y yn I 1 1

40 Adult-a has 3,000 data ponts Cover ype has 100,000 data ponts 10 - stoppng crteron

41 Improved speed and memory consumpton alternatves stll beng nvestgated Gradent project steps can be nserted to move multple constrants per teraton Conjugate gradent can be appled to reduce memory consumpton Problem becomes ncreasngly ll-condtoned as progress made towards soluton Care must be taken to ensure condtons of non-sngularty mantaned despte accuracy of conjugate gradent soluton

(Vapnk, 009) has ntroduced Learnng Usng Prvleged Informaton (SVM+) Makes use of addtonal nformaton avalable durng tranng to speed up converge Examples of prvleged

42 (Vapnk, 009) has ntroduced Learnng Usng Prvleged Informaton (SVM+) Makes use of addtonal nformaton avalable durng tranng to speed up converge Examples of prvleged nformaton Future stock prces Radographer notatons n tumor classfcaton task Addtonal annotatons made n handwrtng recognton task Our method can be appled to ths formulaton

43 Questons?

Support Vector Machines

Support Vector Machines Support Vector Machnes Decson surface s a hyperplane (lne n 2D) n feature space (smlar to the Perceptron) Arguably, the most mportant recent dscovery n machne learnng In a nutshell: map the data to a predetermned