Announcements. Supervised Learning

Transcription

1 Announcements See Chapter 5 of Duda, Hart, and Stork. Tutoral by Burge lnked to on web page. Supervsed Learnng Classfcaton wth labeled eamples. Images vectors n hgh-d space.

2 Supervsed Learnng Labeled eamples called tranng set. Query eamples called test set. Tranng and test set must come from same dstrbuton. Lnearly Separable Classes

3 Lnear Dscrmnants Images represented as vectors,,,. These could be pels But could also be any features Use these to fnd hyperplane defned by vector w and w 0. s on hyperplane: w T + w 0 0. Notaton: a T [w 0, w, ]. y T [,,, ] So hyperplane s a T y0. A query, q, s classfed based on whether a T q > 0 or a T q < 0. Why lnear dscrmnants? Optmal f classes are Gaussan wth same covarances. Lnear separators easer to fnd. Hyperplanes have few parameters, prevents overfttng. Have low VC dmenson. Lnear may seem lke a bg lmtaton But t s not f the features are comple enough 3

4 Eample XOR. An obect s a D bnary vector,y. Class s or,y e.,,0 & 0,. Not lnearly separable n,y space. But s lnearly separable n,y,*,y*y,*y space * + y*y **y > 0 Eample: Naïve Bayes Assume all features are ndependent. Buld optmal Bayesan classfer. For bnary features, two classes, ths produces a lnear classfer. 4

5 5 > + + > + d 0 ln ln ln : Choose Defne : Independence when : Choose q p q p q p q p q p d d Lnearly Separable Classes For one set of classes, a T y > 0. For other set: a T y < 0. Notatonally convenent f, for second class, make y negatve. Then, fndng a lnear separator means fndng a such that, for all, a T y > 0. Note, ths s a lnear program. roblem conve, so descent algorthms converge to global optmum.

6 erceptron Algorthm erceptron Error Functon J y y Y T J a a y y Y Y s set of msclassfed vectors. So update a by: a k + a k + η y Y Smplfy by cyclng through y and whenever one s msclassfed, update a a + cy. Ths converges after fnte # of steps. y erceptron Intuton ak ak+ 6

7 Support Vector Machnes Fnd lnear separator wth mamum margn. Some guarantees ths generalzes well. Can work n hgh-dmensonal space wthout overfttng. Nonlnear map to hgh-dm. space, then fnd lnear separator. Specal trcks allow effcency n rdculously hgh dmensonal spaces. Can handle non-separable classes also. Not as mportant f space very hgh-dmensonal. Mamum Margn Mamze the mnmum dstance from hyperplane to ponts. onts at ths mnmum dstance are support vectors. 7

8 Geometrc Intutons Mamum margn between ponts -> Mamum margn between conve sets p kp ap kap ap or ap + k p + k ap ap ap ap ap 0 k Ths mples ma margn hyperplane s orthogonal to vector connectng nearest ponts of conve sets, and halfway between. 8

9 Why s ma margn good? Intutvely best for nosy data. Guarantees good results n leave-oneout classfcaton f #support vectors small. If you leave out non-support vector, get same hyperplane, and correct classfcaton Intutvely related to Fsher LDA. Computaton: Fndng ma margn classfer s conve Fnd w and b such that: w + b > for one class w + b < for other. Margn s /w. So mnmze <w,w> subect to lnear constrants. Ths s a conve optmzaton problem. 9

10 In w space must fnd pont n conve polytope closest to orgn. If two ponts are local optmum, all conve combnatons of them are too, whch nclude closer ponts. Fast computaton n hgh dmensonal spaces w s lnear combnaton of support vectors. To compute w + b need nner products of and support vectors. Use kernels n whch nner products for hgh dmensonal space computed n low dmensonal space. 0

11 Eample: monomal kernels, z z z z zz z Ths s equvalent to the nner product of,,..., n n z vectors: SVM Summary Ma margn s good, and effcently computed Kernel method allows computatons n rdculously hgh dmensonal spaces Combnaton s what s mportant. Arbtrary lnear separator won t generalze well. Ma margn can generalze n hgh d space.

12 Non-statstcal learnng There are a class of functons that could label the data. Our goal s to select the correct functon, wth as lttle nformaton as possble. Don t thnk of data comng from a class descrbed by probablty dstrbutons. Look at worst-case performance. Ths s CS ey approach. In statstcal model, worst case not meanngful. On-Lne Learnng Let X be a set of obects eg., vectors n a hghdmensonal space. Let C be a class of possble classfyng functons eg., hyperplanes. f n C: X-> {0,} One of these correctly classfes all data. The learner s asked to classfy an tem n X, then told the correct class. Eventually, learner determnes correct f. Measure number of mstakes made. Worst case bound for learnng strategy.

13 VC Dmenson S, a subset of X, s shattered by C f, for any U, a subset of S, there ests f n C such that f s on U and 0 on S-U. The VC Dmenson of C s the sze of the largest set shattered by C. VC Dmenson and worst case learnng Any learnng strategy makes at least VCdmC mstakes n the worst case. If S s shattered by C Then for any assgnment of values to S, there s an f n C that makes ths assgnment. So any set of choces the learner makes for S can be entrely wrong. Alternately, sets of C est where no generalzaton possble based on C- eamples. 3

14 VC Dmenson and SVMs VC dmenson depends on number of support vectors. Eample: suppose support vectors For n ponts, at most n*n classes. To shatter ponts, must have ^n classes. VC dmenson < 5. Ths does not depend on dmenson of space. Smlarly for k support vectors 4