Large Margin Nearest Neighbor Classifiers

Transcription

1 Large Margn earest eghbor Classfers Sergo Bereo and Joan Cabestany Departent of Electronc Engneerng, Unverstat Poltècnca de Catalunya (UPC, Gran Captà s/n, C4 buldng, Barcelona, Span e-al: Abstract. Large argn classfers are coputed to assgn patterns to a class wth hgh confdence. Ths strategy helps controllng the capacty of the learnng devce so good generalzaton s presuably acheved. Two recent exaples of large argn classfers are support vector learnng achnes (SVM [2] and boostng classfers [0]. In ths paper we show that t s possble to copute large-argn axu classfers usng a gradent-based learnng based on a cost functon drectly connected wth ther average argn. We also prove that the use of ths procedure n nearestneghbor ( classfers nduce solutons closely related to support vectors..introducton In learnng pattern recognton, a classfer s constructed for assgnng future observatons to one of the exstng classes based on soe knowledge about the proble gven by the tranng set. Typcally, these learnng achnes use the nuber of sclassfcatons n the tranng set as a cost easure to be optzed durng tranng. Fro a theoretcal pont of vew, ths knd of optzaton process ensures a good generalzaton of the learnng devce once ts capacty (.e. a easure that accounts the coplexty of the learnng achne s controlled. However, recently t has been shown that, we should also take nto account the confdence or argn of the classfcatons n order to guarantee a low generalzaton error. Therefore, the tranng saples ust be assgned to the correct class wth hgh confdence on average durng tranng so the classfer can attan a large argn dstrbuton. The reason why a large argn dstrbuton allows achevng a better control of the capacty of the learnng devce s related to the constrants posed n the soluton that ensure a hgher degree of stablzaton. Two recent exaples of large argn classfers are support vector learnng achnes (SVM [2] and boostng classfers [0]. However, other learnng achnes lke ultlayer perceptrons (MLPs also belong to ths category because the nsaton of the ean squared error (MSE leads n practce to axsng the argn, as [] suggests. In ths paper we show that t s possble to defne another knd of loss functons, whch also axze the argn and s closely related. Besdes, we also deonstrate that the soluton acheved n the context of nearest neghbor classfers (.e. prototypes s connected to support vectors. J. Mra and A. Preto (Eds.: IWA 200, LCS 2084, pp , 200. Sprnger-Verlag Berln Hedelberg 200

2 670 S. Bereo and J. Cabestany In the next secton we ntroduce large argn classfcaton. Secton 3 shows that adaptve soft k- classfers [2][3] and the learn algorth [4] are large argn classfers related wth support vectors. Fnally soe conclusons are gven. 2. Large Margn Classfcaton p Let g ( x be a classfer that assgns an nput pattern x R to one of the c exstng classes. If all the classes have the sae rsk, the perforance of g can be easured wth the probablty of classfcaton error defned as L ( g P{ g( y} = x ( where P s the probablty dstrbuton on Xx{,...,c} and y ndcates the class label of the pattern x. The best possble classfer g *, whch s called the Bayes classfer, nses L(g. Snce P s usually unknown, an eprcal estator of g, g (x, s created by the estate of L(g, defned as (2 Lˆ x ( g ;D = { g ( y} = where (u s the ndcator functon whch s f u s true and 0 otherwse, and D = {, y,...,, y } s a set of exaples called the tranng set. Suppose that g s a axu classfer that uses c dscrnant functons {d (x, c =,...,c}, subect to the constrant d ( = x =, whch deterne the confdence of g n each class. Then, g dscrnates usng the followng rule: g ( x d ( x = ax d ( x = (3 =,...,c The argn of g gven a tranng saple (x,y can be defned as the dfference between the value of the dscrnant functon of the class whch x belongs to and the axal value of the other ones [0],.e. c ( g, x, y = ( y = d ax d = =,...,c Clearly, s on the nterval [-,] and only postve values denote correct classfcatons. As Æ, g classfes wth hgher confdence. The argn s a rando varable and consequently can be analyzed n ters of ts cuulatve dstrbuton functon, whch s called the argn error estate [], ˆ e ( g, γ ; D = { ( g, x, y < γ } = ote that the argn error estate ncludes the sclassfcaton error n the tranng set because ˆ e( g,0;d = Lˆ ( g; D. Snce we are nterested n easurng the argn over the whole tranng set, we can sply obtan the average argn of g on D as (4 (5

3 Large Margn earest eghbor Classfers 67 (6 ( g ; D = ( g, x, y = Equaton (6 can be used as the cost functon to be optzed n the tranng phase of large argn classfers. However note that, n the case of axu classfers, the ncluson of the ter c ( y ax d ( ter then s not necessary snce = x = = once we force the rght dscrnant functon to one, the others autoatcally are c forced to zero due to = d ( x. Therefore, the nzaton of Lˆ = ( g; D = ( y = d c = = s equvalent to the axzaton of Equaton (6. Fgure shows the evoluton of ˆ e( g, γ ; D n test set on the Pa database for the adaptve soft k- classfer [2][3] whch nzes Equaton (7. As the learnng te ncreases, test saples are classfed wth greater argn so ( g ; D s axsed whle the test classfcaton error ˆ e( g,0; decrease. D (7 Fg.. Evoluton of the cuulatve dstrbutons for the Pa test set as the nuber of epochs augents. Accordng to [], the generalzaton error of learnng achnes that axze g ; durng learnng s bounded wth probablty -δ by ( D ( g ˆ e( g, γ ; D r{, δ, fat ( γ } L < + (8 G

4 672 S. Bereo and J. Cabestany where r s a coplexty ter and fat G s a scale-senstve verson of the VC denson called fat-shatterng. Typcally, the coplexty ter r augents as the fat G does. The generalzaton error s nzed when both ters of the rght-hand sde of Equaton (8 are sultaneously nzed. Hence, the nzaton of ˆ e( g, γ ; D for a gven γ, acheved through the axzaton of ( g ; D by the learner, does not guarantee a low generalzaton error snce the coplexty ter r can be arbtrarly large due to the ncrease of the capacty easure fat G. Therefore, fat G ust be controlled. 3. Large Margn n Classfcaton and Support Vectors Suppose we have the lnear-separable 2-class proble of fgure 2. It s possble to copute any lnear classfers that solve the proble. Fgure 2 also shows ten solutons coputed wth the LVQ algorth [7]. Fg.2. A toy two-class proble that s lnear separable. We show the class border of ten lnear classfers (.e. a -nearest-neghbour classfer wth 2 prototypes coputed wth the LVQ algorth. ow, we pose a slope of 45º to the lnear classfer. Agan, any solutons coexst. evertheless, t has been observed [2] that the lnear classfer wth a slope of 45º that acheves a axal separaton (or argn between the extree data ponts (or support vectors of each class controls effectvely ts capacty (.e. ts fat G and consequently s hghly generalzable even f the nput space has a hgh densonalty (see fgure 3. ote that the optal argn (OH hyperplane s ore robust wth respect tranng patterns and paraeters: a slghtly varaton on a test pattern or on the value of the lne wll presuably not affect the classfcaton accuracy []. Consequently, OH s ore relable snce t has the largest argn and then can acheve better generalzaton perforance.

5 Large Margn earest eghbor Classfers 673 Fg.3. The optal argn hyperplane for the toy proble (fgure 2. Ths lne has a slope of 45º and acheves a axal separaton between the extree data ponts of each class. These extree ponts are also known as support vectors. Fgure 4 shows the OH for a separable two-class pattern recognton proble. The applcaton of the Learn algorth [4], whch uses gaussan kernels and the Eucldean dstance etrc, s also shown n ths proble and converges to OH. Learn transfors -nearest-neghbour classfers as axu classfers whose dscrnant functons are based on a local xture odel, whch can be derved as the followng splfcaton of Parzen wndows: K ( W; γ x C ' K W; γ d =,...,c ' = where s the nearest centre of the xture odel to x that belong to class usng W the dstance etrc d and kernel K W; γ = K( d, W ;γ. These centres are n fact the prototypes of the - classfer and are coputed through the nzaton of Equaton (7 usng a gradent-descent algorth. The reason why the lnear classfer coputed wth learn converges to the OH can be explaned coputng the prototypes that nses Equaton (7. When we only have one prototype for each class {, =,,c}, solvng Lˆ ( g ;D = 0 for gaussan kernels yelds w (9

6 674 S. Bereo and J. Cabestany = =,...,c ( y = d d = 0 = 0 C ( x ( y = k d d k = 0 k k = C ( y = d ( d ( y = k d d k = 0 k= k Accordng to Equaton (0, prototypes depend on few tranng saples: only those tranng ponts that have a sgnfcant actvaton of ther correspondng weght functon contrbute to for prototypes. Each prototype are coputed wth the followng subset of tranng data: S Saples belongng to the class whch are near the class border snce the weght functon of these saples s d (-d and reaches ts axu for d =0.5. S Saples belongng to any other class whch are near the border of class snce the weght functon of these saples s d d k and reaches ts axu for d =0.5 and d k =0.5. Snce the nu ponts of Lˆ ( g; D tend to ensure a nu nuber of sclassfcatons, the set of prototypes that solve Lˆ ( g ;D = 0 are fored w wth the sub-set of tranng saples near class borders, that are hard to classfy, that s hard boundary ponts [5] whch are n fact the support vectors of each class. When there s only one prototype for each class learn s equvalent to adaptve soft - classfers so both learnng achnes yeld the sae soluton for the proble n fgure 4. The so-called adaptve soft k- classfer [2][3] s a soft k- rule + a gradent-descent learnng algorth based on nzng Equaton (7. The soft k- rule converts k- ethods as axu classfers whose dscrnant functons are a drect extenson of the crsp k- estates based on the followng use of kernels: ; γ L ( y = k (, x K( x; γ = d =, =,..., c L ( y = k (, x K( x; γ = where { } are the prototypes of the classfer, { } prototypes, L s the nuber of prototypes, ( x (0 ( y are the labels assocated wth the K u ;γ s a bounded and even functon on 2 X that s peaked around 0 wth a localty paraeter γ (e.g. K( u; γ exp( u 2γ and (, x usng the dstance etrc (, x =, k s a functon that takes f s one of the k-nearest-neghbors to x D (e.g. the Eucldean dstance and 0 otherwse. If the tranng set D s used as the set prototypes, the above dscrnant functon estates the posteror class probabltes usng a cobnaton of k- and Parzen estatons [3]. However, the reduced set of prototypes coputed by nzng Equaton (7 typcally exhbts better generalzaton perforance. As Fgure shows, the argn s axzed durng learnng so the dscrnant functons typcally assgn data wth hgh confdence to one of the classes,.e. t assgns values near or

7 Large Margn earest eghbor Classfers But there are soe tranng data near class borders that the classfer assgns wth a saller argn. Aong the, we ght fnd the support vectors. Fg.4. Optal Margn Hyperplane (OH for the toy proble that s lnearly separable. We also show the class border of the classfer coputed wth learn. Observe that t converges to OH. Another addtonal beneft of the gradent-based approach for the coputng of large-argn classfers s related to ts poor behavor as an optzer snce gradentbased algorths are stacked at local na. The under-coputaton of gradent descent algorths prevents over-fttng [6] so capacty can be better controlled. See for nstance MLPs [8]. Fgure 5 shows an exaple usng the Rpley s proble [9] n whch an over-paraeterzed - classfer coputed wth learn does not overft tranng data. The applcaton of learn and adaptve soft k- classfers to real data (e.g. hand-wrtten character recognton s addressed n [2][3][4]. 4. Conclusons In ths paper, we show how to copute large-argn axu classfers usng a gradent-based learnng based on a cost functon drectly related wth the average argn of the classfer on a tranng set. Besdes, we have establshed a connecton between support vectors and large argn nearest-neghbor classfers. However, further work on ths latter topc s needed n order to deterne how close both systes really are.

8 676 S. Bereo and J. Cabestany Fg.4. Rpley s synthetc tranng set wth the Bayes border (sold lne and the class borders coputed wth lear for 2 (dotted lne, 6 (dashed lne and 32 (dashdot lne prototypes. The test error for these classfers was 0.7%, 9.4% and 8.4 % respectvely. ote that the classfer wth 32 prototypes does not over-ft tranng data. References [] Barlett, P. L. (998. The Saple Coplexty of Pattern Classfcaton wth eural etworks: The Sze of the Weghts s More Iportant than the Sze of the etwork, IEEE Transacton on Inforaton Theory, 44, [2] Bereo, S., & Cabestany, J. (999. Adaptve soft k-nearest neghbour classfers. Pattern Recognton, Bref councaton, 32, [3] Bereo, S., & Cabestany, J. (2000a. Adaptve soft k-nearest neghbour classfers. Pattern Recognton, full-length paper, 33, [4] Bereo, S., & Cabestany, J. (2000b. Learnng wth nearest neghbour classfers. To Appear n eural Processng Letters, 3. [5] Brean, L. (998. Half-&-Half Baggng and Hard Boundary Ponts, Techncal Report o.534, Berkley: Unversty of Calforna, Departent of Statstcs. [6] Detterch, T. (997. Machne Learnng Research: Four Current Drectons. AI Magazne, 8, [7] Kohonen, T. (996. Self-organzng Maps, 2nd Edton, Berln: Sprnger-Verlag. [8] Lawrence, S., Gles, C. L. & Tso, A. C. (997. Lessons n neural network tranng: overfttng ay be harder than expected. Proceedngs of AAAI-97, , Menlo Park, CA: AAAI Press. [9] Rpley, D. (994. eural etworks and Methods for Classfcaton, Journal of the Royal Statstcal Socety, Seres B, 56, p [0] Schapre, R.E., Freund, Y., Bartlett, P. & Lee, W.S. (998. Boostng the argn: A new explanaton for the effectveness of votng ethods. The Annals of Statstcs, 26, [] Sola, A. et al. (999. Introducton to Large Margn Classfers, n Sola, A. et al. (Eds. Advances n Large Margn Classfers Boston, MA: MIT Press. [2] Vapnk, V. (998. Statstcal Learnng Theory, ew York: Wley-Interscence.