Support Vector Components Analysis

Transcription

1 and Machne Learnng. Bruges (Belgum), 6-8 Aprl 07, 6doc.com publ., ISBN Support Vector Components Analyss Mchel H. van der Ree, Jos B.T.M. Roerdnk, Chrstophe Phllps3, Gae tan Garraux3, Erc Salmon3 and Marco A. Werng4 - Semotc Labs B.V. Scence Park 40, Amsterdam - The Netherlands - Johann Bernoull Insttute for Mathematcs and Computer Scence Unversty of Gronngen, Njenborgh 9, Gronngen - The Netherlands 3- Cyclotron Research Centre Unversty of Le ge, Alle e du Sx Ao ut 8 B30, Le ge - Belgum 4- Insttute of Artfcal Intellgence and Cogntve Engneerng Unversty of Gronngen, Njenborgh 9, Gronngen - The Netherlands Abstract. In ths paper we propose a novel method for learnng a dstance metrc n the process of tranng Support Vector Machnes (SVMs) wth the radal bass functon kernel. A transformaton matrx s adapted n such a way that the SVM dual objectve of a classfcaton problem s optmzed. By usng a wde transformaton matrx the method can effectvely be used as a means of supervsed dmensonalty reducton. We compare our method wth other algorthms on a toy dataset and on PETscans of patents wth varous Parknsonsms, fndng that our method ether outperforms or performs on par wth the other algorthms. Introducton The Support Vector Machne [] s one of the most popular algorthms for solvng both regresson and classfcaton problems n machne learnng. The algorthm s robust and offers an excellent generalzaton performance, makng t very well suted for small datasets wth many features. One of the drawbacks of SVMs not usng a lnear kernel s that the algorthm s a black box : The model can t be nspected to see what features of the data are decsve for the eventual predcton. In addton, when usng the radal bass functon (RBF) kernel, SVMs are very senstve to a proper scalng of the nput data. The method proposed n ths paper ams to tackle both aforementoned problems. By learnng a quadratc dstance metrc durng SVM tranng the model becomes less senstve to the scalng of data. By forcng the dstance metrc to be of rank or 3, we can vsualze a lower dmensonal representaton of the nput data and make the relatons learned by the SVM more ntellgble. Outlne Secton wll ntroduce the support vector components analyss (SVCA) algorthm. Next, we llustrate how the proposed algorthm can be used as a means of supervsed dmensonalty reducton (SDR). Secton 4 wll cover the 9

2 and Machne Learnng. Bruges (Belgum), 6-8 Aprl 07, 6doc.com publ., ISBN setup and results of experments conducted wth the SVCA and other SDR algorthms. A concluson s presented n Secton 5. Support Vector Components Analyss We have a labeled dataset consstng of P real-valued nput vectors x,..., xp n RN and correspondng class labels c,..., cp. A matrx T RM N defnes a lnear map from RN to RM by mappng x to Tx. We try to optmze T such that when the transformaton s appled to the dataset, class dfferences are emphaszed n the transformed space. Our algorthm tres to maxmze the margns of one-versus-rest Support Vector Machnes n the transformed space. Pror to explanng our approach n more detal, we revew the objectve used n support vector classfcaton.. Support Vector Classfcaton In bnary soft margn lnear support vector classfcaton, tranng conssts of solvng the followng constraned optmzaton problem: w + C ξ w,ξ,b mn () subject to constrants y (w x +b) ξ and ξ 0. Here, w s a weght vector, b s a bas value, (x, y ) s a tranng sample and ts assocated label n {, }, ξ s a so-called slack varable that measures the degree of constrant volaton x and C s a constant determnng the trade-off between margn maxmzaton and error mnmzaton. Introducton of Lagrange multplers α and solvng for the coordnates of a saddle pont allow us to reformulate the prmal objectve and ts constrants as: max Q(α) = α α αj y yj (x xj ) () α,j P subject to constrants 0 α C and α y = 0. Once the α maxmzng () s found, the lnear support vector classfer determnes ts output usng:! f (x) = sgn α (x x) + b. (3) Snce both the dual objectve () and the model output (3) only depend on nner products between patterns, the model can be made non-lnear by usng a kernel functon K(x, x). such as polynomal functons and the radal bass functon.. The SVCA Objectve The basc dea of our algorthm s to learn a projecton matrx T such that the margn between classes n the transformed space s maxmzed. If we tran a oneversus-rest SVM for each class ` n the transformed space, the prmal objectve 30

3 and Machne Learnng. Bruges (Belgum), 6-8 Aprl 07, 6doc.com publ., ISBN of each lnear bnary classfcaton SVM becomes: " # mn J ` (w, ξ) = w + C ξ (4) now subject to constrants y` (w Tx + b) ξ and ξ 0, where we use ξ to denote the vector contanng all ξ s. Correspondngly, the new kernelzed dual objectve s defned as: mn max Q` (α; T) = α α αj y` yj` K(Tx, Txj ) (5) α T,j P subject to constrants 0 α C and α y` = 0. Note that the dual objectve needs to mnmzed w.r.t. T as s also the case n other (mult)-kernel learnng approaches []..3 Tranng Procedure We use the followng procedure to fnd the support vector components : Frst, we solve the quadratc programmng subproblem of fndng the α` that maxmzes the expresson n (5) for each of the one-versus-rest support vector machnes. Snce that expresson s equal to the objectve beng maxmzed n normal support vector machne tranng, we can use tred and tested optmzaton methods such as sequental mnmal optmzaton (SMO) [3] to do so. Then, for the optmzed α` s, we can mnmze the sums of all dual-objectves w.r.t. T usng P stochastc gradent descent. In stochastc gradent descent, we mnmze ` Q` by mnmzng the gradents of sngle examples. Wrtng the P expresson n (5) as Q` = q` wth sngle example terms αj` yj` K(Tx, Txj ) q` = α` α` y` j (6) we fnd the followng dervatve of q` w.r.t. T: q` K(Tx, Txj ) = α` y` αj` yj`. T T j (7) We alternate between optmzng all α` s and T a preset number of tmes. Alternatvely, we can use batch gradent descent orpbatch methods such as reslent ` backprop (RPROP, [4]) and adjust T usng ` Q / T nstead of ts sngle example based estmate. 3 SVCA as Supervsed Dmensonalty Reducton When usng a wde matrx T such that M < N, the SVCA algorthm can be used as a means of supervsed dmensonalty reducton. SDR can have multple 3

4 and Machne Learnng. Bruges (Belgum), 6-8 Aprl 07, 6doc.com publ., ISBN (a) (b) (c) Fgure : The artfcal dataset of concentrc rngs. Shown are scatter plots n whch both features are relevant to the labels (a), only one feature s relevant (b) and none of the features are relevant (c). advantages. When usng an RBF kernel all tranng patterns have to be stored n memory. Wth a wde T, the memory cost of savng these patterns s reduced by a factor M/N. The most nterestng applcaton s when we set M to or 3, so we can vsualze the low-dmensonal representaton of the dataset. Ths can be useful for explorng relatons between and separablty of the classes. Therefore, n ths paper we use M = or 3. Smlar SDR methods learnng a lnear transformaton matrx are neghbourhood components analyss (NCA) [5], local Fsher dscrmnant analyss (LFDA) [6] and Lmted Rank Matrx Learnng Vector Quantzaton (LRaM LVQ) [7]. 4 Experments and Results We report the performance of the SVCA algorthm compared to other SDR algorthms on two datasets: a toy problem of concentrc rngs and FDG-PET scans of patents wth varous Parknsonsms. For SVCA, we use the RPROP algorthm to optmze the transformaton matrx and we use an RBF kernel. 4. Experments on Artfcal Data Inspred by the concentrc rng data n [5], we create an artfcal dataset n the followng way: Frst, we create patterns x... xp n R8 by drawng from N (µ, Σ) where µ = 0 and Σ s the 8 8 dentty matrx. Then we assgn labels p based solely on the dstance from the orgn n the frst two dmensons,.e. x + x. Ths results n classes that take the shape of concentrc rngs n the class-relevant subspace. In total, we create 00 patterns belongng to four dfferent classes. In defnng class boundares, we ensure that each class has about the same number of patterns. Fgure shows the dataset thus generated. We compare the performance of SVCA wth the three other SDR technques mentoned n Secton 3: NCA, LRaM LVQ and LFDA. The algorthms are compared on 00 randomly generated datasets. We fnd that LRaM LVQ and LFDA never succeed n fndng the underlyng structure. For SVCA and NCA, we have 3

5 and Machne Learnng. Bruges (Belgum), 6-8 Aprl 07, 6doc.com publ., ISBN NCA ncorrect correct SVCA ncorrect correct e00 = 30 e0 = 0 e0 = 6 e = 44 Table : Contngency table of SVCA and NCA error rates n the concentrc rng experment. smply counted the number of tmes each algorthm fnds the rght projecton by learnng a transformaton matrx wth M =, resultng n the contngency table shown n table. Under the null hypothess that NCA and SVCA have the same error rate. We computed the number of correctly learned projectons, and used McNemar s test to obtan a p-value of 0.0. SVCA therefore sgnfcantly outperforms NCA and the other algorthms n ths experment. 4. Experments on FDG-PET Scans Here we apply the SDR algorthms to the same set of PET scans as used n [8]. These scans were obtaned n two dfferent locatons between 993 and 009 and are comprsed of 4 Parknson s dsease patents, 3 multple system atrophy patents, 6 progressve supranuclear palsy patents and cortcobasal syndrome patents. Each scan conssts of 53,594 voxels. We preprocess the data usng the Scaled Subprofle Modellng routne [9], leavng us wth projectons onto prncpal components. We retan the frst n prncpal components that explan at least 75% of the varance n the data. Ths procedure s appled n a k-fold fashon, so the number of selected components wll dffer per fold. We predefne 00 splts of the data. In each splt, 0% of the patterns has been randomly assgned to the test set, the rest of the patterns are used for tranng. We report mean test accuraces and ther standard devatons n Table. NCA and LFDA do not provde an explct predcton for new patterns. We have chosen to assgn labels accordng to the nearest neghbor classfcaton n the transformed space, where the number of neghbors was determned through cross-valdaton. Runnng pared t-tests on the dfferent fold error rates, we fnd no sgnfcant dfferences between the varous algorthms. However, we do fnd that for M = 3 the performance of the SDR algorthms rval that of an RBF SVM wth parameters (C, γ) optmzed through cross-valdaton. The results for the SDR algorthms are mpressve snce unlke these algorthms, the RBF SVM does not act on data transformed by a matrx wth lmted rank. M = M =3 SVCA 0.58 ± ± 0. LRMLVQ 0.56 ± ± 0.3 NCA 0.59 ± ± 0. LFDA 0.6 ± ± 0.3 RBF SVM: 0.68 ± 0.3 Table : Average test accuraces and standard devatons of the varous algorthms on 00 test/tran splts on the data from [8]. 33

6 and Machne Learnng. Bruges (Belgum), 6-8 Aprl 07, 6doc.com publ., ISBN Concluson We have presented the novel learnng method SVCA that can be used for both dstance metrc learnng and dmensonalty reducton. In our experment on toy data, we found that SVCA s most lkely to succeed n fndng the rngs hdden n the data, only havng NCA as a true compettor. In [0], we explore the relaton between NCA and SVCA n more detal and fnd that NCA can be seen as dong SVCA wth fxed α values. These results suggest that adaptng the alpha values as we do n SVCA helps n fndng the latent structure n a nosy dataset. The results of the experment on FDG-PET scans do not show any sgnfcant dfferences between the dfferent SDR methods, so SVCA can only be consdered to perform on-par wth the other SDR methods n ths experment. In turn, all SDR algorthms rval the performance of an optmzed RBF SVM whle stll allowng the relatons they dscover n the tranng set to be nspected. In future work, we wll examne the use of non-lnear transformaton functons. Furthermore, t would be very nterestng to ntegrate the SVCA algorthm n the mult-layer SVM archtecture []. Fnally, we would lke to compare our method to other kernel or dstance-functon learnng algorthms. References [] V.N. Vapnk. The Nature of Statstcal Learnng Theory. Sprnger-Verlag, 995. [] A-D. Petersma, L.R.B. Schomaker, and M.A. Werng. Kernel learnng n support vector machnes usng dual-objectve optmzaton. In Proceedngs of the 3rd Belgan-Dutch Conference on Artfcal Intellgence, pages 67 74, 0. [3] J. Platt. Sequental mnmal optmsaton: a fast algorthm for tranng support vector machnes. Techncal Report MSR-TR-98-4, Mcrosoft Research, 998. [4] M. Redmller and H. Braun. A drect adaptve method for faster backpropagaton learnng: The Rprop algorthm. In Proceedngs of the IEEE Internatonal Conference on Neural Networks, pages , 993. [5] J. Goldberger, S. Rowes, G. Hnton, and R. Salakhutdnov. Neghbourhood components analyss. In Advances n Neural Informaton Processng Systems 7, pages MIT Press, 004. [6] M. Sugyama. Dmensonalty reducton of multmodal labeled data by local Fsher dscrmnant analyss. Journal of Machne Learnng Research, 8:07 06, 007. [7] K. Bunte, P. Schneder, B. Hammer, F. Schlef, T. Vllmann, and M. Behl. Lmted rank matrx learnng, dscrmnatve dmenson reducton and vsualzaton. Neural Networks, 6:59 73, 0. [8] G. Garraux, C. Phllps, J. Schrouff, A. Kresler, C. Lemare, Degueldre C., C. Delcour, R. Hustnx, A. Luxen, A. Dese e, and E. Salmon. Multclass classfcaton of FDG PET scans for the dstncton between Parknson s dsease and atypcal parknsonan syndromes. NeuroImage: Clncal, pages , 03. [9] G.E. Alexander and J.R. Moeller. Applcaton of the scaled subprofle model to functonal magng n neuropsychatrc dsorders: a prncpal component approach to modelng bran functon n dsease. Human Bran Mappng, :79 94, 994. [0] M.H. van der Ree. Exploratons n ntellgble classfcaton. Master s thess, Unversty of Gronngen, the Netherlands, 04. [] M.A. Werng and L.R.B. Schomaker. Mult-layer support vector machnes. In J.A.K Suykens, M. Sgnoretto, and A. Argyrou, edtors, Regularzaton, Optmzaton, Kernels, and Support Vector Machnes, chapter 0. Chapman and Hall,