Learning General Gaussian Kernels by Optimizing Kernel Polarization

Transcription

1 Chnese Journal of Electroncs Vol.18, No.2, Apr Learnng General Gaussan Kernels by Optmzng Kernel Polarzaton WANG Tnghua 1, HUANG Houkuan 1, TIAN Shengfeng 1 and DENG Dayong 2 (1.School of Computer and Informaton Technology, Bejng Jaotong Unversty, Bejng , Chna (2.School of Mathematcs and Informaton Engneerng, Zhejang Normal Unversty, Jnhua , Chna Abstract The problem of model selecton for Support vector machnes (SVM wth general Gaussan kernels s consdered. Unlke the conventonal standard sngle scale Gaussan kernels, where all the bass functons have a common kernel wdth, the general Gaussan kernels adopt some lnear transformatons of nput space such that not only the scalng but also the rotaton s adapted. We proposed a gradent-based method for learnng the optmal general Gaussan kernels by optmzng kernel polarzaton. Ths method can fnd a more powerful kernel for a gven classfcaton problem wthout desgnng any classfer. Experments on both synthetc and real data sets demonstrate that tunng of the scalng and rotaton of Gaussan kernels usng our method can yeld better generalzaton performance of support vector machnes. Key words General Gaussan kernels, Kernel polarzaton, Support vector machnes (SVM, Model selecton. I. Introducton Kernel methods such as Support vector machnes (SVM [1,2] have been successfully appled for a wde range of dfferent data analyss problems. The key to success s that kernel methods can be modularzed nto two modules: the frst s to map data nto a feature space, whch s a powerful framework for many dfferent data types; and the second s to use a lnear algorthm n the feature space, whch s effcent and has theoretcal guarantees [2]. The methods take advantage of a problem specfc kernel functon that computes the nner product of nput data ponts n a feature space. Choosng the approprate kernel functon, thereby, the approprate feature space s the crucal step n handlng a learnng task wth kernel machnes. Ths problem s well known as the model selecton ssue for all kernel methods. Often a parameterzed famly of kernel functons s consdered and the problem reduces to fndng an approprate parameter vector for the gven task. Perhaps the most elaborate systematc technque of choosng multple parameters for a gven kernel s gradent-based methods [3 6]. These algorthms terate the followng procedure: the SVM s traned usng current parameter vector to provde Lagrange multplers, the gradent of some generalzaton error bound such as radus-margn bound wth respect to the parameters s calculated and an update rule s learned. However, ths approach has a sgnfcant drawback. In each teraton, t requre the tranng of the learnng machne and the soluton of an addtonal quadratc program to compute the radus of the smallest ball enclosng the tranng data n feature space, whch may result n hgh computatonal expense consumpton for some problems. In ths paper, we propose a smlar method for parameter selecton that does not suffer the drawback descrbed above. In Ref.[7], Baram proposed a new kernel optmalty crteron,.e. kernel polarzaton, for kernel adaptaton. Drawn from physcs, kernel polarzaton means to drve ponts wth dfferent class labels to opposte locatons n the feature space defned by kernel geometry. Independently derved, kernel polarzaton smplfes the kernel-target algnment [8 10] crteron by rddng the latter of ts denomnator, makng the optmzaton problem consderably easer. The sgnfcant property of optmzng kernel polarzaton for model selecton s that t s ndependent of the actual learnng machne used. It makes use of the nformaton from the complete tranng data and can be computed effcently. We adopt the kernel polarzaton crteron to learn the optmal general Gaussan kernels, where not only the scalng but also the rotaton s adapted. The rest of ths artcle s organzed as follows. In Secton II, we gve a bref ntroducton to SVM. In Secton III we frst revew the kernel polarzaton crteron and then gve a further nvestgaton of ts geometrc sgnfcance. The proposed technque for the adaptaton of general Gaussan kernels s presented n Secton IV. Expermental results are presented n Secton V and fnally we draw conclusons wth some future works n Secton VI. II. Support Vector Machnes We consder L 1-norm soft margn support vector machnes for bnary classfcaton. The fundamental concepts of SVM were developed by Vapnk [1]. Let (x, y, 1 l, be the Manuscrpt Receved Jan. 2008; Accepted Oct Ths work s supported by the Natonal Grand Fundamental Research 973 Program of Chna (No.2007CB307100, No.2007CB and the Specalzed Research Foundaton of Doctoral Program of Hgher Educaton of Chna (No

2 266 Chnese Journal of Electroncs 2009 tranng samples, where y {+1, 1} s the label assocated wth nput pattern x X R n. The man dea of SVM s to map the nput patterns to a feature space F and then to separate the transformed data lnearly n F. The transformaton φ : X F s mplctly done by a kernel functon k : X X R, whch computes an nner product n the feature space effcently, that s k(x, x j = φ(x, φ(x j. Patterns are classfed by a decson functon f of the form ( f(x = sgn a y k(x, x + b (1 The coeffcent a are computed by maxmzng the dual optmzaton problem max s.t. a 1 2 j=1 y y ja a jk(x, x j a y = 0 and 0 a C, = 1, 2,, l (2 where C s a regularzaton parameter whch controls the trade-off between maxmzng the target margn and mnmzng the L 1-norm of the margn slack vector of the tranng data. The bas b n Eq.(1 then can be determned based on the soluton a. The patterns x wth a > 0 are called support vectors. The kernel appearng n the problem has to be a postve defnte functon,.e., satsfyng the Mercer s condton. Mercer s condton tells us whether or not a kernel s actually an nner product n a gven space and ensures that the soluton of Eq.(2 produces a global optmum. The well-known kernels nclude lnear kernel, polynomal kernel, Gaussan kernel, sgmod kernel and so on. In the followng we consder the general Gaussan kernels k A(x, x j = exp( (x x j T A(x x j (3 where A s a Postve semdefnte (PSD n n matrx. It s the extenson of the standard sngle scale Gaussan kernels ( x xj 2 k(x, x j = exp (4 2σ 2 where σ s the kernel wdth parameter. III. Kernel Polarzaton Gven a tranng data set (x, y, 1 l, where x X and y {+1, 1}, the kernel functon k : X X R defnes a symmetrc postve defnte kernel matrx (Gram matrx K R l l by K j = k(x, x j. Let y = (y 1,, y l T, the deal kernel matrx would be yy T whch perfectly suts the tranng data. Kernel polarzaton of K s defned as the Frobenus nner product: ϕ = K, yy T = It can be rewrtten as ϕ = j=1 y =y j k(x, x j y y jk(x, x j (5 y y j k(x, x j (6 The kernel polarzaton measures the smlarty of the kernel wth yy T on the tranng data. Obvously, ϕ wll ncrease f the smlarty represented by the kernel s large for nput patterns of the same class and small for patterns from dfferent classes. Ths s the ntutve dea behnd preferrng a kernel wth hgh kernel polarzaton. The ease wth whch kernel polarzaton can be calculated n O(l 2 tme complexty usng only tranng data, pror to any computatonally ntensve tranng SVM, makes t an nterestng qualty crteron for kernel selecton. Employed by SVM, kernel polarzaton for the Gaussan kernel wth sngle scale parameter was found to yeld about the same classfcaton results as exhaustve parameter search. Furthermore, t has been shown that complete kernel polarzaton yelds consstent classfcaton by kernel-sum classfers, whch wll motvate kernel polarzaton from a theoretcal vewpont [7]. Now let us nvestgate further the geometrc sgnfcance of kernel polarzaton wth general Gaussan kernels. we reformulate kernel polarzaton n a parwse manner n feature space. Suppose y 1 = = y l+ = 1 and y l+ +1 = = y l+ +l = 1, where l +(l s the number of samples whose class label s +1( 1, l + + l = l. Note that k A(x, x = 1 for all x X and φ(x φ(x j 2 =k A(x, x 2k A(x, x j + k A(x j, x j =2 2k A(x, x j (7 substtutng Eq.(7 nto Eq.(5 we then get ϕ = l l 2 2l +l where B j = ( B j φ(x φ(x j 2 (8 { 1, y = y j +1, y y j (9 The above representaton gves us a clear nterpretaton of hgh kernel polarzaton: t tres to keep wthn-class data pars close (snce B j s negatve f y = y j and between-class data pars apart (snce B j s postve f y y j. Generally speakng, t s a suffcent condton that havng a hgh ϕ, one can expect good classfcaton performance, as the elements n the same class come close whereas those n dfferent classes go far n the feature space. IV. Implementaton for Kernel Adaptaton By usng the map P P T P = A, the general Gaussan kernels can be equvalently reformulated as k P (x, x j = exp( (x x j T P P T (x x j (10 where P s an arbtrary n n real-valued matrx. Unlke the work done by Glasmachers and Igel [11], where P was restrcted to a postve defnte symmetrc n n matrx. Ths extenson of P can reduce our optmzaton problem to an unconstraned one. Three dfferent kernel parameterzatons are consdered: (1 P = λe, where E s the unt matrx and λ the only parameter to be adapted. Ths case s just the standard Gaussan kernel wth sngle wdth parameter.

3 Learnng General Gaussan Kernels by Optmzng Kernel Polarzaton 267 (2 P s a dagonal matrx. Ths case means ndependent scalngs of the axes are adapted and corresponds to learn dfferent weghts of dfferent axes (features. (3 P s an arbtrary matrx. Ths case means the scalng and rotaton of the nput space s adapted and corresponds to learn possble correlaton among features. Adaptng P changes three propertes of the kernel: the sze, whch s controlled by the determnant of P, s defned as the smallest volume where a certan amount of the kernel s concentrated; the shape s determned by the egenvalues of P ; and the orentaton, when non-dagonal matrces P are allowed. We lst the three cases of P and name them Sngle, Dagonal and Full respectvely n Table 1. Table 1. Three cases of kernel parameterzatons Name Constrant #Parameters Impact on kernel Sngle P = λe 1 sze Dagonal P dagonal n sze & shape Full none n 2 sze, shape & orentaton We propose a gradent ascent algorthm to learn the matrx P by maxmzng the kernel polarzaton. Gven a tranng set (x, y, 1 l, where x X and y {+1, 1}, the kernel polarzaton wth general Gaussan kernels can be wrtten as ϕ(p = k P (x, x jy y j (11 j=1 hence the strategy to learn matrx P can be summarzed to choose P satsfyng P = arg max ϕ(p (12 P The dervatve of the ϕ(p wth respect to P s ϕ(p P = j=1 y y j k P (x, x j P (13 the partal of k P (x, x j s the dervatve of a scalar functon wth respect to the matrx P. As such, the dervatve s a matrx where each element s the dervatve wth respect to each element of P. Frstly, we gve a lemma as follows: Lemma 1 Let U j = x T P P T x j, the frst dervatve of U j wth respect to P s we get Proof Let U j P = (x, xt J + x jx T P (14 x = (x (1, x (2,, x (n T, x j = (x (1 j, x (2 j,, x (n j T P = (p rs n n U j = x T P P T x j = r=1 s=1 t=1 x (r x (t j prspts the frst dervatve of U j wth respect to p rs can be expressed as U j = x (r (x (t j pts + x(r j (x (t p ts. p rs t=1 t=1 Ths expresson s obvously equvalent to the Eq.(14. Secondly, we rewrte Eq.(10 as k P (x, x j = exp( (x T P P T x 2x T P P T x j + x T j P P T x j (15 Accordng to Eqs.(13, (14 and (15, we have ϕ(p P = ( U Uj y y j (k P (x, x j 2 P P + Ujj P (16 Therefore, the gradent of ϕ(p wth respect to P has been successfully fgured out. In the specal case of dagonal P,.e. P = dag(p (1, p (2,, p (n let x = (x (1, x (2,, x (n T, we have ( k P (x, x j = exp ((x (k k=1 + (x (k j p (k 2 Thus the gradent wth respect to p (k s k p(x, x j p (k = 2p (k k p(x, x j((x (k p (k 2 2x (k x (k j (p (k 2 (17 2 2x (k x (k j + (x (k j 2 (18 Fnally, the procedure for learnng the matrx P s summarzed as follows: Procedure 1 (1 Intalze P to a random matrx; (2 Update P such that ϕ(p s maxmzed. Ths can be acheved by a gradent update rule: P P + η P ϕ(p (3 Go to step (2 or stop f a gven stoppng crteron s met; (4 End. where η > 0 s the gradent step and P ϕ(p s the gradent of ϕ(p wth respect to P. In order to smplfy the optmzaton problem, no constrant s placed on P, although Eq.(15 mples that the matrx P P T must be postve defnte. To deal wth ths problem, we ntalze P P T to a postve defnte matrx, for example, we ntalze P to the unt matrx I. We now consder the computatonal complexty of the proposed algorthm. For the case that P s an arbtrary matrx (Full, t can be seen that the tme complexty of Eq.(14 s O(n 3 and Eq.(15 s O(n 2, hence the tme complexty of Eq.(16 s O(l 2 n 3. Therefore, the tme complexty of Procedure 1 s O(kl 2 n 3, where k denotes the teraton tmes of the teratve procedure. Smlarty, the tme complexty of adaptng the dagonal matrx P s O(kl 2 n 2. It can be observed that the overall complexty of our algorthm s somewhat hgh, especally for the arbtrary matrx case. However, t can be easly seen that the computatonal burden manly les n the matrx multplcaton (see Eq.(14. Matrx multplcaton, due to ts fundamental mportance n scence and engneerng, much effort has been devoted to fndng and mplementng fast matrx

4 268 Chnese Journal of Electroncs 2009 multplcaton algorthms [12,13]. As we mentoned above, the standard sequental algorthm has O(n 3 tme complexty n multplyng two n n matrces. Snce Strassen s O(n algorthm was dscovered n 1969, successve progress has been made to develop fast sequental matrx multplcaton algorthms wth tme complexty O(n β, where 2 < β < 3. The current best value of β s less than By some parallel and dstrbuted computng technology, the complexty of such algorthms can be at most reduced to O(log n tme. Therefore, our algorthm can be computed n O(kl 2 log n tme by usng the state-of-the-art matrx multplcaton algorthms. More and detaled dscussons for fast matrx multplcaton are beyond the range of ths artcle. V. Experments In ths secton, we performed experments on two synthetc data sets and three real-world data sets, all of whch are bnary classfcaton task. The synthetc data sets were generated by PRTools 4 [14] and the real-world data sets were selected from the UCI collectons [15]. The two synthetc data sets were nomnated as Dffcult and Hghleyman respectvely, whose data dstrbutons are shown n Fg.1. For the Breast cancer data set, we drectly elmnated the examples that contan mssng attrbute values. The fourth data set LetterIJ was derved from the Letter mage recognton data, we set up a bnary classfcaton task between the letters I and J whch s regarded to be a somewhat challengng problem. For the last data set Irs plants, we set up a bnary classfcaton data set Irs23 between the class Irs verscolour and Irs vrgnca, whch s not lnear dscrmnable. For each data set we parttoned them nto a tranng set and a test set: 30% of the data set serves as tranng set and 70% as test set. The object generaton follows the class pror probabltes. For each of the cases shown n Table 1, we used the tranng set to perform optmal kernel selecton, and then tested the generalzaton performance of SVM usng the test set. We stopped our algorthm when the mprovement of the kernel polarzaton was less than 10 8 and averaged our results over 10 runs. The mplementaton of the SVM classfer was acheved by the software LIBSVM [16]. Fg. 1. Data dstrbutons of two synthetc data sets Table 2 presents the results for three dfferent kernel parameterzatons: Sngle, Dagonal and Full. We lst the classfcaton error and the number of support vectors when gven a slack penalty of C = 1. The average over all data sets s shown n the last row of the table, too. For all fve data sets, we can easly fnd that better results are acheved when the kernel adaptaton s not restrcted to the Sngle case. Furthermore, the Full case leads to better results compared to the Dagonal case. From an average perspectve, the Sngle, Dagonal and Full cases provde respectvely a testng error of 7.60%, 5.85% and 3.77%. There s another remarkable advantage of the Dagonal and Full cases: the number of support vectors decrease. Agan from an average perspectve, the Sngle, Dagonal and Full cases provde respectvely the number of support vectors of 59, 48 and 37. It s well known that the decrease of the number of support vectors can mprove the classfcaton speed n practce. Table 2. Performance results on fve data sets Sngle Dagonal Full Data set Error nsvs Error nsvs Error nsvs Dffcult 10.71% % % 14 Hghleyman 10.71% % % 17 Breast cancer 5.03% % % 30 LetterIJ 5.86% % % 113 Irs % % % 10 Avg. 7.60% % % 37 Fg. 2. Classfcaton error rate on Irs23 data set Fg. 3. Number of support vectors on Irs23 data set Note that parameter C was not optmzed n any way, we vared C over a range C = 1, 10, 30, 50, 70, 100 to test performance on data set Irs23 and showed the results n Fg.2 and Fg.3. For the other data sets, the smlar results were obtaned. It can be found that for all the values of C consdered the Dagonal case as well as the Full case leads to better results compared to the Sngle case: both classfcaton error and number of support vectors decrease. The property that the Full case gves better results than Dagonal case s also preserved. In Ref.[7], kernel polarzaton for the Gaussan kernel wth sngle scale parameter (sphercal kernel has been shown to yeld about the same classfcaton results as exhaustve parameter search. However, t can be seen that only by droppng the restrcton of sngle scale parameter we can acheve a lnear transformaton of the nput space such that n the transformed space Gaussan kernels can perform better. Ths phenomenon s due to the nherent advantage of the optmzaton crteron whch s dscussed n Secton III and the fner data dstrbuton model, n other words a more approprate (separate embeddng, whch was obtaned va adaptaton of the matrx P. In a word, we have emprcally llustrated the effectveness of our method and confrmed further that kernel polarzaton s an effectve crteron for kernel selecton.

5 Learnng General Gaussan Kernels by Optmzng Kernel Polarzaton 269 VI. Concluson The choce of the kernel functon whch determnes mplctly the mappng between the nput space and the feature space s of crucal mportance to kernel methods. In ths study we proposed an approach for learnng general Gaussan kernels, where not only the scalng but also the rotaton s adapted. Ths s based on the possblty of computng the gradent of kernel polarzaton wth respect to a lnear transformaton matrx. The ease wth whch kernel polarzaton can be calculated usng only tranng data, pror to desgnng any classfer, makes t an nterestng crteron for kernel selecton. Furthermore, we presented a further nvestgaton of the geometrc sgnfcance on maxmzng kernel polarzaton, whch ntutvely shows why we choose kernel polarzaton as the crteron for kernel selecton. As for SVM, expermental results on both synthetc and real data sets show our method leads to both an mprovement of the classfcaton accuracy and a reducton of the number of support vectors. Future nvestgaton of ths work wll focus on regresson and clusterng applcatons. Snce we employ the Gaussan kernels throughout, the extensons to other kernel functons may be another nterest n our future work. References [1] V. Vapnk, The Nature of Statstcal Learnng Theory, SprngerVerlag, New York, USA, [2] J. Shawe-Taylor, N. Crstann, Kernel Methods for Pattern Analyss, Cambrdge Unversty Press, Cambrdge, UK, [3] O. Chapelle, V. Vapnk, O. Bousquet, S. Mukherjee, Choosng multple parameters for support vector machnes, Machne Learnng, Vol.46, No.1, pp , [4] S.S. Keerth, Effcent tunng of SVM hyperparameters usng radus/margn bound and teratve algorthms, IEEE Trans. on Neural Networks, Vol.13, No.5, pp , [5] K.M. Chung, W.C. Kao, C.L. Sun, L.L. Wang, C.J. Ln, Radus margn bounds for support vector machnes wth RBF kernel, Neural Computaton, Vol.15, No.11, pp , [6] Chang Qun, Wang Xaolong, Ln Ymeng, Wang Xzhao, D.S. Yeung, Support vector classfcaton and Gaussan kernel wth multple wdths, Acta Electronca Snca, Vol.35, No.3, pp , (n Chnese [7] Y. Baram, Learnng by kernel polarzaton, Neural Computaton, Vol.17, No.6, pp , [8] N. Crstann, J. Shawe-Taylor, A. Elsseeff, J. Kandola, On kernel-target algnment, Proc. of Neural Informaton Processng Systems (NIPS: Natural and Synthetc, Vancouver, Brtsh Columba, Canada, pp , [9] J.B. Pothn, C. Rchard, A greedy algorthm for optmzng the kernel algnment and the performance of kernel machnes, Proc. of 14th European Sgnal Processng Conference (EUSIPCO, Florence, Italy, pp.4 8, [10] C. Lgel, T. Glasmachers, B. Mersch, N. Pfefer, P. Mencke, Gradent-based optmzaton of kernel-target algnment for sequence kernels appled to bacteral gene start detecton, IEEE/ACM Transactons on Computatonal Bology and Bonformatcs, Vol.4, No.2, pp , [11] T. Glasmachers, C. Igel, Gradent-based adaptaton of general Gaussan kernels, Neural Computaton, Vol.17, No.10, pp , [12] K. L, Scalable parallel matrx multplcaton on dstrbuted memory parallel computers, Journal of Parallel and Dstrbuted Computng, Vol.61, No.12, pp , [13] K. Goto, R.V.D. Gejn, Anatomy of hgh-performance matrx multplcaton, ACM Trans. on Mathematcal Software, Vol.34, No.3, pp.12 25, [14] R.P.W. Dun, P. Juszczak, P. Paclk, E. Pekalska, D.D. Rdderet, D.M.J. Tax, PRTools: A Matlab Toolbox for Pattern Recognton, Delft Unversty of Technology, Software avalable at [15] A. Asuncon, D.J. Newman, UCI Machne Learnng Repostory [ Irvne, CA: Unversty of Calforna, School of Informaton and Computer Scence, [16] C.C. Chang, C.J. Ln, LIBSVM: A Lbrary for Support Vector Machnes, Software avalable at tw/ cjln/lbsvm, WANG Tnghua was born n He dd hs undergraduate study at Nanchang Unversty majorng n computer scence durng , and receved the M.S. degree n computer scence n 2006 from Nanchang Unversty. He s studyng for Ph.D. degree at School of Computer and Informaton Technology, Bejng Jaotong Unversty, Chna. Hs current research s focus on machne learnng and data mnng, especally on support vector machnes and kernel-based learnng methods. (Emal: wthbjtu@163.com HUANG Houkuan was born n He dd hs undergraduate study at Pekng Unversty majorng n mathematcs durng , and graduate study n Harbn Engneerng Unversty majorng n appled mathematcs durng He s currently a professor and Ph.D. supervsor at School of Computer and Informaton Technology, Bejng Jaotong Unversty, Chna. Hs current nterests nclude artfcal ntellgence, KDD and mult-agent systems. TIAN Shengfeng was born n He s a professor and Ph.D. supervsor at School of Computer and Informaton Technology, Bejng Jaotong Unversty, Chna. Hs current research s focus on artfcal ntellgence, pattern recognton and network securty, especally on support vector machnes and network ntruson detecton. DENG Dayong was born n He receved the Ph.D. degree n computer scence n 2007 from Bejng Jaotong Unversty. He s currently an assstant professor at School of Mathematcs and Informaton Engneerng, Zhejang Normal Unversty, Chna. Hs current research nterests nclude rough set and data mnng.