On Multiple Kernel Learning with Multiple Labels

Transcription

1 On Multple Kernel Learnng wth Multple Labels Le Tang Department of CSE Arzona State Unversty Janhu Chen Department of CSE Arzona State Unversty Jepng Ye Department of CSE Arzona State Unversty Abstract For classfcaton wth multple labels, a common approach s to learn a classfer for each label. Wth a kernel-based classfer, there are two optons to set up kernels: select a specfc kernel for each label or the same kernel for all labels. In ths work, we present a unfed framework for mult-label multple kernel learnng, n whch the above two approaches can be consdered as two extreme cases. Moreover, our framework allows the kernels shared partally among multple labels, enablng flexble degrees of label commonalty. We systematcally study how the sharng of kernels among multple labels affects the performance based on extensve experments on varous benchmark data ncludng mages and mcroarray data. Interestng fndngs concernng effcacy and effcency are reported. 1 Introducton Wth the prolferaton of kernel-based methods lke support vector machnes SVM, kernel learnng has been attractng ncreasng attentons. As wdely known, the kernel functon or matrx plays an essental role n kernel methods. For practcal learnng problems, dfferent kernels are usually prespecfed to characterze the data. For nstance, Gaussan kernel wth dfferent wdth parameters; data fuson wth heterogeneous representatons [Lanckret et al., 2004b]. Tradtonally, an approprate kernel can be estmated through cross-valdaton. Recent multple kernel learnng MKL methods [Lanckret et al., 2004a] manpulate the Gram kernel matrx drectly by formulatng t as a sem-defnte program SDP, or alternatvely, search for an optmal convex combnaton of multple user-specfed kernels va quadratcally constraned quadratc program QCQP. Both SDP and QCQP formulatons can only handle data of medum sze and small number of kernels. To address large scale kernel learnng, varous methods are developed, ncludng SMOlke algorthm [Bach et al., 2004], sem-nfnte lnear program SILP [Sonnenburg et al., 2007] and projected gradent method [Rakotomamonjy et al., 2007]. It s notced that most exstng works on MKL focus on bnary classfcatons. In ths work, MKL learnng the weghts for each base kernel for classfcaton wth multple labels s explored nstead. Classfcaton wth multple labels refers to classfcaton wth more than 2 categores n the output space. Commonly, the problem s decomposed nto multple bnary classfcaton tasks, and the tasks are learned ndependently or jontly. Some works attempt to address the kernel learnng problem wth multple labels. In [Jebara, 2004], all bnary classfcaton tasks share the same Bernoull pror for each kernel, leadng to a sparse kernel combnaton. [Zen, 2007] dscusses the problem of kernel learnng for mult-class SVM, and [J et al., 2008] studes the case for mult-label classfcaton. Both works above explot the same kernel drectly for all classes, yet no emprcal result s formally reported concernng whether the same kernel across labels performs better over a specfc kernel for each label. The same-kernel-across-tasks setup seems reasonable at frst glmpse but needs more nvestgaton. Mostly, the multple labels are wthn the same doman, and naturally the classfcaton tasks share some commonalty. One the other hand, the kernel s more nformatve for classfcaton when t s algned wth the target label. Some tasks say recognze sunset and anmal n mages are qute dstnct, so a specfc kernel for each label should be encouraged. Gven these consderatons, two questons rses naturally: Whch approach could be better, the same kernel for all labels or a specfc kernel for each label? To our best knowledge, no work has formally studed ths ssue yet. A natural extenson s to develop kernels that capture the smlarty and dfference among labels smultaneously. Ths matches the relatonshp among labels more reasonably, but could t be effectve n practce? The questons above motvate us to develop a novel framework to model task smlarty and dfference smultaneously when handlng multple related classfcaton tasks. We show that the framework can be solved va QCQP wth proper regularzaton on kernel dfference. To be scalable, an SILPlke algorthm s provded. In ths framework, selectng the same kernel for all labels or a specfc kernel for each label are deemed as two extreme cases. Moreover, ths framework allows varous degree of kernel sharng wth proper parameter setup, enablng us to study dfferent strateges of kernel sharng systematcally. Based on extensve experments on benchmark data, we report some nterestng fndngs and explanatons concernng the aforementoned two questons. 1255

2 2 A Unfed Framework To systematcally study the effect of kernel sharng among multple labels, we present a unfed framework to allow flexble degree of kernel sharng. We focus on the well-known kernel-based algorthm SVM, for learnng k bnary classfcaton tasks {f t } k respectvely, based on n tranng samples {x,y t}n,wherets the ndex of a specfc label. Let HK t be the feature space, and φt K be the mappng functon defned as φ t K : φt K x Ht K, for a kernel functon Kt. Let G t be the kernel Gram matrx for the t-th task, namely Gj t = Kt x,x j = φ t K x φ t K x j. Under the settng of learnng multple labels {f t } k usng SVM, each label f t can be seen as learnng a lnear functon n the feature space HK t, such that f t x =sgn w t,φ t K x + bt where w t s the feature weght and b t s the bas term. Typcally, the dual formulaton of SVM s consdered. Let Dα t, G t denote the dual objectve of the t-th task gven kernel matrx G t : Dα t, G t =[α t ] T e 1 2 [αt ] T G t y t [y t ] T α t 1 whereforthetaskf t, G t S+ n denotes the kernel matrx and S+ n s the set of sem-postve defnte matrces; denotes element-wse matrx multplcaton; α t R n denotes the dual varable vector. Mathematcally, mult-label learnng wth k labels can be formulated n the dual form as: max Dα t, G t 2 {α t } k s.t. [α t ] T y t =0, 0 α t C, t =1,,k Here, C s the penalty parameter for allowng the msclassfcaton. Gven {G t } k, optmal {αt } k n Eq. 2 can be found by solvng a convex problem. Note that the dual objectve s the same as the prmal objectve of SVM due to ts convexty equal to the emprcal classfcaton loss plus model complexty. Followng [Lanckret et al., 2004a], multple kernel learnng wth k labels and p base kernels G 1,G 2,,G p can be formulated as: mn {G t } k λ Ω{G t } k + max {α t } k Dα t, G t 3 s.t. [α t ] T y t =0, 0 α t C, t =1,,k G t = θg t, t =1,,k 4 θ t =1, θt 0, t =1,,k 5 where Ω{G t } k s a regularzaton term to represent the cost assocated wth kernel dfferences among labels. To capture the commonalty among labels, Ω should be a monotonc ncreasng functon of kernel dfference. λ s the trade-off parameter between kernel dfference and classfcaton loss. Clearly, f λ s set to 0, the objectve goal s decoupled nto k sub-problems, wth each selectng a kernel ndependently IndependentModel;When λ s suffcently large, the regularzaton term domnates and forces all labels to select the same kernel Same Model. In between, there are nfnte number of Partal Model whch control the degrees of kernel dfference among tasks. The larger λ s, the more smlar the kernels of each label are. 3 Regularzaton on Kernel Dfference Here, we develop one regularzaton scheme such that formula 3 can be solved va convex programmng. Snce the optmal kernel for each label s expressed as a convex combnaton of multple base kernels as n eq. 4 and 5, each θ t essentally represents the kernel assocated wth the t-th label. We decouple the kernel weghts of each label nto two non-negatve parts: θ t = ζ + γ t, ζ,γ t 0 6 where ζ denotes the shared kernel across labels, and γ t s the label-specfc part. So the kernel dfference can be defned as: Ω {G t } k 1 = γ t 7 2 For presentaton convenence, we denote G t α =[α t ] T G t y t [y] T α t 8 It follows that the MKL problem can be solved va QCQP 1. Theorem 3.1. Gven regularzaton as presented n 6 and 7, the problem n 3 s equvalent to a Quadratcally Constraned Quadratc Program QCQP: max [α t ] T e 1 2 s s.t. s s 0, s s 0 s t kλ G t α, =1,,p s t G t α, =1,,p,,,k [α t ] T y t =0, 0 α t C, t =1,,k The kernel weghts of each label ζ,γ t can be obtaned va the dual varables of the constrants. The QCQP formulaton nvolves nk +2varables, k +1p quadratc constrants and Onk lnear constrants. Though ths QCQP can be solved effcently by general optmzaton software, the quadratc constrants mght exhaust memory resources f k or p s large. Next, we ll show a more scalable algorthm that solves the problem effcently. 4 Algorthm The objectve n 3 gven λ s equvalent to the followng problem wth a proper β and other constrants specfed n 3: mn {[α max t ] T e 12 {G t } {α t } [αt ] T G t y t [y t ] T } α t 9 s.t. Ω{G t } k β 10 1 Please refer to the appendx for the proof. 1256

3 Compared wth λ, β has an explct meanng: the maxmum dfference among kernels of each label. Snce p θt = 1, the mn-max problem n 9, akn to [Sonnenburg et al., 2007], can be expressed as: mn g t 11 s.t. θdα t t,g g t, α t St 12 wth St = { α t 0 α t C, [α t ] T y t =0 } 13 Note that the max operaton wth respect to α t s transformed nto a constrant for all the possble α t n the set St defned n 13. An algorthm smlar to cuttng-plane could be utlzed to solve the problem, whch essentally adds constrants n terms of α t teratvely. In the J-th teraton, we perform the followng: 1. Gven θ t and gt from prevous teraton, fnd out new α t J n the set 13 whch volates the constrants 12 most for each label. Essentally, we need to fnd out α t such that p θt Dαt,G s maxmzed, whch bols down to an SVM problem wth fxed kernel for each label: max [α t ] T e 1 α t 2 [αt ] T θg t t y t [y t ] T α t Here, each label s SVM problem can be solved ndependently and typcal SVM acceleraton technques and exstng SVM mplementatons can be used drectly. 2. Gven α t J obtaned n Step 1, add k lnear constrants: θ t Dαt J,G g t, t =1,,k and fnd out new θ t and g t va the problem below: mn g t s.t. θ t Dαt j,g g t, j =1,,J θ t 0, θ t =1, t =1,,k 1 γ t β, θ t = ζ + γ t,ζ,γ t 0 2 Note that both the constrants and the objectve are lnear, so the problem can be solved effcently by general optmzaton package. 3. J = J +1. Repeat the above procedure untl no α s found to volate the constrants n Step 1. So n each teraton, we nterchangeably solve k SVM problems and a lnear program of sze Okp. 5 Expermental Study 5.1 Experment Setup 4 mult-label data sets are selected as n Table 1. We also nclude 5 mult-class data sets as they are specal cases of Table 1: Data Descrpton Data #samples #labels #kernels Lgand Mult-label Bo Scene Yeast USPS Letter Mult-class Yaleface news Segment mult-label classfcaton and one-vs-all approach performs reasonably well [Rfkn and Klautau, 2004]. We report average AUC and accuracy for mult-label and mult-class data, respectvely. A porton of data are sampled from USPS, Letter and Yaleface, as they are too large to handle drectly. Varous type of base kernels are generated. We generate 15 dffuson kernels wth parameter varyng from 0.1 to 6 for Lgand [Tsuda and Noble, 2004]; The 8 kernels of Bo are generated followng [Lanckret et al., 2004b]; 20news uses dverse text representatons [Kolda, 1997] leadng to 62 dfferent kernels; For other data, Gaussan kernels wth dfferent wdths are constructed. The trade-off parameter C of SVM s set to a sensble value based on cross valdaton. We vary the number of samples for tranng and randomly sample 30 dfferent subsets n each setup. The average performance are recorded. 5.2 Experment Results Due to space lmt, we can only report some representatve results n Table 3-5. The detals are presented n an extended techncal report [Tang et al., 2009]. Tr Rato n the frst row denotes the tranng rato, the percentage of samples used for tranng. The frst column denotes the porton of kernels shared among labels va varyng the parameter β n 10, wth Independent and Same model beng the extreme. The last row Dff represents the performance dfference between the best model and the worst one. Bold face denotes the best n each column unless there s no sgnfcant dfference. Below, we seek to address the problems rased n the ntroducton. Does kernel regularzaton yeld any effect? The maxmal dfference of varous models on all data sets are plotted n Fgure 1. The x-axs denotes ncreasng tranng data and y-axs denotes maxmal performance dfference. Clearly, when the tranng rato s small, there s a dfference between varous models, especally for USPS, Yaleface and Lgand. For nstance, the dfference could be as large as 9% when only 2% USPS data s used for tranng. Ths knd of classfcaton wth rare samples are common for applcatons lke object recognton. Data Boand Letter demonstrate medum dfference between 1 2%. But for other data sets lke Yeast, the dfference < 1% s neglgble. The dfference dmnshes as tranng data ncreases. Ths s common for all data sets. When tranng samples are moderately large, kernel regularzaton actually has no much effect. It works only when the tranng samples are few. Whch model excels? Here, we study whch model Independent, Partal or Same excels f there s a dfference. In Table 3-5, the entres n bold 1257

4 Independent Partal Same AUC Computaton Tme Fgure 1: Performance Dfference denote the best one n each settng. It s notced that Same Model tends to be the wnner or a close runner-up most of the tme. Ths trend s observed for almost all the data. Fgure 2 shows the average performance and standard devaton of dfferent models when 10% of Lgand data are used for tranng. Clearly, a general trend s that sharng the same kernel s lkely to be more robust compared wth Independent Model. Note that the varance s large because of the small sample for tranng. Actually, Same Model performs best or close n 25 out of 30 trals. So t s a wser choce to share the same kernel even f bnary classfcaton tasks are qute dfferent. Independent Model s more lkely to overft the data. Partal model, on the other hand, takes the wnner only f t s close to the same model sharng 90% kernel as n Table 4. Mostly, ts performance stays between Independent and Same Model. Why s Same Model better? As have demonstrated, Same Model outperforms Partal and Independent Model. In Table 2, we show the average kernel weghts of 30 runs when 2% of USPS data s employed for tranng. Each column stands for a kernel. The weghts for kernel 8-10 are not presented as they are all 0. The frst 2 blocks represents the kernel weghts of each class obtaned va Independent and Partal Model sharng 50% kernel, respectvely. The row follows are the weghts produced by Same Model. All the models, no matter whch class, prefer to choose K5 and K6. However, Same Model assgns a very large weght to the 6-th kernel. In the last row, we also present the weght obtaned when 60% of data s used for tranng, n whch case Independent Model and Same Model tend to select almost the same kernels. Compared wth others, the weghts of Same Model obtaned usng 2% data s closer to the soluton obtaned wth 60% data. In other words, forcng all the bnary classfcaton tasks to share the same kernel s tantamount to ncreasng data samples for tranng, resultng n a more robust kernel. Whch model s more scalable? Regularzaton on kernel dfference seems to affect margnally when the samples are more than few. Thus, a method requrng less computatonal cost s favorable. Our algorthm conssts of multple teratons. In each teraton, we need to solve k SVMs gven fxed kernels. For each bnary classfcaton problem, combnng the kernel matrx costs Opn 2. The tme complexty of SVM s On η wth η [1, 2.3] [Platt, 1999]. After that, a LP wth 0.67 Independent 20% 40% 60% 80% Same Fgure 2: Performance on Lgand 0 10% 20% 30% 40% 50% 70% Trang Rato Fgure 3: Effcency Comparson Table 2: Kernel Weghts of Dfferent Models K1 K2 K3 K4 K5 K6 K7 C C C C I C C C C C C C C C C P C C C C C C S Tr=60% Opk varables the kernel weghts and ncreasng number of constrants needs to be solved. We notce that the algorthms termnates wth dozens of teratons and SVM computaton domnates the computaton tme n each teraton f p<<n. Hence, the total tme complexty s approxmately OIkpn 2 +OIkn η where I s the number of teratons. As for Same Model, the same kernel s used for all the bnary classfcaton problems and thus requres less tme for kernel combnaton. Moreover, compared wth Partal Model, only Op, nstead of Opk varables kernel weghts, need to be determned, resultng less tme to solve the LP. Wth Independent Model, the total tme for SVM tranng and kernel combnaton remans almost the same as Partal. Rather than one LP wth Opk varables, Independent needs to solve k LP wth only Op varables n each teraton, potentally savng some computaton tme. One advantage of Independent Model s that, t decomposes the problem nto multple ndependent kernel selecton problem, whch can be paralleled seamlessly wth a mult-core CPU or clusters. In Fgure 3, we plot the average computaton tme of varous models on Lgand data on a PC wth Intel P4 2.8G CPU and 1.5G memory. We only plot Partal model sharng 50% kernel to make the fgure legble. All the models yeld sm- 1258

5 Table 3: Lgand Result Tr Rato 10% 15% 20% 25% 30% 35% 40% 45% 50% 60% 70% 80% Independent % % % % Same Dff Table 4: Bo Result Tr Rato 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 20% 30% Independent % % % % % Same Dff Table 5: USPS Result Tr Rato 2% 3% 4% 5% 6% 7% 8% 9% 10% 20% 40% 60% Independent % % % % Same Dff% lar magntude wth respect to number of samples, as we have analyzed. But Same Model takes less tme to arrve at a soluton. Smlar trend s observed for other data sets as well. Same Model, wth more strct constrants, s ndeed more effcent than Independent and Partal model f parallel computaton s not consdered. So n terms of both classfcaton performance and effcency, Same model should be preferred. Partal Model, seemngly more reasonable to match the relatonshp between labels, should not be consdered gven ts margnal mprovement and addtonal computatonal cost. Ths concluson, as we beleve, would be helpful and suggestve for other practtoners. A specal case: Average Kernel Here we examne one specal case of Same Model: average of base kernels. The frst block of Table 6 shows the performance of MKL wth Same Model compared wth average kernel on Lgand over 30 runs. Clearly, Same Model s almost always the wnner. It should be emphaszed that smple average actually performs reasonably well, especally when base kernels are good. The effect s mostly observable when samples are few say only 10% tranng data. Interestngly, as tranng samples ncreases to 60%-80%, the performance wth Average Kernel decreases. However, Same Model s performance mproves consstently. Ths s because Same Model can learn an optmal kernel whle Average does not consder the ncreasng label nformaton for kernel combnaton. A key dfference between Same Model and Average Kernel s that the soluton of the former s sparse. For nstance, Same Model on Lgand data pcks 2-3 base kernels for the fnal soluton whle Average has to consder all the 15 base kernels. For data lke mcroarray, graphs, and structures, some specalzed kernels are computatonally expensve. Ths s even worse f we have hundreds of base kernels. Thus, t s desrable to select those relevant kernels for predcton. Another potental dsadvantage wth Average Kernel s robustness. To verfy ths, we add an addtonal lnear kernel wth random nose. The correspondng performance s presented n the 2nd block of Table 6. The performance of Average Kernel deterorates whereas Same Model s performance remans nearly unchanged. Ths mples that Average Kernel can be affected by nosy base kernels whereas Same Model s capable of pckng the rght ones. 6 Conclusons Kernel learnng for mult-label or mult-class classfcaton problem s mportant n terms of kernel parameter tunng or heterogeneous data fuson, whereas t s not clear whether a specfc kernel or the same kernel should be employed n practce. In ths work, we systematcally study the effects of dfferent kernel sharng strateges. We present a unfed framework wth kernel regularzaton such that flexble degree of kernel sharng s vable. Under ths framework, three dfferent models are compared: Independent, Partal and Same Model. It turns out that the same kernel s preferred for classfcaton even f the labels are qute dfferent. When samples are few, Same Model tends to yeld a more robust kernel. Independent Model, on the contrary, s lkely to learn a bad kernel due to over-fttng. Partal Model, occasonally better, les n between most of the tme. However, the dfference of these models vanshes quckly wth ncreas- 1259

6 Table 6: Same Model compared wth Average Kernel on Lgand Data. The 1st block s the performance when all the kernels are reasonably good; The 2nd block s the performance when a nosy kernel s ncluded n the base kernel set. Tr rato 10% 15% 20% 25% 30% 35% 40% 45% 50% 60% 70% 80% Good Same Kernels Average Nosy Same Kernels Average ng tranng samples. All the models yeld smlar classfcaton performance when samples are large. In ths case, Independent and Same are more effcent. Same Model, a lttle bt surprsng, s the most effcent method. Partal Model, though, asymptotcally bears the same tme complexty, often needs more computaton tme. It s observed that for some data, smply usng the average kernel whch s a specal case of Same Model wth a proper parameter tunng for SVM occasonally gves reasonable good performance. Ths also confrms our concluson that selectng the same kernel for all labels s more robust n realty. However, ths average kernel s not sparse and can be senstve to nosy kernels. In ths work, we only consder kernels n the nput space. It could be nterestng to explore the constructon of kernels n the output space as well. Acknowledgments Ths work was supported by NSF IIS , IIS , CCF , NIH R01-HG002516, and NGA HM We thank Dr. Rong Jn for helpful suggestons. A Proof for Theorem 3.1 Proof. Based on Eq. 4, 5 and 6, we can assume ζ = c 1, γ t = c 2, c 1 + c 2 =1, c 1,c 2 0. Let G t α =[α t ] T G t y t [y] T α t and G t α as n Eq. 8. Then Eq. 3 can be reformulated as: max α t = max α t mn {G t } [α t ] T e 1 2 [α t ] T e 1 2 max ζ,γ t λc2 + G α t λc 2 + ζ + γ t G t α It follows that the 2nd term can be further reformulated as X max ζ G t c 1,c 2,ζ,γ t α + γ t Gt α λc2 = max P max ζ G t c 1,c 2 ζ =c α 1 + P max γ t γt =c Gt α kλc2 2 = max c 1 max G t c 1,c α + X k c 2 max G t α 2 kλc2 = max c 1 +c 2 =1 c 1 max " G t α X k +c2 " = max max G t α, X k # max G t α kλ # max G t α kλ By addng constrants as s s 0,s s t kλ s 0 G t α, =1,,p s t G t α, =1,,p,,,k We thus prove the Theorem. References [Bach et al., 2004] Francs R. Bach, Gert R. G. Lanckret, and Mchael I. Jordan. Multple kernel learnng, conc dualty, and the smo algorthm. In ICML, [Jebara, 2004] Tony Jebara. Mult-task feature and kernel selecton for svms. In ICML, [J et al., 2008] S. J, L. Sun, R. Jn, and J. Ye. Mult-label multple kernel learnng. In NIPS, [Kolda, 1997] Tamara G. Kolda. Lmted-memory matrx methods wth applcatons. PhD thess, [Lanckret et al., 2004a] Gert R. G. Lanckret, Nello Crstann, Peter L. Bartlett, Laurent El Ghaou, and Mchael I. Jordan. Learnng the kernel matrx wth semdefnte programmng. JMLR, 5, [Lanckret et al., 2004b] Gert R. G. Lanckret, et al. Kernelbased data fuson and ts applcaton to proten functon predcton n yeast. In PSB, [Platt, 1999] John C. Platt. Fast tranng of support vector machnes usng sequental mnmal optmzaton [Rakotomamonjy et al., 2007] Alan Rakotomamonjy, Francs R. Bach, Stephane Canu, and Yves Grandvalet. More effcency n kernel learnng. In ICML, [Rfkn and Klautau, 2004] Ryan Rfkn and Aldebaro Klautau. In defense of one-vs-all classfcaton. JMLR, 5, [Sonnenburg et al., 2007] Sren Sonnenburg, Gunnar Rtsch, Chrstn Schfer, and Bernhard Schlkopf. Large scale multple kernel learnng. JMLR, 7: , [Tang et al., 2009] Le Tang, Janhu Chen, and Jepng Ye. On multple kernel learnng wth multple labels. Techncal report, Arzona State Unversty, [Tsuda and Noble,2004] Koj Tsuda and Wllam Stafford Noble. Learnng kernels from bologcal networks by maxmzng entropy. Bonformatcs, 20: , [Zen,2007] Alexander Zen. Multclass multple kernel learnng. In ICML,