Multiple Instance Learning via Multiple Kernel Learning *

Transcription

1 The Nnth nternatonal Syposu on Operatons Research and ts Applcatons (SORA 10) Chengdu-Juzhagou, Chna, August 19 23, 2010 Copyrght 2010 ORSC & APORC, pp ultple nstance Learnng va ultple Kernel Learnng * Bng Yang Qan L Lng Jng Lng Zhen College of Scence, Chna Agrcultural Unversty, Bejng , P.R. Chna. Abstract n ths paper, we forulated a novel ethod to solve the classfcaton proble wthn the ultple nstance learnng (L) contexts by ultple kernel learnng. Despte the large nuber of SV odels, there are only a few odels that can solve general L probles well. To prove the classfcaton precson of SV ethod wth regard to L proble, ths paper ntroduced ultple kernel learnng ethod to the process of ultple nstance learnng, and proposed a new SVW odel (K-SV), whch based on the -SV odel. The soluton for ths odel was presented, and soe nuercal experents on benchark data were taken nto ths paper too. Coputatonal results on a nuber of datasets ndcate that the proposed algorth s copettve wth other SV ethods. Keywords ultple nstance learnng, Support vector achnes, ultple kernel learnng, ult-class SV 1 ntroducton Lteratures [1 3] gave us an ntroducton about ultple nstance classfcaton probles. n ths paper we gave a ethod to solve the proble whch s entoned n lterature [10]. The proble to consst of classfyng postve and negatve bags of ponts n the n-densonal real space R n on the followng bass s consdered. A bag s classfed as a postve bag f one or ore nstances n that bag are postve, otherwse t s classfed as a negatve bag. Ths proble was frstly analyzed by T. G. Detterch et al. [1] n the pharc actvty s predcton n the 90s, 20century. T. G. Detterch consdered every olecule as a bag n ther analyss, and every low power shape represented an nstance n the bag. Ths s the orgn of ultple nstance learnng (L). The L proble has been exsted for a long whle; however, t s not a sudden result of pharc actvty s predcton. Prevous achne learnng [4, 5] ddn t take ths knd of probles property nto consderaton forally, and the proble hasn t been exactly * Ths work s supported by the Key Project of Natonal Natural Scence Foundaton of Chna (No ), the Natonal Natural Scence Foundaton of Chna (No ), the Chnese Unverstes Scentfc Fund(No ),and the Graduate Basc Research Foundaton of Chna Agrcultural Unversty Funded Project (No ). Correspondng author. E-al address: jnglng_student@163.co.

2 ultple nstance Learnng va ultple Kernel Learnng 161 defned untl T. G. Detterch s work. Later the analyss of ths proble aroused a nuber of achne learnng researchers nterest and a lot of research works have been done. Varous ethods for ultple nstance classfcaton probles have been proposed, ncludng nteger prograng [6], expectaton axzaton [7], kernel forulatons [8], and lazy learnng [7]. Ray and Craven [9] provde an eprcal coparson of several ultple nstance classfcaton algorths and ther non-ultple-nstance counterparts. The classcal SV ethods to solve L proble are -SV and -SV, whch proposed by S. Andrews [6]. Based on ther work, ths paper added ultple kernel ethods to the classcal SV ethods, and gave a novel forulaton for L proble. eanwhle the strengths and the weakness about ths new ethod have been dscussed. Andrews et al. extend the sngle kernel SV, whle we begn wth the ultple kernels SV [10]. The use of the ultple kernels SV allows us to get a better descrpton of data s dstrbutng as opposed to sngle kernel SV. We nclude results n Sect. 4 whch deonstrate that ultple kernel ethods are uch ore coputatonally effcent and faster than classcal ethods. The paper s organzed as follows. n Sect. 2 we gve a revew of soe nterrelated concepts. n Sect. 3 we ntroduce our forulaton of the ultple nstance classfcaton probles and state ts propertes. n Sect. 4 we present our nuercal tests on fve datasets. Secton 5 concludes the paper. 2 The background about SV ethod for ultple nstance learnng and ultple kernel learnng As the background of ths paper, ths secton wll ntroduce -SV whch s the classcal SV ethod for L proble and the standard ultple kernel SV ethod. 2.1 Support vector achnes for ultple nstance learnng n ths part, a bref revew about classcal SV ethods for ultple nstance learnng, -SV and -SV, wll be shown. And we are gong to ntroduce the basc dea, odel and solvng algorth respectvely. Andrews et al. [6] have prevously nvestgated extendng support vector achnes to the ultple nstance classfcaton probles usng xed-nteger prograng. They use nteger varables ether to select the class of ponts n postve bags or to dentfy one pont n each postve bag as a wtness pont that ust be placed on the postve sde of the decson boundary. Each of these representatons leads to a natural heurstc for approxately solvng the resultng xed-nteger progra. S. Andrews gave an alternatve way [6] of applyng axu argn deas whch s the an deas of SV to the L settng. They extend the noton of a argn fro ndvdual patterns to set of patterns. t s natural to defne the functonal argn of a bag wth respect to a hyperplane by Y ax(, ) w x b. Therefore based on the ths noton of a bag argn, the SV odel has been redefned nto

3 162 The 9th nternatonal Syposu on Operatons Research and ts Applcatons -SV n wb,, 1 w 2 2 C s. t. Y ax( w, x b) 1 0 For solvng ths progra, unfoldng the ax operatons by ntroducng one nequalty constrant for per nstance has been done. For negatve bags, the nequalty constrans can be read as w, x b 1,, where Y 1. For postve bags, [6] ntroduces a selector varable s whch denotes the nstance selected as the postve wtness n per postve bag B. eanwhle they gave two ethods to select s fro B, -SV and -SV. Both of -SV and -SV are casted as xed-nteger progras. They wll be shown n algorth1 and algorth 2 n the followng, respectvely. Algorth1 -SV algorth ntalze y Y for REPEAT Copute SV soluton w, b for data wth puted labels Copute outputs f ( W, X) b for all X n postve bags Set y sgn( f) for every, Y 1 FOR (every postve bag B ) F ( (1 y) / 2 0 ) * Copute arg ax f Set y * 1 END END WHLE (puted labels have changed) OUTPUT(w, b) Algorth2 -SV algorth ntalze X x / for every postve bag B REPEAT Copute QP soluton w, b for data set wth postve exaples{ X : Y 1 } Copute outputs f ( w, x) b for all x n postve bags Set X Xs( ), s( ) arg ax f for every Y, 1 WHLE ( selector varables s () have changed) OUTPUT(w, b) 2.2 ultple kernel learnng -SV odel by Andrews et al. extend the classcal sple kernel SV, whle we begn wth the ultple kernels SV [11]. n ths part, we wll gve an ntroducton about ultple kernels SV s basc dea, odel and an coputng Algorth. ultple kernels learnng (KL) as at sultaneously learnng a kernel and l the assocated predctor n supervsed learnng settngs. Let x, y 1 s the learnng set, where x belongs to soe nput space X and y s the target value for pattern x. For kernel algorths n SV, the soluton of the learnng proble s of the for l f ( x) K( x, x ) b 1 * * * * where and b are soe coeffcents to be learned fro exaples, whle K(, ) s a gven postve defnte kernel assocated wth a reproducng kernel Hlbert space (RKHS) H. n soe stuatons, a learnng practtoner ay be nterested n ore flexble

4 ultple nstance Learnng va ultple Kernel Learnng 163 odels. Recent applcatons have shown that usng ultple kernels nstead of a sngle one can enhance the nterpretablty of the decson functon and prove perforances [11]. n such cases, a convenent approach s to consder that the kernel K( x, x ') s actually a convex cobnaton of bass kernels: K( x, x') d K ( x, x'), 1 wth d 0, d 1 1 where s the total nuber of kernels. Each bass kernel K ay ether use the full set of varables descrbng x or subsets of varables steng fro dfferent data sources [11]. Alternatvely, the kernels K can sply be classcal kernels (such as Gaussan kernels) wth dfferent paraeters. Wthn ths fraework, the proble of data representaton through the kernel s then transferred to the choce of weghts d. n the SV ethodology, the decson functon s of the for l f ( x) y d K b, where the optal paraeters 1 1 obtaned by solvng the dual of the followng optzaton proble [10] : n { },,, 2 f C H f b d d s. t. y f ( x ) y b 1 0 d 1, d 0 d, and b are The KL forulaton ntroduced by Bach et al. [12] and further developed by Sonnenburg et al. [13] conssts n solvng an optzaton proble expressed above. Nowadays an effectve ethod to solve ths optzaton proble s proposed by Alan Rakotoaonjy et al. n 2008 [11]. The an algorth wll be shown n algoth3 n followng. Algorth3 Sple KL algorth 1 Set d for 1,..., Whle stoppng crteron not et do Copute Jd ( ) by usng an SV solver wth K d K Copute J d for 1,... and descent drecton D (12). Set arg ax d, J 0, d d, D D Whle J J ( d) do {descent drecton update} d d, { D 0} D D v argn d / D, d / D d d axd ax v v, D D Dv, D v 0 Copute End whle Lne search along D for d d D End whle J by usng an SV solver wth ax K d K [0, ] {calls an SV solver for each tral value}

5 164 The 9th nternatonal Syposu on Operatons Research and ts Applcatons 3 K-SV Classfcaton odel and Algorth ultple kernel SV s used for soe stuatons where a achne learnng practtoner ay be nterested n ore flexble odels. We can expect ultple kernel learnng wll has a better perforance n L proble for two reasons. Obvously, snce the hghly coplcated descrpton about real object n L that the specal proble, a flexble odel s necessary for the learnng task. eanwhle the enhance about the nterpretablty of the decson functon, ore effectble coputaton and hgher predcaton accuracy not only can be expected, but also are our hopes n L. Therefore, t s sgnfcant to add ultple kernel ethod to L proble. n ths secton, the odel and algorth of K-SV wll be gven. n K-SV ethod, we also defned the functonal argn of a bag wth respect to a hyperplane by Y ax(, ) w x b. Based on ths rules, the nequalty constrants n ultple kernel SV can be changed for solvng L proble. Therefore the K-SV odel can be expressed a new optzaton proble showed n followng: K-SV 1 1 n 2 d { f }, b,, d 2 H s. t. Y ax( f ( x ) b) 1 0 d 1, d 0 Snce the frst constrant n our ultple nstance forulaton contans the ax operatons. We also unfolded ths ax operaton as [6]. For negatve bags, the nequalty constrant can be read as w, x b 1,, where Y 1. For postve bags, a selector varable s whch denotes the nstance selected as the postve nstance n per postve bag B wll be gave. For d, s and,b, alternately copute one set of varables when hold other sets. Ths leads to the successve soluton of K-SV progras that underly our algorth whch we specfy now. f C Algorth4 K-SV Algorth ntalze y Y for REPEAT Copute K-SV soluton K dk( xn, x ),, b for data set wth puted labels Copute outputs FOR (every postve bag B ) l 1 f ( d K ( x, x )) b n n1 1 F ( (1 y) / 2 0 ) * Copute arg ax f Set y* 1 END END WHLE (puted labels have changed) OUTOUT ( d,, b ) for all x n postve bags

6 ultple nstance Learnng va ultple Kernel Learnng 165 n practce, coputng the K-SV Algorth 4 ay be faster than classcal -SV when you should change your kernel to check ore kernel Hlbert space (RKHS). f the nuber of kernel you should copute s N, the classcal SV ethods wll copute ther Algorth 5 tes. And the K-SV te s equal to ts nuber of teratons te. n [11], a lot of nuercal testng had been done to copare whch s faster. ultple kernel learnng often has the better perforances. 4 Nuercal Experents n ths secton, soe nuercal experents wll be done for testng the K-SV s capabltes n L proble. To evaluate the capabltes of K-SV ethod, we have perfored soe experents on benchark data. n ths paper, we reported results on 5 datasets, two fro the UC achne learnng repostory [14], and three fro [6]. Detaled nforaton about these datasets s suarzed n Table 1. We use the datasets fro [6] to evaluate our ultple kernel classfcaton algorths. These three datasets are fro an age annotaton task n whch the goal s to deterne whether or not a gven anal s present n an age. The two datasets fro the UC repostory [14] are the usk datasets, whch are coonly used n ultple nstance classfcaton. Table 1 Descrpton of the datasets used n the experents. Elephant, Fox and Tger datasets are used n [6], whle usk-1 and usk-2 are avalable fro [14]. +Bags denotes the nuber of postve bags n each dataset, whle +nstances denotes the total nuber of nstances n all the postve bags. Slarly, Bags and nstances denote correspondng quanttes for the negatve bags Data set +bag +nstances -bag -nstances features Elephant Fox Tger usk usk We copare our ultple kernels classfcaton algorth to the -SV and -SV [6] on three age datasets. Snce Andrews et al. also report results on Zhang and Goldan s expectaton axzaton approach E-DD [7] on these datasets [6] ; we nclude those results here as well. Table 2 reports results coparng K-SV to -SV, -SV and E-DD. Accuracy results for -SV, -SV and E-DD were taken fro [6]. Accuracy for K-SV was easured by averagng ten ten-fold cross valdaton runs. The ultple kernels for K-SV were selected by 10 dfferent Gaussan kernel, kernel paraeters for 2-5 to 2 4. The paraeters C for K-SV were selected fro the set {2 = 5,..., 4} by ten-fold cross valdaton on each tranng saples of the age datasets.

7 166 The 9th nternatonal Syposu on Operatons Research and ts Applcatons Table 2 K-SV, -SV [6], -SV [6] and E-DD [7] testng accuracy used averaged over ten ten-fold cross valdaton experents. The datasets are those used by Andrews et al. n [6]. Best accuracy on each dataset s n bold. Data set K-SV -SV -SV E-DD Elephant 81.8% 82.2% 81.4% 78.3% Fox 58.7% 58.2% 57.8% 56.1% Tger 84.0% 78.4% 84.0% 72.1% n order to evaluate the dfference between the algorths ore precsely, we used the Fredan test [17] on the results reported n Table 2. The Fredan test s a nonparaetrc test that copares the average ranks of the algorths, where the algorth wth the hghest accuracy on a dataset s gven a rank of 1 on that dataset, and the algorth wth the worst accuracy s gven a rank of 5. Therefore the average rank was 1.3 for K-SV, 1.5 for -SV, 2.3 for -SV, and 4 for E-DD. The better perforance by K-SV expressed on KL proble was clearly showed. Table 3 K-SV, -SV [6], -SV [6], E-DD [7], DD [15], -NN [16], APR [1], and K [8] ten-fold testng accuracy on the usk-1 and usk-2 datasets. Best accuracy s n bold. Dataset K -SV -SV E-DD DD -NN APR K -SV usk % 87.4% 77.9% 84.8% 88.0% 88.9% 92.4% 91.6% usk % 83.6% 84.3% 84.9% 84.0% 82.5% 89.2% 88.0% Table 3 gves ten-fold cross valdaton accuracy results for K-SV usng the sae test ethod on the usk-1 and usk-2 datasets whch are avalable fro the UC repostory [14]. n table 3, we can see that K-SV got the best accuracy n all SV ethods, but soe sple ethod lke APR ethods, got the better result on the contrary. t showed that K-SV wll just have a better perforance on the coplcated dataset of whch the actng feature and ts recprocty n classfcaton s not very clear, but for the ordnary datasets of whch the actng feature and ts recprocty s clearly enough, t s proper to perfor soe sple ethods. Obvously, ths s n lne wth the prncple of Occa's razor; eanwhle t can suggest us the type of dataset for whch K-SV ethod s proper to be used. 5 Concluson and Outlook Ths paper has ntroduced a atheatcal prograng forulaton of the ultple nstance probles that has used ultple kernel learnng. Results on prevously publshed datasets ndcate that our approach s effectve at soe stuaton where a achne learnng practtoner ay be nterested n ore flexble odels. Furtherore, ultple kernels learnng often cost less than sple kernel for learnng n ultple kernel Hlbert space, and coputng the K-SV aybe faster than classcal sple kernel ethod n practce. proveents n the atheatcal

8 ultple nstance Learnng va ultple Kernel Learnng 167 prograng forulaton and evaluaton usng a wde varety of datasets and algorths, such as those n [17], are prosng avenues of future research. References [1] Detterch, T.G., Lathrop, R.H., Lozano-Perez, T.: Solvng the ultple-nstance proble wth axs-parallel rectangles. Artf. ntell. 89, (1998) [2] Auer, P.: On learnng fro ult-nstance exaples: eprcal evaluaton of a theoretcal approach. n: Proceedngs of 14th nternatonal Conference on achne Learnng, pp organ Kaufann, San ateo (1997) [3] Long, P.., Tan, L.: PAC learnng axs algned rectangles wth respect to product dstrbutons fro ultple nstance exaples. ach. Learn. 30(1), 7 22 (1998) [4] Fredan J H, Stuetzle W. Projecton pursut regresson. Journal of the Aercan Statstcal Assocaton, 1981, 76(376): [5] Lndsay R, Buchanan B, Fegenbau E, Lederberg J. Applcatons of Artfcal ntellgence to Organc Chestry: The Dendral Project, New York, NY: cgraw-hll, [6] Andrews, S., Tsochantards,., Hofann, T.: Support vector achnes for ultple-nstance learnng. n: Becker, S., Thrun, S., Oberayer, K. (eds.) Advances n Neural nforaton Processng Systes 15, pp T Press, Cabrdge (2003) [7] Zhang, Q., Goldan, S.A.: E-DD: an proved ultple-nstance learnng technque. n: Neural nforaton Processng Systes 2001, pp T Press, Cabrdge (2002) [8] Gartner, T., Flach, P.A., Kowalczyk, A., Sola, A.J.: ult-nstance kernels. n: Saut, C., Hoffann, A. (eds.) Proceedngs of 19th nternatonal Conference on achne Learnng, pp organ Kaufann, San ateo (2002) [9] Ray, S., Craven,.: Supervsed versus ultple nstance learnng: an eprcal coparson. n: Proceedngs of 22nd nternatonal Conference on achne Learnng, Bonn, Gerany, vol. 119, pp Assoc. Coput. ach., New York (2005) [10] O.L. angasaran, E.W. Wld: ultple nstance Classfcaton va Successve Lnear Prograng. n: J Opt Theory Appl (2008) 137: [11] Alan Rakotoaonjy, Francs R. Bach, St éphane Canu, Yves Grandvalet: SpleKL. n: Journal of achne Learnng Research 9 (2008) [12] F. Bach, G. Lanckret, and. Jordan: ultple kernel learnng, conc dualty, and the SO algorth. n: Proceedngs of the 21st nternatonal Conference on achne Learnng, pages 41 48, 2004a. [13] S. Sonnenburg, G. Ratsch, C. Schafer, and B. Scholkopf.: Large scale ultple kernel learnng. n: Journal of achne Learnng Research, 7(1): , [14] urphy, P.., Aha, D.W.: UC achne Learnng Repostory (1992). [15] aron, O., Ratan, A.L.: ultple-nstance learnng for natural scene classfcaton. n: 15 th nternatonal Conference on achne Learnng, San Francsco, CA. organ Kaufann, San ateo(1998) [16] Raon, J., De Raedt, L.: ult-nstance neural networks, n: Proceedngs of CL Workshop on Attrbute-Value and Relatonal Learnng (2000) [17] Ray, S., Craven,.: Supervsed versus ultple nstance learnng: an eprcal coparson. n: Proceedngs of 22nd nternatonal Conference on achne Learnng, Bonn, Gerany, vol. 119, pp Assoc. Coput. ach., New York (2005)