Multilabel Classification with Meta-level Features

Transcription

1 Multlabel Classfaton wth Meta-level Features Sddharth Gopal Carnege Mellon Unversty Pttsburgh PA 523 Ymng Yang Carnege Mellon Unversty Pttsburgh PA 523 ABSTRACT Effetve learnng n mult-label lassfaton (MLC) requres an approprate level of abstraton for representng the relatonshp between eah nstane and multple ategores. Current MLC methods have been foused on learnng-to-map from nstanes to ranked lsts of ategores n a relatvely hgh-dmensonal spae. The fne-graned features n suh a spae may not be suffently expressve for haraterzng dsrmnatve patterns, and worse, make the model omplexty unneessarly hgh. Ths paper proposes an alternatve approah by transformng onventonal representatons of nstanes and ategores nto a relatvely small set of lnk-based meta-level features, and leveragng suessful learnng-to-rank retreval algorthms (e.g., SVM-MAP) over ths redued feature spae. Controlled experments on multple benhmark datasets show strong empral evdene for the strength of the proposed approah, as t sgnfantly outperformed several state-of-the-art methods, nludng Rank-SVM, ML-kNN and IBLR-ML (Instane-based Logst Regresson for Mult-label Classfaton) n most ases. Categores and Subet Desrptors I.5.2 [PATTERN RECOGNITION] Desgn Methodology; Classfer desgn and evaluaton; H..0 [INFORMATION SYSTEMS]: General; General Terms Algorthms, Desgn, Expermentaton, Performane Keywords Mult-label lassfaton; model desgn; learnng to rank; omparatve evaluaton. INTRODUCTION Mult-label lassfaton (MLC) refers to the problem of nstane labelng where eah nstane may have more than one orret label. MLC has a broad range of applatons. For example, a news artle ould belong to both tops polts and eonoms, and also ould be related to Chna and USA as the regonal ategores. An mage (pture) ould have flower as the obet type, yellow and red as the Permsson to make dgtal or hard opes of all or part of ths work for personal or lassroom use s granted wthout fee provded that opes are not made or dstrbuted for proft or ommeral advantage and that opes bear ths note and the full taton on the frst page. To opy otherwse, or republsh, to post on servers or to redstrbute to lsts, requres pror spef permsson and/or a fee. SIGIR 0, July 9 23, 200, Geneva, Swtzerland. Copyrght 200 ACM /0/07 $0.00. olors, and outdoor as the bakground ategory. A omputer trouble report ould be smultaneously related to a hardware falure, a software problem, an urgeny-level ategory, a regonal ode, and so on. MLC s tehnally hallengng as t goes beyond the sope of well-studed two-way lassfers, suh as bnary Support Vetor Mahnes (SVM), Naïve Bayes probablst lassfers, et. Approahes to MLC typally redue the problem nto two subproblems: the frst s learnng to rank ategores wth respet to eah nput nstane, and the seond s learnng to plae a threshold on eah ranked lst for a yes/no deson per ategory. The frst subproblem s the most hallengng part and therefore has been the entral fous n MLC. A varety of approahes has been developed and an be roughly dvded nto two types: bnary-lassfer based methods versus global optmzaton methods, and the latter an be further dvded nto model-based and nstane-based methods. Bnary-lassfer based methods are the smplest. A representatve example s to use a standard SVM ( bnary-svm ) [7] to learn a sorng funton for eah ategory ndependently from the sorng funtons for other ategores. Other knds of bnary lassfers ould also be used for suh a purpose, suh as logst regresson, Naïve Bayes probablst lassfers, boostng algorthms, neural networks et. In the testng phase, the ranked lst of ategores s obtaned for eah test nstane by sorng eah ategory ndependently and then sortng the sores. Bnarylassfer based methods have been ommonly used due to ther smplty, but also have been rtzed for the lak of global optmzaton n ategory sorng. These methods are ommon baselnes n benhmark evaluatons of stronger methods n MLC. Elsseeff and Weston [5] proposed a large-margn approah, Rank-SVM, whh s a representatve example of model-based methods. Unlke onventonal bnary SVM whh maxmzes the margn for eah ategory ndependently, Rank-SVM maxmzes the sum of the margns for all ategores smultaneously, ondtoned on partal-order onstrants. That s, rankng the relevant ategores of eah tranng nstane hgher than the rrelevant ategores s an explt optmzaton rteron n ths method. The sorng funton s parameterzed by the weghts of nput features for every ategory; thus, the number of parameters n Rank-SVM s the produt of the number of ategores and the sze of the feature set. In other words, the model omplexty (measured usng the number of free parameters) s the same as that of m bnary-svms where m s the number of target ategores. Experments by the authors of Rank- SVM show that ths method sgnfantly outperformed bnary SVM n gene lassfaton on a mro-array dataset (namely the yeast dataset ). Zhang and Zhou [25] proposed an nstane-based approah whh s named as Mult-label k-nearest Neghbor (ML-kNN). Cheng and Hullermeer proposed another varant alled Instane-

2 2 Based Logst Regresson (IBLR) [3]. Mult-label versons of knn have been studed n text ategorzaton for a long tme and ommonly used as strong baselnes n benhmark evaluatons [2][20][22]. ML-kNN and IBLR are relatvely new varants whh are smlar n the sense that both use Euldean dstane to dentfy the k nearest neghbors for eah test nstane, but dffer n how the loal probabltes are estmated for ategores. ML-kNN gves an equal weght to eah label ourrene n the neghborhood of the nput nstane whle IBLR vares the weght of eah label aordng to how dstant t s to the test nstane. Further, ML-kNN assumes ndependene among ategory ourrenes whle IBLR expltly models parwse dependenes among ategores usng logst regresson. IBLR also makes ombned use of nstane-based features (suh as the smlarty sore of eah neghbor) and onventonal features (suh as words n the test doument) n the logst regresson. The evaluatons by the authors of ML-kNN showed ts superor performane over Rank-SVM, BoosTexter [4] and Adaboost.MH [5], and the experments by the authors of IBLR showed further performane mprovement by IBLR over the results of ML-kNN on multple datasets [3]. MLC methods, nludng the ones dsussed above, have been foused on learnng-to-rank n a relatvely hgh-dmensonal spae. In Rank-SVM for text ategorzaton the features are words n the tranng-set doument voabulary. In IBLR for text ategorzaton the feature set s the unon of word-level features and knn-based features (used to model the nterdependenes among ategores). Both methods learn a lnear funton per ategory, so the total number of features s md where d s the feature-set sze and m s the number of ategores. In mahne learnng, t s generally understood that when the number of model parameters s unneessarly large, the model tends to overft tranng data and does not generalze well on test data. To what extent would ths be an ssue n urrent MLC methods? Further, an we fnd a better soluton for MLC by transformng lower-level features to hgher-level ones, and then learnng to rank more effetvely n the redued spae? Thorough nvestgaton s needed to answer these questons, and suh s the prmary fous of ths paper. Spefally, we address these questons by the followng means: ) We propose a gener framework that allows automated transformaton of a onventonal data representaton (suh as a bag of words per nstane and a set of tranng nstanes per ategory) nto meta-level features. Ths enables a broad range of learnng-to-rank algorthms n nformaton retreval (IR) to be deployed for rankng ategores n MLC. 2) We use SVM-MAP, a large-margn method for optmzng ranked lsts of douments wth respet to the Mean Average Preson n IR, as the hoe of algorthm n ths paper to llustrate effetve MLC learnng wth meta-level features. For onvenene we all ths nstantaton of our framework SVM-MAP-MLC n dstnton from the use of SVM-MAP n ad-ho nformaton retreval. 3) We ondut ontrolled experments on multple benhmark datasets to evaluate SVM-MAP-MLC n omparson wth other state-of-the-art methods, nludng Rank-SVM, MLkNN, IBLR and Bnary SVM. 4) We provde strong empral evdene for the strengths of the proposed method, wth p-values at the % level or smaller on all the datasets n statstal sgnfane tests for omparng our approah wth the other state-of-the-art methods. The rest of the paper s organzed as follows. Seton 2 ntrodues the meta-level features for MLC. Seton 3 desrbes the method for ategory rankng optmzaton,.e., SVM-MAP- MLC and the strategy for optmzng the threshold on eah ranked lst. Seton 4 presents desgn of ontrolled experments. Seton 5 reports the man results. Seton 6 summarzes our fndngs and onluson. 2. META-LEVEL FEATURES FOR MLC Followng a standard notaton n mahne learnng, we defne X as the nput spae (of all possble nstanes), Y as the output spae (of all possble ranked lsts of target varables), and T {( x, y ), ( x 2, y 2),,( x n, y n)} as a tranng set of n pars of x X and y Y. The learnng-to-rank problem s generally defned as to fnd the optmal mappng f : X Y gven T. Now we defne a transformaton from x X to z Z wth the followng propertes: ) The feature-set sze of Z should be relatve small. 2) The features n z Z should be hghly nformatve about how eah nstane s related to multple ategores, and should dsrmnate relevant nstanes from rrelevant nstanes wth respet to any possble subset of ategores. 3) The transformaton from x X to z Z should be learnable based on a labeled tranng set and automatally omputable for eah test nstane on the fly. 4) The transformed tranng data should allow a broad range of learnng-to-rank algorthms to be used for MLC optmzaton wth respet to varous loss funtons. Based on these rtera we defne vetor z as the onatenaton of the sub-vetors below, whh are defned for eah test nstane x 0 and ategory C for,2,, m as: v x, ) d ( x, x ), d( x, x ),, d( x, x ) L ( k, a k- dmensonal vetor where x knn x, ) s the th ( 0 nearest neghbor (,2,, k ) of x 0 among the members of n the tranng set, and d( x, x0 ) x x0 2 s the L -norm of the dfferene between the two vetors ; 2 v x, ) d '( x, x ), d'( x, x ),, d'( x, x ) L ( k 0, a k- dmensonal vetor where d' ( x, x0 ) x x0 s the L -norm of the dfferene between the two vetors; For rare ategores, sne the number of tranng nstanes n eah ategory s small, there mght not be k nearest neghbors. In suh a ase we duplate the furthest neghbor so that the subvetor reahes sze k.

3 3 v x, ) os( x, x ),os( x, x ),,os( x, x ) os( k 0 k-dmensonal vetor whose elements are the osne smlarty of the orrespondng vetor pars; v x, ) d ( x, x ),os( x, x )) mod ( vetor where, a, a 2-dmensonal x s the entrod (vetor average) of all the postve tranng examples n ategory. Of ourse the lsted features are not neessarly exhaustve for all possbly nformatve features but rather a set of onrete examples for llustratng the prnple n our approah. The number of features n vetor z s 3k 2 per ategory, and the total of ( 3k 2) m for m ategores. Parameter k an be tuned on a held-out valdaton dataset whh s typally n the range from 0 to 00. The sze of suh a meta-level feature set s muh smaller than those typally found n urrent MLC methods. More mportantly, these meta-level features are ndued n a supervsed manner, takng the labels of tranng nstanes nto aount. Also, the meta-level features make a ombned use of loal nformaton (through knn-based features) and global nformaton (through ategory entrods) n the tranng set. Vetor z Z synthetally represents a pattern for eah nstane about how t s related to multple ategores. Fgure llustrates the onept geometrally n a 2-D spae. For smplty we only plot the L2-norm lnks n ths graph and omt the other types of lnks. Fgure : The lnk-based representaton of one partular nstane (the dot n the enter) n relaton to multple ategores s shown. Eah ategory s represented usng ts postve examples (ponts n the same olor) n the tranng set and ts entrod (trangle). Eah nstane s represented usng the average lnks (.e., the dstane to eah ategory entrod, shown by thk lnes) and multple sngle lnks (.e., the dstane to eah of the k nearest neghbors, shown by the thn lnes) per ategory. These lnks together portray an nformatve pture about how the nstane s related to multple ategores. 3. OUR APPROACH: SVM-MAP-MLC 3. Learnng to rank for MLC Our goal now s to solve the mappng the transformed tranng set T MLC f MLC {( z, y), ( z2, y2),,( z, y n n : Z Y based on z z( x ) Z s an transformed nput vetor whose where elements are meta-level features, and y Y )} s a true ranked lst of ategores for the nput 2. Any learnng-to-rank algorthm ould be deployed n prnple; among the suessful ones, we hoose SVM-MAP n the remander of ths paper. SVM-MAP s orgnally desgned for rankng douments wth respet ad-ho queres where ad-ho means that queres ould be any ombnaton of words, not fxed n advane. The learnng-to-rank problem s to fnd the optmal mappng f IR : Q Y where Q the nput spae (of all possble queres), Y as the output spae (of all possble ranked lsts of douments), and optmal means to mnmze the tranng-set loss as well as the model omplexty. A varety of learnng-to-rank algorthms have been developed reently n mahne learnng for ad-ho retreval, wth dfferent loss funtons and model seleton rtera. SVM-MAP [24] s desgned to maxmze the Mean Average Preson (MAP) of ranked lsts, whh s a ommon metr for method omparson n IR evaluatons. Methods fousng on other optmzaton rtera nlude a multvarate verson of SVM [8] that maxmzes ROC-Area for lassfaton, MCRank and AdaRank [] [9] that use boostng algorthms to maxmze the Normalzed Dsounted Cumulated Gan (NDCG) of ranked lsts, and so on. Most learnng-to-rank algorthms n ad-ho retreval rely on a shared representaton between queres and douments,.e., a bag-of-words per query and per doument. Suh a shared representaton faltates a natural way to ndue features for dsrmnatng the relevane of query-doument pars. For example, SVM-MAP uses a onventonal searh engne (Indr) to produe the osne smlarty and language-model based sores for eah query-doument par, dsretzes those sores nto bns, and treats the bns as the features n a dmenson-redued vetor spae where eah query-doument par wth relevane udgment s treated as a tranng nstane. Optmzng the mappng from queres to ranked lsts of douments therefore redues to the learnng of feature weghts n the dmenson-redued vetor spae. In order to apply SVM-MAP to MLC, or to apply any learnng-to-rank retreval algorthm to MLC n general, we need to fnd dsrmnatve features to represent nstane-ategory pars and the one-to-many mappng from eah nstane to ategores. The meta-level features we ntrodued n the prevous seton are exatly desgned for suh a purpose, allowng a broad range of learnng-to-rank algorthms n IR to be deployed for MLC. The ategory-spef lnks ( 3k 2 per ategory) n vetor z (Seton 2) and the orrespondng label (yes or no wth respet to the ategory) s treated as an nstane-ategory par n the tranng set. We fous 2 There may be more than one true ranked lst for an nstane. That s, any lst that plaes all the relevant ategores above all the rrelevant ategores s truly orret.

4 4 on SVM-MAP n ths paper beause t expltly optmzes MAP whh s one of the prmary metrs we use n our evaluaton of MLC methods (Seton 4.3). SVM-MAP and other learnng-to-rank retreval methods have not been used for MLC before, to our knowledge. We name our novel applaton of SVM-MAP as SVM-MAP-MLC, n ontrast to ts onventonal use n ad-ho nformaton retreval. 3.2 Learnng to threshold for MLC In order to enable the system to make lassfaton desons n MLC, we need to apply a threshold to the ranked lst of ategores for eah test nstane. A varety of thresholdng strateges have been studed n the lterature [2],[5], among whh we hoose the lnear regresson approah proposed n [5]. Unlke bnary-svm where the natural hoe of threshold s zero and probablst lassfers (suh as logst regresson, Naïve Bayes, IBLR, et.) where the default hoe of threshold s 0.5, SVM-MAP-MLC produes nonprobablst sores to rank ategores wth partal-order preferenes. Obvously, nether 0 nor 0.5 s the approprate optmal threshold on a ranked lst of ategores gven an nstane. The strategy proposed n Rank-SVM [5] s desgned for optmzng the threshold ondtoned on eah ranked lst. That s, the system uses a tranng set to learn a lnear mappng from an arbtrary ranked lst of ategores to the optmal threshold as: g : L T m where L R s the spae of all possble vetors of system-sored ategores, and T R s the spae of all possble thresholds. The optmal mappng s defned as the lnear-least-squared-ft (LLSF) soluton gven a tranng set of ranked lsts wth the optmal threshold per lst. The optmal threshold gven a lst s defned as the threshold that mnmzes total error,.e. the sum of type-i errors (the false postves) and type-ii errors (the false negatves). Suh a tranng set an be automatally generated by ) learnng a SVM- MAP-MLC model to sore all ategores ondtoned on eah nput nstane, and 2) fndng the optmal threshold for eah vetor of sored ategores gven an nstane. The LLSF funton s appled n the testng phase, to the system-sored ategores ondtoned on eah test nstane. The resultng threshold s then appled to the orrespondng ranked lst of ategores: the ategores above or at the threshold reeve a yes deson, and the ategores below the threshold reeve a no deson. We modfed the orgnal method by resalng sores of eah nstane to make t n the unt norm. 4. EVALUATION DESIGN 4. Methods for Comparson We onduted ontrolled experments to evaluate SVM-MAP- MLC 3 n omparson wth the followng methods: Bnary-SVM s a standard verson of SVM for one-versus-rest lassfaton, and a ommon baselne n omparatve evaluaton of lassfers (nludng MLC methods) [5] [9] 4. Rank-SVM, the method proposed by [5], s representatve among the model-based methods whh expltly optmze ranked lsts of ategores for MLC 5. IBLR, the nstane-based method reently proposed by [3], s a new method n MLC evaluatons. The method has two versons, one usng knn-based features only (IBLR-ML) and another (IBLR-ML+) usng world-level features n onunton wth knn- based features. We tested both versons and found that IBLR-ML performed onsstently better than IBLR-ML+, whh agrees wth the onluson by the authors of IBLR [3]. We therefore use the results of IBLR-ML for method omparson n the rest of the paper. We used the Weka [7] mplementaton provded by the authors. ML-kNN, the nstane-based method proposed by [25] s another strong baselne n MLC evaluatons [3]. We used the publly avalable Weka mplementaton [7] of ths method n our experments. All the systems produe a ranked lst of ategores gven a test nstane. The ranked lsts an be dretly evaluated usng rank-based metrs, and ndretly evaluated based on bnary lassfaton desons (yes or no on eah ategory) after applyng a threshold to eah ranked lst. The hoe of thresholdng strategy depends on the nature of eah lassfaton method. In Bnary-SVM, for eah ategory, we used the onventonal threshold of zero to obtan a yes/no deson for that ategory. In ML-kNN, IBLR-ML and Rank-SVM we follow the same thresholdng strateges as proposed by the authors. Spefally, for ML-kNN and IBLR we set the threshold to 0.5, and for Rank-SVM we use the lnear least squares ft soluton as the threshold, as proposed n [5]. 4.2 Datasets We used fve datasets 6 n our experments, namely emoton, sene, yeast, Reuters-2578 and Cteseer. These datasets form a representatve sample aross dfferent felds and they vary n tranng-set sze and feature-spae sze. All the datasets exept Cteseer have been used n prevous evaluatons of MLC methods, wth onventonal tran-test splts. We follow suh onventons n order to make our results omparable to the prevously publshed ones. Table summarzes the datasets statsts: Emotons [6] s a mult-label audo dataset, n whh eah nstane s a song, ndexed usng 72 features suh as ampltude, beats per mnute et. The songs have been lassfed under sx moods suh as sad/lonely, relaxng/alm, happy/pleased, amazed/surprsed, angry/aggressve and quet/stll. Sene [] s an mage dataset. The mages are ndexed usng a set of 294 features whh deompose eah mage nto smaller bloks and represent the olor of eah blok. The mages are lassfed based on the senery (Beah, Sunset et) they portray. 3 We used the publly avalable SVM-MAP software at as the ore algorthm. 4 We used the publly avalable SVMlght software pakage at n our experments. 5 We thank the authors of ML-kNN for sharng ther mplementaton of Rank-SVM. 6 The emotons, sene and yeast datasets were obtaned from

5 5 Dataset Name Table. Dataset Charatersts Tranng Sze Testng Sze #Categores Avg #Categores per nstane #Features Emotons Sene Yeast Cteseer Reuters Yeast dataset [5] s a bomedal dataset. Eah nstane s a gene, represented usng a vetor whose features are the mro-array expresson levels under varous ondtons. The genes are lassfed nto 4 dfferent funtonal lasses. Cteseer s a set of researh artles we olleted from the Cteseer web ste 7. Eah artle s ndexed usng the words n ts abstrat as the features, wth a feature-set sze of 4,60. We use the top level of 7 ategores n the Cteseer lassfaton herarhy as the labels n ths dataset, and randomly splt 80% of the orpus nto tranng and the rest as testng nstanes. The dataset wll be made publly avalable along wth the publaton of ths paper. Reuters-2578 s a benhmark dataset n text ategorzaton evaluatons. The nstanes are Reuters news artles durng the perod 987 to 99, and labeled usng 90 topal ategores. We follow the same tran-test splt n [2]. 4.3 Metrs We selet two standard metrs for evaluatng ranked lsts, and two standard metrs for evaluatng lassfaton desons. Mean Average Preson (MAP) [8] s a popular metr n tradtonal IR evaluatons for omparng ranked lsts. It s defned as the average of the per-nstane (or per-ranked-lst) Average preson (AP) over all test nstanes. Let D x,, n be the test set of nstanes, L be the ranked lst of ategores for a spef nstane, r () be the rank of ategory n lst L, and R be the set of ategores relevant to nstane x. MAP s defned as: n MAP ( D) AP( x ) n AP( x ) R R ' R, s. t., r ( ') r ( ) r ( ) Rankng Loss (RankLoss) s a popular metr for omparng MLC methods n rankng ategores [4][3][5][25]. It measures the average number of tmes an rrelevant ategory s ranked above a relevant ategory n a ranked lst as: ( RankLoss( x ), ') R R, s. t. r ( ) r ( ') R R 7 Mro-averaged F s a onventonal metr for evaluatng lassfers n ategory assgnments to test nstanes [0],[22],[23]. The system-made desons on test set D wth respet to a spef ategory C,, m an be dvded nto four groups: True Postves ( TP ), False Postves ( FP ), True Negatves ( TN ) and False Negatves ( FN ), respetvely. The orrespondng evaluaton metrs are defned as: C TP Global Preson P TP FP Global Reall R Mro-averaged C C C TP TP FN F 2 PR P R Maro-averaged F s also a onventonal metr for evaluatng lassfers n ategory assgnments, defned as: Category-spef Preson Category-spef Reall Maro-averaged F TP P TP FP TP R TP FN m C 2P R P R Both mro-averaged F and maro-averaged F are nformatve for method omparson. The former gves the performane on eah nstane an equal weght n omputng the average; the latter gves the performane on eah ategory an equal weght n omputng the average. We hoose the evaluaton measures so as to evaluate the performane of both the rankng algorthms as well as the thresholdng strateges. MAP and RankLoss measure how well a system ranks the ategores; Mro-F and Maro-F evaluate the effetveness of the thresholdng strateges for makng lassfaton desons. 4.4 Expermental Settng Detals All parameters are tuned to optmze MAP and the tunng s done through a fve-fold ross valdaton on the tranng set for eah orpus. The tuned parameters nlude the number of nearest

6 6 neghbors n SVM-MAP-MLC, ML-kNN and IBLR-ML, and the regularzaton parameter n SVM-MAP-MLC, Bnary-SVM, Rank-SVM. For the number of nearest neghbors we tred values from 0 to 00 wth the nrements of 0; for the regularzaton parameter we tred 20 dfferent values between 0 to0. For IBLR, on Cteseer and Reuters-2578, the best hoe for the number of nearest neghbors through ross valdaton was found to be 300 and 90 respetvely. Feature seleton was not performed on any of the datasets for any method. For term weghtng n Cteseer and Reuters douments, we used a onventonal TF-IDF sheme (namely lt ) [22]. On the Emotons dataset, eah feature was resaled to a unt varane representaton sne we observed that the orgnal values of the features are not omparably saled. 5. RESULTS The results of all the systems on the fve datasets are summarzed n Table 2. The methods wth the best sores are hghlghted n bold, and the relatve ranks among the methods on eah dataset n 3 6 a spef metr are provded nsde parentheses. We report the value of -RankLoss nstead of RankLoss ust to make the sores onsstent to eah other,.e., hgher values are better. The total rank of eah system s provded at the bottom lne of the table. 5. Comparatve Analyss Let us all eah lne n Table a ase, whh orresponds to a partular dataset and a spef metr. SVM-MAP-MLC s the best n 8 out of the 20 ases whle Bnary-SVM s the best on the two remanng ases. That s, the proposed approah onsstently outperformed other methods n most ases. Comparng Rank-SVM wth bnary-svm, both are largemargn methods but the former outperformed the latter on 2 out of the 20 ases. These results are onsstent wth the prevously reported evaluaton on the Yeast dataset [5], showng some suess of Rank-SVM by renforng partal-order preferenes among ategores. Comparng Rank-SVM wth SVM-MAP-MLC, on the other hand, the latter outperformed the former n 9 out of the 20 ases although both methods have partal-order preferenes n ther obetve funtons for optmzaton. The advaned Table 2. Results Summary: Eah method s evaluated on fve datasets usng four metrs. The bold-faed numbers ndate the best system on a partular dataset gven the metr; the numbers n parentheses are the ranks of the systems aordngly. Dataset Metr SVM-MAP-MLC ML-kNN Rank-SVM Bnary-SVM IBLR-ML Emotons MAP () (3) (5) (4) (2) -Rankloss () (3) (4) (5) (2) Mro-F () (3) (4) (5) (2) Maro-F () (3) (4) (5) (2) Sene MAP () 0.85 (5) (4) (3) (2) -Rankloss () (5) (2) (3) (4) Mro-F () (3) 0.66 (5) (4) (2) Maro-F () (3) (5) (4) (2) Yeast MAP () (3) (4) (5) (2) -Rankloss () (4) (3) (5) (2) Mro-F () (5) (2) (4) (3) Maro-F () (4) (3) (5) (2) Cteseer MAP () (4) (2) (3) (5) -Rankloss () (3) (2) (5) (4) Mro-F () (4) (2) (3) (5) Maro-F () (3) (2) (4) (5) Reuters MAP (2) (4) (3) () (5) Rankloss () (4) (3) (2) (5) Mro-F (3) (4) (2) () (5) Maro-F 0.545() (5) (3) (2) (4) Rank Total

7 7 Table 3. P-Values n sgned-rank tests for omparng SVM-MAP-MLC wth other methods ( * denotes lower performane of SVM-MAP-MLC). Dataset/Method ML-kNN Rank-SVM Bnary-SVM IBLR-ML Emotons e e Sene 6.975e e-04 Yeast.706e e e Cteseer 9.85e e e e-26 Reuters e * 3.399e-72 performane of SVM-MAP-MLC, evdentally, omes from the use of meta-level features nstead of the onventonal features as that n Rank-SVM. Comparng SVM-MAP-MLC, ML-kNN and IBLR-ML, these methods have one property n ommon: they are ether fully nstane-based or partally nstane-based leveragng knn-based features. IBLR-ML outperformed ML-kNN n 3 out of the 20 ases n our experments; ths s more or less onsstent wth the prevous report by [3] n terms of the relatve performane of the two methods. Nevertheless, both IBLR-ML and ML-kNN underperformed SVM-MAP-MLC n all the 20 ases, showng the advantage of usng of the meta-level features n the learnng-torank framework wth SVM-MAP. Comparng Rank-SVM wth ML-kNN, the former outperformed the latter n 3 out of the 20 ases. The relatve performane of these two methods ompared to eah other s dfferent from the prevously reported evaluaton n [25] where ML-kNN outperformed Rank-SVM on average. In order to larfy ths ssue, we ompared the performane of Rank-SVM wth dfferent values of ts regularzaton parameter whh ontrols the balane between the tranng-set loss and model omplexty. We found Rank-SVM performed suboptmally wth the default parameter settng,, and performed better when ths parameter was tuned through ross valdaton. Our results of Rank-SVM are based on the properly tuned parameters, and ths should explan why Rank-SVM performed stronger than ML-kNN n our experments. Comparng Rank-SVM wth IBLR-ML, eah method outperformed the other n 0 out of the 20 ases. Thus the two methods have omparable performane; both are strong baselnes for method omparson n MLC. 6. SIGNIFICANCE TESTS We used the Wloxon sgned-rank test to ompare the performane of eah method wth that of SVM-MAP-MLC. Sgned-rank s a non-parametr statstal test for parwse omparson of methods, and a better alternatve to the pared t-test when the performane sores are not normally dstrbuted. Due to the spae lmt of the paper, we only present the tests based on the MAP sores of the systems. Eah test nstane s treated as a random event, and the average preson sores of systemgenerated ranked lsts over all test nstanes are used to ompare eah par of systems. The null hypothess s that the system beng ompared wth SVM-MAP-MLC s equally good; the alternatve s that SVM-MAP-MLC s better. The p-values are presented n Table 3 where the null hypothess s reeted wth strong evdene n most ases. That s, the performane dfferene s statstally sgnfant n 7 out of the 20 tests f usng % p-value as the threshold. We dd not use ANOVA tests for mult-set omparson of systems beause suh tests have a normal-dstrbuton assumpton about the data whh s napproprate for the performane sores we use for system omparson. The Fredman test has been reently advoated for omparng lassfers on multple datasets [4] [6], whh does not have the normal assumpton. However, t treats eah dataset as a random event, and requres a relatvely large number of datasets for meanngful testng. It s not reommended to use the Fredman test when the number of datasets s 0 or less Experments wth Feature Subsets In order to analyze the usefulness of dfferent types of meta-level features (lnks), we onduted experments wth the followng ombnatons: usng the sngle lnks n L -norm only, usng the sngle lnks n L2 -norm only, usng the sngle lnks n osne smlarty, and usng all the lnks -- nludng those n L -norm, L2 -norm, osne smlarty and the two average lnks per ategory. Fgure 2 ompares the performane of SVM-MAP-MLC under these ondtons. The dfferent types of lnks have omplementary effets: the sngle lnks n L -norm are more useful n the datasets (Emotons, Sene and Yeast) whose feature- Fgure 2. SVM-MAP-MLC wth dfferent feature sets (Performane n MAP, -RankLoss, Mro-F, Maro-F) 8 Ths s aordng to personal ommunaton wth the author of [4].

8 8 set szes are relatvely small than they are on the datasets (Cteseer and Reuters-2578) whose feature-set szes are large. On the other hand, osne-smlarty based sngle lnks have a dfferent performane pattern; sngle lnks n L2 -norm have omparable performane aross the datasets. Usng all the features together performs the best, showng that SVM-MAP-MLC s able to assgn approprate weghts to dfferent features, and mprove the robustness of ts predtons by makng a ombned use of dfferent features types. 7. CONCLUDING REMARKS In ths paper we produed a new approah for learnng to rank ategores n mult-label lassfaton. By ntrodung meta-level features that effetvely haraterze the one-to-many relatonshp from nstanes to ategores n MLC, and by formulatng the ategory rankng problem as a standard ad-ho retreval problem, our framework allows a broad range of learnng-to-rank retreval algorthms to be deployed for MLC optmzaton wth respet to varous performane metrs. Usng SVM-MAP-MLC as a spef nstantaton of ths framework, and wth ontrolled experments on multple benhmark datasets, the strength of the proposed approah s strongly evdent, as t sgnfantly outperformed all the state-of-the-art methods (Rank-SVM, MLkNN and IBLR-ML) beng evaluated n our experments. We hope ths study provdes useful nsghts nto how to enhane the performane of MLC methods by mprovng the representaton shemes for nstanes, ategores and ther relatonshps, and by reatvely leveragng dmensonalty reduton. A lne of future researh would be to explore our framework wth other learnng-to-rank algorthms, usng dfferent dmensonalty reduton tehnques (suh as SVD or LDA), and for dfferent optmzaton metrs (suh ah NDCG and other types of loss funtons). ACKNOWLEDGEMENTS Ths work s supported, n part, by the Natonal Sene Foundaton (NSF) under grant IIS_ Any opnons, fndngs, onlusons or reommendatons expressed n ths materal are those of the authors and do not neessarly reflet the vews of the sponsors. REFERENCES [] M.R. Boutell, J.Luo, X.Shen and C.M. Brown. Learnng multlabel sene lassfaton. Pattern Reognton 2004, pages [2] Robert H. Creey, Br M. Masand, Stephen J. Smth, Davd L. Waltz: Tradng MIPS and Memory for Knowledge Engneerng. Communatons of the ACM 992, pages (992) [3] W.Cheng and E.Hüllermeer. Combnng Instane-Based Learnng and Logst Regresson for Multlabel Classfaton. Journal of Mahne Learnng Researh 2009, pages [4] J. Demsar. Statstal omparsons for lassfers over multple data sets. Journal of Mahne Learnng Researh 2006, pages -30. [5] A Elsseff and J. Weston. A kernel method for mult-labeled lassfaton. Advanes n Neural Informaton Proessng Systems 2002, pages [6] S Garıa and F. Herrera. An extenson on statstal omparsons of lassfers over multple data sets for all parwse omparsons. Journal of Mahne Learnng Researh 2008, pages [7] M. Hall, E. Frank, G. Holmes, B. Pfahrnger, P. Reutemann and I.H. Wtten; The WEKA Data Mnng Software: An Update. SIGKDD Exploratons, Volume, [8] T. Joahms: A support vetor method for multvarate performane measures. Internatonal Conferene on Mahne Learnng 2005, pages [9] T. Joahms, Makng large-sale SVM Learnng Pratal. Advanes n Kernel Methods - Support Vetor Learnng, MIT- Press, 999. [0] D.D. Lews, R.E. Shapre, J.P. Callan and R.Papka. Tranng algorthms for lnear text lassfers. ACM SIGIR 996, pages [] F. L and Y. Yang. A loss funton analyss for lassfaton methods n text ategorzaton. Internatonal Conferene on Mahne Learnng 2003, pages [2] P L, C Burges, Q Wu, JC Platt, D Koller, Y Snger and S Rowes. MRank: Learnng to rank usng multple lassfaton and gradent boostng. Advanes n Neural Informaton Proessng Systems [3] S Har-Peled, D Roth and D Zmak. Constrant lassfaton: a new approah to multlass lassfaton and rankng. Advanes n Neural Informaton Proessng Systems 2002, pages [4] R.E. Shapre and Y. Snger. BoosTexter: A boostng-based system for text ategorzaton. Journal of Mahne learnng Researh 2000, pages [5] R.E. Shapre and Y. Snger. Improved boostng algorthms usng onfdene-rated predtons. Journal of Mahne learnng Researh 999, pages [6] K. Trohds,G. Tsoumakas, G. Kallrs, I. Vlahavas. Multlabel lassfaton of mus nto emotons Internatonal Conferene on Mus Informaton Retreval [7] V. Vapnk, The nature of statstal learnng theory, Sprnger verlag, New York, [8] E.Voorhees Overvew of TREC 2002, NIST Speal Publaton SP [9] Jun Xu and Hang L. AdaRank: a boostng algorthm for nformaton retreval. ACM SIGIR 2007, pages [20] Y. Yang: Expert Network: Effetve and Effent Learnng from Human Desons n Text Categorzaton and Retreval. ACM SIGIR 994, pages [2] Y. Yang. A Study of thresholdng strateges for text ategorzaton. ACM SIGIR 200, pages [22] Y. Yang. An evaluaton of statstal approahes for text lassfaton. Informaton Retreval 999. Vol, No. ½, pages [23] Y. Yang and J.O. Pederson. A omparatve study of features seleton n text ategorzaton. Internatonal Conferene on Mahne Learnng 997, pages [24] Ysong Yue, T. Fnley, F. Radlnsk and T. Joahms. A Support Vetor Method for Optmzng Average Preson. ACM SIGIR Conferene [25] ML Zhang and ZH Zhou. ML-kNN: A lazy learnng approah to mult-label learnng. Pattern Reognton 2007, pages