MLKD s Participation at the CLEF 2011 Photo Annotation and Concept-Based Retrieval Tasks

Transcription

1 MLKD s Partcpaton at the CLEF 2011 Photo Annotaton and Concept-Based Retreval Tasks Eleftheros Spyromtros-Xoufs, Konstantnos Sechds, Grgoros Tsoumakas, and Ioanns Vlahavas Dept of Informatcs Arstotle Unversty of Thessalonk Thessalonk 54124, Greece {espyrom,sechds,greg,vlahavas}@csd.auth.gr Abstract. We partcpated both n the photo annotaton and conceptbased retreval tasks of CLEF For the annotaton task we developed vsual, textual and mult-modal approaches usng mult-label learnng algorthms from the Mulan open source lbrary. For the vsual model we employed the ColorDescrptor software to extract vsual features from the mages usng 7 descrptors and 2 detectors. For each combnaton of descrptor and detector a mult-label model s bult usng the Bnary Relevance approach coupled wth Random Forests as the base classfer. For the textual models we used the boolean bag-of-words representaton, and appled stemmng, stop words removal, and feature selecton usng the ch-squared-max method. The mult-label learnng algorthm that yelded the best results n ths case was Ensemble of Classfer Chans usng Random Forests as base classfer. Our mult-modal approach was based on a herarchcal late-fuson scheme. For the concept based retreval task we developed two dfferent approaches. The frst one s based on the concept relevance scores produced by the system we developed for the annotaton task. It s a manual approach, because for each topc we manually selected the topcs and manually set the strength of ther contrbuton to the fnal rankng produced by a general formula that combnes topc relevance scores. The second approach s based solely on the sample mages provded for each query and s therefore fully automated. In ths approach only the textual nformaton was used n a query-by-example framework. 1 Introducton ImageCLEF s the cross-language mage retreval track run annually snce 2003 as part of the Cross Language Evaluaton Forum (CLEF) 1. Ths paper documents the partcpaton of the Machne Learnng and Knowledge Dscovery (MLKD) group of the Department of Informatcs of the Arstotle Unversty of Thessalonk at the photo annotaton task (also called vsual concept detecton and annotaton task) of ImageCLEF

2 Ths year, the photo annotaton task conssted of two subtasks. An annotaton task, smlar to that of ImageCLEF 2010, and a new concept-based retreval task. Data for both tasks come from the MIRFLICKR-1M mage dataset [1], whch apart from the mage fles contans Flckr user tags and Exchangeable Image Fle Format (Exf) nformaton. More nformaton about the exact setup of the data can be found n [4]. In the annotaton task, partcpants are asked to annotate a test set of 10,000 mages wth 99 vsual concepts. An annotated tranng set of 8,000 mages s provded. Ths mult-label learnng task [8] can be solved n three dfferent ways accordng to the type of nformaton used for learnng: 1) vsual (the mage fles), 2) textual (Flckr user tags), 3) mult-modal (vsual and textual nformaton). We developed vsual, textual and mult-modal approaches for ths task usng mult-label learnng algorthms from the Mulan open source lbrary [9]. In ths task, the relatve performance of our textual models was qute good, but that of our vsual models was bad (our group does not have expertse on computer vson), leadng to an average mult-modal (and overall) performance. In the concept-based retreval task, partcpants were gven 40 topcs consstng of logcal connectons between the 99 concepts of the photo annotaton task, such as fnd all mages that depct a small group of persons n a landscape scenery showng trees and a rver on a sunny day, along wth 2 to 5 examples mages of each topc from the tranng set of the annotaton task. Partcpants were asked to submt (up to) the 1,000 most photos for each topc n ranked order from a set of 200,000 unannotated mages. Ths task can be solved by manual constructon of the query out of the narratve of the topcs, followed by automatc retreval of mages, or by a fully automated process. We developed a manual approach that explots the mult-label models traned n the annotaton task and a fully automated query-by-example approach based on the tags of the example mages. In ths task, both our manual and automated approaches ranked 1st n all evaluaton measures by a large margn. The rest of ths paper s organzed as follows. Sectons 2 and 3 descrbe our approaches to the annotaton task and concept-based retreval task respectvely. Secton 4 presents the results of our runs for both tasks. Secton 5 concludes our work and poses future research drectons. 2 Annotaton Task Ths secton presents the vsual, textual and mult-modal approaches that we developed for the automatc photo annotaton task. There were two (eventually three) evaluaton measures to consder for ths task: a) mean nterpolated average precson (MIAP), b) example-based F-measure (F-ex), c) semantc R-precson (SR-Precson). In order to optmze a learnng approach based on each of the ntal two evaluaton measures and type of nformaton, sx models should be bult. However, there were only fve runs allowed for ths task. We therefore decded to perform model selecton based on the wdely-used mean average precson (MAP) measure for all types of nformaton. In partcular, MAP was estmated

3 usng an nternal 3-fold cross-valdaton on the 8,000 tranng mages. Our multmodal approach was submtted n three dfferent varatons to reach the total number of fve submssons. 2.1 Automatc Annotaton wth Vsual Informaton We here descrbe the approach that we followed n order to learn mult-label models usng the vsual nformaton of the mages. The flowchart of ths approach s shown n Fg. 1. x p( c x) j j Fg. 1. Automatc annotaton usng vsual nformaton. As our group does not have expertse n computer vson, we largely followed the color-descrptor extracton approach descrbed n [6,7] and used the accompanyng software tool 2 for extractng vsual features from the mages. Harrs-Laplace and Dense Samplng were used as pont detecton strateges. Furthermore seven dfferent descrptors were used: SIFT, HSV-SIFT, HueSIFT, OpponentSIFT, C-SIFT, rgsift and RGB-SIFT. For each one of the 14 combnatons of pont detecton strategy and descrptor, a dfferent codebook was created n order to obtan a fxed length representaton for all mages. Ths s also known as the bag-of-words approach. The k-means clusterng algorthm was appled to 250,000 randomly sampled ponts from the tranng set, wth the codebook sze (k) fxed to 4096 words. Fnally, we employed hard assgnment of ponts to clusters. Usng these 4,096-dmensonal vector representatons along wth the ground truth annotatons gven for the tranng mages we bult 14 mult-label tranng datasets. After expermentng wth varous mult-label learnng algorthms we found that the smple Bnary Relevance (BR) approach coupled wth Random Forests as the base classfer (number of trees = 150, number of features = 40) yelded the best results. In order to deal wth the mbalance n the number of postve and negatve examples of each label we used nstance weghtng. The weght of the examples of the mnorty class was set to (mn+maj)/mn and the weght of the examples of the majorty class was set to (mn + maj)/maj, where mn s the number of examples of the mnorty class and maj the number of examples of the majorty class. We also expermented wth sub-samplng, but the results were worse than nstance weghtng. 2 Avalable from

4 Our approach concludes wth a late fuson scheme that averages the output of the 14 dfferent mult-label models that we bult. 2.2 Automatc Annotaton wth Flckr User Tags We here descrbe the approach that we followed n order to learn mult-label models usng the tags assgned to mages by Flckr users. The flowchart of ths approach s shown n Fg. 2. Preprocessng Tags of Image Stemmer Stop Words 27,323 features Feature Selecton 4,000 features x Mult-label learnng algorthm p( c x) j j Fg. 2. Automatc annotaton usng Flckr user tags. An ntal vocabulary was constructed by takng the unon of the tag sets of all mages n the tranng set. We then appled stemmng to ths vocabulary and removed stop words. Ths led to a vocabulary of approxmately stems. The use of stemmng mproved the results, despte that some of the tags were not n the Englsh language and that we used an Englsh stemmer. We further appled feature selecton n order to remove r or redundant features and mprove effcency. In partcular, we used the χ 2 max crteron [3] to score the stems and selected the top 4000 stems, after expermentng wth a varety of szes (500, 1000, 2000, 3000, 4000, 5000, 6000 and 7000). The mult-label learnng algorthm that was found to yeld the best results n ths case was Ensemble of Classfer Chans (ECC) [5] usng Random Forests as base classfer. ECC was run wth 15 classfer chans and Random Forests wth 10 decson trees, whle all other parameters were left to ther default value. The approach that we followed to deal wth class mbalance n the case of vsual nformaton (see the prevous subsecton), was followed n ths case too. 2.3 Automatc Annotaton wth a Mult-Modal Approach Our mult-modal approach s based on a late fuson scheme that combnes the output of the 14 vsual models and the sngle textual model. The combnaton s not an average of these 15 models, because n that case the vsual models would domnate the fnal scores. Instead, we follow a herarchcal combnaton scheme. We separately average the 7 vsual models of each pont estmator and then combne the output of the textual model, the Harrs-Laplace average and the Dense Samplng average, as depcted n Fg. 3. The motvaton for ths scheme was the three dfferent vews of the mages that exsted n the data (Harrs-Laplace, Dense Samplng, user tags) as explaned n the followng two paragraphs.

5 Harrs Laplace Ensemble Model Sngle Model usng Harrs Laplace & SIFT Codebook Sngle Model usng Harrs Laplace & HSV-SIFT Codebook Sngle Model usng Harrs Laplace & HueSIFT Codebook Sngle Model usng Harrs Laplace & OpponentSIFT Codebook Averagng pc ( x) j j Sngle Model usng Harrs Laplace & C-SIFT Codebook Sngle Model usng Harrs Laplace & rgsift Codebook Sngle Model usng Harrs Laplace & RGB-SIFT Codebook Dense Samplng Ensemble Model Sngle Model usng Dense Samplng & SIFT Codebook Sngle Model usng Dense Samplng & HSV-SIFT Codebook x Image Sngle Model usng Dense Samplng & HueSIFT Codebook Sngle Model usng Dense Samplng & OpponentSIFT Codebook Averagng pc ( x) j j Averagng/ Arbtrator pc ( x) j j Sngle Model usng Dense Samplng & C-SIFT Codebook Sngle Model usng Dense Samplng& rgsift Codebook Sngle Model usng Dense Samplng& RGB-SIFT Codebook Flckr users tags Model pc ( x) j j Fg. 3. Automatc annotaton wth a mult-modal approach We can dscern two man categores of concepts n photo annotaton: objects and scenes. For objects, Harrs-Laplace performs better because t gnores the homogeneous areas, whle for scenes, Dense Samplng performs better [6]. For example, two of the concepts where Dense Samplng acheves much hgher Average Precson (AP) from Harrs-Laplace are Nght and Macro, whch are abstract, whle the nverse s happenng n concepts Fsh and Shp, whch correspond to thngs (organsms, objects) of partcular shape. Furthermore, we observe that the vsual approach performs better n concepts, such as Sky, whch for some reason (e.g. lack of user nterest for retreval by ths concept) do not get tagged. On the other hand the textual approach performs much better when t has to predct concepts, such as Horse, Insect, Dog and Baby that typcally get tagged by users. Table 1 shows the average precson for 10 concepts, half of whch sut much better the textual models and half the vsual models. Two varatons of ths scheme were developed, dfferng n how the output of the three dfferent vews s combned. The frst one, named Mult-Modal-Avg, used an averagng operator, smlarly to the one used at the lower levels of the herarchy. The second one, named Mult-Modal-MaxAP, used an arbtrator functon to select the best one out of the three outputs for each concept, accordng to nternal evaluaton results n terms of average precson. Our thrd mult-modal submsson, named Mult-Modal-MaxAP-RGBSIFT, was a prelmnary verson of Mult-Modal-MaxAP, where only the RGBSIFT descrptor was used.

6 Table 1. Average precson for 10 concepts, half of whch sut much better the textual models and half the vsual models. Concept Textual Vsual Concept Textual Vsual Arplane Trees Horse Clouds Brd Sky Insect Overexposed Dog Bg Group Thresholdng The mult-label learners used n ths work provde us wth a confdence score for each concept. Ths s fne for an evaluaton wth MIAP and SR-Precson, but does not suffce for an evaluaton wth example-based F-measure, whch requres a bpartton of the concepts nto and r ones. Ths s a typcal ssue n mult-label learnng, whch s dealt wth a thresholdng process [2]. We used the thresholdng method descrbed n [5], whch apples a common threshold across all concepts and provdes a close approxmaton of the label cardnalty (LC) of the tranng set to the predctons made on the test set. The threshold s calculated usng the followng formula: t = argmn {t 0.00,0.05,...,1.00} LC(D tran ) LC(H t (D test )) (1) where D tran s the tranng set and H t s a classfer whch has made predctons on a test set D test under threshold t. 3 Concept-Based Retreval Task We developed two dfferent approaches for the concept-based retreval task. The frst one s based on the concept relevance scores produced by the system we developed for the annotaton task. It s a manual approach, because for each topc we manually selected the topcs and manually set the strength of ther contrbuton to the fnal rankng produced by a general formula that combnes topc relevance scores. The second one s based solely on the sample mages provded for each query and s therefore fully automated. 3.1 Manual Approach Let I = 1,..., 200, 000 be the collecton of mages, Q = 1,..., 40 the set of topcs and C = 1,..., 99 the set of concepts. We frst apply our automated mage annotaton system to each mage I and obtan a correspondng 99- dmensonal vector S = [s 1, s2,..., s99 ] wth the relevance scores of ths mage to each one of the 99 concepts. For effcency reasons, we used smplfed versons of

7 our vsual approach, takng nto account only models produced wth the RGB- SIFT descrptor, whch has been found n the past to provde better results compared to other sngle color descrptors [7]. Then, based on the descrpton of each of the 40 queres, we manually select a number of concepts that we consder related to the query, ether postvely or negatvely. Formally, for topc q Q let P q C denote the set of concepts that are postvely related to q and N q C the set of concepts that are negatvely related to q, P q N q =. For each concept c n P q N q, we further defne a real valued parameter m c q 1 denotng the strength of relevance of concept c to q. The larger ths value, the stronger the nfluence of concept c to the fnal relevance score. For each topc q and mage, the scores of the concepts are combned usng (2). S q, = (s c ) m c q (1 s c ) m c q (2) c P q c N q Fnally, for each topc, we arrange the mages n descendng order accordng to the overall relevance score and we retreve a fxed number of mages (n our submssons we retreved 250 and 1,000 mages). Note that for each topc, the selecton of related concepts and the settng of values for the m c q parameters was done usng a tral-and-error approach nvolvng careful vsual examnaton of the top 10 retreved mages, as well as more relaxed vsual examnaton of the top 100 retreved mages. Two examples of topcs and correspondng combnaton of scores follow. Topc 5: rder on horse. Here we lke to fnd photos of rders on a horse. So no sculptures or pantngs are. The rder and horse can be also only n parts on the photo. It s mportant that the person s rdng a horse and not standng next to t. Based on the descrpton of ths topc and expermentaton, we concluded that concepts 75 (Horse) and 8 (Sports) are postvely related (rder on horse), whle concept 63 (Vsual Arts) s negatvely related (no sculptures or pantngs). We therefore set P 5 = {8, 75}, N 5 = {63}. All concepts were set to equal strength for ths topc: m 8,5 = m 63,5 = m 75,5 = 1. Topc 24: funny baby. We lke to fnd photos of babes lookng funny. The baby should be n the man focus of the photo and be the reason why the photo looks funny. Photos presentng funny thngs that are not related to the baby are not. Based on the descrpton of ths topc and expermentaton, we concluded that concepts 86 (Baby), 92 (Funny) and 32 (Portrat) are postvely related. We therefore set P 24 = {32, 86, 92}, N 24 =. Based on expermentaton the concept Funny was gven twce the strength of the other concepts, we set m 32,24 = m 86,24 = 1 and m 92,24 = 2. For some topcs, nstead of explctly usng the score of a group of nterrelated concepts we consdered ntroducng a vrtual concept wth score equal to the maxmum of ths group of concepts. Ths slght adaptaton of the general rule of (2), enhances ts representaton capabltes. The followng example clarfes ths adaptaton.

8 Topc 32: underexposed photos of anmals. We lke to fnd photos of anmals that are underexposed. Photos wth normal llumnaton are not. The anmal(s) should be more or less n the man focus of the mage. Based on the descrpton of ths topc and expermentaton, we concluded that concepts 44 (Anmals), 34 (Underexposed), 72 (Dog), 73 (Cat), 74 (Brd), 75 (Horse), 76 (Fsh) and 77 (Insect) are postvely related, whle concept 35 (Neutral Illumnaton) s negatvely related. The sx last specfc anmal concepts were grouped nto a vrtual concept, say concept 1001, wth score, the maxmum of the scores of these sx concepts. We then set P 32 = {34, 44, 1001}, N 32 = {35} and m 34,32 = m 44,32 = m 1001,32 = m 35,32 = 1. Fgure 4 shows the top 10 retreved mages for topcs 5, 24 and 32, along wth the Precson@10 for these topcs. 3.2 Automated Approach Apart from the narratve descrpton, each topc of the concept-based retreval task was accompaned by a set of 2 to 5 mages from the tranng set whch could be consdered for the topc. Usng these examples mages as queres we developed a Query by Example approach to fnd the most mages n the retreval set. The representaton followed the bag-of-words model and was based on the Flckr user tags assgned to each mage. To generate the feature vectors, we appled the same method as the one used for the annotaton task. Thus, each mage was represented as a 4000-dmensonal feature vector where each feature corresponds to a tag from the tranng set whch was selected by the feature selecton method. A value of 1/0 denotes the presence/absence of the tag n the tags accompanyng an mage. To measure the smlarty between the vectors representng two mages we used the Jaccard smlarty coeffcent whch s defned as the total number of attrbutes where two vectors A and B both have a value of 1 dvded by the the total number of attrbutes where ether A or B have a value of 1. Snce more than one mages where gven as examples for each topc, we added ther feature vectors n order to form a sngle query vector. Ths approach was found to work well n comparson to other approaches, such as takng only one of the example mages as query or measurng the smlarty between a retreval mage and each example mage separately and then returnng the mages from the retreval set wth the largest smlarty score to any of the queres. We attrbute ths to the fact that by addng the feature vectors, a better representaton of the topc of nterest was created whch could not be possble f only one mage (wth possbly nosy or very few tags) was consdered. As n the manual approach, we submtted two runs, one returnng the 250 and one the 1000 most smlar mages from the retreval set (n descendng smlarty order). Fgure 5 shows the top 10 retreved mages, along wth the Precson@10 for the followng topcs: Topc 10: sngle person playng a muscal nstrument. We lke to fnd pctures (no pantngs) of a person playng a muscal nstrument. The person

9 can be on stage, off stage, nsde or outsde, sttng or standng, but should be alone on the photo. It s enough f not the whole person or nstrument s shown as long as the person and the nstrument are clearly recognzable. Topc 12: snowy wnter landscaper. We lke to fnd pctures (photos or drawngs) of whte wnter landscapes wth trees. The landscape should not contan human-made objects e.g. houses, cars and persons. Only snow on the top of a mountan s not, the landscape has to be fully covered n (at least lght) snow. Topc 30: cute toys arranged to a stll-lfe. We lke to fnd photos of toys arranged to a stll-lfe. These toys should look cute n the arrangement. Smple photos of a collecton of toys e.g. n a shop are not. We see that the 10 retreved mages for topc 30 are better than those of topcs 12 and 10. Ths can be explaned by notcng that topc 12 s a dffcult one, whle the tags of the example mages for topc 10 are not very descrptve/nformatve. 4 Results We here brefly present our results, as well as our relatve performance compared to other groups and submssons. Results for all groups, as well as more detals on the data setup and evaluaton measures can be found n [4]. 4.1 Annotaton Task The offcal results of our runs are llustrated n Table 2. We notce that n terms of MIAP, the textual model s slghtly better than the vsual, whle for the other two measures, the vsual model s much better than the textual. Among the mult-modal varatons, we notce that averagng works better than arbtratng, and as expected usng all descrptors s better than usng just the RGB- SIFT one. In addton, we notce that the mult-modal approach sgnfcantly mproves over the MIAP of the vsual and textual approaches, whle t slghtly decreases/ncreases the performance of the vsual model n the two examplebased measures. Ths may partly be due to the fact that we performed model selecton based on MAP. Table 2. Offcal results of the MLKD team n the annotaton task. Run Name MIAP F-measure SR-Precson Textual Vsual Mult-Modal-Avg Mult-Modal-MaxAP-RGBSIFT Mult-Modal-MaxAP

10 Table 3 shows the rank of our best result compared to the best results of other groups and compared to all submssons. We dd qute good n terms of textual nformaton, but qute bad n terms of vsual nformaton, leadng to an overall average performance. Lack of computer vson expertse n our group may be a reason for not beng able to get results out of the vsual nformaton. Among the three evaluaton measures, we notce that overall we dd better n terms of MIAP, slghtly worse n terms of F-measure, and even worse n terms of SR-Precson. The fact that model selecton was performed based on MAP defntely played a role for ths result. Table 3. Rank of our best result compared to the best results of other teams and compared to all submssons n the annotaton task. Approach Team Rank Submsson Rank MIAP F-Measure SR-Prec MIAP F-Measure SR-Prec Vsual 9th/15 5th/15 9th/15 25th/46 12th/46 17th/46 Textual 3rd/7 2nd/7 3rd/7 3rd/8 2nd/8 4th/8 Mult-modal 5th/10 5th/10 7th/10 9th/25 7th/25 15th/25 All 5th/18 7th/18 10th/18 9th/79 19th/79 31st/ Concept-Based Retreval Task In ths task, partcpatng systems were evaluated usng the followng measures: Mean Average Precson (MAP), Precson@10, Precson@20, Precson@100 and R-Precson. The offcal results of our runs are llustrated n Table 4. We frst notce that the frst 5 runs, whch retreved 1000 mages, lead to better results n terms of MAP and R-Precson compared to the last 5 runs, whch retreved 250 mages. Obvously, n terms of Precson@10, Precson@20 and Precson@100, the results are equal. Among the manual runs, we notce that the vsual models perform qute bad. We hypothesze that a lot of concepts that favor textual rather than vsual models, as dscussed n Sect. 2, appear n most of the topcs. The textual and mult-modal models perform best, wth the Mult-Modal-Avg model havng the best result n 3 out of the 5 measures. The automated approach performs slghtly better than the vsual model of the manual approach, but stll much worse than the textual and mult-modal manual approaches. As expected, the knowledge that s provded by a human can clearly lead to better results compared to a fully automated process. However, ths s not true across all topcs, as can be seen n Table 5, whch compares the results of the best automated and manual approach for each ndvdual topc. We can see there that the automated approach performs better on 9 topcs, whle the manual on 31.

11 Table 4. Offcal results of the MLKD team n the concept-based retreval task. Run Name MAP P@10 P@20 P@100 R-Prec Manual-Vsual-RGBSIFT Automated-Textual Manual-Textual Manual-Mult-Modal-Avg-RGBSIFT Manual-Mult-Modal-MaxAP-RGBSIFT Manual-Vsual-RGBSIFT Automated-Textual Manual-Textual Manual-Mult-Modal-Avg-RGBSIFT Manual-Mult-Modal-MaxAP-RGBSIFT Table 5. Comparson of AP for each topc between automated and manual approach Topc Automated Manual Topc Automated Manual MAP Table 6 shows the rank of our best result compared to the best results of other groups and compared to all submssons. Both our manual and our automated approach ranked 1st n all evaluaton measures.

12 Table 6. Rank of our best result compared to the best results of other teams and compared to all submssons n the annotaton task. Team Rank Submsson Rank Confguratons MAP P@10 P@20 P@100 R-Prec MAP P@10 P@20 P@100 R-Prec Automated 1st/4 1st/4 1st/4 1st/4 1st/4 1st/16 1st/16 1st/16 1st/16 1st/16 Manual 1st/3 1st/3 1st/3 1st/3 1st/3 1st/15 1st/15 1st/15 1st/15 1st/15 All 1st/4 1st/4 1st/4 1st/4 1st/4 1st/31 1st/31 1st/31 1st/31 1st/31 5 Conclusons and Future Work Our partcpaton to the very nterestng photo annotaton and concept-based retreval tasks of CLEF 2011, led to a couple of nterestng conclusons. Frst of all, we found out that we need the collaboraton of a computer vson/mage processng group to acheve better results. In terms of mult-label learnng algorthms, we notced that bnary approaches worked qute well, especally when coupled wth the strong Random Forests algorthm and class mbalance ssues are taken nto account. We also reached to the concluson, that we should have performed model selecton separately for each evaluaton measure. We therefore suggest that n future versons of the annotaton task, the allowed number of submssons should be equal to the number of evaluaton measures multpled by the number of nformaton types, so that there s space n the offcal results for models wth all knds of nformaton. There s a lot of room for mprovements n the future, both n the annotaton and the very nterestng concept-based retreval task. In terms of textual nformaton, we ntend to nvestgate the translaton of non-englsh tags. We would also lke to nvestgate other herarchcal late fuson schemes, such as an addtonal averagng step for the two dfferent vsual modaltes (Harrs-Laplace, Dense Samplng) and more advanced arbtraton technques. Other thresholdng approaches for obtanng bparttons s another nterestng drecton for future study. Acknowledgments We would lke to acknowledge the student travel support from EU FP7 under grant agreement no (PetaMeda Network of Excellence). References 1. Huskes, M.J., Thomee, B., Lew, M.S.: New trends and deas n vsual concept detecton: The mr flckr retreval evaluaton ntatve. In: MIR 10: Proceedngs of the 2010 ACM Internatonal Conference on Multmeda Informaton Retreval. pp ACM, New York, NY, USA (2010)

13 2. Ioannou, M., Sakkas, G., Tsoumakas, G., Vlahavas, I.: Obtanng bparttons from score vectors for mult-label classfcaton. Tools wth Artfcal Intellgence, IEEE Internatonal Conference on 1, (2010) 3. Lews, D.D., Yang, Y., Rose, T.G., L, F.: Rcv1: A new benchmark collecton for text categorzaton research. J. Mach. Learn. Res. 5, (2004) 4. Nowak, S., Nagel, K., Lebetrau, J.: The clef 2011 photo annotaton and conceptbased retreval tasks. In: Workng Notes of CLEF 2011 (2011) 5. Read, J., Pfahrnger, B., Holmes, G., Frank, E.: Classfer chans for mult-label classfcaton. In: Proc. 20th European Conference on Machne Learnng (ECML 2009). pp (2009) 6. van de Sande, K.E.A., Gevers, T.: Unversty of Amsterdam at the Vsual Concept Detecton and Annotaton Tasks, The Informaton Retreval Seres, vol. 32: ImageCLEF, chap. 18, pp Sprnger (2010) 7. van de Sande, K.E.A., Gevers, T., Snoek, C.G.M.: Evaluatng color descrptors for object and scene recognton. IEEE Transactons on Pattern Analyss and Machne Intellgence 32(9), (2010) 8. Tsoumakas, G., Kataks, I., Vlahavas, I.: Mnng mult-label data. In: Mamon, O., Rokach, L. (eds.) Data Mnng and Knowledge Dscovery Handbook, chap. 34, pp Sprnger, 2nd edn. (2010) 9. Tsoumakas, G., Spyromtros-Xoufs, E., Vlcek, J., Vlahavas, I.: Mulan: A java lbrary for mult-label learnng. Journal of Machne Learnng Research (JMLR) 12, (July )

14 Topc 5 P@10 = 0.8 Topc 24 P@10 = 0.2 Topc 32 P@10 = 0.5 r r r r r r r r r r r r r r r Fg. 4. Retreved mages for topcs 5, 24 and 32 usng manual retreval. Images come from the MIRFLICKR-1M mage dataset [1].

15 Topc 10 = 0.3 Topc 12 = 0.2 Topc 30 = 1.0 r r r r r r r r r r r r r r r Fg. 5. Retreved mages for topcs 10, 12 and 30 usng automated retreval. Images come from the MIRFLICKR-1M mage dataset [1].