Efficient Large-Scale Image Annotation by Probabilistic Collaborative Multi-Label Propagation

Transcription

1 Effcent Large-Scale Image Annotaton by Probablstc Collaboratve Mult-Label Propagaton Xangyu Chen, Yadong Mu, Shucheng Yan, and Tat-Seng Chua NUS Graduate School for Integratve Scences and Engneerng, School of Computng, Department of Electrcal and Computer Engneerng Natonal Unversty of Sngapore, Sngapore {chenxangyu, elemy, eleyans, ABSTRACT Annotatng large-scale mage corpus requres huge amount of human efforts and s thus generally unaffordable, whch drectly motvates recent development of sem-supervsed or actve annotaton methods. In ths paper we revst ths notorously challengng problem and develop a novel multlabel propagaton scheme, whereby both the effcacy and accuracy of large-scale mage annotaton are further enhanced. Our nvestgaton starts from a survey of prevous graph propagaton based annotaton approaches, wheren we analyze ther man drawbacks when scalng up to large-scale datasets and handlng mult-label settng. Our proposed scheme outperforms the state-of-the-art algorthms by makng the followng contrbutons. 1) Unlke prevous approaches that propagate over ndvdual label ndependently, our proposed large-scale mult-label propagaton LSMP) scheme encodes the tag nformaton of an mage as a unt label confdence vector, whch naturally mposes nter-label constrants and manpulates labels nteractvely. It then utlzes the probablstc Kullback-Lebler dvergence for problem formulaton on mult-label propagaton. 2) We perform the mult-label propagaton on the so-called hashng-based l 1 -graph, whch s effcently derved wth Localty Senstve Hashng approach followed by sparse l 1-graph constructon wthn the ndvdual hashng buckets. 3) An effcent and convergency provable teratve procedure s presented for problem optmzaton. Extensve experments on NUS- WIDE dataset both lte verson wth 56k mages and full verson wth 270k mages) well valdate the effectveness and scalablty of the proposed approach. Categores and Subject Descrptors H.3.1 [Informaton Storage and Retreval]: Analyss and Indexng-ndexng methods General Terms Algorthms, Performance, Expermentaton Content Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, to republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. MM 10, October 25 29, 2010, Frenze, Italy. Copyrght 2010 ACM /10/10...$ Keywords Image Annotaton, Collaboratve Mult-label Propagaton 1. INTRODUCTION For many applcatons lke mage annotaton, especally n large-scale settng, annotatng tranng data s often very tme-consumng and tedous. Sem-supervsed learnng SSL) lends tself as an effectve technque, through whch users only need to annotate a small amount of mage data, and other unlabeled data can work together wth these labeled data for learnng and nference. In ths paper we are partcularly nterested n effcent graph-based mult-label propagaton n large-scale settng. It s known that graph s a natural representaton for label propagaton, wheren each vertex corresponds to a unque mage and any edge connectng the two vertces ndcates certan relatons between the mages. Unlke generatve modelng methods, graph modelng focuses on nonparametrc local structure dscovery, rather than a pror probablstc assumptons. For the transducton task on partally labeled data known as sem-supervsed learnng n lterature), graph-based methods usually demonstrate the state-of-the-art performance than other SSL algorthms [24]. Generally, there are three crucal subtasks n graph-based algorthms: 1) graph constructon; 2) the choce of loss functon; and 3) the choce of regularzaton term. As argued n [23], graph constructon s supposed to be more domnatng than the other two factors n terms of performance. Unfortunately, t s also the area that s most nadequately studed. In Secton 2.2, we propose a novel hashng-based scheme for effcent large-scale graph constructon. The solutons to the last two subtasks may affect the fnal accuracy as well as the proper optmzaton strategy thus the convergence speed). As reported n [8], early work on semsupervsed learnng can only handle unlabeled samples. Consequently, a large number of recent endeavors has been devoted to the scalablty to large-scale datasets. Several recent large scale algorthms e.g. [11, 6]) plug graph Laplacan based regularzers nto transductve support vector machnes TSVM) to obtan better transducton capablty. The work n [11] solves a graph transducton problem wth 650, 000 samples. The whole objectve functon s optmzed va the stochastc gradent descent. Whle the method n [6] suggests a tranng method usng the concave-convex procedure CCCP), whch brngs scalablty mprovement on large-scale dataset. The work n [19]

2 Fgure 1: Flowchart of our proposed scheme for mult-label propagaton. Step-0 and step-1 are the proposed hashng-based l 1 -graph constructon scheme, whch perform neghborhood selecton and weght computaton respectvely; Step-2 s the probablstc mult-label propagaton based Kullback-Lebler dvergence. solves the largest graph-based problem to date, where there are about 900, 000 samples ncludng both labeled and unlabeled data). By usng a sparsfed manfold regularzer and formulatng as a center-constraned mnmum enclosng ball problem, ths method produces sparse solutons wth low tmeandspacecomplextesandcanbeeffcentlysolvedby the core vector machne CVM). The semnal work n [17] s most smlar to our work n ths paper. Unlke prevous approaches, ths method models the mult-class label confdence vector as a probablstc dstrbuton, and utlzes the Kullback-Lebler KL) dvergence to gauge the parwse dscrepancy. The underlyng phlosophy s that such soft regularzaton term wll be less vulnerable to nosy annotaton or outlers. Here we adopt the same representaton and dstance measure, yet n a dfferent scenaro.e. mult-label mage annotaton), thus demandng new soluton. Several algorthms were recently proposed to explot the nter-relatons among dfferent labels [12]. For example, Q et al. [15] proposed a unfed Correlatve Mult-Label CML) framework to smultaneously classfy labels and model correlatons between them. Chen et al. [4] formulated ths problem as a sylvester equaton, whch s smlar to [22]. They frst constructed two graphs at the sample level and category level assocated wth a quadratc energy functon respectvely, and then obtan the labels of the unlabeled mages by mnmzng the combnaton of the two energy functons. Lu et. al. [13] utlzed constraned nonnegatve matrx factorzaton CNMF) to optmze the consstency between mage smlarty and label smlarty. Unfortunately, most of the aforementoned algorthms are of hgh complexty and unsutable to scale up to the large-scale datasets. Most exstng work n the lne of graph-based label propagaton suffer or partally suffer) from these dsadvantages: 1) they consder each tag ndependently when handlng multlabel propagaton problem, 2) the derved labels for one mage are not rankable, and 3) the graph constructon process s tme-consumng. And most recent large-scale algorthms focus on the sngle label case, but the scalablty to large number of labels s unclear. To address the above ssues, we proposed a new large-scale graph-based mult-label propagaton approach by mnmzng the Kullback-Lebler dvergence of the mage-wse label confdence vector and ts propagated verson va the so-called hashng-based l 1- graph, whch s effcently derved wth Localty Senstve Hashng approach followed by sparse l 1-graph constructon wthn the ndvdual hashng buckets. Fnally, an effcent and convergency provable teratve procedure s presented for problem optmzaton. The major contrbutons of our proposed scheme can be summarzed as follows: We propose a probablstc collaboratve mult-label propagaton formulaton for large-scale mage annotaton, whch s founded on Kullback-Lebler dvergence based label smlarty measurement and scalable l 1 - graph constructon. We also propose a novel hashng-based scheme for effcent large-scale graph constructon. Localty senstve hashng [10, 1, 14] s utlzed to speed up the canddate selecton of smlar neghbors for one mage, whch makes the l 1 -graph constructon process scalable.

3 The remander of ths paper s organzed as follows. In Secton 2, we elaborate on the proposed probablstc collaboratve mult-label propagaton LSMP) algorthm. Secton 3 presents analyss on algorthmc complexty and convergence propertes. Expermental results on both mddlescale and large-scale mage datasets are reported n Secton 4. Secton 5 concludes ths work along wth future work dscusson. 2. OUR PROPOSED SCHEME 2.1 Scheme Overvew Our proposed large-scale mult-label propagaton framework ncludes three concatenatng parts: 1) An effcent k-nearest-neghbor k-nn) search based on localty senstve hashng LSH) approach; 2) sparse l 1 -graph constructon wthn hashng buckets; and 3) mult-label propagaton based on Kullback-Lebler dvergence. Fgure 1 gves an llustraton of the algorthmc ppelne. 2.2 Hashng-based l 1 -Graph Constructon The frst step of the proposed framework s the constructon of an drected weghted graph G =< V, E>, where the cardnalty of the node set V s m = l + u denote the labeled and unlabeled data respectvely), and the edge set E V V descrbes the graph topology. Let V l and V u be the sets of labeled and unlabeled vertces respectvely. G can be equvalently represented by a weght matrx W = {w j } R m m. To effcently handle the large-scale data, we enforce the constructed graph to be sparse. The weght between two nodes w j s nonzero only when j N, where N denotes the local neghborhood of the -th mage. The graph constructon can thus be decomposed nto two sub-problems: 1) how to determne the neghborhood of a datum; and 2) how to compute the edge weght w j Neghborhood Selecton For the frst problem, the conventonal strateges n prevous work can be roughly dvded nto two categores: k-nearest-neghbor based neghborhood: w j s nonzero only f x j s among the k-nearest neghbors to the - th datum. Obvously, graphs constructed n ths way may ensure a constant vertex degree, avodng overdense sub-graphs and solated vertces. ɛ-ball neghborhood: gven a pre-specfed dstance measure between two nodes d Gx, x j) and a threshold ɛ. Any vertex x j that satsfes d Gx, x j) ɛ wll be ncorporated n the neghborhood of the vertex x,resultng n nonzero w j. Itseasytoobservethatthe weght matrx of the constructed graph s symmetrc. However, for some vertces beyond a dstance from the others, there s probably no edge connectng to other vertces. Although domnatng the graph-based learnng lterature, the above two schemes are both computaton-ntensve on large-scale dataset, snce a lnear scan s requred to processasnglesampleandtheoverallcomplextyson 2 )n s the number of all samples). For a typcal mage data set to annotate, there are mages, from each of whch hgh-dmensonal features are extracted. A nave mplementaton based on ether of these two schemes usually takes several days to accomplsh graph constructon, whch s defntely unaffordable n terms of effcacy. Instead, n our mplementaton we use the localty-senstve hashng LSH) to enhance the effcacy on large-scale data sets. The basc dea of LSH s to store proxmal samples nto the same bucket, whch greatly saves the retreval tme at the expense of addtonal storage of hash bts. LSH s a recently proposed hashng algorthm famly. The most attractve property of LSH s the theoretc guarantee that the collson probablty of two samples.e., projected nto the same bucket) s proportonal to ther smlarty n feature space. The most popular LSH approach reles on random projecton followed by a threshold-based bnarzaton. Formally, gven a random projecton drecton v, the whole dataset s spltted nto two half-spaces, accordng to the rule hx )= Booleanv T x > 0). The hash table typcally conssts of k ndependent bts, namely the fnal hash bts are obtaned va sequental concatenaton Hx )= h 1 x ),...,h k x ). In the retreval phase, the k-nn canddate set can be safely confned to be the buckets whose Hammng dstances to the query sample are below a pre-specfed small threshold. Pror nvestgaton at the theoretc aspect reveals that a sublnear retreval complexty s feasble by the LSH method, whch s a crucal acceleraton for the scenaro of large-scale mage search. Note that n our mplementaton, LSH s run for multple tmes n all the experments, and the neghborhoods are the combned to avod the case of solated subgraphs Weght Computaton A proper nter-sample smlarty defnton s the core for graph-based label propagaton. The message transmtted from the neghborng vertces wth hgher weghts wll be much stronger than the others. Generally, the more smlar a sample s to another sample, the stronger the nteracton thus larger weght) exsts between them. Below are some popular ways to calculate the parwse weghts: Unweghted k-nn smlarty: The smlarty w j between x and x j s 1 f x j s among the k-nn of x ; otherwse 0. For undrected graph, the weght matrx s symmetrc and therefore w j = w j s enforced. Exponentally weghted smlarty: Forallchosenk-NN neghbors, ther weghts are determned as below: ) w j =exp dg x,x j ), 1) σ 2 where d Gx,x j) s the ground truth dstance and σ s a free parameter to control the decay rate. Weghted lnear neghborhood smlarty [16, 20]: In ths scheme sample x s assumed to be lnearly reconstructed from ts k-nn. The weghts are obtaned va solvng the followng optmzaton problem: mn x w w jx j 2. 2) j j N Typcally addtonal constrants are gven to w j. For example, n [20], the constrants w j 0and j wj = 1 are mposed. In our mplementaton, we adopt a scheme smlar to the dea n [16, 20], based on the lnear reconstructon assumpton. Moreover, pror work [18] reveals that mnmzng the

4 l 1 norm over the weghts s able to suppress the nose contaned n data. The constructed graph s non-parametrc and s comparably more robust than the other graph constructon strateges. Meanwhle, the graph constructed by datum-wse one-vs-all sparse reconstructon of samples can remove consderable label-unrelated lnks between those semantcally unrelated samples to reduce the ncorrect nformaton for label propagaton. Suppose we have an over-determned system of lnear equatons: [ ] x1 x 2 x k w = x, 3) where x s the feature vector of the -th mage to be reconstructed, w s the vector of the unknown reconstructon coeffcents. Let X R d k be a data matrx, each column of whch corresponds to the feature vector of one of ts k- NN. In practce, there are probably noses n the features, and a natural way to recover these elements and provde a robust estmaton of w s to formulate x = Xw +ξ, where ξ R d s the sparse nose term. We can then solve the followng l 1-norm mnmzaton problem wth respect to both reconstructon coeffcents and feature nose: arg w, ξ mn ξ 1 4) s.t. x = Xw + ξ, w 0, w 1 =1. Ths optmzaton problem s convex and can be transformed nto a general lnear programmng problem. There exsts a globally optmal soluton, and the optmzaton can be solved effcently usng many avalable l 1 -norm optmzaton toolboxes lke l 1 -MAGIC [3]. 2.3 Problem Formulaton Let M l = {x,r } l be the set of labeled mages, where x s the feature vector of the -th mage and r s a multlabel vector ts entry s set to be 1 f t s assgned wth the correspondng label, otherwse 0). Let M u = {x } l+u =l+1 be the set of unlabeled mages, and M = {M l,m u} s the entre data set. The graph-based mult-label propagaton s ntrnscally a transductve learnng process, whch propagates the labels of M l to M u. For each x, we defne the probablty measure p over the measurable space Y,Y). Here Y s the σ-feld of measurable subsets of Y and Y N the set of natural numbers) s the space of classfer outputs. Y = 2 yelds bnary classfcaton whle Y > 2 mples mult-label. In ths paper, we focus on the mult-label case. Hereafter, we use p and r for the -th mage, both of whch are subject to the multnomal dstrbutons, and p y) s the probablty that x belongs to class y. As mentoned above, {r j,j V l } encodes the supervson nformaton of the labeled data. If t s assgned a unque label by the annotator, r j becomes the so-called one-hot vector only the correspondng entry s 1, the rest s 0). In case beng assocated wth multple labels, r j s represented to be a probablstc dstrbuton wth multple non-zero entres. We propose the followng crteron to gude the propagaton of the supervson nformaton, whch s based on the concept of KL dvergence defned on two dstrbutons: D 1p) = l ) m D KL r p + μ D KL p j N) w jp j ), 5). The frst term n D 1p) trgger a heavy penalty f the estmated value p devates from the pre-specfed r. Note that unlke most tradtonal approaches, there s no constrant for the rgd equvalence between p and r. Such a relaxaton s able to mtgate the bad effect of nosy annotatons. The second term of D 1 and the optmal soluton p =arg p mn D 1 p). Here D KL r p ) denotes the KL dvergence between r and p, whose formal defnton for the dscrete case s expressed as D KL r p )= y r y)log r y) p y) stems from the assumpton that p can be lnearly reconstructed from the estmatons of ts neghbors, thus penalzng the nconsstency between the p and ts neghborhood estmaton. Unlke prevous works [20] usng squared-error optmal under a Gaussan loss assumpton), the adopted KL-based loss penalzes relatve error rather than absolute error n the squared-error case. In other words, they can be regarded as the regularzaton terms from pror supervson and local coherence respectvely. μ s a free parameter to balance these two terms. If μ, w j 0, then D 1 p) sconvextheproofsgven n Appendx I). Snce no closed-form soluton s feasble, standard numercal optmzaton approaches such as nteror pont methods IPM) or method of multplers MOM) can be used to solve the problem. However, most of these approaches guarantee global optma yet are trcky to mplement e.g., an mplementaton of MOM to solve ths problem would have seven extraneous parameters) [17]. Instead, we utlze a smple alternatng mnmzaton method n ths work. Alternatng mnmzaton s an effectve strategy to optmze functons of the form fx, y) where x, y are two sets of varables. In many cases, smultaneous optmzng over x and y s computatonally ntractable or unstable, whle optmzng over one set of varables wth the other fxed s relatvely easer. Formally, a typcal alternatng mnmzaton loops over two sub-problems,.e., x t) =arg x mn fx, y t 1) ) and y t) =arg y mn fx t),y). An example for alternatng optmzaton s the well-known Expectaton-Maxmzaton EM) algorthm. Note that D 1 n Equaton 5) s not amenable to alternatng optmzaton. We further propose a modfed verson by ntroducng a new group of varables {q },whch s shown as below: D 2 p, q) = l m D KL r q )+μ D KL p +η j N ) w j q j ) m D KLp q ). 6) In the above, a thrd measure q s ntroduced to decouple the orgnal term μ m DKL p ) j N) wjpj. q can actually be regarded as a relaxed verson of p. To enforce consstency between them, the thrd term m D KLp q ) s ncorporated. 2.4 Part I: Optmze p wth q Fxed Wth {q, =1...m} fxed, the optmzaton problem s reduced to the followng form: p = arg p mn D 2p, q) 7) s.t. p y) =1, p 0,. y The above constraned optmzaton problem can be easly

5 transformed nto an unconstraned one usng the Lagrange multpler: m p = arg p mn D 2p, q)+ p y)). 8) λ 1 y For brevty, let L p D 2 p, q) + m λ 1 y p y)). Recall that any locally optmal solutons should be subject to the zero frst-order dervatve,.e., L p p y) = μ log p y)+1 log j N ) w jq jy) ) +η log p y)+1 log q y) ) λ = 0. 9) From Equaton 9), t s easly verfed that let γ = μ+η): μ log j N ) p y) =exp w ) jq j y)+ηlog q y) γ + λ. γ Recall that λ sthelagrangecoeffcentforthe-th sample and unknown. Based on the fact y py) =1,λ can be elmnated and fnally we obtan the updatng rule: ) μ exp log w γ j q j y)) + η log q γ y) j N ) p y) = ). 10) y exp μ log w γ j q j y)+ η log q γ y) j N ) 2.5 Part II: Optmze q wth p Fxed The other step of the proposed alternatng optmzaton s to update q wth p fxed. Unfortunately, t proves that the same trck used n subsecton 2.4 cannot be appled to the optmzaton of q, due to the hghly non-lnear term ) log j N w jq j y). To ensure that q s stll a vald probablty vector after updatng, we set the updatng rule as: q new = q old + Uh, 11) where the column vector of matrx U R d d 1) s constraned to be summed 0. Denote e to be a column vector wth ts all entres equal to 1, then we have e T U = 0. An alternatve vew of ths relatonshp s that U s the complementary subspace of the one spanned by 1 n e,thus UU T = I 1 n eet also holds. Vector h n each teraton should be carefully chosen so that the updated value of q new results n a non-trval decrease of the overall objectve functon. Denote L q D 2 p, q) and the value of q at the t-th teraton as q t),wehave L h q t) ) Lq q t) + U T h) = U T L q h q q =q t). 12) Note that n each teraton h s typcally ntalzed as 0, thus h = α L h q t) ) s a canddate descent drecton α s a parameter to control the step sze). By substtutng t nto Equaton 11), we obtan the followng updatng rule: q t+1) = q t) αuu T L q q q =q t) = q t) αi 1 n eet ) Lq. 13) q q =q t) Algorthm 1 Probablstc Collaboratve Mult-Label Propagaton 1: Input:An drected weghted sparse graph G =< V, E> of the whole mage dataset M = {M l,m u}, where M l = {x,r } l s the labeled mage set and M u = {x } l+u =l+1 s the set of unlabeled mages. x s the feature vector of the -th mage and r s a mult-label confdence vector for x. 2: Output: The convergent probablty measures p and q. 3: Intalzaton: Randomly ntalze {p 0, y p y) =1} and {q 0, y q y) =1}. 4: for p and q are not convergent do 5: Optmze p wth q Fxed: p y) = ) μ exp γ log w j q j y))+ η γ log q y) j N ) ). y exp μ γ log w j q j y)+ η γ log q y) j N ) 6: Optmze q wth p Fxed: q t+1) = q t) range defned n Equaton 16). 7: end for αi 1 n eet ) L q q, where α les n the Fgure 2: The dstrbuton of the number of nearest neghbors denote as k) n our proposed LSMP. In ths way, the pursut of the descent drecton wth respect to q s transformed nto an equvalent problem takng h as varable, whch s further solved by calculatng Lq q. For completeness, we lst the concrete value of an entry of L q q : L q q = ry) y) q μ w kp k y) y) k: N j N k k w kj q η py) jy) q. 14) y) One practcal ssue s the feasble regon of parameter α. An arbtrary α probably cannot ensure that the updated p t+1) n Equaton 13) stays wthn the range [0, 1]. A proper value of α should ensure: 0 q αuu T L q 1. 15) q q =q t) Denote v = UU T L q q q =q t).itseasytoverfythat { { }} q y) 0 α mn max vy), q y) 1, ɛ. 16) vy) In practce, α can be adaptvely determned from q t).the whole process of optmzaton s llustrated n Algorthm 1. The resultant p s adopted to nfer the mage tags, as t connects both r and q.

6 x 106 Average Precson AP) EGSSC LNP k NN Objectve Functon Value The value of k Fgure 3: The performance of three baselne algorthms wth respect to the number of nearest neghbors denote as k). 3. ALGORITHMIC ANALYSIS 3.1 Computatonal Complexty Overall speakng, the computatonal complexty of the proposed algorthm conssts of two components: the cost of hashng-based l 1 -graph constructon, and the cost of KLbased label propagaton. The effcacy of tradtonal graph constructon as n [21, 18] hnges on the complexty of k- NN retreval, whch s typcally On 2 )n s the number of mages) for a nave lnear-scan mplementaton. Our proposed LSH-based scheme guarantees a sublnear complexty by aggregatng vsually smlar mages nto the same buckets, greatly reducng the cardnalty of the set of canddate neghbors. Formally, recent work ponts out the lower bound of LSH s only slghtly hgh than On logn)), whch drastcally reduces the computatonal overhead of graph constructon compared wth tradtonal On 2 )complexty. On the other hand, for our proposed KL-guded label propagaton procedure, t has On kl) computaton n each teraton, where k denotes the averaged number of nearest neghbors for a graph vertex and l s the total number of labels. Actually, most label propagaton methods based on local confdence exchange have the same complexty. The consumed tme n real calculaton manly hnges on the value of k. In Fgure 2 we plot the dstrbuton of k obtaned va the proposed l 1-regularzed weght computaton, whch reaches ts peek value around k = 35. Ths small k value ndcates that l 1 penalty term s able to select much compacter reconstructon bass for a vertex. In contrast, to obtan nearly optmal performance, prevous works usually take k>100 see Fgure 3). In mplementaton, we fnd that the subtle reduce of k results n a drastc reduce of the runnng tme see more detals n the expermental secton). 3.2 Algorthmc Convergence The above two updatng procedures are terated untl converged. For the experments on NUS-WIDE dataset, generally about 50 teratons are requred for the convergency of the soluton. An exemplar convergency curve s shown n Fgure EXPERIMENTS To valdate the effectveness of our proposed approach on large-scale mult-label datasets, we conduct extensve experments on the real-world mage dataset NUS-WIDE [5], whch contans 269,648 mages accompaned wth totally Iteraton Number Fgure 4: Convergence curve of our proposed Algorthm on NUS-WIDE dataset. 5,018 unque tags. Images n ths dataset are crawled from the photo sharng webste Flckr by usng ts publc API. The underlyng mage dversty and complexty make t a good testbed for large-scale mage annotaton experments. Moreover, a subset of NUS-WIDE known as NUS-WIDE-Lte) obtaned after nosy tag removal s also publcly avalable. We provde quanttatve study on both the lte dataset and the full NUS-WIDE dataset, wth an emphass on the comparson wth fve state-of-the-art related algorthms n terms of accuracy and computatonal cost. 4.1 Datasets NUS-WIDE [5]: The dataset contans 269,648 mages and the assocated 5,018 tags. For evaluaton, we construct two mage pools from the whole dataset: the pool of labeled mages s comprsed of 161,789 mages whlst the rest are used for the pool of unlabeled mages. For each mage, an 81-D label vector s mantaned to ndcate ts relatonshp to 81 dstnct concepts tghtly related to tags yet relatvely hgh-level). Moreover, to testfy the performance stablty of varous algorthms, we vary the percentage of labeled mages selected from the labeled mage pool n mplementaton t s varyng from 10% to 100% ncreased by a step of 10%. We ntroduce the varable τ [0, 1] for t). The sampled labeled mages are then amalgamated wth the whole set of unlabeled mages 107,859 n all). We extract multple types of local vsual features from the mages 225-D blockwse color moments, 128-D wavelet texture and 75-D edge drecton hstogram). NUS-WIDE-Lte: As stated above, ths dataset s a lte verson of the whole NUS-WIDE database. It conssts of 55,615 mages randomly selected from the NUS-WIDE dataset. And the labels of each mage are also lke those of NUS- WIDE, an 81-D label vector s set to ndcate ts relatonshp to 81 dstnct concepts. As done on NUS-WIDE, three types of local vsual features are also extracted for ths dataset. We randomly select about half of the mages as labeled and theresttobeunlabeled. Agan,weusethesamesamplng strategy on the labeled set to perform the stablty test. 4.2 Evaluaton Crtera and Baselnes In the experments, fve baselne algorthms as shown n Table 1 are evaluated for comparatve study. Amongst them, the support vector machnes SVM) s orgnally developed to solve bnary-class or mult-class classfcaton problem. Here we use ts mult-class verson by adoptng the one-vsone method. The selected baselnes ncludes several stateof-the-art algorthms for sem-supervsed learnng. The ln-

7 Table 1: The Baselne Algorthms. Name Methods KNN k-nearest Neghbors [9] SVM Support Vector Machne [6] LNP Lnear Neghborhood Propagaton [20] EGSSC Entropc Graph Sem-Supervsed Classfcaton [17] SGSSL Sparse Graph-based Sem-supervsed Learnng [18] ear neghborhood propagaton LNP) [20] bases on a lnearconstructon crteron to calculate the edge weghts of the graph, and dssemnates the supervson nformaton by a local propagaton and updatng process. The EGSSC [17] s an entopc graph-regularzed sem-supervsed classfcaton method, whch s based on mnmzng a Kullback-Lebler dvergence on the graph bult from k-nn Gaussan smlarty as ntroduced n Sub-secton and The SGSSL [18] s a sparse graph-based method for sem-supervsed learnng by harnessng the labeled and unlabeled data smultaneously, whch consders each label ndependently. The crtera to compare the performance nclude Average Precson AP) for each label or concept) and Mean Average Precson MAP) for all labels. The former s a well-known gauge wdely used n the feld of mage retreval, whlst the latter s developed to handle the mult-class or mult-label cases. Forexample,nourapplcatonMAPsobtaned by averagng the APs on 81 concepts. All experments are conducted on a common desktop PC equpped wth Intel dual-core CPU frequency: 3.0 GHz) and 32G bytes physcal memory. For the experments on NUS-WIDE-Lte, the proposed method s compared wth all the fve baselne algorthms. Whle on the NUS-WIDE, the results from SGSSL s not reported due to ts ncapablty to handle dataset n such large scale. 4.3 Experment-I: NUS-WIDE-LITE 56k) In ths experment, we compare the proposed algorthm wth fve baselne algorthms. The results wth varyng numbers of labeled mages controlled by the parameter τ) are presented n Fgure 5. Below are the parameters and the adopted values for each method: for KNN, there s only one parameter k for tunng, whch stands for the number of nearest neghbors and s trvally set as 500. For SVM algorthm, we adopt the RBF kernel. For ts two parameters γ and C, wesetγ =0.6 andc = 1 n experments after fne tunng. For LNP algorthm, one parameter α s adjusted, whch s the fracton of label nformaton that each mage receves from ts neghbors. The optmal value s α = 0.95 n our experments. There are three parameters μ, ν and β n EGSSC, where μ and ν are used for weghtng the Kullback-Lebler dvergence term and Shannon entropy term respectvely and β ensures the convergence of the two smlar probablty measures. The optmal values are set as μ =0.1, ν =1andβ = 2 here. For our proposed algorthm, we set μ =10andη = 5. MAP of these sx methods s llustrated n Fgure 6. Our observatons from Fgure 5 are descrbed as follows: Our proposed algorthm LSMP outperforms the other baselne algorthms sgnfcantly when selectng dfferent proportons of labeled set. For example, wth 10 percent of labeled mages selected, LSMP has an m- Fgure 5: The results of the comparson of LSMP and the fve baselnes wth varyng parameter τ on NUS-WIDE-Lte dataset. provement 16.6% over SGSSL, 58.5% over EGSSC, 107.6% over LNP, 137.2% over SVM, and 154.5% over KNN. The mprovement s supposed to stem from the fact that our proposed algorthm encodes the label nformaton of each mage as a unt confdence vector, whch mposes extra nter-label constrants. In contrast, other methods ether consder the vsual smlarty graph only, or consders each label ndependently. Wth the ncreasng number of labeled mages, the performances of all algorthms consstently ncrease. When τ 0.6, the algorthm SGSSL outperforms the other two state-of-art algorthms LNP and EGSSC sgnfcantly. However, when τ>0.6, the mprovement of SGSSL over the others s lower. The proposed method keeps hgher MAP value than other fve methods over all values of τ. Recall that the proposed algorthm s a probablstc collaboratve mult-label propagaton algorthm, wheren p y) expresses the probablty for the -th mage to be assocated wth the y-th label. A drect applcaton for ths probablstc mplcaton s the tag rankng task. Some exemplar results of tag rankng are shown n Fgure Experment-II: NUS-WIDE 270k) In ths experment, we compare the proposed LSMP algorthm wth four state-of-the-art algorthms on the large-scale NUS-WIDE dataset for mult-label mage annotaton. As n prevous experments, we modulate the parameter τ to vary the percentage of the labeled mages used n the experments and carefully tune the optmal parameters n each method for far comparson. For KNN, the optmal value s k = For SVM algorthm, we set λ =0.8 andc =2. ForLNP method, the optmal value s α = In the experment of EGSSC, the best values are μ =0.5, ν =1andβ =1. For our proposed LSMP algorthm, μ =15andη =8. The results of all algorthms are shown n Fgure 8 and the results wth respect to each ndvdual concept are presented n Fgure 9. From Fgure 9, we can observe that On the large-scale real-world mage dataset, the proposed algorthm outperforms other algorthms sgnfcantly at all values of τ. For example, when τ = 0.1, LSMP has an mprovement 53.5% over EGSSC, 112.6% over LNP, 197.2% over SVM, and 220.5% over

8 Fgure 6: The comparson of APs for the 81 concepts usng sx methods wth τ =1. Wth the ncreasng parameter τ, the performances of all algorthms also ncrease. When τ 0.6, the algorthm EGSSC outperforms LNP sgnfcantly, but for τ>0.6, the mprovement of EGSSC than LNP s neglgble. The proposed method LSMP also keeps hgher MAP value than all baselnes over all feasble values of τ smlar to the case on NUS-WIDE-LITE, whch valdates the robustness of our proposed algorthm. We also provde the recorded runnng tme for dfferent algorthms on NUS-WIDE, as shown n Table 2. A salent effcacy mprovement can be observed from our proposed method. Fgure 8: The results of the comparson of LSMP and the four baselnes wth varyng parameter τ on NUS-WIDE. KNN. Compared wth the performance on NUS-WIDE- Lte, the best performance of LSMP n NUS-WIDE s 0.193, whch s smaller than the MAP value n the Lte verson. The performance degradaton s prmarly attrbuted to the ncrease of data scale the sze of labeled mage pool n NUS-WIDE s 170K, whle for the Lte verson t s only 27K). 5. CONCLUSION In ths paper we propose and valdate an effcent largescale mage annotaton method. Our contrbutons le n both the hashng-accelerated l 1-graph constructon, and KLdvergence orented soft loss functon and regularzaton term n graph-based modelng. The optmzaton framework utlzes the nter-label relatonshp and fnally returns a probablstc label vector for each mage, whch s more robust to noses and can be used for tag rankng. The proposed algorthm s expermented on several publcly-avalable mage benchmarks bult for mult-label annotaton, ncludng the

9 Fgure 7: The tags rankng results of LSMP n NUS-WIDE-LITE. Table 2: Executng tme unt: hours) comparson of dfferent algorthmson the NUS-WIDE dataset. Algorthms Graph Constructon Tme Label Estmaton Tme Total Tme KNN SVM LNP EGSSC LSMP ever-known largest NUS-WIDE data set. We shows ts superorty n terms of both accuracy and effcacy. Our future work wll follow two drectons: 1) extend the mage annotaton datasets to web-scale and further testfy the scalablty of our proposed method; 2) develop more elegant algorthms for KL-based label propagaton whch shows better convergent speed. 6. ACKNOWLEDGMENTS Ths research was supported by Research Grants NRF2007IDM- IDM on NRF/IDM Program and by AcRF Ter-1 Grant of R , Sngapore. 7. REFERENCES [1] A. Andon and P. Indyk. Near-optmal hashng algorthms for approxmate nearest neghbor n hgh dmensons. Commun. ACM, 511): , February [2] S. Boyd and L. Vandenberghe. Convex Optmzaton. Cambrdge Unversty Press, [3] E.J.Candès, J. K. Romberg, and T. Tao. Robust uncertanty prncples: exact sgnal reconstructon from hghly ncomplete frequency nformaton. IEEE Transactons on Informaton Theory, 522): , February [4] G. Chen, Y. Song, F. Wang, and C. Zhang. Sem-supervsed mult-label learnng by solvng a sylvester equaton. In SIAM Internatonal Conference on Data Mnng, [5] T.-S. Chua, J. Tang, R. Hong, H. L, Z. Luo, and Y.-T. Zheng. Nus-wde: A real-world web mage database from natonal unversty of sngapore. In CIVR, July [6] R.Collobert,F.H.Snz,J.Weston,andL.Bottou. Large scale transductve svms. Journal of Machne Learnng Research, 7: , September [7] T.M.CoverandJ.A.Thomas.Elements of Informaton Theory. Wley Seres n Telecommuncatons, [8] O. Delalleau, Y. Bengo, and N. Le Roux. Effcent non-parametrc functon nducton n sem-supervsed learnng. In Proceedngs of the Tenth Internatonal Workshop on Artfcal Intellgence and Statstcs, pages , [9] R. Duda, D. Stork, and P. Hart. Pattern Classfcaton. JOHN WILEY, [10] P. Indyk and R. Motwan. Approxmate nearest neghbors: Towards removng the curse of dmensonalty. In Proceedngs of the Symposum on Theory Computng, [11] M. Karlen, J. Weston, A. Erkan, and R. Collobert. Large-scale manfold transducton. In ICML, [12] D. Lu, X.-S. Hua, L. Yang, M. Wang, and H. jang Zhang. Tag rankng. In WWW, [13] Y. Lu, R. Jn, and L. Yang. Sem-supervsed mult-label learnng by constraned non-negatve matrx factorzaton. In AAAI, [14] Y. Mu, J. Shen, and S. Yan. Weakly-supervsed hashng n kernel space. In CVPR, [15] G.-J.Q,X.-S.Hua,Y.Ru,J.Tang,T.Me,and H.-J. Zhang. Correlatve mult-label vdeo annotaton. In MM, [16] S.T.Rowes and L.K.Saul. Nonlnear dmensonalty reducton by locally lnear embeddng. Scence, 290: , [17] A. Subramanya and J. Blmes. Entropc graph regularzaton n non-parametrc sem-supervsed classfcaton. In NIPS, [18] J. Tang, S. Yan, R. Hong, G.-J. Q, and T.-S. Chua. Inferrng semantc concepts from communty-contrbuted mages and nosy tags. In MM, [19] I. W. Tsang and J. T. Kwok. Large-scale sparsfed manfold regularzaton. In NIPS, [20] F. Wang and C. Zhang. Label propagaton through lnear neghborhoods. In ICML, June [21] J. Yuan, J. L, and B. Zhang. Explotng spatal

10 Fgure 9: The comparson of APs for the 81 concepts wth τ =1.0 on NUS-WIDE. context constrants for automatc mage regon annotaton. In MM, [22] Z.-J.Zha,T.Me,J.Wang,Z.Wang,andX.-S.Hua. Graph-based sem-supervsed learnng wth multple labels. Journal of Vsual Communcaton and Image Representaton, 202):97 103, February [23] X. Zhu. Sem-supervsed learnng wth graphs. Carnege Mellon Unversty, [24] X. Zhu. Sem-Supervsed Learnng Lterature Survey. Carnege Mellon Unversty, APPENDIX: Convexty of D 1 p) and D 2 p, q) PROOF: The convexty of D 1p) s obvous f D KLr p ) and D KLp j N) wjpj) prove convex. Consequently, to justfy the convexty of D 1 p), frst we elaborate on the convexty of KL dvergence defned on two probablty mass functons, whch has already been studed n the felds of both nformaton theory [7] and convex optmzaton [2]. Specfcally, for D KLp q) defned on two pars of probablty mass functons p 1,q 1)andp 2,q 2), the convexty of D KL equvalently mples the followng fact: D KL λp 1 +1 λ)p 2 λq 1 +1 λ)q 2 ) λd KL p 1 q 1 ) +1 λ)d KL p 2 q 2 ), 17) clear by applyng the log-sum nequalty [7],.e., n ) n a log a n n a log a, b b on both the left and rght sdes of the followng nequalty: D KL λp 1 +1 λ)p 2 λq 1 +1 λ)q 2 )= λp 1 y)+1 λ)p 2 y)) log λp 1y)+1 λ)p 2 y) λq 1 y)+1 λ)q 2 y). y It s easly verfed that D KL λp 1 +1 λ)p 2 λq 1 +1 λ)q 2 ) λp λp1y) 1y)log λq + 1 λ)p2y) 1 λ)p 2y)log y 1y) 1 λ)q y 2y) = λd KLp 1 q 1)+1 λ)d KLp 2 q 2). 18) Thus D KLr p )sconvex. And lkewse the convexty of D KLp j N) wjpj) can be justfed, observng that j N k ) w jp j s a convex, lnear combnaton of several varables. Hence D 1p) sconvex. Usng the smlar trcks above, D 2 p, q) s also demonstrated to be convex. where λ [0, 1]. The correctness of the above nequalty s