IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY

Transcription

1 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY Clc Predcton for Web Image Reranng Usng Multmodal Sparse Codng Jun Yu, Member, IEEE, Yong Ru, Fellow, IEEE, and Dacheng Tao, Senor Member, IEEE Abstract Image reranng s effectve for mprovng the performance of a text-based mage search. However, exstng reranng algorthms are lmted for two man reasons: 1) the textual meta-data assocated wth mages s often msmatched wth ther actual vsual content and 2) the extracted vsual features do not accurately descrbe the semantc smlartes between mages. Recently, user clc nformaton has been used n mage reranng, because clcs have been shown to more accurately descrbe the relevance of retreved mages to search queres. However, a crtcal problem for clc-based methods s the lac of clc data, snce only a small number of web mages have actually been clced on by users. Therefore, we am to solve ths problem by predctng mage clcs. We propose a multmodal hypergraph learnng-based sparse codng method for mage clc predcton, and apply the obtaned clc data to the reranng of mages. We adopt a hypergraph to buld a group of manfolds, whch explore the complementarty of dfferent features through a group of weghts. Unle a graph that has an edge between two vertces, a hyperedge n a hypergraph connects a set of vertces, and helps preserve the local smoothness of the constructed sparse codes. An alternatng optmzaton procedure s then performed, and the weghts of dfferent modaltes and the sparse codes are smultaneously obtaned. Fnally, a votng strategy s used to descrbe the predcted clc as a bnary event (clc or no clc), from the mages correspondng sparse codes. Thorough emprcal studes on a large-scale database ncludng nearly 330K mages demonstrate the effectveness of our approach for clc predcton when compared wth several other methods. Addtonal mage reranng experments on realworld data show the use of clc predcton s benefcal to mprovng the performance of promnent graph-based mage reranng algorthms. Index Terms Image reranng, clc, manfolds, sparse codes. I. INTRODUCTION DUE to the tremendous number of mages on the web, mage search technology has become an actve and challengng research topc. Well-recognzed mage search engnes, Manuscrpt receved February 22, 2013; revsed August 31, 2013 and February 20, 2014; accepted March 6, Date of publcaton March 11, 2014; date of current verson March 31, Ths wor was supported n part by the Natonal Natural Scence Foundaton of Chna under Grant , n part by ARC under Grants FT and DP , n part by the Program for New Century Excellent Talents n Unversty under Grant NCET , and n part by the Hong Kong Scholar Programme under Grant XJ The assocate edtor coordnatng the revew of ths manuscrpt and approvng t for publcaton was Prof. Mar H.-Y. Lao. J. Yu s wth the School of Computer Scence and Technology, Hangzhou Danz Unversty, Hangzhou , Chna (e-mal: yujun@hdu.edu.cn). Y. Ru s wth Mcrosoft Research Asa, Peng, Chna (e-mal: yongru@mcrosoft.com). D. Tao s wth the Centre for Quantum Computaton and Intellgent Systems, Faculty of Engneerng and Informaton Technology, Unversty of Technology, Sydney, Ultmo, NSW 2007, Australa (e-mal: dacheng.tao@uts.edu.au). Color versons of one or more of the fgures n ths paper are avalable onlne at Dgtal Object Identfer /TIP such as Bng [1], Yahoo [2] and [3], usually Google use textual meta-data ncluded n the surroundng text, ttles, captons, and URLs, to ndex web mages. Although the performance of text-based mage retreval for many searches s acceptable, the accuracy and effcency of the retreved results could stll be mproved sgnfcantly. One major problem mpactng performance s the msmatches between the actual content of mage and the textual data on the web page [4]. One method used to solve ths problem s mage re-ranng, n whch both textual and vsual nformaton are combned to return mproved results to the user. The ranng of mages based on a text-based search s consdered a reasonable baselne, albet wth nose. Extracted vsual nformaton s then used to re-ran related mages to the top of the lst. Most exstng re-ranng methods use a tool nown as pseudo-relevance feedbac (PRF) [34], where a proporton of the top-raned mages are assumed to be relevant, and subsequently used to buld a model for re-ranng. Ths s n contrast to relevance feedbac, where users explctly provde feedbac by labelng the top results as postve or negatve. In the classfcaton-based PRF method [35], the top-raned mages are regarded as pseudo-postve, and low-raned mages regarded as pseudo-negatve examples to tran a classfer, and then re-ran. Hsu et al. [36] also adopt ths pseudo-postve and pseudo-negatve mage method to develop a clusterng-based re-ranng algorthm. The problem wth these methods s the relablty of the obtaned pseudo-postve and pseudo-negatve mages s not guaranteed. PRF has also been used n graph-based re-ranng [37] and Bayesan vsual re-ranng [38]. In these methods, low-ran mages are promoted by recevng renforcement from related hgh-ran mages. However, these methods are lmted by the fact that rrelevant hgh-ran mages are not demoted. Therefore, both explct and mplct re-ranng methods suffer from the unrelablty of the orgnal ranng lst, snce the textual nformaton cannot accurately descrbe the semantcs of the queres. Instead of related textual nformaton, user clcs have recently been used as a more relable measure of the relatonshp between the query and retreved objects [5], [6], snce clcs have been shown to more accurately reflect the relevance [7]. Joachms et al. [39] conducted an eye-tracng experment to observe the relatonshp between the clced lns and the relevance of the target pages, whle Shoouh et al. [8] nvestgated the effect of reorderng web search results based on clc through search effectveness. In the case of mage searchng, clcs have proven to be very relable [7]; 84% of clced mages were relevant compared IEEE. Personal use s permtted, but republcaton/redstrbuton requres IEEE permsson. See for more nformaton.

2 2020 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 to 39% relevance of documents found usng a general web search. Based on ths fact, Jan et al. [9] proposed a method whch utlzes clcs for query-dependent mage searchng. However, ths method only taes clcs nto consderaton and neglects the vsual features whch mght mprove the retreved mage relevance to the query. In another study, Jan and Varma [10] proposed a Gaussan regresson model whch drectly concatenates the clcs and varous vsual features nto a long vector. Unfortunately the dversty of multple vsual features was not taen nto consderaton. Accordng to commercal search engne analyss reports, only 15% of web mages are clced by web users. Ths lac of clcs s a problem that maes effectve clc-based re-ranng challengng for both theoretcal studes and real-world mplementaton. In order to solve ths problem, we adopt sparse codng to predct clc nformaton for web mages. Sparse codng s a popular sgnal processng method and performs well n many applcatons, e.g. sgnal reconstructon [11], sgnal decomposton [12], and sgnal denosng [13]. Although orthogonal bases le Fourer or Wavelets have been wdely adopted, the latest trend s to adopt an overcomplete bass, n whch the number of bass vectors s greater than the dmensonalty of the nput vector. A sgnal can be descrbed by a set of overcomplete bases usng a very small number of nonzero elements [18]. Ths causes hgh sparsty n the transform doman, but many applcatons need ths compact representaton of sgnals. In computer vson, sgnals are mage features, and sparse codng s adopted as an effcent technque for feature reconstructon [14] [16]. It has been wdely used n many dfferent applcatons, such as mage classfcaton [14], face recognton [15], mage annotaton [17], and mage restoraton [13]. In ths paper, we formulate and solve the problem of clc predcton through sparse codng. Based on a group of web mages wth assocated clcs (nown as a codeboo), and a new mage wthout any clcs, sparse codng s utlzed to choose as few basc mages as possble from the codeboo n order to lnearly reconstruct a new nput mage whle mnmzng reconstructon errors. A votng strategy s utlzed to predct the clc as a bnary event (clc or no clc) from the sparse codes of the correspondng mages. The overcomplete characterstc of the codeboo guarantees the sparsty of the reconstructon coeffcents. However, n addton to sparsty, the overcompleteness of the codeboo causes loss n the localty of the features to be represented. Ths results n smlar web mages beng descrbed by totally dfferent sparse codes, and unstable performance n mage reconstructon; clcs are thus not predcted successfully. In order to address ths ssue, one feasble soluton s to add an addtonal localty preservng term to the formulaton of sparse codng. Laplacan sparse codng (LSC) [19], n whch a localty-preservng Laplacan term s added to the sparse code, maes the sparse codes more dscrmnatve whle mantanng the smlarty of features, and enhancng the sparse codng s robustness. However, LSC [19] can only handle sngle feature mages; n practce, web mages are usually descrbed by multple features. For nstance, commercal search engnes extract and Fg. 1. Example mages and ther clc number accordng to the queres of bull and Whte Tger. preserve dfferent features such as color hstograms, edge drecton hstograms, and SIFTs. Two categores of methods are used to deal wth multmodal data: early fuson and late fuson [20], [50]. They dffer n the way they ntegrate the results from feature extracton on varous modaltes. In early fuson, feature vectors are connected from dfferent modaltes as a new vector. However, ths concatenaton does not mae sense due to the specfc characterstcs of each feature. In late fuson, the results obtaned by learnng for each modalty are ntegrated, but these fused results from late fuson may not be satsfactory snce results for each modalty mght be poor, and assgnng approprate weghts to dfferent modaltes s dffcult. In ths paper we propose a novel method named multmodal hypergraph learnng-based sparse codng for clc predcton, and apply the predcted clcs to re-ran web mages. Both strateges of early and late fuson of multple features are used n ths method through three man steps. Frst, we construct a web mage base wth assocated clc annotaton, collected from a commercal search engne. As shown n Fg. 1, the search engne has recorded clcs for each mage. Fg. 1(a), (b), (e), and (f) ndcate that the mages wth hgh clcs are strongly relevant to the queres, whle Fg. 1(c), (d), (g), and (h) present non-relevant mages wth zero clcs. These two components form the mage bases. Second, we consder both early and late fuson n the proposed objectve functon. The early fuson s realzed by drectly concatenatng multple vsual features, and s appled n the sparse codng term. Late fuson s accomplshed n the manfold learnng term. For web mages wthout clcs, we mplement hypergraph learnng [29] to construct a group of manfolds, whch preserves local smoothness usng hyperedges. Unle a graph that has an edge between two vertces, a set of vertces are connected by the hyperedge n a hypergraph. Common graph-based learnng methods usually only consder the parwse relatonshp between two vertces, gnorng the hgher-order relatonshp among three or more vertces. Usng ths term can help the proposed method preserve the local smoothness of the constructed sparse codes. Fnally, an alternatng optmzaton procedure s conducted to explore the complementary nature of dfferent modaltes. The weghts of dfferent modaltes and the sparse codes are smultaneously obtaned usng ths optmzaton strategy. A votng strategy s then adopted to predct f an nput mage wll be clced or not, based on ts sparse code.

3 YU et al.: CLICK PREDICTION FOR WEB IMAGE RERANKING 2021 The obtaned clc s then ntegrated wthn a graph-based learnng framewor [37] to acheve mage re-ranng. In summary, we present the mportant contrbutons of ths paper: Frst, we effectvely utlze search engne derved mages annotated wth clcs, and successfully predct the clcs for new nput mages wthout clcs. Based on the obtaned clcs, we re-ran the mages, a strategy whch could be benefcal for mprovng commercal mage searchng. Second, we propose a novel method named multmodal hypergraph learnng-based sparse codng. Ths method uses both early and late fuson n multmodal learnng. By smultaneously learnng the sparse codes and the weghts of dfferent hypergraphs, the performance of sparse codng performs sgnfcantly. We conduct comprehensve experments to emprcally analyze the proposed method on real-world web mage datasets, collected from a commercal search engne. Ther correspondng clcs are collected from nternet users. The expermental results demonstrate the effectveness of the proposed method. The rest of ths paper s organzed as follows. In Secton II, we brefly revew some related wor. The proposed method of multmodal hypergraph learnng-based sparse codng s presented n Secton III. Secton IV presents our expermental results. In Secton V we apply clcs to mage re-ranng, and fnally, n secton VI we draw our conclusons. II. RELATED WORK A. Multmodal Learnng for Web Images We can assume that each web mage s descrbed by t vsual features as x (1), x (2),...,x (t). A normal method for handlng multmodal [ features s to drectly ] concatenate them nto a long vector x (1), x (2),...,x (t), but ths representaton may reduce the performance of algorthms [20], especally when the features are ndependent or heterogeneous. It s also possble that the structural nformaton of each feature may be lost n feature concatenaton [20]. In [20], the methods of multmodal feature fuson are classfed nto two categores, namely early fuson and late fuson. It has been shown that f an SVM classfer s used, late fuson tends to result n better performance [20]. Wang et al. have [30] provded a method to ntegrate graph representatons generated from multple modaltes for the purpose of vdeo annotaton. Geng et al. [31] have ntegrated graph representatons usng a ernelzed learnng approach. Our wor ntegrates multple features nto a graph-based learnng algorthm for clc predcton. B. Graph-Based Learnng Methods Graph-based learnng methods have been wdely used n the felds of mage classfcaton [21], ranng [22] and clusterng. In these methods, a graph s bult accordng to the gven data, where vertces represent data samples and edges descrbe ther smlartes. The Laplacan matrx [23] s constructed from the graph and used n a regularzaton scheme. The local geometry of the graph s preserved durng the optmzaton, and the functon s forcefully smoothed on the graph. However, a smple graph-based method cannot capture hgherorder nformaton. Unle a smple graph, a hyperedge n a hypergraph lns several (two or more) vertces, and thereby captures ths hgher-order nformaton. Hypergraph learnng has acheved excellent performance n many applcatons. For nstance, Shashua [24] utlzed the hypergraph for mage matchng usng convex optmzaton. Hypergraphs have been appled to solve problems wth multlabel learnng [25] and vdeo segmentaton [26]. Tan et al. [27] have provded a sem-supervsed learnng method named HyperPror to classfy gene expresson data, by usng bologcal nowledge as a constrant. In [28], a hypergraph-based mage retreval approach has been proposed. In ths paper, we construct the hypergraph Laplacan usng the algorthm presented n [29]. III. MULTIMODAL HYPERGRAPH LEARNING-BASED SPARSE CODING FOR CLICK PREDICTION Here we present defntons of multmodal hypergraph learnng-based sparse codng for clc predcton, and defne mportant notatons used n the rest of the paper. Captal letters, e.g. X, represent the database of web mages. Lower case letters, e.g. x, represent mages and x s the th mage of X. Superscrpt (), e.g.x () and x (), represents the web mage s feature from the th modalty. A multmodal mage database wth n{ mages[ and m representatons ] } can be represented as: X = X () = x () 1,...,x() n R m t n. Fg. 2 llustrates =1 the detals of the proposed framewor. Frst, multple features are extracted to descrbe web mages. Second, from these features, we construct multple hypergraph Laplacans, and perform sparse codng based on the ntegraton of multple features. Meanwhle, the local smoothness of the sparse codes s preserved by usng manfold learnng on the hypergraphs. The sparse codes of mages, and the weghts for dfferent hypergraphs, are obtaned by smultaneous optmzaton usng an teratve two-stage procedure. A votng strategy s adopted to predct the clc as a bnary event (clc or no clc) from the obtaned sparse codes. Specfcally, the non-zero postons n sparse code represent a group of mages, whch are used to reconstruct the mages. If more than 50% of the mages have clcs, then the mage s predcted as clced. Otherwse, the mage s predcted as not clced. Fnally, a graph-based schema [37] s conducted wth the predcted clcs to acheve mage re-ranng. Some mportant notatons are presented n Table I. A. Defnton of Hypergraph-Based Sparse Codng Gven an mage x { x R d}, and web mage bases wth assocated clcs as A =[a 1, a 2,...,a s, ] ( A R d s),sparse codng can buld a lnear reconstructon of a gven mage x by usng the bases n A: x = c 1 a 1 + c 2 a 2 + +c s a s = Ac. The reconstructon coeffcent vector c for clc predcton s

4 2022 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 Fg. 2. The framewor of multmodal hypergraph learnng-based sparse codng for clc predcton. Frst, multple features are extracted from both the nput mages and mage bases. Second, multple hypergraph Laplacans are constructed, and the sparse codes are bult. Meanwhle, the localty of the obtaned sparse codes s preserved by usng manfold learnng on hypergraphs. Then, the sparse codes of the mages and the weghts for dfferent hypergraphs are obtaned by smultaneously optmzaton through an teratve two-stage optmzaton procedure. A votng strategy s used to acheve clc data propagaton. Fnally, the obtaned sparse codes are ntegrated wth the graph-based schema for mage re-ranng. TABLE I IMPORTANT NOTATIONS AND THEIR DESCRIPTIONS sparse, meanng that only a small proporton of entres n c are non-zero. c 0 can be denoted as the number of nonzero entres for vector c, and sparse codng can be descrbed as: mn c 0 s.t. x = Ac. However, the mnmzaton of ths problem s NP-hard. It has been proven n [18] that the mnmzaton of l 1 -norm approxmates the sparsest near-soluton. Therefore, most studes normally descrbe the sparse codng problem as the mnmzaton of l 1 -norm of the reconstructon coeffcents. The objectve of sparse codng can be defned as [32]: mn c x Ac 2 + α c 1. (1) The reconstructon error s represented by the frst term n (1), and the second term s adopted to control the sparsty of sparse codes c. α s the tunng parameter used to coordnate sparsty and reconstructon error. By usng the sparse codng method, the web mages are represented ndependently, and smlar web mages can be descrbed as totally dfferent sparse codes. One reason for ths s the loss of the localty nformaton n equaton (1). Therefore, to preserve the localty nformaton, the hypergraph Laplacan s utlzed n (1). We adopt V, E to represent the vertex set, and the hyperedge set as G = (V, E). The hyperedge weght vector s w. Here, each hyperedge e s assgned a weght w (e ).A V E ncdence matrx H denotes G wth the followng elements: { 1, f v e H (v, e) = (2) 0, f v / e. Based on H, the vertex degree of each vertex v V s d (v) = e E w (e) H (v, e), (3) and the edge degree of hyperedge e E s δ (e) = v V H (v, e). (4) D v and D e are used to denote dagonal matrces of vertex degrees and hyperedge degrees, respectvely. Let W denote the dagonal matrx of the hyperedge weghts. The value of each hyperedge s weght s set accordng to the rules used n [28]. Frst, we construct the V V affnty matrx

5 YU et al.: CLICK PREDICTION FOR WEB IMAGE RERANKING 2023 accordng to j = exp ( / v v j σ 2 ), where σ s the average dstance among all vertces. Then, the weght for each hyperedge s calculated as W = j.thee s a vertex v j e set, whch s composed of K nearest neghbors of vertex v. The unnormalzed hypergraph Laplacan matrx [29] can be defned as follows: L = D v HWD 1 e H T. (5) Therefore, we want the sparse codes of mages wthn the same hyperedge to be smlar to each other. Summng up the parwse dstances between the sparse codes wthn each hyperedge by w (e) / δ (e), the hypergraph-based sparse codng can be formulated as mn x Ac 2 c 1,...,c n + α c 1 + β 2 e E (p,q) e w (e) cp c q 2, (6) δ (e) where c s a vector of sparse codng. Hence, the web mages connected by the same hyperedge are encoded as smlar sparse codes usng ths formulaton. The smlarty among these web mages wthn the same hyperedge s preserved. We denote X = [x 1, x 2,...,x n ], and C = [c 1, c 2,...,c n ]. Equaton (6) can be rewrtten as mn X C AC 2 F + α c 1 + βtr ( CLC T ). (7) B. Multmodal Feature Combnatons In real applcatons, mages are descrbed by multmodal { features. [ Gven a dataset ] wth } multple features: X = X () = x () 1,...,x() n R m t n, n whch each representaton X () s a feature matrx from vew, we can formulate =1 the objectve functon based on (7) as: mn x ( j) c 1,...,c n A ( j) 2 c + α c 1 + βtr C λ j L ( j) C T, (8) where L ( j) s the constructed hypergraph Laplacan matrx for [ the jth vew, and ] λ j s the correspondng weght. A ( j) = a ( j) j) j) 1, a( 2,...,a( s, ( A ( j) R d s) s a specfed codeboo for jth vew. Eq. (8) can be rewrtten as mn X ( j) A ( j) C 2 + α c 1 C,λ F + βtr C λ j L ( j) C T s.t. λ j = 1 λ j > 0. (9) The object of Equaton (9) s to fnd a sparse lnear reconstructon of the gven mages n X by usng multple bases from dfferent features. The reconstructon coeffcents c for each mage are sparse, whch means that only a small fracton of entres c are non-zero. We present mplementaton detals of (9) n Secton III.C and Secton III.D. We propose an alternatng optmzaton procedure wth two stages to solve the problem n (9). Frst, we fx the weghts λ, and optmze C. Then we fx C, and optmze λ. These two stages are terated untl the objectve functon converges. C. Implementatons for Sparse Codes Instead of optmzng the entre sparse code matrx C smultaneously, we optmze each c sequentally untl the whole C converges. To optmze c, we should fx all the left sparse codes c p (p = ), and the weghts λ. Therefore, we can obtan L = t λ j L ( j). The optmzaton of (9) can be rewrtten wth respect to c as follows: mn Q (c ) + α c 1 c, (10) where Q (c ) = x () A () 2 c ( +β c T =1 ( C L ) ( + C L) T c c T L c ). (11) To solve the problem n (11), some dervatons are conducted on the frst term t x () A () 2 c of (11): =1 =1 x () A () 2 c = = = =1 =1 ( =1 ( x () A () c ) T ( x () A () c ) ( ( ) ) x ()T T ( ) A () c x () A () c x ()T x () ( ) Tx ( ) ) x()t A () c A () () TA c + A () c () c = x Ac 2, (12) [ ] where x = x (1) ;...; x(t) R (m 1+m 2 + +m t ) 1 and A = [ A (1) ; ;A (t)] R (m 1+m 2 + +m t ) N. Accordng to the method n [32], we adopt the feature-sgn search algorthm to solve c. Fg. 3 shows the detals of ths algorthm. We should enhance that n order to speed up the convergence of sparse codes. We ntalze the sparse codes wth the results of general sparse codng. After we complete the optmzaton of c, C wll be updated accordngly. In our experments, C converges n only a few teratons. In Fg. 3, the matrx C s the submatx after removng the th column of matrx C, and the vector L, s defned to be the subvector formed by removng the th of vector L. ϒ c and ϒ c c n Fg. 3 are calculated as ϒ c = Q (c ( ) ) = 2 A T Ac A T x +βc L +β L c, c, ϒ c c = 2 Q (c ) (c ) 2 = 2 ( ) A T A + β L I. (13)

6 2024 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 Fg. 3. Algorthm detals of mplementatons for sparse codes. D. Implementatons for Obtanng Weghts In the second stage, ( we( fx C to ) update ) λ. Therefore, n (9), only the term tr C λ j L ( j) C T can affect the objectve functon s value, and Eq. (9) can be rewrtten as mn tr C λ j L ( j) C T λ s.t. λ j = 1 λ j > 0. (14) The soluton to λ n (14) s λ p = 1, relatng to the mnmum tr ( CL ( j) C T ) over dfferent modaltes, and λ p = 0otherwse. It ndcates that only one modalty wll be selected. Hence, ths soluton cannot effectvely explore the complementary characterstcs of dfferent modaltes. To solve ths problem, the method n [33] s utlzed. The λ j n (14) s replaced by λ z j wth z > 1. Therefore, t λ j reaches ts mnmal value when λ j = 1/t related to t λ j = 1,λ j > 0. In ths case, equaton (14) can be reformulated as mn λ s.t. tr C λ z j L( j) C T λ j = 1 λ j > 0, (15) where z > 1. The constrant λ j = 1 s taen nto consderaton through the Lagrange multpler, and the objectve functon n (15) can be rewrtten as (λ, ς)=tr C λ z j L( j) C T ς λ j 1. (16)

7 YU et al.: CLICK PREDICTION FOR WEB IMAGE RERANKING 2025 TABLE II DETAILS OF THE REAL-WORLD WEB QUERIES DATASETS t can be guaranteed that our method obtans state-of-art performance of tme complexty. Fg. 4. The detals of the alternatng algorthm to obtan sparse codes and optmal weghts. From (λ, ς), we can mae the partal dervatve wth respect to λ j and ς zero. Thus, λ j can be obtaned as ( ( 1/tr CL ( j) C T )) 1/(z 1) λ j = ( ( 1/tr CL ( j) C T )). (17) 1/(z 1) The Laplacan matrx L ( j) s postve semdefnte, so we have λ j 0. When C s fxed, the global optmal of λ j can be obtaned from (17). The algorthm detals are lsted n Fg. 4. E. Tme Complexty Analyss We suppose the { experment [ s conducted ] on the dataset } wth mage bass A = A () = a () t 1,...,a() N 1 R m N 1 { [ ] } and =1 mage set X = X () = x () t 1,...,x() N 2 R m N 2.The =1 tme complexty of conductng the alternatng algorthm to obtan the sparse codes for X s calculated from two parts: The calculaton of L ( j) for vsual manfolds: the tme complexty for ths part s O (( t ) =1 m (N2 ) 2). The calculaton for alternatng algorthm to obtan sparse codes and optmal weghts: for the update of λ, the tme complexty s O ( t (N 2 ) 2). For the sparse codng part, we adopt effcent sparse codng algorthm [32] to calculate sparse codes c for each x, andc converges n only a few teratons. We adopt to ndcate the tme complexty of effcent sparse codng. Therefore, the tme complexty of ths part s O (T 1 (N 2 )). Therefore, the entre tme complexty of the proposed algorthm s O (( t ) =1 m (N2 ) 2 + ( t (N 2 ) 2 +T 1 (N 2 ) ) ) T 2, where T2 s the number of teratons n alternatve optmzaton. T 1 s less than three and T 2 s less than 5 n all experments. Accordng to the demonstratons n [32], the tme complexty of effcent sparse codng s lower than some state-of-art sparse codng methods ncludng QP solver, a modfed verson of LARS [47], graftng [48], and Chen et al. s nteror pont method [49]. Therefore, IV. EXPERIMENTAL RESULTS AND DISCUSSION To demonstrate the effectveness of the proposed method, we conduct experments on a real-world dataset wth mages collected from a commercal search engne. We compare performance of the proposed method wth representatve algorthms, such as sngle hypergraph learnng-based sparse codng [19], sngle graph learnng-based sparse codng [19], regular sparse codng [41] and the -nearest neghbor ( NN) algorthm. The experments are conducted n two stages. In the frst stage, we compare our method wth the others for clc predcton. In the second stage, we conduct experments to test the senstvty of the parameters. The detals are provded below. A. Dataset Descrpton We use real-world Web Queres dataset, whch contans 200 dverse representatve queres collected from the query log of a commercal search engne. In total, t contans 330,665 mages. Table II provdes detals of the real-world web query datasets ncludng the query number for each category and some examples. We select ths dataset to assess our method for clc predcton for two man reasons. Frst, the web queres and ther related mages orgnate drectly from the nternet, and the queres are manly hot (.e. current) queres that have appeared frequently over the past sx months. Second, ths dataset contans real clc data, mang t easy to evaluate whether our method accurately predcts clcs on web mages. The labels of mages n the dataset are assgned accordng to ther clc counts. The mages are categorzed nto two categores: the mages of whch the clc count s larger than zero and the mages of whch the clc count s zero. We represent each mage by extractng fve dfferent vsual features from the mages ncludng: bloc-wse color moments (CM), the HSV color hstogram (HSV), color autocorrelogram (CO), wavelet texture (WT) and face feature. B. Experment Confguraton To evaluate the performance of the proposed method for clc predcaton, we compare the followng seven methods, ncludng the proposed method: 1. Multmodal hypergraph learnng-based sparse codng (MHL). Parameters α and β n (9) are selected by fvefold cross valdaton. The neghborhood sze n the hyperedge generaton process and the value of z n (15) are tuned to the optmal values.

8 2026 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 TABLE III PERFORMANCE COMPARISONS OF CLASSIFICATION ACCURACY (%) FOR CLICK PREDICTIONS WITH A FIXED SIZE OF IMAGE BASES, AND VARIED SIZE OF TEST IMAGE SETS.THE COMPARISON INCLUDES A COMPARISON OF MHL, MGL, SHL, SGL, SC, KNN AND GP. THE SIZE OF THE TEST IMAGE SET IS VARIED FROM AMONG [5%, 10%, 15%, 20%, 25%], AND THE SIZE OF THE IMAGE BASE IS FIXED AT 75%. THE RESULTS SHOWN IN BOLD ARE SIGNIFICANTLY BETTER THAN OTHERS 2. Multmodal graph learnng-based sparse codng (MGL). Followng the framewor of (9), we adopt a smple graph [40] to replace the hypergraph. The parameters α and β n (9) are determned usng fve-fold cross valdaton. The neghborhood sze and the value z are tuned to optmal values. 3. Sngle hypergraph learnng-based sparse codng (SHL) [19]. The framewor n (7) s adopted for each sngle vsual feature separately. The average performance of SHL-SC s reported and we name t SHL(A). In addton, we concatenate vsual features nto a long vector and conduct SHL-SC on t. The results are denoted as SHL(L). The parameters n ths method are tuned to optmal values. 4. Sngle graph learnng wth sparse codng (SGL) [19]. We adopt a smple graph [40] to replace the hypergraph n (7). The performance of SGL(A) and SGL(L) are recorded. The parameters are tuned to ther optmal values. 5. Regular sparse codng (SC). The sparse codng s drectly conducted on each vsual feature separately usng Lasso [41]. The average performance of SC s reported, and denoted as SC(A). In addton, we conduct sparse codng on the ntegrated long vector and record the results as SC(L). 6. The -nearest neghbor algorthm (KNN). To provde the baselne performance for the experment, we adopt KNN for each vsual feature. Ths s a method that classfes a sample by fndng ts closest samples n the tranng set. In ths experment, each parameter s tuned to the optmal value. The KNN(A) and KNN(L) are reported. 7. The Gaussan Process regresson [10]: Ths method dentfes a group of clced mages and conducts dmensonalty reducton on concatenated vsual features. A Gaussan Process regressor s traned on the set of clced mages and s then used to predct clc counts for all mages. Ths method s named GP n the expermental results. We randomly select mages to form the mage bases and test mages. Snce dfferent queres contan a dfferent number of mages, t would be napproprate to fnd a fxed number settng for dfferent queres. Therefore, we choose dfferent percentages of mages to form the mage bases. Specfcally, the experments are separated nto two stages: the sze of test mage set s fxed at 5%, and the sze of mage base s vared from among [10%, 30%, 50%, 70%, 90%]; the sze of mage base s fxed at 75%, and the sze of the test mage s vared from among [5%, 10%, 15%, 20%, 25%]. Besdes, we conduct experments to show the effects of dfferent parameters. For all methods, we ndependently repeat the experments fve tmes wth randomly selected mage bases and report the averaged results. C. Results on Clc Predcton The performance of MHL was compared wth other varous methods. We performed MHL, MGL, SHL, SGL and SC to obtan sparse codes for the nput mages, and the votng strategy was utlzed to predct whether the mages would be clced or not. We obtaned the classfcaton accuracy (%) as an estmate of the result of clc predcton. Table III lsts the estmated average classfcaton accuracy for the dfferent methods. We used 75% mages from each query to form the mage base. The experments were conducted under fve dfferent condtons, where the proporton of nput mages vared n the range of 5-25%. Accordng to the expermental results, we observe that nearly all the used methods effectvely mprove on baselne comparatve results. Our method, MHL, acheved the best results for clc predcton, wth the hypergraphbased method performng better than other sngle graph-based methods. The hgh-order nformaton preserved by hypergraph constructon s benefcal to preservng local smoothness. Compared wth a normal graph, the use of the hypergraph can effectvely mprove clc predcton performance. In addton, we observe that multmodalty methods (MHL and MGL) outperformed sngle modalty methods (SHL and SGL). Ths suggests that the multmodalty desgn n equaton (9), and ts correspondng alternatng optmzaton algorthm are effectve n obtanng optmal classfcaton results. Another nterestng fndng s that the graph-based learnng methods (MHL, MGL, SHL and SGL) perform better than regular sparse codng (SC). The graph and hypergraph-based regularzers perform effcently n obtanng excellent sparse codng results. Addtonally, we found that MHL s stable when the sze of the mage test set ncreases from 5% to 25%. Table IV compares the performance of dfferent methods when the sze of the test mage set s fxed at 5%, and the sze of the tranng mage bases are vared n the range of 10-90%. In general, MHL had the best performance. We observed that the classfcaton accuracy of MHL ncreases from 64.8% to 66.9% when the sze of the mage base ncreases from 10% to 90% ndcatng that a small mage base can adversely affect clc predcton performance.

9 YU et al.: CLICK PREDICTION FOR WEB IMAGE RERANKING 2027 TABLE IV PERFORMANCE COMPARISONS OF CLASSIFICATION ACCURACY (%) FOR CLICK PREDICTIONS WITH VARIED SIZE OF IMAGE BASES, AND FIXED SIZE OF TEST IMAGE SETS.THE COMPARISON INCLUDES MHL, MGL, SHL, SGL, SC, KNN AND GP. THE SIZE OF THE IMAGE BASE IS VARIED FROM AMONG [10%, 30%, 50%, 70%, 90%], AND THE SIZE OF THE TEST IMAGE SET IS FIXED AT 5%. THE RESULTS SHOWN IN BOLD ARE SIGNIFICANTLY BETTER THAN OTHERS Fg. 5. Average classfcaton accuracy (%) wth dfferent parameters of α, β, K and z. The szes of mage bases and test mage sets are fxed at 10% and 5%, respectvely. (a) Classfcaton Accuracy versus the values of alpha. (b) Classfcaton Accuracy versus the values of beta. (c) Classfcaton Accuracy versus the values of K. (d) Classfcaton Accuracy versus the values of z. Another nterestng fndng n Tables III and IV s the sparse codng method dose not perform better than KNN. The reason s that the overcompleteness of sparse codng causes loss n the localty of the features. Smlar web mages wll be descrbed by totally dfferent sparse codes, and the performance n clc data predctons s unstable. In order to address ths ssue, an addtonal localty-preservng term wth a laplacan matrx s added nto the sparse codng formulaton. The expermental results n Tables III and IV show that SHL and SGL perform better than KNN. D. The Effect of Changng Parameter Values In Fg. 5, we show the senstvty of parameters α, β, K, and z n graph-based sparse codng algorthms. In these experments, we have fxed the percentage of the mage base to 10%, and the percentage comprsng the testng mage set to 5%. We frst fxed β to β opt, and vared α wth [ 10 2 α opt, 10 1 α opt,α opt, 10 1 α opt, 10 2 ] α opt.the average classfcaton accuraces of the methods are shown n Fg. 5(a). We have then fxed α to α opt, and vared β wth [ 10 2 β opt, 10 1 β opt,β opt, 10 1 β opt, 10 2 ] β opt. The average classfcaton accuraces are shown n Fg. 5(b). From these fgures we see that MHL performs best. From 10 2 α opt to 10 1 α opt, these methods perform stably, as shown n Fg. 5(a). However, from 10 1 α opt to 10 2 α opt, the performance degrades severely. In Fg. 5(b), these methods are stable when β ncreases from 10 2 α opt to 10 2 α opt. In Fg. 5(c), we observe that the hypergraph-based methods

10 2028 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 Fg. 6. Performance comparsons of classfcaton accuracy (%). The comparson s conducted among MHL, SHL(BOF) and SHL(LLC). The sze of the mage bases s fxed at 75% and the sze of the test mage set s vared from among [5%, 10%, 15%, 20%, 25%]. (MHL, SHL(A), and SHL(L)) obtan the hghest performance when s fxed at 10. The graph-based methods (MGL, SGL(A) and SGL(L)) have the best performance when K s set to 5. In Fg. 5(d), we have vared the parameter z from 2 to 6, and observed that the classfcaton accuraces of MHL and MGL are hghest when z s 5. E. Dscusson About Features We adopt conventonal features n experments and the results are presented n Secton IV.C. Recently, some stateof-the-art features have been proposed, such as the Bag of Features (BOF) based on SIFT descrptons [45], and the localty-constraned lnear codng (LLC) [46], whch s a SPM le descrptor. LLC has obtaned state-of-the-art performance n mage classfcaton. In ths part, the features of 1024 dmensonal BOF and dmensonal LLC wth spatal bloc structure [1 2 4] are extracted for mages. The expermental results are presented n Fg. 6. The comparson s conducted among MHL, SHL(BOF) and SHL(LLC). The sze of the mage bases s fxed at 75% and the sze of the test mage set s vared from among [5%, 10%, 15%, 20%, 25%]. The expermental results demonstrate that our proposed method- MHL performs better than BOF and LLC. It ndcates that the multmodal learnng adopted n MHL s effectve n enhancng the classfcaton performance. F. Expermental Results on Scene Recognton In ths part, we demonstrate that MHL performs well by conductng mage classfcaton experments on the standard dataset of Scene 15 [43], whch contans 1500 mages that belong to 15 natural scene categores: bedroom, CALsuburb, ndustral, tchen, lvngroom, MITcoast, MITforest, MIThghway, MITnsdecty, MITmountan, MITopencountry, MITstreet, MIT-tallbuldng, PARoffce, and store. Fve dfferent features are adopted to descrbe the scenes, ncludng Color Hstogram (CH), Edge Drecton Hstogram (EDH), SIFT [45], Gst [44] and Localty-constraned Lnear Codng (LLC) [46]. In our experments, the labeled sample mages n the bases are randomly selected. Snce the sze of each class vares sgnfcantly, t s napproprate to fnd a fxed number for Fg. 7. Performance comparsons of classfcaton accuracy (%) for scene recognton. The comparson s conducted on MHL, MGL, SHL, SGL, SC and KNN. (a) The sze of the mage bases s fxed at 75% and the sze of the test mage set s vared from among [5%, 10%, 15%, 20%, 25%] (b) the sze ofthetestmagesetsfxedat5%andtheszeofthemagebasesvared from among [10%, 30%, 50%, 70%, 90%]. Fg. 8. The comparsons of the average NDCG measurements. From these results, we can see that the clc-based method, whch combnes the multmodal vsual features and clc nformaton, outperforms other methods. dfferent classes. Therefore, we fx a percentage number, whch s used for choosng mages from dfferent classes to form the mage bases and testng mages. For the experments on scene recognton, we compare the performance of sx methods: Multmodal hypergraph learnng-based sparse codng (MHL), Multmodal graph learnng-based sparse codng (MGL), Sngle hypergraph learnng-based sparse codng (SHL), Sngle graph learnng wth sparse codng (SGL), Regular sparse codng (SC) and -nearest neghbor algorthm (KNN). The detals of these sx methods have been presented n Secton IV.B. As shown n Fg. 7(a), our proposed method MHL outperforms other

11 YU et al.: CLICK PREDICTION FOR WEB IMAGE RERANKING 2029 Fg. 9. The top 10 mages n the commercal ranng lst (baselne) and the re-ranng lsts obtaned usng the graph-based method and clc-based method for the queres Butterfly and Musc. The orders of the mages are provded wth green, blue and red ndcatng relevance scales 2, 1 and 0 respectvely. From the fgure, t s clear that the clc-based method obtans the best results. (a) Results of mage re-ranng for query Butterfly. (b) Results of mage re-ranng for query Musc. methods n all cases. Ths demonstrates that the proposed method can perform well on standard datasets. V. APPLICATION OF THE ALGORITHM FOR IMAGE RERANKING To evaluate the effcacy of the proposed method for mage re-ranng, we conduct experments on a new dataset consstng of two subsets A and B. Subset A ncludes 330,665 mages wth 200 queres used n Secton IV, and subset B contans mages usng the same 200 queres. Images n subset A contan assocated clc data, and are used to form the mage base for sparse codng. The mages of subset B contan the orgnal ranng nformaton from a popular search engne, and thus we can easly evaluate whether our approach s able to mprove performance over the search engne algorthm. In subset B, each mage was labeled by the human oracle accordng to ts relevance to the correspondng query, as ether not relevant, relevant, or hghly relevant. We utlze scores of 0, 1, and 2 to ndcate the three relevance levels, respectvely. We use the popular graph-based re-ranng [38] scheme n the desgn of our experment. Accordng to ths framewor, a ranng score lst, y = [s 1, s 2,...,s n ] T, s a vector of ranng scores, whch corresponds to an mage set X = {x 1, x 2,...,x n }. The purpose of graph-based re-ranng s to calculate a new ranng score lst through learnng, based on the mages vsual features. Therefore, the re-ranng process can be formulated as a functon f : s = f (X, s ), n whch s =[s1, s 2,...,s n ]T s the ntal ranng score lst. Hence, the graph-based re-ranng can be formulated as a regularzaton framewor as follows: arg mn s { (s) + γ Remp (s) }, (18) n whch (s) s the regularzaton term that maes the ranng scores of vsually smlar mages close, and R emp (s) s an emprcal loss. γ s a tradeoff parameter to balance the

12 2030 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 emprcal loss and the regularzer. Accordng to [38], (18) can be rewrtten as: arg mn s T Ls + γ s s, (19) s n whch L s a Laplacan matrx. Snce we can calculate a group of weghts λ j for each modalty, the Laplacan n (19) s assgned wth L = t λ j L ( j). Based on the graph-based framewor, we compare two methods whch create ntal scores dfferently. In the frst method, the ntal score s s obtaned usng a tradtonal algorthm [38], whch assocates s wth the poston τ usng heurstc strateges: s = 1 τ n. Here, n s the number of mages n ths query. We name t the graph-based method. In the second method, the predcted clc nformaton s consdered along wth the poston. Therefore, we name ths the clc-based method. For a specfed mage, fts predcted to be clced by the user, then the ntal score wll be calculated as: s = ( 1 τ n + 1)/ 2. On the other hand, f the mage s not predcted to be clced the score wll be: s = ( 1 τ )/ n 2. The ranng results of the commercal search engne are provded as the baselne, whch we name the commercal baselne. In order to evaluate the re-ranng performance for each query, we apply the normalzed dscounted cumulated gan (NDCG) [42], whch s a standard measurement n nformaton retreval. For a gven query, the NDCG at poston P can be calculated as: p 2 l() 1 NDCG@p = Z p log(1 + ), (20) =1 where p s the consdered depth, l() s the relevance level of the th mage n the refned ranng lst, and Z p s a normalzaton constant; ths maes NDCG@p 1 for a perfect ranng. For the Web Queres dataset, Z p was calculated based on the labels provded. To compute the overall performance, NDCGs were averaged over all queres for each dataset. Fg. 8 shows average NDCG scores obtaned by dfferent methods at depths of [3, 5, 10, 20, 50]. Accordng to these results we can see that both the graph-based method and clc-based method can effectvely mprove baselne results. It ndcates that the graph-based learnng method performs well for web mage re-ranng. The clc-based method, whch ntegrates all vsual features and clc data, acheved the best results. Specfcally, t outperformed the graph-based method at all depths. It shows that by ncludng the clc data, semantc gaps can be brdged and the performance of mage re-ranng mprove. Fg. 9 provdes the top ten returned mages obtaned by the three methods for two example queres. The blocs wth green, blue and red ndcate mages relevance scales 2, 1 and 0 respectvely. In the frst and second rows of Fg. 9(a) and (b), there are many rrelevant and low relevant mages, but the mages of the thrd row are all relevant to the query. It clearly shows that clc data s effectve n coordnatng the ranng results. VI. CONCLUSION In ths paper we propose a new multmodal hypergraph learnng based sparse codng method for the clc predcton of mages. The obtaned sparse codes can be used for mage re-ranng by ntegratng them wth a graph-based schema. We adopt a hypergraph to buld a group of manfolds, whch explore the complementary characterstcs of dfferent features through a group of weghts. Unle a graph that has an edge between two vertces, a set of vertces are connected by a hyperedge n a hypergraph. Ths helps preserve the local smoothness of the constructed sparse codes. Then, an alternatng optmzaton procedure s performed and the weghts of dfferent modaltes and sparse codes are smultaneously obtaned usng ths optmzaton strategy. Fnally, a votng strategy s used to predct the clc from the correspondng sparse code. Expermental results on real-world data sets have demonstrated that the proposed method s effectve n determnng clc predcton. Addtonal expermental results on mage re-ranng suggest that ths method can mprove the results returned by commercal search engnes. REFERENCES [1] Y. Gao, M. Wang, Z. J. Zha, Q. Tan, Q. Da, and N. Zhang, Less s more: Effcent 3D object retreval wth query vew selecton, IEEE Trans. Multmeda, vol. 13, no. 5, pp , Oct [2] S. Clnchant, J. M. Renders, and G. Csura, Trans-meda pseudo relevance feedbac methods n multmeda retreval, n Proc. CLEF, 2007, pp [3] L. Duan, W. L, I. W. Tsang, and D. Xu, Improvng web mage search by bag-based reranng, IEEE Trans. Image Process., vol. 20, no. 11, pp , Nov [4] B. Geng, L. Yang, C. Xu, and X. Hua, Content-aware Ranng for vsual search, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2010, pp [5] B. Carterette and R. Jones, Evaluatng search engnes by modelng the relatonshp between relevance and clcs, n Proc. Adv. Neural Inf. Process. Syst., 2007, pp [6] G. Dupret and C. Lao, A model to estmate ntrnsc document relevance from the clcthrough logs of a web search engne, n Proc. ACM Int. Conf. Web Search Data Mnng, 2010, pp [7] G. Smth and H. Ashman, Evaluatng mplct judgments from mage search nteractons, n Proc. WebSc. Soc., 2009, pp [8] M. Shoouh, F. Scholer, and A. Turpn, Investgatng the effectveness of clcthrough data for document reorderng, n Proc. Eur. Conf. Adv. Inf. Retr., 2008, pp [9] V. Jan and M. Varma, Random wals on the clc graph, n Proc. ACM SIGIR Conf. Res. Develop. Inf. Retr., 2007, pp [10] V. Jan and M. Varma, Learnng to re-ran: Query-dependent mage re-ranng usng clc data, n Proc. Int. Conf. World Wde Web, 2011, pp [11] E. Candès, J. Romberg, and T. Tao, Stable sgnal recovery from ncomplete and naccurate measurements, Commun. Pure Appl. Math., vol. 59, no. 8, pp , [12] A. Ersson and A. van den Hengel, Effcent computaton of robust low-ran matrx approxmatons n the presence of mssng data usng the l1 norm, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2010, pp [13] J. Maral, F. Bach, J. Ponce, G. Sapro, and A. Zsserman, Non-local sparse models for mage restoraton, n Proc. IEEE Int. Conf. Comput. Vs., Oct. 2009, pp [14] J. Yang, K. Yu, Y. Gong, and T. Huang, Lnear spatal pyramd matchng usng sparse codng for mage classfcaton, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2009, pp [15] S. Gao, I. Tsang, and L. T. Cha, Kernel sparse representaton for mage classfcaton and face recognton, n Proc. Eur. Conf. Comput. Vs., 2010, pp [16] J. Wrght, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, Robust face recognton va sparse representaton, IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp , Feb [17] C. Wang, S. Yan, L. Zhang, and H. Zhang, Mult-label sparse codng for automatc mage annotaton, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2009, pp

13 YU et al.: CLICK PREDICTION FOR WEB IMAGE RERANKING 2031 [18] D. Donoho, For most large underdetermned systems of lnear equatons, the mnmal l1-norm soluton s also the sparsest soluton, Dept. Statst., Stanford Unv., Stanford, CA, USA, Tech. Rep., [19] S. Gao, I. Tsang, and L. Cha, Laplacan sparse codng, hypergraph Laplacan sparse codng, and applcatons, IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp , Jan [20] C. Snoe, M. Worrng, and A. Smeulders, Early versus late fuson n semantc vdeo analyss, n Proc. ACM Int. Conf. Multmeda, 2005, pp [21] D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Scholopf, Learnng wth local and global consstency, n Proc. Int. Conf. Neural Inf. Process. Syst., 2004, pp [22] X. He, W. Y. Ma, and H. J. Zhang, Learnng an mage manfold for retreval, n Proc. ACM Int. Conf. Multmeda, 2004, pp [23] M. Beln and P. Nyogy, Laplacan egenmaps for dmensonalty reducton and data representaton, Neural Comput., vol. 15, no. 6, pp , [24] R. Zass and A. Shashua, Probablstc graph and hypergraph matchng, n Proc. Int. Conf. Comput. Vs. Pattern Recognt., 2008, pp [25] L. Sun, S. J, and J. Ye, Hypergraph spectral learnng for mult-label classfcaton, n Proc. Int. Conf. Know. Dscovery Data Mnng, 2008, pp [26] Y. Huang, Q. Lu, and D. Metaxas, Vdeo object segmentaton by hypergraph cut, n Proc. Int. Conf. Comput. Vs. Pattern Recognt., 2009, pp [27] Z. Tan, T. Hwang, and R. Kuang, A hypergraph-based learnng algorthm for classfyng gene expresson and array CGH data wth pror nowledge, Bonformatcs, vol. 25, no. 21, pp , [28] Y. Huang, Q. Lu, S. Zhang, and D. Metaxas, Image retreval va probablstc hypergraph ranng, n Proc. Int. Conf. Comput. Vs. Pattern Recognt., 2010, pp [29] D. Zhou, J. Huang, and B. Schölopf, Learnng wth hypergraphs: Clusterng, classfcaton, and embeddng, n Proc. Neural Inf. Process. Syst., 2006, pp [30] M. Wang, X. S. Hua, R. Hong, J. Tang, G. Q, and Y. Song, Unfed vdeo annotaton va multgraph learnng, IEEE Trans. Crcuts Syst. Vdeo Technol., vol. 19, no. 5, pp , May [31] B. Geng, C. Xu, D. Tao, L. Yang, and X. S. Hua, Ensemble manfold regularzaton, n Proc. IEEE Int. Conf. Comput. Vs. Pattern Recognt., Jun. 2009, pp [32] H. Lee, A. Battle, R. Rana, and A. Y. Ng, Effcent sparse codng algorthms, n Proc. Adv. Neural Inf. Process. Syst., 2006, pp [33] M. Wang, X. S. Hua, X. Yuan, Y. Song, and L. R. Da, Optmzng multgraph learnng: Towards a unfed vdeo annotaton scheme, n Proc. ACM Int. Conf. Multmeda, 2007, pp [34] Y. Lv and C. Zha, Postonal relevance model for pseudo-relevance feedbac, n Proc. Int. Conf. Res. Develop. Inf. Retr., 2010, pp [35] R. Yan, A. Hauptmann, and R. Jn, Multmeda search wth pseudorelevance feedbac, n Proc. Int. Conf. Image Vdeo Retr., 2003, pp [36] W. Hsu, L. Kennedy, and S. Chang, Vdeo search reranng va nformaton bottlenec prncple, n Proc. ACM Int. Conf. Multmeda, 2006, pp [37] W. Hsu, L. Kennedy, and S. Chang, Vdeo search reranng through random wal over document-level context graph, n Proc. ACM Int. Conf. Multmeda, 2007, pp [38] X. Tan, L. Yang, J. Wang, X. Wu, and X. Hua, Bayesan vsual reranng, IEEE Trans. Multmeda, vol. 13, no. 4, pp , Aug [39] T. Joachms, L. Grana, B. Pan, H. Hembrooe, F. Radlns, and G. Gay, Evaluatng the accuracy of mplct feedbac from clcs and query reformulatons n web search, ACM TOIS, vol. 25, no. 2, pp. 1 6, [40] D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Scholopf, Learnng wth local and global consstency, n Proc. Neural Inf. Process. Syst., 2004, pp [41] R. Tbshran, Regresson shrnage and selecton va the lasso, J. R. Statst. Soc., Seres B, vol. 58, no. 8, pp , [42] K. Jarveln and J. Kealanen, Cumulated gan-based evaluaton of r technques, ACM Trans. Inf. Syst., vol. 20, no. 4, pp , [43] S. Lazebn, C. Schmd, and J. Ponce, Beyond bags of features: Spatal pyramd matchng for recognzng natural scene categores, n Proc. IEEE Int. Conf. Comput. Vs. Pattern Recognt., Jan. 2006, pp [44] A. Olva and A. Torralba, Modelng the shape of the scene: A holstc representaton of the spatal envelope, Int. J. Comput. Vs., vol. 42, no. 3, pp , [45] D. Lowe, Dstnctve mage features from scale-nvarant eyponts, Int. J. Comput. Vs., vol. 60, no. 2, pp , [46] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, Localtyconstraned lnear codng for mage classfcaton, n Proc. Comput. Vs. Pattern Recognt., 2010, pp [47] B. Efron, T. Haste, I. Johnstone, and R. Tbshran, Least angle regresson, Ann. Statst., vol. 32, no. 2, pp , [48] S. Perns and J. Theler, Onlne feature selecton usng graftng, n Proc. ICML, 2003, pp [49] S. Chen, D. Donoho, and M. Saunders, Atomc decomposton by bass pursut, SIAM J. Sc. Comput., vol. 20, no. 1, pp , [50] C. Xu, D. Tao, and C. Xu, A survey on mult-vew learnng, Neural Comput. Appl., vol. 23, no. 7 8, pp , Jun Yu (M 13) receved the B.Eng. and Ph.D. degrees from Zhejang Unversty, Zhejang, Chna. He s currently a Professor wth the School of Computer Scence and Technology, Hangzhou Danz Unversty. He was an Assocate Professor wth the School of Informaton Scence and Technology, Xamen Unversty. From 2009 to 2011, he was wth Sngapore Nanyang Technologcal Unversty. From 2012 to 2013, he was a Vstng Researcher wth Mcrosoft Research Asa. Over the past years, hs research nterests nclude multmeda analyss, machne learnng, and mage processng. He has authored and co-authored more than 50 scentfc artcles. He has (Co-)Chared for several specal sessons, nvted sessons, and worshops. He served as a Program Commttee Member or revewer of top conferences and prestgous journals. He s a Professonal Member of the IEEE, ACM, and CCF. Yong Ru (F 10) s currently a Senor Drector at Mcrosoft Research Asa, leadng research effort n the areas of multmeda search, nowledge mnng, and socal and urban computng. As a fellow of the IEEE, IAPR, and SPIE, and a Dstngushed Scentst of ACM, he s recognzed as a leadng expert n hs research areas. He holds more than 50 U.S. and Internatonal patents. He has authored 16 boos and boo chapters, and more than 100 referred journal and conference papers. Hs publcatons are among the most cted hs top fve papers have been cted more than 6000 tmes and hs h-index s 43. He s the Assocate Edtor-n-Chef of the IEEE MULTIMEDIA MAGAZINE, an Assocate Edtor of the ACM Transactons on Multmeda Computng, Communcaton and Applcatons, a foundng Edtor of the Internatonal Journal of Multmeda Informaton Retreval, and a foundng Assocate Edtor of the IEEE ACCESS. He was an Assocate Edtor of the IEEE TRANSACTIONS ON MULTIMEDIA from 2004 to 2008, the IEEE TRANS- ACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGIES from 2006 to 2010, the ACM/Sprnger Multmeda Systems Journal from 2004 to 2006, and the Internatonal Journal of Multmeda Tools and Applcatons from 2004 to He also serves on the Advsory Board of the IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING. Heson Organzng Commttees and Program Commttees of numerous conferences, ncludng ACM Multmeda, IEEE Computer Vson and Pattern Recognton, IEEE European Conference on Computer Vson, IEEE Asan Conference on Computer Vson, IEEE Internatonal Conference on Image Processng, IEEE Internatonal Conference on Acoustcs, Speech, and Sgnal Processng, IEEE Internatonal Conference on Multmeda and Expo, SPIE ITCom, Internatonal Conference on Pattern Recognton, and Conference on Image and Vdeo Retreval (CIVR). He s a General Co-Char of CIVR 2006, ACM Multmeda 2009, and ICIMCS 2010, and a Program Co-Char of ACM Multmeda 2006, Pacfc Rm Multmeda (PCM) 2006, and IEEE ICME He s/was on the Steerng Commttees of ACM Multmeda, ACM ICMR, IEEE ICME, and PCM. He s the foundng Char of ACM SIG Multmeda Chna Chapter. Dr. Ru receved the B.S. degree from Southeast Unversty, the M.S. degree from Tsnghua Unversty, and the Ph.D. degree from the Unversty of Illnos at Urbana-Champagn. He also holds a Mcrosoft Leadershp Tranng Certfcate from Wharton Busness School, Unversty of Pennsylvana.

14 2032 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 5, MAY 2014 Dacheng Tao (M 07 SM 12) s a Professor of Computer Scence wth the Centre for Quantum Computaton and Informaton Systems and the Faculty of Engneerng and Informaton Technology, Unversty of Technology, Sydney. He manly apples statstcs and mathematcs for data analyss problems n computer vson, machne learnng, multmeda, data mnng, and vdeo survellance. He has authored and co-authored more than 100 scentfc artcles at top venues, ncludng IEEE TRANSACTIONS ON PAT- TERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE Internatonal Conference on Artfcal Intellgence and Statstcs, IEEE COMPUTER VISION AND PATTERN RECOGNITION, and IEEE Internatonal Conference on Data Mnng (ICDM), wth the best theory/algorthm paper runner-up award n IEEE ICDM 07.