Manifold-Ranking Based Keyword Propagation for Image Retrieval *

Transcription

1 Manfold-Rankng Based Keyword Propagaton for Image Retreval * Hanghang Tong,, Jngru He,, Mngjng L 2, We-Yng Ma 2, Hong-Jang Zhang 2 and Changshu Zhang 3,3 Department of Automaton, Tsnghua Unversty, Bejng 00084, Chna 2 Mcrosoft Research Asa, 49 Zhchun Road, Bejng 00080, Chna {walkstar98, hejngru98}@mals.tsnghua.edu.cn, 2 {mjl, wyma, hjzhang}@mcrosoft. 3 zcs@tsnghua.edu.cn ABSTRACT In ths paper, a novel keyword propagaton method s proposed for mage retreval based on a recently developed manfold-rankng algorthm. In contrast to exstng methods whch tran a bnary classfer for each keyword, our keyword model s constructed n a straghtforward manner by explorng the relatonshp among all mages n the feature space n the learnng stage. In relevance feedback, the feedback nformaton can be naturally ncorporated to refne the retreval result by addtonal propagaton processes. In order to speed up the convergence of the uery concept, we adopt two actve learnng schemes to select mages durng relevance feedback. Furthermore, by means of keyword model update, the system can be self-mproved constantly. The updatng procedure can be performed on-lne durng relevance feedback wthout extra off-lne tranng. Systematc experments on a general-purpose mage database consstng of 5,000 Corel mages valdate the effectveness of the proposed method. Keywords Content based mage retreval, keyword propagaton, manfoldrankng, relevance feedback, actve learnng *. INTRODUCTION The ntal mage retreval methods are based on keyword annotaton and can be traced back to the 970 s [2, 5]. In such approaches, mages are frst annotated manually wth keywords, and then retreved by ther annotatons. As long as the annotaton s accurate and plete, keywords can accurately represent the semantcs of mages. However, t suffers from several man dffcultes, e.g., the large amount of manual labor reured to annotate the whole database, and the nconsstency among dfferent annotators n percevng the same mage [3]. Moreover, although t s possble to extract keywords for Web mages from ther surroundng text, such extracton mght be far from accurate and plete [3]. To overe these dffcultes, an alternatve scheme, contentbased mage retreval (CBIR) was proposed n the early 990 s, whch makes use of low-level mage features nstead of the keyword features to represent mages, such as color [4,, 4], texture [8, 0, 7], and shape [2, 2]. Its advantage over keyword based mage retreval les n the fact that feature extracton can be performed automatcally and the mage s own * Ths work was performed at Mcrosoft Research Asa. The frst two authors contrbute eually to ths paper. content s always consstent [3]. Despte the great deal of research work dedcated to the exploraton of an deal descrptor for mage content, ts performance s far from satsfactory due to the wellknown gap between vsual features and semantc concepts,.e., mages of dssmlar semantc content may share some mon low-level features, whle mages of smlar semantc content may be scattered n the feature space [3]. In order to narrow or brdge the gap, a great deal of work has been performed n the past years, such as explorng more powerful lowlevel feature representaton, seekng for more sutable metrc for perceptual smlarty measurement [3] and so on. Furthermore, many efforts have been made to effcently utlze the strengths of both keyword based and content based methods n mage retreval. Those methods can be categorzed nto: on-lne, off-lne and ther bnaton [, 6, 9]. Most, f not all, of exstng on-lne methods make use of relevance feedback (RF). For example, Lu et al proposed n [9] usng a semantc network and relevance feedback based on vsual features to enhance keyword based retreval and update the assocaton of keywords wth mages. Among others, a key ssue n relevance feedback s the learnng strategy. One of the most effectve learnng technues used n RF s support vector machnes (SVM) [8], whch ams to create a classfer that separates the relevant and rrelevant mages and generalzes well on unseen examples. Furthermore, to speed up the convergence to the target concept, actve learnng methods, such as SVM actve [6], are also utlzed to select the most nformatve mages. However, one major problem wth SVM and SVM actve s the nsuffcency of labeled examples, whch mght brng great degradaton to the performance of the traned classfer. On the other hand, Chang et al n [] proposed an off-lne method to perform keyword propagaton based on classfcaton. In ther work, startng from a small porton of manually labeled mages n the database, an ensemble of bnary classfers was traned for multple soft annotatons, whch n turn asssts a user to fnd relevant mages rapdly va keyword. Jng et al n [6] further extended ths work n: ) bnng relevance feedback to refne the retreval result; and 2) ntroducng labelng vector to on-lne collect tranng samples and off-lne update the keyword models. However, there stll exst some lmtatons and drawbacks: ) ther method wll not work f only postve example s provded n relevance feedback; 2) the way they bne the nformaton from relevance feedback s somewhat heurstc; 3) actve learnng s not consdered n relevance feedback; 4) the rato of ntal manually labeled mages s relatvely hgh (ten percent n ther experment), whch s stll a heavy burden especally when the

2 database s large. In ths paper, we propose a novel method to support keyword propagaton for mage retreval based on a recently developed manfold rankng algorthm [9, 20]. Ths work s motvated by our prevous success n applyng ths algorthm n the scenaro of uery by example (QBE) [3] and can be vewed as ts counterpart n the scenaro of uery by keyword (QBK). Frstly, n our method, the keyword model s constructed by explorng the relatonshp among all the mages n the feature space, n contrast to nductve methods whch only use the labeled mages to tran an ensemble of bnary classfers [, 6]. Secondly, our method provdes a very natural way to ncorporate the nformaton from relevance feedback to refne the retreval result. Moreover, to maxmally mprove the performance of propagaton process, actve learnng s nvestgated to select mages n relevance feedback. Fnally, our method also supports accumulaton and memorzaton of knowledge learnt from user-provded relevance feedback by means of keyword model update. Dfferent from [6], n whch an extra off-lne tranng procedure s needed, our update procedure can be performed on-lne durng relevance feedback sessons. The manfold rankng algorthm [9, 20] s ntally proposed to rank the data ponts or to predct the labels of unlabeled data ponts along ther underlyng manfold by analyzng ther relatonshp n Eucldean space. In [3], we ntroduced t n the scenaro of QBE and found out that by ncorporatng unlabeled data n the learnng process and explorng ther relatonshp wth labeled data, ths method outperforms exstng classfcaton-based ones (such as SVM) by a large margn. Motvated by ths, we further apply t to keyword propagaton n ths paper, hopng that t wll stll outperform exstng methods n the scenaro of QBK. Lke n [3], the algorthm frst constructs a weghted graph usng each data pont as a vertex. Next, the keyword model s ntalzed as a keyword matrx wth postve scores of the labeled mages n the correspondng postons. Then these scores are teratvely propagated to nearby ponts va the graph. Fnally each mage n the database wll be gven a score vector, the element of whch ndcates the relevance of the gven mage to the correspondng keyword (a larger score ndcatng hgher relevance). By rankng all the mages accordng to ther relevance to a gven keyword, t can assst a user to fnd relevant mages uckly va keywords. In relevance feedback, f the user only marks relevant examples, they serve as the uery set wth respect to the uery keyword, and ther nfluence can be calculated by an addtonal propagaton process. On the other hand, f both relevant and rrelevant examples are avalable, ther nfluence wll be propagated respectvely n dfferent manners: the effect of negatve examples s suppressed due to the asymmetry between relevant and rrelevant mages. Ths on-lne nformaton, when bned wth the ntal keyword model, wll help to mprove the retreval result. To maxmally mprove the propagaton performance, actve learnng can be also ncorporated n relevance feedback. To be specfc, we wll examne two dfferent schemes developed n [3]: ) to select the most postve mages; and 2) to select the most postve and nconsstent mages. Another mportant ssue wth keyword propagaton s how to accumulate and memorze knowledge learnt from user-provded relevance feedback so that the retreval system can be selfmproved constantly. To acheve ths goal, the keyword model should be updated perodcally. By careful analyss, we reach the concluson that n our method, such update procedure can be performed on-lne durng relevance feedback so that no extra offlne tranng s needed. The organzaton of the paper s as follows. In Secton 2, we brefly revew the two versons of manfold-rankng algorthm and ts applcaton to mage retreval n the scenaro of QBE. We descrbe the constructon of the keyword model usng manfoldrankng algorthm wth some analyss n Secton 3. In Secton 4, the ntal retreval and followng feedback process of QBK scenaro s presented, and the actve learnng schemes are also dscussed. The keyword model updatng s addressed n Secton 5. In Secton 6, we provde systematc expermental results. Fnally, we conclude the paper n Secton RELATED WORK 2. Manfold Rankng Algorthm The manfold rankng algorthm s a sem-supervsed learnng algorthm whch explores the relatonshp among all the data ponts [9, 20]. It has two versons for dfferent tasks: to rank data ponts and to predct the labels of unlabeled data ponts. For the rankng task, t can be formulated as: gven a set of ponts m χ = { x,, x, x+,, xn},the frst ponts are the ueres whch form the uery set; the remanng ponts are to be ranked accordng to ther relevance to the ueres. Let d : χ χ denote a metrc on χ whch assgns each par of ponts x and x j a dstance d( x, x j), and f : χ denote a rankng functon whch assgns to each pont x a rankng score f. Fnally, we defne a vector [,, T y = y y n ] correspondng to the uery set, n whch y = f x s a uery, and y = 0 otherwse. The procedure of rankng the data ponts n [20] can be gven as follows: Algorthm Rankng data ponts. Sort the par-wse dstances among ponts n ascendng order. Repeat connectng the two ponts wth an edge accordng to the order untl a connected graph s obtaned. 2. Form the affnty matrx W defned by 2 2 Wj = exp d ( x, xj ) 2σ f there s an edge lnkng x and x j. Let W = Symmetrcally normalze W by /2 /2 S = D WD n whch D s the dagonal matrx wth (, )-element eual to the sum of the th row of W. f t + = αsf t + α y untl convergence, f 0 = y. 4. Iterate ( ) ( ) ( ) where α s a parameter n [ 0, ) and ( ) * 5. Let f denote the lmt of the seuence { f () t } each pont x accordng to ts rankng scores * f.. Rank

3 For the task of predctng the labels of unlabeled data ponts, t can be formulated as: gven a set of ponts m = x,, x, x,, x and a label set { } {,, c} χ l l+ n ζ =, the frst l ponts x ( l) are labeled as y ζ ; and the remanng ponts x ( l + u n) are to be labeled. u Defne a n c matrx F correspondng to a classfcaton on the dataset χ by labelng each pont x wth y = argmax j c F, j. We also defne a n c matrx Y = [ Y,, Y c ] wth Y, j= f x s labeled as y = j and Y, j= 0 otherwse. The procedure of predctng labels s ute smlar wth that of rankng the data ponts [9]: Algorthm 2 Predctng labels.-3. The same as Algorthm. 4. Iterate F ( t+ ) = αsf( t) + ( α) Y untl convergence, F 0 = Y. 5. Let where α s a parameter n [ 0, ) and ( ) * F denote the lmt of the seuence { F () t } each pont x wth * argmax j c F, j y =.. Label An ntutve descrpton of the above two algorthms s: a weghted graph s frst formed whch takes each data pont as a vertex; a postve score s assgned to each uery whle zero to the remanng ponts; all the ponts then spread ther scores to the nearby ponts va the weghted graph; the spread process s repeated untl a global stable state s reached, and all the ponts wll have ther own scores accordng to whch they wll be ranked or to be labeled. 2.2 Applcaton for Image Retreval n the Scenaro of QBE In [3], we have appled Algorthm to mage retreval n the scenaro of QBE. Its key ponts are summarzed as follows: In the ntal uery stage n the scenaro of QBE, there s only one uery n the uery set. The resultant rankng score of an unlabeled mage s n proporton to the probablty that t s relevant to the uery, wth large rankng score ndcatng hgh probablty. In relevance feedback, f the user only marks relevant examples, the algorthm can be easly generalzed by addng these ly labeled mages nto the uery set; on the other hand, f examples of both labels are avalable, they are treated dfferently: relevant mages are also added to the uery set, whle for rrelevant mages, we desgned three schemes based on the observaton that postve examples should make more contrbuton to the fnal rankng score than negatve ones. To maxmally mprove the rankng result, we also developed three actve learnng methods for selectng mages n each round of relevance feedback. Namely, ) to select the most postve mages; 2) to select the most nformatve mages; and 3) to select the most postve and nconsstent mages. 3. KEYWORD MODEL CONSTRUCTION 3. Notaton Our keyword model s actually an n c matrx F = [ F,, F c ], where n s the total number of mages n the database and c s the total number of keywords. Each mage n the database corresponds to a row and each keyword corresponds to a column. The element F, ( =,, n; =,, c) of the keyword model denotes the relevance of mage x to keyword ndcatng hgh relevance). K (large value 3.2 The Keyword Propagaton Process To construct such keyword model, we need to manually label a small porton of mages n the database, and then propagate ther labels (keywords) to the unlabeled ones. It can be seen that Algorthm 2 can perform ths task well. However, we wll make some modfcatons as follows: Mult-labels for a sngle mage are supported. If an mage s gven more than one keyword n the manually labelng stage, all the correspondng elements n Y are assgned. The weghted graph n step s constructed as: calculate the K nearest neghbors for each pont; connect two ponts wth an edge f they are neghbors. The reason for ths modfcaton s to ensure enough connecton for each pont whle preservng the sparse property of the weghted graph. Snce L dstance can better approxmate the perceptual dfference between two mages than other popular Mnkowsk dstances when usng ether color or texture representaton or both [3], t s adopted to defne the edge weghts n W: m ( ) W = exp x x σ () j l jl l l= where x l and x jl are the lth dmenson of x and x j respectvely; m s the dmensonalty of the feature space; and σ l s a postve parameter that reflects the scope of dfferent dmensons. Step 5 n Algorthm 2 s gnored for the purpose of soft annotaton. 3.3 Analyss We make a short analyss of the keyword propagaton process by Algorthm 2 wth respect to ts transductve learnng; multrankng and ncremental learnng nature Transductve Learnng Nature The theorem n [9] guarantees that the seuence F () t converges to (from now on, we wll omt the mark * ) F = β( αs) Y (2) { } where β = α. Although F can be expressed n a closed form, for large scale problems, the teraton algorthm s preferable due to putatonal effcency. Usng Taylor expanson and omttng the constant coeffcent β, we have

4 ( α ) F = I S Y 2 2 = ( I + αs + α S + ) Y = Y + αsy + αs αsy + ( ) From the above euaton, we can grasp the dea of the algorthm from a transductve learnng pont of vew. F can be regarded as the sum of a seres of nfnte terms. The frst term s smply the score of ntal labels Y, the second term s to spread the scores of the ntal labeled mages to ther nearby ponts, the thrd term s to further spread the scores, and so on. Thus the effect of unlabeled mage s gradually ncorporated. Dfferent from exstng methods, such as [, 6], n whch keyword propagaton s performed by tranng an ensemble of bnary classfers, n our method, t s performed n a much more straghtforward way. Whle those nductve methods am to tran a classfer usng labeled mages whch generalzes well on unlabeled mages, our method s a transductve method and explores the unlabeled mages n the learnng stage. By dong so, we hope t wll outperform the exstng nductve ones Mult-Rankng Nature Snce Y = [ Y,, Y c ], and F = [ F,, F c ], the followng fact wll hold: F = β( αs) Y ( =,, c) (4) Defnng the ntal uery set Q for each keyword (3) K : f a gven mage s labeled as keyword K, t s added nto Q ( =,, c). It can be seen that Y s the correspondng vector as defned n Algorthm for Q. By dong so, we make a brdge between the two versons of manfold-rankng algorthm. The keyword propagaton by Algorthm 2 can be vewed as a mult-rankng process: each keyword has ts own ntal uery set, propagates ts nfluence by step 4 of Algorthm ndependently; and bnes the results altogether Incremental Learnng Nature Here, we explore the ncremental learnng nature of the keyword propagaton process. Snce t can be vewed as a mult-rankng process, we only focus on one specfc keyword K. Let Q and Y be the ntal uery set and the correspondng vector, respectvely. The rankng vector F can be puted as E.4. Suppose that we get some labeled examples for K. Let these examples pose a uery set Q and defne ts correspondng vector y. Addng Q nto Q, we get a bned uery set Q and ts correspondng vector y. The rankng vector wth respect to K should be updated by rerunnng Algorthm on Q, and the seuence { f () t } converges to ( ) = β α (5) f I S y Note that Q = { Q, Q } and y = Y + y. Thus E.5. can be converted to ( ) ( ) f = β I αs Y + β I αs y = F + f where ( ) f = β I αs y. It can be seen from the above euaton that the algorthm provdes a natural way to ncorporate the ly labeled examples: propagate ther nfluence and smply add the result nto the orgnal rankng vector. 4. QUERY BY KEYWORD 4. Intal Retreval Result After the keyword model s constructed, each mage x ( =,, n) n the database corresponds to a row n the matrx, ndcatng ts relevance to dfferent keywords; whle each keyword K corresponds a column n the matrx, n, F = [ F,, F ] T, ndcatng the relevance of dfferent mages to that keyword. Thus, the smlarty score of mage x wth respect to the uery keyword K can be expressed as: { } { } (6) S = F,,, n ;,, c (7) The ntal retreval result s gven by sortng the mages n the decreasng order of ther smlarty scores. As pont out n [], when the uery s not n the keyword set, uery expanson s needed to translate the ntal uery. However, we wll skp the detals of ths ssue n ths paper. 4.2 Relevance Feedback Benefted from ts ncremental learnng nature, our method provdes a natural way to ncorporate the addtonal nformaton from users to refne the smlarty score n relevance feedback. For a uery keyword K, ts ntal rankng vector s F. Let all examples from users feedback pose two uery sets: Q + for postve examples and Q for negatve ones. We also defne ther correspondng vectors y + and y as Algorthm except that the element of y s set to - f the correspondng mage s a negatve example. Usng Algorthms, the effect of these two uery sets can be wrtten as: + + f = β( I αs) y (8) f = β( I αs) y By the ncremental learnng nature we analyzed n 3.3.3, the smlarty score of mage x wth respect to K s updated as: + { } { } S = F, + f + γ f,, n ;,, c (9) where f + and f th are the elements of f + and f, respectvely, and γ [0,]. Note that the effect of negatve

5 examples s suppressed by γ, the dea of whch can be traced back to [3]: due to the asymmetry between relevant and rrelevant mages, the postve and negatve examples should be treated dfferently. Generally speakng, postve examples should make more contrbuton to the overall smlarty score than negatve ones. Here the parameter γ controls the suppresson extent: the smaller γ s; the less mpact negatve examples wll have on the overall smlarty score. When only postve examples are avalable from the user s feedback or when we consder only the relevant mages, we smply set γ = 0 and the smlarty score s updated as: + { } { } S = F, + f,, n ;,, c (0) 4.3 Actve Learnng Contrary to passve learnng, n whch the learner randomly selects some unlabeled mages and asks the user to provde ther labels, actve learnng selects mages accordng to some prncple, hopng to speed up the convergence to the uery concept. Ths scheme has been proven to be effectve n mage retreval by prevous research work [7, 6]. In [3], we have developed three actve learnng methods based on dfferent prncples, and each of them has ts counterpart n the scenaro of QBK: The frst method s to select the unlabeled mages wth the largest S,.e., the most postve mages, whch s wdely used n prevous research work [5, 8]. The motvaton behnd ths smple scheme s to ask the user to valdate the judgment of the current system on mage relevance. The second method s to select the unlabeled mages wth the + smallest F, + f + γ f. Snce the value of F, and f + ndcates the relevance of an unlabeled mage determned by ntal labels and postve examples, respectvely, whle the absolute value of γ f ndcates the rrelevance of an unlabeled mage determned by negatve examples, a small value of + F, + f + γ f means that the mage s judged to be relevant by the same degree as t s judged to be rrelevant, therefore, t can be consdered an nconsstent one. From the perspectve of nformaton theory, such mages are most nformatve. The thrd method tres to take the advantage of the above two schemes by selectng the nconsstent mages whch are also ute smlar to the uery (most postve and nconsstent). To be specfc, we select unlabeled mages wth the largest + + F, + f F, + f + γ f. The scheme can be explaned ntutvely as follows: the selected mages should not only provoke maxmum dsagreement among labeled examples (small + F, + f + γ f ), they must also be relatvely confdently judged as a relevant one by the ntal labels and the postve examples (large F, + f + ). In [3], we found out by experments that the second scheme s not as effectve as the other two. So n ths paper, we wll only adopt the frst (most postve) and the thrd (most postve and nconsstent) schemes. For keyword propagaton, another nterestng matter wth actve learnng s how to select the mages for labelng n the stage of ntal keyword model constructon. However, we wll not address ths ssue n ths paper and wll leave t to future work. 5. KEYWORD MODEL UPDATE An mportant ssue wth keyword propagaton s how to accumulate and memorze knowledge learnt from user-provded relevance feedback so that the retreval system can be selfmproved constantly. In [6], Jng et al ntroduced labelng vectors to collect examples provded by the users and ther keyword model (an ensemble of bnary SVM classfers) s perodcally updated by an off-lne tranng procedure. It s very easy to ncorporate such updatng procedure n our method. Remember that each column F ( =,, c) n our keyword model corresponds to a keyword K. Takng the multrankng nature of keyword propagaton, the updatng procedure s performed on one column by one as follows: Frstly, consder postve examples only. For a uery keyword ts j, + + j, ntal rankng vector s F. Let Q ( j =,, N ) and y + denote the ensemble of the uery sets and correspondng vectors from varous postve feedback sessons, where N + s the total number of feedback sessons used to update the keyword model. Usng Algorthm, ther effect can be wrtten as: ( ) j, + j, f β I αs + + = y ( j =,, N ) () Takng advantage of the ncremental learnng nature agan, the fnal updatng process can be denoted as: + N j, + (,, ) (2) j= F F + f for = c If both postve examples and negatve examples are consdered, k, j, let Q ( k =,, N ) and y denote the ensemble of the uery sets and correspondng vectors from varous negatve feedback sessons, where N s the total number of negatve feedback sessons used to update the keyword model. By a smlar analyss, we reach the followng updatng procedure: + N N j, + k, F F + f + γ f ( for =,, c) (3) j= k= k where ( ), k, f β I αs = y ( k =,, N ), and γ [0,] s the controllng parameter as dscussed n secton If only postve examples are avalable or consdered, γ = 0. In both cases, the ensemble of correspondng vectors actually j, + plays a role of labelng vector as n [6]. Moreover, note that f and k, f are actually what we get durng relevance feedback, thus the keyword model can be updated on-lne durng the relevance feedback sessons, and there s no extra off-lne tranng process.

6 0.4 SVM Manfold-Rankng SVM Manfold-Rankng Precson 0.3 Precson Percentage of tranng data (%) (a) Precson vs. scope (a) P20 vs. the percentage of tranng data 0.3 SVM Manfold-Rankng 0.2 SVM Manfold-Rankng Recall Recall Percentage of tranng data (%) (b) Recall vs. scope Fgure. Comparson of the ntal retreval result between manfold-rankng and SVM. Only % of mages were labeled for tranng. (b) R20 vs. the percentage of tranng data Fgure 2. Systematc parson between manfold-rankng and SVM under dfferent sze of tranng data. 6. EXPERIMENTAL RESULT 6. Experment Desgn We have evaluated the proposed method wth a general-purpose mage database of 5,000 mages from COREL. In our experments, one percent (much less than that of [6]: ten percent) of all mages n the database are randomly selected for manual annotaton and used to tran the ntal keyword model. Currently, an mage s labeled wth only one keyword,.e. the name of the category that contans t. Some categores are sunset, mountan, eagle, beach, subsea anmal. Totally, there are 50 keywords representng all mages n the database. Images from the same (dfferent) category are consdered relevant (rrelevant). The precson vs. scope curve s used to evaluate the performance of varous methods. We use each keyword as a uery. Consderng the randomness of ntal labels, we run 50 tmes of labelng and tranng for each uery and the average retreval result s recorded. Fnally, we average the results over the total 50 ueres. Image feature has a great mpact on the performance of mage retreval system. However, n ths paper, our major concern s relatve performance parson and therefore we do not perform careful feature selecton. In our current mplementaton, the features that we use to represent each mage nclude color hstogram [4] and wavelet feature [7]. Color hstogram s obtaned by uantzng the HSV color space nto 64 bns. To calculate the wavelet feature, we frst perform 3-level Daubeches wavelet transform to the mage, and then calculate the frst and second order moments of the coeffcents n Hgh/Hgh, Hgh/Low and Low/Hgh bands at each level. There are fve parameters left to be set n the algorthm: K, α, σ l, γ and the teraton steps. As ponted out n [3], the algorthm s not very senstve to the number of neghbors. In ths paper, we set K = 20. The other four parameters are consstent wth what we dd n [3], that s, α = 0.99, σ l = 0.05, γ =, and the number of teraton steps s 50.

7 6.2 Intal Retreval Result Frstly, the ntal retreval s evaluated. The precson (recall) vs. scope curve s shown n Fgure. In order to perform a systematc evaluaton, we vary the percentage of tranng data and pare the average precson (P20) and recall (R20) of top 20 retreved mages wth that by SVM [6]. The precson (recall) vs. the percentage of tranng data curve s shown n Fgure 2. From the fgures, t can be seen that our manfold-rankng based method outperforms the classfcaton-based one by a large margn, especally when only a small number of mages were labeled for tranng. The mprovement s very sgnfcant from the practcal pont of vew. 6.3 Relevance Feedback In ths case, we fx the total number of mages that are marked by the user to 20, but vary the tmes of feedback and the number of feedback mages each tme accordngly. The bnatons used n ths experment nclude: feedback wth 20 mages each tme; 2 feedbacks wth 0 mages each tme; and 4 feedbacks wth 5 mages each tme. When both postve and negatve examples are avalable, passve scheme (select randomly) and two actve schemes (to select most postve and to select most postve and nconsstent) are pared. When only postve examples are avalable, the most postve and nconsstent scheme s skpped. Note that, n the frst round of relevance feedback, the most postve and nconsstent scheme s not provoked, and the most postve mages are selected for users labels. In all experments, we fnd out that both of our actve schemes help to mprove the retreval performance by a large margn, whle the passve scheme makes lttle mprovement. Here, we only present the results after 2 feedbacks wth 0 mages each tme n Fgure 3 and the ntal result s also gven as a reference. 6.4 Keyword Model Update To collect tranng data for the updatng process, each of the 50 keywords s used as the uery once. In ths case, users feedback processes are smulated as follows. For a uery mage, 5 teratons of user-and-system nteracton were carred out. At each teraton, 5 most postve mages are labeled by the user. Both postve and negatve examples are consdered. The ntal retreval result after updatng the keyword model s presented n Fgure 4 (a), together wth that wthout the updatng procedure as a reference. The effect of updatng process on subseuent relevance feedback sessons s also evaluated and the retreval results (one feedback wth 0 mages) wth and wthout updatng process are shown n the Fgure 4 (b). It can be seen that the updatng process enables the proposed system to self-mprove progressvely. 7. CONCLUSION In ths paper, we have proposed a novel method to support keyword propagaton for mage retreval. Ths work s an extenson to our prevous work n the scenaro of QBE and can be vewed as ts counterpart n the scenaro of QBK. Startng from a very small porton of labeled mages, a keyword model s constructed by the manfold-rankng algorthm and all the mages n the database are softly annotated. Dfferent from exstng methods whch rely on labeled data to tran an ensemble of bnary classfers, ours s a transductve one whch explores the relatonshp among all labeled and unlabeled mages n the learnng stage. Such keyword model serves as a brdge that connects the semantc keyword space wth the low-level feature space. The nformaton from relevance feedback can be naturally ncorporated to refne the retreval result; and the nfluence of postve examples and negatve ones are treated dfferently. Two actve schemes are adopted to accelerate the convergence to the uery concept. The proposed keyword model can be updated onlne wthout extra off-lne tranng process. Experments on a general-purpose mage database consstng of 5,000 Corel mages demonstrate the effectveness of the proposed method. 8. ACKNOLEDGEMENTS Ths work s supported by the project ( ) of the Natonal Natural Scence Foundaton of Chna. 9. REFERENCE [] Chang, E., et al. CBSA: content-based soft annotaton for multmodal mage retreval usng Bayes pont machnes. IEEE Trans on Crcuts and Systems for Vdeo Technology, Volume 3, Number, pp.26-38, January [2] Chang, S.K. and Hsu, A. Image nformaton systems: where do we go from here? IEEE Trans. on Knowledge and Data Engneerng, 4(5), Oct [3] He J.R., L M., Zhang H.J., Zhang C.S., and Tong H.H. Manfold-rankng based mage retreval. To Appear n Proc. ACM Internatonal Multmeda Conference, [4] Huang, J., et al. Image ndexng usng color correlograms. Proc. IEEE Conf. on Computer Vson and Pattern Recognton, pp , 997. [5] Jng F., L M., Zhang H.J., and Zhang B. An effectve regon-based mage retreval framework. Proc. ACM Internatonal Multmeda Conference, [6] Jng F., L M., Zhang H.J., and Zhang B. Keyword propagaton for mage retreval. Proc. IEEE Internatonal Symposum on Crcuts and Systems, 2004 [7] L, B., Chang, E., and L, C.S. Learnng mage uery concepts va ntellgent samplng. Proc. IEEE Int. Conf. on Multmeda & Expo, pp , 200. [8] Lu, F., and Pcard, R.W. Perodcty, drectonalty, and randomness: Wold features for mage modelng and retreval. IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 8, 996. [9] Lu, Y., et al. A unfed framework for semantcs and feature based relevance feedback n mage retreval systems. Proc. ACM Internatonal Multmeda Conference, [0] Manjunath, B.S., and Ma, W.Y. Texture features for browsng and retreval of mage data. IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 8, pp , 996. [] Pass, G. Comparng mages usng color coherence vectors. Proc. 4th ACM Int. Conf. on Multmeda, pp , 997. [2] Schmd, C., and Mohr, R. Local grayvalue nvarants for mage retreval. IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 9, pp , 997. [3] Shen, H.T., et al. Gvng meanngs to WWW mages. Proc. 4th ACM Int. Conf. on Multmeda, 2000.

8 [4] Swan, M., and Ballard, D. Color ndexng. Int. Journal of Computer Vson, 7(): -32, 99. [5] Tamura, H. and Yokoya, N. Image database systems: a survey. Pattern Recognton, Vol. 7, No., 984. [6] Tong, S. and Chang, E. Support vector machne actve learnng for mage retreval. Proc. 9th ACM Int. Conf. on Multmeda, 200. [7] Wang, J.Z., Wederhold, G., Frschen, O., and Sha, X.W. Content-based mage ndexng and searchng usng Daubeches wavelets. Int. Journal of Dgtal Lbrares, vol., no. 4, pp , 998. [8] Zhang, L., Ln, F., and Zhang, B. Support Vector Machne learnng for mage retreval. Proc. IEEE Int. Conf. on Image Processng, vol. 2, pp , 200. [9] Zhou, D., et al. Learnng wth local and global consstency. 8th Annual Conf. on Neural Informaton Processng Systems, [20] Zhou, D., et al. Rankng on data manfolds. 8th Annual Conf. on Neural Informaton Processng System, [2] Zhou, X.S., Ru, Y., and Huang, T. Water-Fllng: a novel way for mage structural feature extracton. Proc. IEEE Int. Conf. on Image Processng, vol. 2, pp , 999. Intal Result 0.75 Randomly Select Most Postve Befor Update After Update Precson Precson (a) Only postve examples are consdered (a) Intal retreval result Precson Intal Result Randomly Select Most Postve Most Postve and Inconsstent Precson 0.75 Befor Update After Update (b) Both postve and negatve examples are consdered. Fgure 3. Comparson of dfferent relevance feedback schemes (2 feedbacks wth 0 mages at each). (b) Relevance feedback (one feedback wth 0 mages) Fgure 4. Comparson of performance mprovement of the keyword model update process.