A Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems

Transcription

1 A Unfed Framework for Semantcs and Feature Based Relevance Feedback n Image Retreval Systems Ye Lu *, Chunhu Hu 2, Xngquan Zhu 3*, HongJang Zhang 2, Qang Yang * School of Computng Scence Smon Fraser Unversty Burnaby, B.C., Canada, V5AS6 {yel,qyang}@cs.sfu.ca 2 Mcrosoft Research Chna 5F, Bejng Sgma Center Bejng 00080, Chna {-chhu,hjzhang}@mcrosoft.com 3 Department of Computer Scence Fudan Unversty Shangha , Chna 98005@fudan.edu.cn ABSTRACT The relevance feedback approach to mage retreval s a powerful technque and has been an actve research drecton for the past few years. Varous ad hoc parameter estmaton technques have been proposed for relevance feedback. In addton, methods that perform optmzaton on mult-level mage content model have been formulated. However, these methods only perform relevance feedback on the low-level mage features and fal to address the mages semantc content. In ths paper, we propose a relevance feedback technque, Fnd, to take advantage of the semantc contents of the mages n addton to the low-level features. By formng a semantc network on top of the keyword assocaton on the mages, we are able to accurately deduce and utlze the mages semantc contents for retreval purposes. The accuracy and effectveness of our method s demonstrated wth expermental results on real-world mage collectons. Keywords relevance feedback, mage semantcs, mage retreval, multmeda database.. ITRODUCTIO Wth the ncreasng avalablty of dgtal mages, automatc mage retreval tools provde an effcent means for users to navgate through them. Even though tradtonal methods allow the user to post queres and obtan results, the retreval accuracy s severely lmted because of the nherent complexty of the mages for users to descrbe exactly. The more recent relevance feedback approach, on the other hand, reduces the needs for a user to provde accurate ntal queres by estmatng the user s deal query usng the postve and negatve examples gven by the user. The current relevance feedback based systems estmate the deal query parameters on only the low-level mage features such as color, texture, and shape. These systems work well f the feature vectors can capture the essence of the query. For example, f the user s searchng for an mage wth complex textures havng a partcular combnaton of colors, ths query would be extremely dffcult to descrbe but can be reasonably represented by a combnaton of color and texture features. Therefore, wth a few postve and negatve examples, the relevance feedback system wll be able to return reasonably accurate results. On the other hand, f the user s searchng for a specfc object that cannot be suffcently represented by combnatons of avalable feature vectors, these relevance feedback systems wll not return many relevant results even wth a large number of user feedbacks. To address the lmtatons of the current relevance feedback systems, we propose a framework that performs relevance feedback on both the mages semantc contents represented by keywords and the low-level feature vectors such as color, texture, and shape. The contrbuton of our work s twofold. Frst, t ntroduces a method to construct a semantc network on top of an mage database and uses a smple machne learnng technque to learn from user queres and feedbacks to further mprove ths semantc network. In addton, we propose a framework n whch semantc and low-level feature based relevance feedback can be seamlessly ntegrated. Ths paper s organzed as follows. In Secton 2, we wll provde an overvew of the current state of the art relevance feedback systems. In Secton 3, we wll present the detals of our work. Secton 4 wll descrbe the Fnd mage retreval system that we have mplemented based on the proposed method and provde expermental evaluatons showng ts effectveness n mage retreval. Concludng remarks wll be gven n Secton RELATED WORK One of the most popular models used n nformaton retreval s the vector model [, 8, 9]. Varous effectve retreval technques have been developed for ths model and among them s the method of relevance feedback. Most of the prevous relevance feedback research can be classfed nto two approaches: query pont movement and re-weghtng [3]. The query pont movement method essentally tres to mprove the estmate of the deal query pont by movng t towards good examples pont and away from bad example ponts. The frequently used technque to teratvely mprove ths estmaton s the Roccho s formula gven below for sets of * Ths work was performed at Mcrosoft Research Chna.

2 relevant documents D R and non-relevant documents D gven by the user. Q' = αq + β ( R' D' R D ) γ ( ' D' D ) where α, β, and γ are sutable constants; R and are the number of documents n D R and D respectvely. Ths technque s mplemented n the MARS system [6]. Experments show that the retreval performance can be mproved consderably by usng relevance feedback [, 8, 9]. The central dea behnd the re-weghtng method s very smple and ntutve. The MARS system mentoned above mplements a slght refnement to the re-weghtng method call the standard devaton method [6]. Snce each mage s represented by an dmensonal feature vector, we can vew t as a pont n an dmensonal space. Therefore, f the varance of the good examples s hgh along a prncple axs j, then we can deduce that the values on ths axs s not very relevant to the nput query so that we assgn a low weght w j on t. Therefore, the nverse of the standard devaton of the j th feature values n the feature matrx s used as the basc dea to update the weght w j. Recently, more computatonally robust methods that perform global optmzaton have been proposed. The MndReader retreval system desgned by Ishkawa et al. [3] formulates a mnmzaton problem on the parameter estmatng process. Unlke tradtonal retreval systems whose dstance functon can be represented by ellpses algned wth the coordnate axs, the MndReader system proposed a dstance functon that s not necessarly algned wth the coordnate axs. Therefore, t allows for correlatons between attrbutes n addton to dfferent weghts on each component. A further mprovement over ths approach s gven by Ru and Huang [7]. In ther CBIR system, t not only formulates the optmzaton problem but also takes nto account the mult-level mage model. All the approaches descrbed above perform relevance feedback at the low-level feature vector level, but faled to take nto account the actual semantcs for the mages themselves. The nherent problem wth these approaches s that the low-level features are often not as powerful n representng complete semantc content of mages as keywords n representng text documents. In other words, applyng the relevance feedback approaches used n text nformaton retreval technologes to lowlevel feature based mage retreval wll not be as successful as n text document retreval. In vewng ths, there have been efforts on ncorporatng semantcs n relevance feedback for mage retreval. The framework proposed n [4] attempted to embed semantc nformaton nto a low-level feature based mage retreval process usng a correlaton matrx. In ths effectve framework, semantc relevance between mage clusters s learnt from user s feedback and used to mprove the retreval performance. As we shall show later, our proposed method ntegrates both semantcs and low-level features nto the relevance feedback process n a new way. Only when the semantc nformaton s not avalable, our method s reduced to one of the prevously descrbed low-level feedback approaches as a specal case. 3. THE PROPOSED METHOD There are two dfferent modes of user nteractons nvolved n typcal retreval systems. In one case, the user types n a lst of () keywords representng the semantc contents of the desred mages. In the other case, the user provdes a set of examples mages as the nput and the retreval system wll try to retreve other smlar mages. In most mage retreval systems, these two modes of nteracton are mutually exclusve. We argue that combnng these two approaches and allow them to beneft from each other yelds a great deal of advantage n terms of both retreval accuracy and ease of use of the system. In ths secton, we descrbe a method to construct a semantc network from an mage database and present a smple machne learnng algorthm to teratvely mprove the system s performance over tme. In addton, we descrbe a framework n whch the prevously constructed semantc network can be seamlessly ntegrated wth low-level feature vector based relevance feedback. 3. Semantc etwork The semantc network s represented by a set of keywords havng lnks to the mages n the database. Weghts are assgned to each ndvdual lnk. Ths representaton s shown pctorally as follows. mage mage 2 mage 3 mage M w 2 w w 2 w n w 3 w 22 wnk keyword keyword 2 keyword Fgure : Semantc network The lnks between the keywords and mages provde structure for the network. The degree of relevance of the keywords to the assocated mages semantc content s represented as the weght on each lnk. It s clear that an mage can be assocated wth multple keywords, each of whch wth a dfferent degree of relevance. Keyword assocatons may not be avalable at the begnnng. There are several ways to obtan keyword assocatons. The frst method s to smply manually label mages. Ths method may be expensve and tme consumng. To reduce the cost of manual labelng, we utlze the Internet and ts countless number of users. One possble way to do that may be to mplement a crawler to go to dfferent webstes to download mages. We store the nformaton such as the fle name and the ALT tag strng wthn the IMAGE tags of the HTML fles as keywords assocated wth the downloaded mage. Also, the lnk strng and the ttle of the page may be somewhat related to the mage. We assgn weghts to these keyword lnks accordng to ther relevance. Heurstcally, we lst ths nformaton n the order of descendng relevance: the lnk strng, the ALT tag strng, the fle name, and the ttle of the page. Another approach to ncorporate addtonal keywords nto the system would be to utlze the user s nput queres. Whenever the user feeds back a set of mage beng relevant to the current query, we add the nput keywords nto the system and lnk them wth these mages. In addton, snce the user tells us that these mages are relevant, we can confdently assgn a large weght on each of the newly created

3 lnks. Ths effectvely suggests a very smple votng scheme for updatng the semantc network n whch the keywords wth a majorty of user consensus wll emerge as the domnant representaton of the semantc content of ther assocated mages. 3.2 Semantc Based Relevance Feedback Semantc based relevance feedback can be performed relatvely easly compared to ts low-level feature counterpart. The basc dea behnd t s a smple votng scheme to update the weghts w j assocated wth each lnk shown n Fgure wthout any user nterventon. The weght updatng process s descrbed below.. Intalze all weght w j to. That s, every keyword has the same mportance. 2. Collect the user query and the postve and negatve feedback examples. 3. For each keyword n the nput query, check to see f any of them s not n the keyword database. If so, add them nto the database wthout creatng any lnks. 4. For each postve example, check to see f any query keyword s not lnked to t. If so, create a lnk wth weght from each mssng keyword to ths mage. For all other keywords that are already lnked to ths mage, ncrement the weght by. 5. For each negatve example, check to see f any query keywordslnkedwtht. Ifso,setthenewweght w j =w j /4. If the weght w j on any lnk s less than, delete that lnk. It can be easly seen that as more queres are nputted nto the system, the system s able to expand ts vocabulary. Also, through ths votng process, the keywords that represent the actual semantc content of each mage wll receve a large weght. The weght w j assocated on each lnk of a keyword represents the degree of relevance n whch ths keyword descrbes the lnked mage s semantc content. For retreval purposes, we need to consder another aspect. The mportance of keywords that have lnks spreadng over a large number of mages n the database should be penalzed. Therefore, we suggest the relevance factor r k of the k th keyword assocaton be computed as follows. M rk = wk (log 2 +) d where M s the total number of mages n the database, w k = w mn f m = and 0 otherwse, and d s the number of lnks th keyword has. 3.3 Integraton wth Low-Level Feature Based Relevance Feedback Snce [7] summarzed a general framework n whch all the other low-level feature based relevance feedback methods dscussed n Secton 2 can be vewed as ts specal cases, n ths secton, we show how the semantc relevance feedback method can be seamlessly ntegrated wth t. To expand the framework summarzed n [7] to nclude semantc feedback, notce that the nputs to t are a query vector q assocated wth the th feature, an element vector π=[π,...π ] that represents the degree of relevance for each of the nput (2) tranng samples, and a set of tranng vectors x n for each feature. As shown n [7], the deal query vector q * for feature s the weghted average of the tranng samples for feature gven by q π X T T * = n = π n where X s the K tranng sample matrx for feature, obtaned by stackng the tranng vectors x n nto a matrx. The optmal weght matrx W *sgvenby K W * = (det( C )) C where C s the weghted covarance matrx of X.Thats C rs = n= π ( x n nr q )( x n= r π n ns q s ) r, s =, KK We can see from the above equatons that the crtcal nputs nto the system are x n and π. Intally, the user nputs these data to the system. However, we can elmnate ths frst step by automatcally provdng the system wth ths ntal data. Ths s done by searchng the semantc network for keywords that appear n the nput query. From these keywords, we can follow the lnks to obtan the set of tranng mages (duplcate mages are removed). The vectors x n can be computed easly from the tranng set. To compute the degree of relevance vector π, we can use the followng formula. π = α M M rj j= where M s the number of query keywords lnked to the tranng mage, r jk s the relevance factor of the j th keyword assocated wth mage, andα >s a sutable constant. We can see that the degree of relevance of the th mage ncreases exponentally wth the number of query keywords lnked to t. In the current mplementaton of our system, we have expermentally determned that settng α to 2.5 gves the best result. To ncorporate the low-level feature based feedback and rankng results nto hgh-level semantc feedback and rankng, we defne a unfed dstance metrc functon G j to measure the relevance of any mage j wthn the mage database n terms of both semantc and low-level feature content. The functon G j s defned usng a modfed form of the Roccho s formula as follows. I Gj = log( + π j) Dj + β + Sjk R k A R I2 γ + Sjk (7) 2 k A where D j s the dstance score computed by the low-level feedback n [7], R and are the number of postve and negatve feedbacks respectvely, I s the number of dstnct keywords n common between the mage j and all the postve feedback (3) (4) (5) (6)

4 mages, I 2 s the number of dstnct keywords n common between the mage j and all the negatve feedback mages, A and A 2 are the total number of dstnct keywords assocated wth all the postve and negatve feedback mages respectvely, and fnally S j s smply the Eucldean dstance of the low-level features between the mages and j. We have replaced the frst parameter α n Roccho s formula wth the logarthm of the degree of relevance of the j th mage. The other two parameters β and γ are assgned a value of.0 n our current mplementaton of the system for the sack of smplcty. However, other values can be gven to emphasze the weghtng dfference between the last two terms. Usng the method descrbed above, we can perform the combned relevance feedback as follows.. Collect the user query keywords 2. Use the above method to compute x n and π and nput them nto the low-level feature relevance feedback component to obtan the ntal query results. 3. Collect postve and negatve feedbacks from the user 4. Update the semantc network wth the method gven n secton Update the weghts of the low-level feature based component usng the methods dscussed n [7] 6. Compute the new x n and π and nput nto the low-level feedback component 7. Compute the rankng score for each mage usng equaton 7 and sort the results. 8. Show new results and go to step 3 Usually the values of x n are computed beforehand n a preprocessng step. We can see that usng ths approach, our system learns from the user s feedback both semantcally and n a feature based manner. In addton, t can be easly seen that our method degenerates nto the method of Ru and Huang [7] when no semantc nformaton s avalable. We wll show n the next secton how our system deals wth nput queres that have no assocated mages from the semantc network. Also, next secton wll present some expermental results to confrm the effectveness of ths approach. 3.4 ew Image Regstraton Addng new mages nto the database s a very common operaton under many crcumstances. For retreval systems that entrely rely on low-level mage features, addng new mages smply nvolves extractng varous feature vectors for the set of new mages. However, snce our system utlzes keywords to represent the mages semantc contents, the semantc contents of the new mages have to be labeled ether manually or automatcally. In ths secton, we present a technque to perform automatc labelng of new mages. In paper [5], a method was presented whch automatcally classfy mages nto only two categores, ndoor and outdoor, based on both text nformaton and low-level feature. There s currently no algorthm avalable to automatcally determne the semantc content of arbtrary mages accurately. We mplemented a scheme to automatcally label the new mages by guessng ther semantc contents usng low-level features. The followng s a smple algorthm to acheve ths goal.. For each category n the database, compute the representatve feature vectors by determnng the centrod of all mages wthn ths category. 2. For each category n the database, fnd the set of representatve keywords by examnng the keyword assocaton of each mage n ths category. The top keywords wth largest weght whose combned weght does not exceed a prevously determned threshold τ are selected and added nto the lst the representatve keywords. The value of the threshold τ s set of 40% of the total weght as dscussed n secton For each new mage, compare ts low-level feature vectors aganst the representatve feature vectors of each category. The mages are labeled wth the set of representatve keywords from the closest matchng category wth an ntal weght of.0 on each keyword. Because the low-level features are not enough to present the mages semantcs, some or even all of the automatcally labeled keywords wll nevtably be naccurate. However, through user queres and feedbacks, semantcally accurate keywords labels wll emerge. Anther problem related to automatc labelng of new mages s the automatc classfcaton of these mages nto predefned categores. We solve ths problem wth the followng algorthm.. Put the automatcally labeled new mages nto a specal unknown category. 2. At regular ntervals, check every mage n ths category to see f any keyword assocaton has receved a weght greater than a threshold ξ. If so, extract the top keywords whose combned weght does not exceed the threshold τ. 3. For each mage wth extracted keywords, compare the extracted keywords wth the lst of representatve keywords from each category. Assgned each mage to the closest matchng category. If none of the avalable categores result n a meanngful match, leave ths mage n the unknown category. The keyword lst comparson functon used n step 3 of the above algorthm can take several forms. The deal functon would take nto account the semantc relatonshp of keywords n one lst wth those of the other lst. However, for the sake of smplcty, our system only checks for the exstence of keywords from the extracted keyword lst n the lst of representatve keywords. 4. EXPERIMETAL RESULTS We have presented a framework n whch semantc and low-level feature based feedback can work together to acheve greater retreval accuracy. In ths secton, we wll descrbe the mage retreval system Fnd that we have mplemented usng ths framework and show some expermental results. 4. The Fnd Retreval System The Fnd mage retreval system mplements the framework dscussed n ths paper. It s a web based retreval system n whch multple users can perform retreval tasks smultaneously at any gven tme.

5 The Fnd system supports three modes of nteracton: keyword based search, search by example mages, as well as browsng the entre mage database usng a pre-defned category herarchy. The man user nterface s shown n Fgure 2. When the user enters a keyword-based query, the system nvokes the combned relevance feedback mechansm dscussed n Secton 3.3. The result page s shown n Fgure 3. Fgure 2: Man user nterface. The user s able to select multple mages from ths page and clck on the Feedback button to gve postve and negatve feedback to our system. The mages wth blue background ndcate a postve feedback whle mages wth a red background ndcate a negatve feedback. Images wth gradent background are not consdered n the relevance feedback process. The system presents 240 mages for each query. The frst 00 mages are actually retreved usng the algorthm outlned n Secton 3. The next 20 mages are randomly selected from each category. The fnal 20 mages are randomly selected regardless of categores. The purpose of presentng the randomly selected mages would be to gve the user a new startng pont f none of the mages actually retreved by our system can be consdered relevant. ew search results wll be presented to the user as soon as the Feedback button s pressed. At any pont durng the retreval process, the user can clck on the Vew lnk to vew a partcular mage n ts orgnal sze, or clck on the Smlar lnk to perform an example based query. One pont of detal to note s that f the user enters a set of query keywords that cannot be found n the semantc network, the system wll smply output the mages n the database one page at a tme to let the user browse through and select the relevant mages to feedback nto the system. 4.2 Results Here are some expermental results that we have gathered from our system to valdate some smple assumptons and demonstrate ts effectveness. Because we are nterested n examnng how the semantc network evolves wth an ncreasng number of user feedbacks, we select a very clean but roughly labeled mage set as our startng pont. The dataset that we have chosen s from the Corel Image Gallery. We have selected 2,000 mages and manually classfed them nto 60 categores. One assumpton we have made n the desgn of the system s that a sgnfcant porton of the total weght of all the keyword assocatons wth an mage s concentrated on a subset of keywords that are relevant to the semantc content of the mage. Ths relatonshp s shown n Fgure 4 wth the x axs beng the number of keywords assocated wth the mage and the y axs beng the average percentage of the total weght that are assgned to relevant keywords. Weght contrbuted by relevant keywords umber of keyword assocatons Fgure 3: The query result page Fgure 4: Keyword relevance VS keyword count.

6 To obtan the graph shown n Fgure 4, we have asked human subjects to examne the keyword assocaton on the mages havng 2 to 7 keywords assocated and pck out the relevant keywords. These keyword assocatons are obtaned from the user query usng the method descrbed n Secton 3. We have also verfed that the keywords wth large weghts are ndeed the relevant keywords selected by the users. From the plot of Fgure 4, we can see that as the number of keyword assocatons ncrease, the percentage of the weght contrbuted by the relevant keywords levels off to approxmately 40%. We therefore conjecture that f we rank the keywords n descendng order of ther assocated weght and select the top few that contrbute no more than 40% of the total weght, the selected keywords wll be an accurate representaton of the semantc meanng of the mage. The verfcaton of ths conjecture s currently on the lst of our future works. Fgure 5 shows the performance of our system n terms of precson and recall. We performed eght random queres on our system. We ensured that none of the query keywords are labeled on any of the mages and that there are exactly 00 mages wth the correct semantc content n our mage database. Snce we have used exactly 00 mages as our ground truth for each query and that we only actually retreve 00 mages, the value of precson and recall s the same. Therefore, we have used the term Accuracy to refer to both n our plot. Accuracy umber of relevance feedback Fgure 5: System performance. As we can see from the results, our system acheves on average 80% retreval accuracy after just 4 user feedback teratons and over 95% after 8 teratons for any gven query. In addton, we can clearly see that more relevant mages are beng retreved as the number of user feedbacks ncrease. Unlke some earler methods where more user feedback may even lead to lower retreval accuracy, our method proves to be more stable. In addton to verfyng the effectveness of our system through the performance measure shown n Fgure 5, we have also compared t aganst other state of the art mage retreval systems. We have chosen to compare our method wth the retreval technque used n the CBIR system [7]. The comparson s made through 8 sets of random queres wth 0 feedback teratons for each set of query and the number of correctly retreved mages s counted after each user feedback. The average accuracy s then plotted aganst the number of user feedbacks. The result s shown n Fgure 6. Accuracy umber of relevance feedback Fgure 6: Performance comparson. Fnd CBIR It s easly seen from the above result that by combnng semantc level feedback wth low-level feature feedback, the retreval accuracy s mproved substantally. 5. COCLUSIO In ths paper, we have presented a new framework n whch semantcs and low-level feature based relevance feedbacks are combned to help each other n achevng hgher retreval accuracy wth lesser number of feedback teratons requred from the user. The novel feature that dstngushed the proposed framework from the exstng feedback approaches n mage database s twofold. Frst, t ntroduces a method to construct a semantc network on top of an mage database and uses a smple machne learnng technque to learn from user queres and feedbacks to further mprove ths semantc network. In addton, a scheme s ntroduced n whch semantc and low-level feature based relevance feedback s seamlessly ntegrated. Expermental evaluatons of the proposed framework have shown that t s effectve and robust and mproves the retreval performance of CBIR systems sgnfcantly. We have chosen to use the approach summarzed n [7] as our low-level feature based feedback component. However, t can be easly demonstrated that ths framework s general enough to allow any low-level feedback method to be ncorporated. As a future work, we wll study the possblty to ncorporate the approaches proposed n [2, 4] to further mprove the performance of the Fnd system. 6. REFERECES [] Buckley, C., and Salton, G. Optmzaton of Relevance Feedback Weghts, n Proc of SIGIR 95. [2] Cox, I.J., Mller, M.L., Mnka, T.P., Papathornas, T.V., Yanlos, P.. The Bayesan Image Retreval System, PcHunter: Theory, Implementaton, and Psychophyscal Experments IEEE Tran. On Image Processng, Volume 9, Issue, pp , Jan [3] Ishkawa, Y., Subramanya R., and Faloutsos, C., Mndreader: Query Databases Through Multple Examples, In Proc. of the 24 th VLDB Conference, (ew York), 998. [4] Lee, C., Ma, W. Y., and Zhang, H. J. Informaton Embeddng Based on user s relevance Feedback for Image Retreval, Techncal Report HP Labs, 998.

7 [5] Paek S., Sable C.L., Hatzvassloglou V., James A.,Schffman B.H., Chang S. F., Mckeown K.R, Integraton of Vsual and Text-Based Approaches for the Content Labelng and Classfcaton of Photographs, SIGIR 99. [6] Ru, Y., Huang, T. S., and Mehrotra, S. Content-Based Image Retreval wth Relevance Feedback n MARS, n Proc. IEEE Int. Conf. on Image proc., 997. [7] Ru, Y., Huang, T. S. A ovel Relevance Feedback Technque n Image Retreval, ACM Multmeda, 999. [8] Salton, G., and McGll, M. J. Introducton to Modern Informaton Retreval, McGraw-Hll Book Company, 983. [9] Shaw, W. M. Term-Relevance Computaton and Perfect Retreval Performance Informaton processng and Management.