Improving Web Image Search using Meta Re-rankers

Transcription

1 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: Improvng Web Image Search usng Meta Re-rankers B.Kavtha 1, N. Suata 2 1 Department of Computer Scence and Engneerng, Chtanya Bharath Insttute of Technology,Hyderabad, INDIA 2 Department of Computer Scence and Engneerng,Srdev Women s Engneerng College,Hyderabad, INDIA Abstract: The prevous methods for mage search rerankng suffer from the unrelablty of the assumptons under whch the ntal text based mage search result s employed n the rerankng process. In our proposed system, prototype-based rerankng method s suggested address ths problem n scalable fashon. Ths typcal assumpton that the top-mages n the text-based search result are equally relevant s relaxed by lnkng the relevance of the mages to ther ntal rank postons. The number of mages s employed by the ntal search result as the prototypes that serve to vsually represent the query and that are subsequently used to construct Meta re-rankers. For applyng dfferent Meta re-rankers to an mage from the ntal result, then the re-rankng scores are generated, whch are then aggregated by usng a lnear model to produce the fnal relevance score and the new rank poston for an mage n the re-ranked search result. It s mprovng the performance over the text-based mage search engne. Key concepts: Prototype based Meta re-ranker, Text based search Image Re-rankng. 1. Introducton The exstng web mage search engnes, ncludng Google, Bng and Yahoo retreve and rank mages mostly based on the textual nformaton assocated wth the mage n the hostng web pages, such as the ttle and the surroundng text. Whle text-based mage rankng s often effectve to search for relevant mages, the precson of the search result s largely lmted by the msmatch between the true relevance of an mage and ts relevance nferred from the assocated textual descrptons. To mprove the precson of the text-based mage search rankng and vsual rerankng has been proposed to refne the search result from the textbased mage search engne by ncorporatng the nformaton conveyed by the vsual modalty. Based on the mages n the ntal result, vsual prototypes are generated that are vsually representng the query. Each of the prototypes s used to construct a Meta reranker to produce a rerankng score for any other mage from the ntal lst. Fnally, the scores from all the Meta rerankers are aggregated together usng a lnear rerankng model to produce the fnal relevance score for an mage and to defne ts poston n the reranked results lst. The lnear rerankng model s learned n a supervsed fashon to assgn approprate weghts to dfferent meta rerankers. Snce the learned model weghts are related to the ntal text-based rank poston of the correspondng mage and not to the mage tself, then the rerankng model s queryndependent and can be generalzed across queres. Fgure 1. A hgh-level overvew of web mage Search engne 2. Proposed work To mprove the performance of searchng mages vsual search rerankng s very good opton. 4 steps are needed n our module text rankng, Prototype 71 IJDCST

2 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: generaton, and Meta Re-Ranker and also Re- Rankng Result. Text Rankng: Intal search s text based search. We need mage search engne to submt query from user. In the search engne query s n text format. It s the text based Image search n whch we get the mage rankng on the bases of text query whch we gve. Prototype generaton: In prototype generaton phase we create a rules for mage rerankng on whch further mages has been reranked. In ths we examne the vsual smlartes. From the top L mage set prototypes are generated usng vsual smlartes. These prototypes are used as an nput to the meta ranker. Meta Rankng: In meta rerankng, multple set prototype technque s used. Ths technque s used to compute the rankng score. The computed rerankng score gve as an nput to rerankng model to estmate ultmate rerankng score. In re-rankng module use of K-means clusterng s benefcal. The K-means handles the rankng problem. Thus the basc dea s for decomposng a rankng n to a set off par-wse preferences and then to reduce the rankng-learnng problem nto a par wse classfcaton problem. Rerankng Result: In ths step we get fnal reranked mages n prototype based rankng. In ths paper we proposed a prototypebased rerankng framework, whch constructs meta rerankers correspondng to vsual prototypes score of a gven mage taken from the ntal textbased search result. Fgure 2: Archtecture of Image re-rankng. 3. Image Rerankng Framework The proposed prototype-based rerankng method conssts of two steps. Onlne: In the onlne part, when a textual query s submtted to the mage search engne by a user, ntal search s performed usng any contemporary text-based search technque. Then, the vsual prototypes are generated and for each prototype a meta reranker s constructed. So, for each of the top N mages n the ntal search result, an L-dmensonal score vector s obtaned comprsng the scores from all meta rerankers when appled to that mage. Fnally, the score vector s used as nput to a rerankng model, whch s already turn to offlne to estmate the rankng scores n the reranked mage search lst. representng the textual query and learns the weghts of a lnear rerankng model to combne the results of ndvdual meta rerankers and produces the rerankng 72 IJDCST

3 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: Offlne: Fgure 3 Image Rerankng Framework The offlne component s devoted to learnng the rerankng model from user-labeled tranng data. Snce the learned model wll be used for rerankng the text-based search results, the tranng set s constructed from these results through the followng steps. Frst, several representatve queres sampled from the query log are selected. Then, usng these queres the top mages are retreved from the textbased mage search engne and downloaded for processng. Fnally for each query-mage par, people are nvted to label the relevance between them to form the ground-truth. After the tranng data s collected, score vector can be computed from the meta rerankers, as mentoned n the onlne part, for each mage and the correspondng query. Then the rerankng model s learned and stored n the memory to be used n the onlne part for respondng to user s submtted queres. Learnng the rerankng model: The lnear rerankng model has learned by estmatng the weghts of the combned scores comng from dfferent meta rerankers. Ths problem can be addressed usng a learnng-to-rank method, by regardng ths score vector as the rankng feature of an mage. Rankng K-means s among the most popular learnng to rank algorthms. Ths algorthm s wdely used K-means clusterng s to handle a rankng problem. The basc dea has to decompose a rankng nto a set of par wse preferences and then to reduce the rankng-learnng problem nto a par-wse classfcaton problem. The basc dea was to decompose a rankng nto a set of par-wse preferences and then to reduce the rankng-learnng problem nto a par-wse classfcaton problem. Standard effcent approaches are to learnng K- means clusterng, such that a sequental mnmal optmzaton, t can be drectly employed for learnng the Rankng K-means. Moreover, the fast algorthm, e.g., the cuttng-plane algorthm, can be adopted to speed up the tranng of a lnear Rankng K-means. The reason s why the learned rerankng model descrbed above can be generalzed across queres beyond those used for the tranng was that the model weghts are not related to specfc mages but to ther rank postons n the text-based search result. The separaton of the model weghts from specfc mages s the key to ensure that there rankng model only needs to be learned once and then t can be appled to any arbtrary query. The exstng learnng s to-rerank methods, ncludng the supervsed-rerankng and query-relatve classfer, desgn the rerankng model based on the hand desgned rankng features defned at a hgher abstracton level or on the ordered vsual words, respectvely. When compared to them, the prototypebased learnng to rerank method learns how lkely the mages at each of the ranked poston n by text-based result are to be relevant to the query. 4. Constructng Meta Rerankers One of the key steps n the Multple Set Prototype mage rerankng method s the constructon of meta 73 IJDCST

4 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: rerankers. The computed scores are used to nput for the rerankng model to estmate the ultmate rankng scores to determne the rank poston of the mages n the reranked result. There are three types to construct meta rerankers, dependng on how the prototypes are generated from the ntal text-based search result. Sngle mage prototype, Multple average prototype and multple set prototype are the three algorthms for constructng meta rerankers. Sngle-Image Prototype: A straghtforward way to generate a set of A prototype s to select top mages from the text based result, as llustrated n fg: 4. If we denote ths set as then the meta reranker can be bult smply based on the vsual smlarty S(.) between the prototype mage I to be reranked as S P S P L 1, and the to the lnear rerankng model n order to compute the defntve rankng score for mage I : R S ( I ) Where W are the ndvdual weghts from the model weght vector W. Multple average prototypes: Prototype 1 W xs( I P can be construct by frst selectng the top L mages n the ntal search result lst and then by cumulatvely averagng the features of all mages ranked startng from the topmost poston to the poston, as llustrated n Fg. 4. In other words, the prototype P 1 L 1, P s ) ( 2) P can be defned as I (3) Then, ths prototype can be employed to compute the scores of ndvdual meta rerankers by agan computng the vsual smlarty between a prototype and the mage to be reranked: M I / P SI, P......( 4) M S Fgure 4 Sngle-Image prototype s s I / P SI, P (1 ) The score vector aggregatng the values (1) from all meta rerankers s then used as nput 74 IJDCST

5 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: The multple-set prototype s a more flexble representaton, whch can support the development of more types of Meta rerankers. Gven a multple-set prototype P can learn a vsual classfer by Fgure 5 Multple Average prototype Multple set prototype: The multple-set prototype P MS at rank s defned as a bag of mages ranked from the topmost poston to the rank, as llustrated n Fg.6. regardng all the mages n P as postve samples, whch s then employed as meta reranker and the predcton score s used as the meta rerankng score. Snce a dscrmnatve learnng method s usually more effectve for learnng a vsual model, there s K- means n ths paper. However, t needs not only postve samples but also negatve samples. The Meta reranker wth a multple-set prototype can be defned as follows: M MS ( I / P arg max MS ) p( I ( P MS Where θ s the learned model and / ) (6) / ) ( 7) Here s the analyss of the propertes of the rerankng method based on the multple-average prototype. By usng the dot product as the smlarty measure, a correspondng meta reranker, leads to the followng expresson: P MS Fgure 6 Multple set prototype I (5 ) J 1... The multple-average prototype s the average of features for the mages n the multple-set prototype and can be seen as a specal case of ths prototype. R L 1 ( I ) L xs( I I 1 w x 1 k1, S( I, I )....(8) where L w = k (10) k k The above expressons transform the model based on a multple average prototype on to the model based on a sngle mage prototype, however, wth dfferent weghts. It states that the rankng n the text-based search result represents the orderng of the mportance for each ndvdual mage to be used as a prototype for rerankng. In other words, there rankng k ) (9) 75 IJDCST

6 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: based on a multple-average prototype wll rely more on the ntal text based result than that based on a sngle mage prototype. Weghts for ndvdual mages by the rerankng based on a multple-average prototype wll declne gradually wth the decreasng ranks. Ths may make ths rerankng model less aggressve and more robust than the one based on a sngle mage prototype. Meanwhle, t makes the rerankng model learned by the multple-average prototype-based rerankng method hardly over-fttng to the tranng queres % 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% P P 5. Comparson and Results a) Comparson TABLE I Performance comparson of varous rerankng methods. Method Precson (Mean Avg. Precson) Text based Rankng 4.39% Pseudo Relevance 15.64% Feed Back Supervsed 16.87% Rerankng Sngle Prototype 19.16% Multple Average 17.57% Prototype Multple Set 23.60% Prototype Multple Set 25.45% Prototype +Text 76 IJDCST

7 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: Performance comparson of Prototype-Set and Textbaselne from the search engne. The query s arranged n the ascendng order of the performance of Text-baselne. Generally Image search engnes apparently provde an effortless route. But they are presently lmted by returned mage precson. Our obectve n ths work s to re rank a large number of mages of a partcular class automatcally, and to acheve ths wth the hgh precson. Image clusters for each topc are formed by selectng mages where nearby text s top ranked by the topc. The user then parttons the clusters nto postve and negatve for the class. Then second, mages and the assocated text from these clusters are used as exemplars to tran a classfer based on votng on vsual (shape, color, and texture) and text features. Fg 5.1 Query Image P comparson of Prototype-Sngle and Prototype- Average methods. b) Results Query Image When we are searchng for an mage n search engnes, the correspondng mages are loaded. Among them uncategorzed mages are also spotted. However, by producng such databases contanng a large number of mages and wth hgh precson s stll an arduous manual task. Fg 5.2 Gvng Query 77 IJDCST

8 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: Download Assocate Images, Url Parsng We are usng Google search engne to downloadng mages from the Web. Image Search gves a very low precson (only about 4 percent) and s not used for the harvestng experments. Ths low precson s probably due to the fact that Google selects many mages from Web gallery pages whch contan mages of all sorts. Google s able to select the nclass mages from those pages, e.g., the ones wth the obect-class n the flename; however, f we use those Web pages as seeds, the overall precson greatly decreases. Therefore, we only use Web Search and Google Images, whch are merged nto one data set per obect class. Table 2 lsts the 18 categores downloaded and the correspondng statstcs for nclass and non-class mages. The overall precson of the mages downloaded for all 18 classes s about 29 percent. textual attrbutes whose presence s a strong ndcaton of the mage content. The goal s to re rank the retreved mages. Each feature s treated as bnary: True f t contans the query word (e.g., pengun) and False otherwse. To rerank mages for one partcular class (e.g., pengun), we do not employ the whole mages for that class. Instead, we tran the classfer usng all avalable annotatons except the class we want to rerank. Fg 5.4 Re-Rankng Fg 5.3 Downloadng Images Apply Re-Rankng Algorthm Now descrbe the re rankng of the returned mages based on text and metadata alone. Here, we follow and extend the method proposed by usng a set of Fg 5.5 Applyng Re-rankng K-means Implementaton K-means clusterng s used to form the cluster of smlar mages. Ths technque wll flter PNG or GIF format mages. Based on threshold value the re rankng process wll be done. 78 IJDCST

9 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: Fg 5.6 K-means Implementaton 6. Concluson Ths paper proposes a prototype based mage rerankng by usng K-means clusterng, whch constructs Meta rerankers correspondng to vsual prototypes representng the textual query and learns the weghts of a lnear rerankng model to combne the results of ndvdual Meta rerankers and produce the rerankng score of a gven mage taken from the ntal text-based search result. It mproves the performance by 25.48% over the text-based search result by combnng prototypes and textual rankng features. The natural extenson of the approach descrbed n ths paper would be to apply the proposed methods to learn concept models from mage search engnes n a semautomatc fashon. Compared to the fully automatc methods, then the sem-automatc approach could learn then the concept models for any arbtrary concept much better and wth only lttle human supervson. Whle the proposed methods have proved effectve for rerankng mage search results, There was envson of two drectons for future work to further mprove the rerankng performance. Frst, It could be could further speed up the Prototype-Set method varant whle decreasng the precson degradaton. Snce top mages are ncrementally added nto the multple-set prototypes to tran the meta rerankers, one of the possble approaches n ths drecton s to utlze the onlne learnng algorthms. Next the second, although It assume that the rank poston s generally correlated wth the relevance value of the mage found there, and whle our results show that ths assumpton can be regarded vald n a general case, stll the devatons from ths expectaton can occur for ndvdual quered. One possble approach here would be to automatcally estmate the query-relatve relablty and accuracy of each meta-reranker and then ncorporate t nto the rerankng model. The another approach may be to learn the rerankng models for dfferent query classes. ACKNOWLEDGMENT We wsh to acknowledge the efforts of Pantech Soluton Pvt ltd., Hyderabad, for gudance whch helped us work hard towards producng ths research work. 7. References [1] Lnun Yang, and Alan Hanalc, Prototype based Image search Rrrankng, IEEE Transactons on Mltmeda, vol. 14, no. 3, June [2] L. Yang and A. Hanalc "Supervsed rerankng for web mage search", Proc. ACM Multmeda, 2010 [3] R. Fergus, L. Fe-Fe, P. Perona and A. Zsserman "Learnng obect categores from Google's mage search", Proc. ICCV, [4] W. H. Hsu, L. S. Kennedy and S.-F. Chang "Vdeo search rerankng va nformaton bottleneck prncple", Proc.ACM Multmeda, [5] M. Frtz and B. Schele "Decomposton, dscovery and detecton of vsual categores usng topc models", Proc. CVPR, [6] W. H. Hsu, L. S. Kennedy and S.-F. Chang "Vdeo search rerankng through random walk over 79 IJDCST

10 VOLUME-1, ISSUE-V (Aug-Sep 2013) IS NOW AVAILABLE AT: document-level context graph", Proc. ACM Multmeda, [7] Y. Jng and S. Balua "Vsualrank: Applyng pagerank to large-scale mage search", IEEE Trans. Pattern Anal. Mach. Intell, vol. 30, no. 11, pp [8] X. Tan, L. Yang, J. Wang, Y. Yang, X. Wu and X.-S. Hua "Bayesan vdeo search rerankng", Proc. ACM Multmeda, [9] Y. Lu, T. Me, X.-S. Hua, J. Tang, X. Wu and S. L "Learnng to vdeo search rerank va pseudo preference feedback", Proc. ICME, [10] F. Schroff, A. Crmns and A. Zsserman "Harvestng mage databases from the web", Proc. ICCV, IJDCST