Relevance Feedback for Image Retrieval

Transcription

1 Vashal D Dhale et al, / (IJCSIT Internatonal Journal of Computer Scence and Informaton Technologes, Vol 4 (2, 203, Relevance Feedback for Image Retreval Vashal D Dhale, Dr A R Mahaan, Prof Uma Thakur Dept of Computer Scence, PIET, Nagpur Unversty, Nagpur (MS, Inda Abstract In recent years the relevance feedback technology s regarded n content based mage retreval The dea s to adopt the system to the specfc user preferences makng more mportant weghts or features that reflect the actual user need n order to acheve hgher precson Therefore we can defne relevance feedback as the process by whch human and computer nteract n order to automatcally adust the query to the real user preferences The dea s to adust the query selecton crtera to better approxmate real user need usng the result retreved by the orgnal query To be more proftable, relevance feedback technques were ncorporated nto CBIR such that more precse result can be obtaned by takng users feedback nto account Query mage Feature Extracton II ARCHITECTURE N Feedback algorthm U/R Y Keywords Relevance feedback, CBIR, Feature Extracton Smlarty measure Retreval results User feedback I INTRODUCTION Relevance feedback [] s a method to enhance the system search effect It studes from the real nteractve process of the user and the search system, then dscovers and captures user's actual search ntenton, and modfes the search strategy of system, Image retreval based on relevance feedback s an unceasngly repeated and gradually advanced processes The nteracton between the system and the user enables the retreval to approach the user s expectaton, and fnally acheves the requests Image Retreval s becomng a doman of ncreasng and crucal mportance n the new nformaton based socety, as a part of Informaton Retreval (IR feld Image retreval has been addressed n varous ways [], [2], [3], [4], [5], snce wth the ncrease of Internet bandwdth and CPU speed the use of mages n the orld de eb has become prevalent Informaton sharng has ncreasngly become a common phenomenon among the users of today s hgh speed network Due to advancements n the dgtal photography technology, large storage capacty and hgh speed networks, storng large amount of mages has become possble Dgtal mages fnd a wde range of applcaton n the medcne, scence, mltary and securty purposes etc Therefore there s a need for an effcent way for mage retreval There are dfferent ways to retreve the mages n CBIR A bg challenge n CBIR s the semantc gap between the low level feature and the hgh level concept In order to reduce the gap between the low level feature and hgh level concepts, relevance feedback was ntroduced nto CBIR [5], [6] Recently, many researchers began to consder the RF s a classfcaton or learnng problem That user provdes postve and/or negatve examples, and the system learns from such examples to separate all data nto relevant and rrelevant groups Low Level Features Feature Extracton Image DB Fg A general descrpton of standard mage retreval Fg shows a general descrpton of standard mage retreval from database usng relevance feedback These features can be classfed as global features and local features The most commonly used features are color, texture, and shape They are all applcaton ndependent The basc dea of relevance feedback s to shft the burden of fndng the rght query formulaton from the user to the system In order to make ths true, the user has to provde system wth some nformaton, so that system can perform well n answerng the orgnal query To retreve the mage from the database, we frst extract feature vectors from mages (the features can be shape, color, texture etc, then store feature vectors nto another database for future use hen gven query mage, we smlarly extract ts feature vectors, and match those features wth database mage features If the dstance between two mages feature vectors s small enough; we consder the correspondng mage n the database smlar to the query hen searchng more generc mage databases, one way of dentfyng what the user s lookng for n the current retreval sesson (the target of the user s by ncludng the user n the retreval loop For ths, the sesson s dvded nto several consecutve rounds; at every round the user provdes feedback regardng the retreval results, eg by qualfyng mages returned as ether \relevant" or \rrelevant" (relevance feedback or RF n the followng; from ths feedback, the engne learns the vsual features of the mages and returns mproved results to the user The RF mechansm mplemented n a search wwwcstcom 39

2 Vashal D Dhale et al, / (IJCSIT Internatonal Journal of Computer Scence and Informaton Technologes, Vol 4 (2, 203, engne should attempt to mnmze the amount of nteracton between the user and the engne requred for reachng good results In fact, RF was frst ntroduced for the retreval of text documents n [7] The ease wth whch the relevance of an mage can be evaluated and the persstent dffculty of dealng wth the semantc gap n CBIR explans the rapd development of RF for mage retreval snce the early work n [8], [9], [0] III OBJECTIVE OF THE PROBLEM The frst and most frequent obectve conssts n fndng mages that share some specfc characterstc the user has n mnd The case of target search studed n [] and [2], where the user s lookng for that partcular sngle mage she has n mnd, was further dstngushed from the more common category search A complementary but less frequent use of RF was ntroduced n [3] and conssts n defnng a class of mages and extendng textual annotatons of some mages n the class to the others In the explore and search for some relevant tems, the user has a rather vague pror notaton of relevance and reles on the exploraton of the mage based to classfy t In the retreve most tems n the set of relevant one, the user would lke to fnd all or most of the mage that share some specfc characterstc she has n mnd IV IMAGE REPRESENTATION The representaton of ndvdual mages also has an mpact on the RF mechansm employed In exstng work on the CBIR wth RF, two dfferent representaton schemes were used for the mages: Most of the tme, the global vsual appearance of the mages s descrbed usng a combnaton of global sgnatures ncludng color, texture and shape nformaton Images are then represented by fxedlength vectors n a descrpton space In some publcatons, such as [4], [5] an mage s consdered to be a set of regons obtaned by an automatc segmentaton Every regon can be descrbed by color, texture and shape Addtonally, some nformaton regardng the confguraton of regons can be avalable An mage s then represented as varable length collecton of regon sgnatures, possbly ncludng confguraton nformaton From user feedback concernng entre mages, the search engne s also expected to learn whch regons are mportant for the current search sesson and whch regons to gnore V GENERAL ASSUMPTIONS One can customze an RF mechansm f one knows the characterstcs of the scenaro, of the target applcaton and of ts users The dscrmnaton between relevant" and rrelevant" mages must be possble wth the avalable mage descrptors 2 There s some relatvely smple relaton between the topology of the descrpton space and the characterstc shared by the mages the user s searchng for 3 Relevant" mages are a small part of the entre mage database 4 hle part of the early work on RF assumed that the user could (and would be wllng to provde a rather rch feedback, ncludng \relevance notes" for many mages, the current assumpton s that ths feedback nformaton s scarce: the user wll only mark a few \relevant" mages as postve and some very dfferent mages as negatve VI RELEVANCE FEEDBACK MECHANISMS The RF mechansm mplemented n a search engne must operate n real tme It s expected to maxmze the rato between the qualty of the retreval results and the amount of nteracton between the user and the system An RF mechansm has two components: a learner and a selector At every feedback round, the user marks (part of the mages returned by the search engne as relevant or rrelevant The learner explots ths nformaton to reestmate the target of the user th the current estmaton of the target, the selector chooses other mages that are dsplayed by the nterface of the search engne; the user s asked to provde feedback on these mages durng the next round The task of the learner s partcularly dffcult n the context of RF for several reasons The amount of tranng data s very low, usually much lower than the number of dmensons of the descrpton space There are usually much fewer postve examples than negatve examples recent work on RF often reles on support vector machnes In RF, SVMs appear to be the learners of choce for several reasons: The decson functon of an SVM allows both the defnton of a fronter and the rankng of mages 2 SVMs are very flexble 3 SVMs are usually less senstve than densty-based learners to the mbalance between postve and negatve examples n the tranng data 4 SVMs allow fast learnng for medum-szed databases hle most of the exstng work usng SVMs for RF concentrates on 2-class SVMs[6], [7] that must learn to dscrmnate postve and negatve examples, -class SVMs were also put forward n [8] n order to learn from postve examples only class SVMs are able to estmate the support of the dstrbuton of postve examples A Idea of Relevance Feedback The relevance feedback mechansm had been ntroduced nto the mage retreval system The relevance feedback technology adusts the search automatcally accordng to the user s relevant feedback of the precedng retreval result The basc steps of user s relevance feedback are as follows System search for the query mage gven by user 2 The user compares the retreval result returned from the system wth own demands 3 System analyses the character whch can ndcate user s retreval am best automatcally from the feedback nformaton produced by the user, adusts the smlarty method, then carres on the retreval agan, repeats the step 2 wwwcstcom 320

3 Vashal D Dhale et al, / (IJCSIT Internatonal Journal of Computer Scence and Informaton Technologes, Vol 4 (2, 203, Relevance feedback technology s the man doman n current mage retreval research The am of relevance feedback s to study from the real nteracton between the user and the retreval system It dscovers and captures user's actual search ntenton, and modfes the search strategy of system, thus obtans the search result whch talles as precse as possble wth the user s actual demand B Image Retreval Mode of Relevance Feedback l Stage To extract the characterstc needed n the content-based mage retreval, we consder the relevant problem and the user relevance feedback problem therefore must use a set of reasonable retreval models to carry on to t, and then make the retreval results may rely on e explan the mage retreval model based on relevance feedback as follows Frst, we should defne the obect mage models; an obect mage model I may be represented as: I I( D, F, R ( D s the raw mage data, eg the mage n JPEG form, etc F { f } s the low level feature assocated wth mage obect R { r } s the expresson of a certan characterstc f, the color hstogram and the color matrx are the expresson way of color characterstc, each characterstc expresson r s possbly a vector whch s composed by many components, can be wrtten n the followng form: r { r, r 2, r 3 r k } (2 where k s the length of the vector The obect model supports multple representatons wth dynamcally updated weghts to accommodate the content n the mage obect The mage characterstc weght value exsts n each level of the model,, and k corresponds to the mage characterstc f, the expresson r, and the components r k The am of relevance feedback s to search for proper weght value Based on Relevance Feedback retreval process s as follows: Intalze the weght values [,, k ] nto O, whch s a set of no bas weghts That s every entty s ntally of the same mportance O I (3 O J (4 k k K (5 here I s the number of mage characterstc, J s the number of representaton for feature f, K s the dmenson of vector r 2 Dvde the query obect Q provded by the user nto group of mage characterstcs f accordng to the weght 3 thn each characterstc f can be dvded nto the correspondng expresson r accordng to the weght 4 In certan characterstcs expresson r, the smlarty between the mage I and the query mage Q s calculated accordng to the correspondng smlarty algorthm m and the weght value k : Sr ( m ( r, w k (6 5 Each representaton smlarty value are then merge nto feature smlarty value: S( f w S( r (7 6 The total smlarty S between the mage I and the query mage Q can be obtaned by combnng ndvdual S( f : S w S( f (8 7 All mages n the database are arrange accordng to ther smlarty, then return the frst N mages whch are most smlar to the mage user need 8 For each of the retreved obect, the user marks t as relevant or rrelevant accordng to hs nformaton need 9 The system updates the weghts accordng to user s feedback opnon and go to step 2 C Updates eght value accordng to User s Feedback In the computer-based retreval system, the expresson and the weght value of mage vson characterstc s defnte, and n relevance feedback-based retreval system, t s necessary to update the weght value dynamcally, and express the rch mage content by many knds of characterstcs expressons s corresponded to the weght of dfferent characterstc vector, and t reflects the dfferent attenton to varous characterstcs n total smlarty, the adustment to may follow the formula herenafter accordng to user's relevance feedback wwwcstcom 32

4 Vashal D Dhale et al, / (IJCSIT Internatonal Journal of Computer Scence and Informaton Technologes, Vol 4 (2, 203, Let T s the aggregate of the frst N mages, whch are the most smlar ones that determned by the total smlarty S S s the relevance score feedback by the user = 3 Hghly relevant =, Relevant S = 0, No opnon = -, Not relevant = -3, Hghly not relevant (9 For each r set T o be the most smlar mage search Rank Retreved Image Rank 3 Retreved Image Rank 2 Retreved Image Rank 4 Retreved Image mage, whch determned accordng to Sr (, set 0 and then adust the weght as follows: S f To TM (0 Others Suppose that there are M mages n the database, put the expresson vectors r of extremely related mage together nto a M x K matrx and each column of ths matrx s a r k sequence The recprocal of standard devaton of ths sequence s preferable estmaton to weght, k k VII EXPERIMENTAL ANALYSIS Gven an mage query Q and an mage database S, retreve from S ' those mages Q whch contan Q accordng to some noton of smlarty Fgure whch dsplays an example query mage and ts relevant answer set Fgure 2 shows the mage of such answer set and ther respectve answer rank retreved wthn the top 0 matches Query Image Answer Set Fg 2 A Query mage and ts relevant answer Set Rank 5 Retreved Image Rank 7 Retreved Image Fg 3 Rank of the relevant mages obtaned VIII CONCLUSIONS Relevance feedback s a powerful technque n order to mprove the performance of mage retreval It s an open research area to the researcher to reduce the semantc gap between low-level features and hgh level concepts In ths paper we have shown, for the frst tme, how relevance feedback can be used to mprove the performance of CBIR e presented a relevance feedback based technque, whch s based on re-weghtng scheme that assgns penaltes to each of database mages and updates those of all relevant mages usng both the postve and negatve examples dentfed by the user The user's feed-back s used to refne the mage smlarty measure by weghtng the dstances between the query and the database mage REFERENCES [] Y Ru, TSHuang, S Mehrotra, Content based mage retreval wth relevance feedback [2] Nblak et al, The QBIC proect: Queryng mages by content usng color, texture, and shape, n Proc of SPIE, vol 908, 73-82, 993 [3] M Flckner et al, Query by Image and Vdeo Content: The QBIC System, IEEE Computer, 28, 9, 23-3, 995 [4] R Bach et al, The Vrage Image Search Engne: An open framework for mage management, n Proc of SPIE, vol 2670, 76-87, 996 [5] JR Smth, SF Chang, Vsual SEEK: a fully automated contentbased mage query system, Proc of ACM Multmeda'96, 996 [6] M Ioka," A method of defnng the smlarty of mages on the bass of color nformaton," Techncal Report RT-0030, IBM Research, Tokyo Research Laboratory Nov, 989 [7] Gerard Salton Automatc Informaton Organzaton and Retreval, McGraw-Hll, 968 [8] T Kurta and T Kato Learnng of personal vsual mpresson for mage database systems In Second Intl Conf on Document Analyss and Recognton, pages , 993 [9] Rosalnd Pcard, Thomas P Mnka, and Martn Szummer Modelng user subectvty n mage lbrares In IEEE Int Conf On Image Processng, pages , 996 [0] Yong Ru, Thomas S Huang, Mchael Ortega, and Sharad Mehrotra Relevance feedback: a power tool n nteractve content-based mage retreval IEEE Transactons on Crcuts and Systems for Vdeo Technology, 8(5: , 998 [] Ingemar J Cox, Matthew L Mller, Thomas P Mnka, Thomas Papathomas, and Peter N Yanlos The Bayesan retreval system, wwwcstcom 322

5 Vashal D Dhale et al, / (IJCSIT Internatonal Journal of Computer Scence and Informaton Technologes, Vol 4 (2, 203, PcHunter: theory, mplementaton and psychophyscal experments IEEE Transactons on Image Processng, 9(:20-37, January 2000 [2] Ingemar J Cox, Matthew L Mller, Stephen M Omohundro, and Peter N Yanlos An optmzed nteracton strategy for Bayesan relevance feedback In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, pages IEEE Computer Socety, 998 [3] Ye Lu, Chunhu Hu, Xngquan Zhu, Hong-Jang Zhang, and Qang Yang A unted framework for semantcs and feature based relevance feedback n mage retreval systems In Proceedngs of the 8th ACM Internatonal Conference on Multmeda, pages 3{37 ACM Press, 2000 [4] Feng Jng, Mngng L, Hong-Jang Zhang, and Bo Zhang Learnng regon weghtng from relevance feedback n Image retreval In Proceedngs of the IEEE Internatonal Conference on Acoustcs, Speech and Sgnal Processng, 2002 [5] Feng Jng, Mngng L, Le Zhang, Hong-Jang Zhang, and Bo Zhang Learnng n regon-based mage retreval In Proceedngs of the IEEE Internatonal Symposum on Crcuts and Systems, 2003 [6] Pengyu Hong, Q Tan, and Thomas S Huang Incorporate support vector machnes to content-based mage retreval wth relevant feedback In Proceedngs of the 7th IEEE Internatonal Conference on Image Processng, September 2000 [7] Smon Tong and Edward Chang Support vector machne actve learnng for mage retreval In Proceedngs of the 9th ACM nternatonal conference on Multmeda, pages 07-8 ACM Press, 200 [8] Yunqang Chen, Xang Sean Zhou, and Thomas S Huang One-class SVM for learnng n mage retreval In Proceedngs of the IEEE Internatonal Conference on Image Processng (ICIP'0, 200 wwwcstcom 323