Optimal Adaptive Learning for Image Retrieval

Transcription

1 Optimal Adaptive Leaning fo Image Retieval ao Wang Dept of Compute Sci and ech singhua Univesity Beijing 00084, P. R. China Yong Rui Micosoft Reseach One Micosoft Way Redmond, WA 9805, USA Shi-Min Hu Dept of Compute Sci and ech singhua Univesity Beijing 00084, P. R. China Abstact Leaning-enhanced elevance feedback is one of the most pomising and active eseach diections in ecent yea s content-based image etieval. Howeve, the existing appoaches eithe equie pio knowledge of the data o consume high computation cost, making them less pactical. o ovecome these difficulties and motivated by the successful histoy of optimal adaptive filtes, in this pape, we pesent a new appoach to inteactive image etieval. Specifically, we cast the image etieval poblem in the optimal filteing famewok, which does not equie pio knowledge of the data, suppots incemental leaning, is simple to implement and achieves bette pefomance than the state-of-the-at appoaches. o evaluate the effectiveness and obustness of the poposed appoach, extensive expeiments have been caied out on a lage heteogeneous image collection with 7,000 images. We epot pomising esults on a wide vaiety of queies.. Intoduction In the past decade, both technology push (e.g., multimedia data analysis and machine leaning [3,,7,9,,4,5]) and application pull (e.g., vaious national digital libay initiatives [,6]) have contibuted to the polifeation of image etieval techniques [5,0]. Howeve, even afte yeas of extensive eseach, helping uses to find thei desied images accuately and quickly is still fa fom satisfactoy. In the ealy yeas of content-based image etieval (CBIR), most of eseaches devote thei eseach effot in finding the best visual featue o the best similaity measue [0,0]. Howeve, accoding to ecent use study esults [4], what aveage uses eally want ae the systems that suppot queies based on high-level concepts (e.g., cas and apples), not low-level featues (e.g., ed colo blob with hoizontal edges). he difficulty is that, with today s technology, it is almost impossible to map fom the low-level featues to high-level concepts without eithe automated image undestanding o inteactive human help [5]. Fully automated image undestanding is still a fa cy in the aeas of atificial intelligence and compute vision. Howeve, one of the most impotant distinctions between today s CBIR systems and 970 s fully automated image undestanding systems is that human uses ae aleady a pat of the CBIR systems. While it is not possible to achieve fully automated image content undestanding given today s technology, it is possible to leveage uses knowledge to find bette mappings between low-level featues and high-level concepts. Motivated by this obsevation, seveal teams intoduce the elevance feedback paadigm into CBIR systems in the middle nineties [4,,6]. he basic idea behind elevance feedback is to use a leaning mechanism that adapts image featues and similaity measues to best eflect high-level concepts, based on use povided examples. Relevance feedback is fist developed in text-based infomation etieval (IR) eseach community [8]. But because image undestanding needs moe uses guidance than text undestanding, elevance feedback is gaining a lot of momentum in CBIR in ecent yeas [5]. Diffeent leaning mechanisms esult in diffeent elevance feedback techniques. Fo example, some leaning mechanisms assume linea similaity models [7,6] while othes use nonlinea ones [9,]. Following the pioneeing wok in [4,,6], vaious elevance feedback appoaches have been poposed. he most epesentative ones ae the pobabilistic Bayesian appoach [,8,3,4], tansductive leaning appoach (e.g., disciminate EM) [5], boosting appoach [], kenel appoximation appoach (e.g., suppot vecto machines) [] and optimization leaning (OPL) appoach [7]. his list is by no means exhaustive. Instead, it is intended to show epesentative appoaches taken in elevance feedback. While the vaious appoaches have advanced the state-of-the-at elevance feedback techniques in CBIR, many of them suffe fom one o moe of the following limitations: he leaning pocess has its best stength only if pio knowledge about the data distibution o a lage taining set is available [4,3,,4,6]. If uses do not give sufficient feedback examples duing the etieval pocess, only a sub-optimal solution can be achieved [7]. he convegence speed is quite slow and/o the computation cost is quite high [7,3,6]. All the above limitations pevent the existing appoaches fom being fully deployed in pactical systems. o ovecome these difficulties, in this pape we popose a new elevance feedback technique based on adaptive filteing. Adaptive filtes have been successfully used fo moe than

2 fou decades in vaious eseach aeas including signal pocessing, telecommunication, system identification, and automatic contol [9]. he optimality and efficiency of the adaptive filtes is ooted in the optimal estimation theoy, which allows the filte to automatically adapt to the minimum mean squae eo (MMSE) solution efficiently. Consideing the human vision system as an intelligent signal filte, we can cast elevance feedback into the optimal adaptive filte s famewok. By doing so, we can leveage a matued field with many excellent techniques and solve the leaning poblem in a pincipled way. Least mean squae (LMS) algoithm and ecusive least squae (RLS) algoithm ae the two best known techniques to appoximate the optimal Wiene filte solution. Both LMS and RLS ecusively computes the optimal filte s weights, esulting in simple implementation, fast convegence ate, and good pefomance. he est of the pape is oganized as follows. In Section, befoe we go into the details of the poposed appoach, we fist intoduce vaious impotant concepts and notations. Fo elated wok in Section 3, we focus on the optimization leaning appoach (OPL) [7]. It is one of the best appoaches available and the one that we will compae against in this pape. In Section 4, we give detailed desciption of the poposed appoaches based on optimal adaptive filtes. Specifically, we descibe optimal Wiene filte solutions, and develop new elevance feedback techniques based on LMS and RLS algoithms. We futhe discuss how to solve the leaning ode issue encounteed in CBIR, and give computation complexity compaison between OPL, LMS, and RLS. o evaluate the etieval pefomance of the poposed appoaches, extensive expeiments ove a lage heteogeneous image collection with 7,000 images ae epoted in Section 5. Concluding emaksaegiveninsection6.. Concepts and notations Befoe we go into details of the pape, it is beneficial fo us to fist intoduce and define some impotant concepts and notations that will be used extensively late in the pape. Let M be the total numbe of images in the image database. We use F m = [ f m, f m,..., f mk ] to denote the featue vecto of the m th image, m =,,, M, whee K is the numbe of elements in the featue vecto. Similaly, we use F = f, f,..., f ] to denote the featue vecto fo the q [ q q qk quey image q. We futhe define a diffeence vecto between image m, m =,,, M, and the quey image q as: X ( m) = [ f m f q, K, f mk f qk ] () whee x-y epesents the diffeence between x and y. Because diffeent featue elements may have diffeent contibution to the peception of image content, diffeent weights can be associated with diffeent featue elements to eflect this effect [7]. he oveall distance between image m, m =,,, M, and quey q can theefoe be calculated as m) = X ( m) W X ( m), Euclidean metic, L nom m) = W X ( m), cityblock metic, L nom depending on if we want to use L o L distances. Fo L distance, W is a weight matix, while fo L distance, W is a weight vecto. Sofawehavediscussedhowtocomputethedistance between two images, but in CBIR similaity is nomally used instead. o convet between distance and similaity, we adopt the appoach poposed in [7]. Assuming the distance m), m =,,, M, obeys the Gaussian distibution of N(0,³ ), the similaity π(m) between image m and quey image q is the likelihood of this distibution, with π(m) = 0 being the least simila and π(m) = being the most simila: m) π ( m) = exp( ), π ( m) [0,] σ () m) = σ ln( π ( m)) 3. Related wok Among the existing appoaches [4,7,8,9,,3,7,,3,6] intoduced in Section, we choose the OPL appoach developed in [7] as the appoach to compae against, due to the following easons: OPL is the one of the best techniques available. Unlike some peviously poposed ad hoc appoaches [6], it fomulates the elevance feedback in a vigoous optimization famewok and solves the paametes in a pincipled way; OPL deives explicit optimal solutions fo the weights, making it faste than many othe existing appoaches; Unlike many othe appoaches that ae only tested on pe-selected queies ove small data sets, the OPL appoach has been tested with a wide vaiety of queies ove a lage heteogeneous image collection [7]. he OPL appoach deives the explicit optimal weights W by using the Lagange multiplies and the L distance nomal [7]: / K (det( C )) C, det( C) 0 W = (3) diag(/ σ,/ σ,.../ σ K ), othewise he tem C is the weighted K-by-K covaiance matix of the featue vecto of the feedback examples. hat is, N π ( ( X ( X n n ( ) s ) = Cs =,, s =, K, K N (4) π ( n= whee N is the numbe of positive feedback examples and π( is the similaity of image n, n =,,, N, given by the use. When computing W, the OPL appoach switches between a full matix and a diagonal matix, depending on

3 the elationship between the numbe of feedback examples N and the length of the featue vecto K. WhenN>K,the OPL uses the full matix fom to take advantage of lage feedback examples. When N<K, the OPL uses diagonal matix to avoid noisy paamete estimation. he OPL appoach has many advantages ove othe existing appoaches including MARS [6] and Mindeade [7]. Howeve, it still has the following difficulties: Calculating det(c i )andc - i is quite expensive, i.e., equies 3 O( I ( K ) ) opeations, which is not desiable i i fo pactical image etieval applications; When the numbe of feedback examples N is small, e.g., N<K, the weights W educes fom a full matix to a diagonal matix, which esults in sub-optimal solutions. Moe impotantly, OPL is a batch leaning appoach which equies all the examples ae given at the same time befoe it can lean. When an additional feedback example is pesented, thee is no easy way to incementally incopoate the new example without e-computing the weights. o addess these difficulties, in the next section, we popose an adaptive-filte-based leaning appoach, which conveges quickly to cuent optimal solution, avoids expensive computation, and uses an incemental ecusive leaning paadigm. 4. Leaning with adaptive filtes Fist, we would like to exam how human etieve images. he human vision system can be consideed as a, potentially non-linea, signal filte (see Figue ). In ou paticula algoithm, we stat with linea filtes. But the analogy still applies. Fo quey image q and feedback image n, n =,,, N, the input to the filte is the diffeence vecto X (,andthe output fom the filte is the distance. he poblem of CBIR would have been solved if we knew the human vision system s esponse function to X (. Fotunately, based on the adaptive filte theoy, it is possible fo us to constuct an adaptive filte to simulate the esponse function of the human vision system. he input to the adaptive filte is the same as the input to the human vision system, i.e., X (, X ( n ) Visual System + y( - Adaptive Filte Figue Human vision system model and the output of the adaptive filte is y(. By compaing y( against, wecanobtainaneosignal, which can then be used to dive the adaptive filte moving towads the human vision system s esponse function (see Figue ). In the est of this section, we fist give the optimal Wiene solution, then develop two elevance feedback techniques basedonlmsandrls.wefuthediscusshowtosolve the leaning ode issue encounteed in CBIR, and give computation complexity compaison between OPL, LMS, and RLS. 4.. Optimal Wiene filte Given a wide-sense stationay (WSS) input signal X( and desied output signal, the Wiene filte is the optimal stochastic filte that minimizes the vaiance of the eo [9]: min E[ e ] = E[( ) ] W = E[( = E[ N ] N N l= 0 W ( l) X ( k l)) ] N l= 0 W ( l) E[ X ( k l)] + W ( l) W ( m) E[ X ( k l) X ( k m)] l= 0 m= 0 = E[ ] P W + W RW whee N is the length of the Wiene filte, W = [ W (0), W (), L, W ( N )] is the filte coefficient, and R (0) () () (0) = M M ( N ) ( N ) P = [ (0), (), L, ( l) = E[ X ( k l)] L ( N ) L ( N ), O M L (0) ( N )], ( l m) = E[ X ( X ( k + ( l m)), dx dx dx he gadient of E[e ] with espect to the filte coefficient is = ( E[ e ])/ W = P + RW (5) by setting the gadient to zeo, we aive at the optimal Wiene solution: W = R P (6) his solution is geat in theoy, but in eality we do not know the statistics of the signals apioi. Fotunately, we can estimate the statistics on the fly while we ae computing the optimal solution. wo of such techniques ae LMS and RLS, with LMS appoximating the steepest gadient descent and RLS appoximating R and P diectly. 4.. Least mean squae solution (LMS) Because we do not know R and P in advance, the gadient descent appoach can be used to solve the non-linea optimization poblem min E[ e ]. At each iteation, we W compute the gadient and move the solution towads the dx

4 steepest gadient descent diection, i.e.: ( ( = E( e ) / W ] = P + RW ( n+ ) ( ( W = W µ ( (7) (8) whee µ is the step size and n is the iteation index. Note that the above algoithm involves the calculation of E[e ], which is expensive to compute. One of the geatest ideas developed in adaptive filte theoy is to appoximate the tue gadient = ( E[ e ]) / W by its noisy estimate ˆ = ([ e ]) / W. his esults in the LMS algoithm, the fist pactical adaptive filte algoithm developed fou decades ago by Widow and Hoff [4]. oday it is still the most widely used algoithm because of its simplicity, little computation and geat pefomance [9]. We next give a complete LMS algoithm developed fo elevance feedback in CBIR. [Pocedue : LMS elevance feedback algoithm] (A) Initialization: Choose step size 0 < µ <, and set the filte coefficients to W ( 0) = [/ K,/ K,...,/ K] (9) (B) Fo each n =,,, N,. Compute the distance y( based on the cuent weights: y( = W ( X ( (0). Compute the eo signal = = σ ln( π ( ) Note that compaed with standad Wiene filtes, we have an exta step to convet fom the similaity π( to distance. 3. Compute the updated weights µ W ( = W ( + X ( () a + X ( X ( Whee a is a small positive constant to avoid denominato to be 0. his LMS-based elevance feedback algoithm is elegant in theoy, easy to implement and equies little computation Recusive least squae algoithm (RLS) Instead of appoximating the gadient, RLS attempts to appoximate the R and P diectly. It uses the famous matix invese equation [9]: A = B + CF C A = B BC( F + C BC) C B to simplify the computation of R. Fo a detailed deivation of the RLS algoithm, please efe to Appendix A. Compaed with LMS, RLS have the following featues: Because LMS uses the noisy gadient to appoximate the tue gadient, it conveges fast at initial steps and gadually slows down when close to final solution. RLS, on the othe hand, estimates R and P diectly at each iteation, thus esulting in faste oveall convegence. Howeve, RLS s faste convegence speed is at the cost of moe computation. In addition, when taining examples ae not sufficient, estimating R and P can be poblematic. his may actually esult in a slowe convegence than LMS. A detailed compaison between LMS and RLS is given in Section 5. We next give the complete RLS algoithm developed fo elevance feedback in CBIR. [Pocedue : RLS elevance feedback algoithm] (A) Initialization: W (0) = [/ K, / K, L, / K ], 0) = δ I () whee Q is the invese of the signal covaiance matix and δ is a small positive constant. (B) Fo each n =,,, N,. Compute the distance y( based on the cuent weights using Equation (9).. Compute the eo signal: = = σ ln( π ( ) 3. Compute the gain vecto: X ( K( = (3) / π ( + X ( X ( 4. Compute the updated weights: W ( = W ( + K( (4) 5. Compute the invese coelation matix: = X ( (5) 4.4. Adaptive filtes in CBIR So fa we have discussed the LMS and RLS algoithms designed fo elevance feedback in CBIR. Howeve, one detail is left untouched: in the oiginal adaptive filtes, signals X( and aive in a sequential ode, while in CBIR, thee is no explicit ode fo feedback examples. he two most obvious appoaches we can take fo this odeing issue ae the fowad odeing appoach and the backwad odeing appoach. Let set S contain the N feedback examples in the ode of inceasing similaity to the quey image. hat is, the fist image in the set has the lagest similaity to the quey image and the last image in the set has the smallest similaity to the quey image. he fowad appoach is to lean the feedback examples fom the fist to the last, and the backwad appoach is to lean the examples fom the last to the fist. Because both LMS and RLS ae incemental leaning algoithms, we expect the backwad appoach to be moe advantageous: its leaning samples ae oganized in a coase-to-fine fashion. Just like the hieachical pyamid appoach in optical flow computation [], the backwad appoach simulates a hieachical algoithm to avoid local minimum and to speed up convegence. It saves the best example at the last to fine-tunethepaametes.wegivedetailedcompaisonof the fowad and backwad leaning odes in Section 5.

5 4.5. Computation complexity Given the high computation cost involved in most of today s elevance feedback techniques [7,6], one of ou motivations to develop the adaptive-filte-based appoach is its efficiency. he OPL appoach is aleady one of the most efficient elevance feedback appoaches available, but it 3 still equies O ( K + NK ) computations fo each elevance feedback iteation [7]. If we exam the LMS and RLS algoithms, they only need O(NK) and O( NK ) computations, espectively [9]. As we have discussed in Section 4., LMS is extemely efficient, which is linea in both the numbe of feedback examples N and the featue vecto length K. Futhemoe, unlike OPL which is a batch leaning algoithm, both LMS and RLS ae incemental leaning algoithms. hat is, when the n th feedback example becomes available, they can lean it incementally fom example n-, without e-executing the whole algoithms. o illustate the diffeence in thei computation complexity, let s plug in some eal-wold numbes with K = 34, N = 0. he LMS, RLS, and OPL equie 680, 30, and computations, espectively. Both LMS and RLS ae moe efficient than OPL. It is woth mentioning that LMS is exceptionally efficient, whose computation count is two odes of magnitude less than RLS o OPL. 5. Expeiments In the pevious section, we have shown the advantages of LMS and RLS in theoy, e.g., optimality, incemental leaning and low computation complexity. In this section, we would like to exam thei etieval pefomance (e.g., accuacy and obustness) via expeiments. Specifically, we would like to answe the following questions: Which algoithm is bette, LMS, RLS o the existing OPL and unde what condition? Which sequencing ode is bette fo adaptive filtes, fowad o backwad and why? 5.. Data set Fo all the expeiments epoted in this section, the Coel image collection is used as the test data set. We choose this data set due to the following consideations: It is a lage-scale data set. Compaed with the data sets used in some systems that contain only a few hunded images, the Coel data set includes 7,000 images. It is heteogeneous. Unlike the data sets used in some systems that ae all textue images o ca images, the Coel data set coves a wide vaiety of content fom animals and plants to human society and natual sceney. It is pofessional-annotated. Instead of using the less meaningful low-level featues as the evaluation citeion, the Coel data set has human annotated gound tuth. he whole image collection has been classified into distinct categoies by Coel pofessionals and thee ae 00 images in each categoy. Because aveage uses want to etieve image based on high-level concepts, not low-level featues [4,5], the gound tuth we use in the expeiments is based on high-level categoies. hat is, we conside a etieved image as a elevant image only if it is in the same categoy as the quey image. his is a much moe difficult task to tackle, but this is what aveage uses want [4], thus ou ultimate goal. he Coel data set have also been used in othe systems and elatively high etieval pefomance has been epoted. Howeve, those systems only use pe-selected categoies with distinctive visual chaacteistics (e.g., cas vs. mountains). In ou expeiments, no pe-selection is made. We believe only in this manne can we obtain an objective evaluation of diffeent etieval techniques. 5.. Queies Some existing systems only test a few pe-selected quey images. It is not clea if those systems will still pefom well on othe not-selected images. o faily evaluate the etieval pefomance between LMS, RLS, and OPL, we andomly geneated 400 queies fo each etieval condition. Fo all the expeiments epoted in this section, they ae the aveage of all the 400 quey esults Visual featues In ou cuent system, we use thee visual featues: colo moments, wavelet-based textue and wate-fill edge featue. Fo colo moments, we choose to use the HSV colo space because of its similaity to human vision peception of colo [5]. Fo each of the thee colo channels, we extact the fist two moments (e.g., mean and standad deviatio and use them as the colo featue. Fo the wavelet-based textue, the oiginal image is fed into a Daubechies-4 wavelet filte bank [5], and decomposed into the thid level, esulting 0 de-coelated sub-bands. Out of the 0 sub-bands, 9 of them ae detailed bands captuing the chaacteistics of the oiginal image at diffeent scales and oientations. he last band is the smoothed band, which is a sub-sampled aveage image of the oiginal image. Fo each sub-band, we extact the standad deviation of the wavelet coefficients and theefoe have a textue featue Figue. Use inteface of the system

6 vecto of length 0. his wavelet-based featue has been poven to be quite effective in modeling image textue featues [5,6]. he last visual featue we use is a ecently developed wate-fill edge featue [7]. he oiginal image is fist fed into the Canny edge detecto to geneate its coesponding edge map. he edge map is then consideed as a netwok of tunnels. Vitual wate is then poued into the tunnels. By measuing maximum filling time, maximum fok count, etc., this algoithm captues impotant edge chaacteistics of the oiginal image. It extacts 8 elements in total [7]. One thing woth pointing out is that the adaptive filteing famewok we poposed in this pape is an open famewok. hat is, the algoithm woks egadless of which visual featues ae used. We use the above thee visual featues just fo illustation pupose, moe advanced featue (e.g., egion layout, etc. [5]) can eadily be incopoated into the system Pefomance measues he most widely used pefomance measues fo infomation etieval ae pecision (P) and ecall (Re)[8]. P is defined as the numbe of etieved elevant objects (i.e., N) ove the numbe of total etieved objects, say the top 0 images. Re is defined as the numbe of etieved elevant objects ove the total numbe of elevant objects in the image collection (in the Coel data set case, 99). he pefomance of an "ideal" system is to have both high P and Re. Unfotunately, P and Re ae conflicting citeia and cannot be at high values at the same time. Because of this, instead of using a single value fo P and Re, there) cuve is nomally used to chaacteize the pefomance of a etieval system System desciption We have developed a pototype system based on the poposed optimal adaptive filteing appoaches. he system is witten in C++ and unning on Windows N platfom. Its inteface is shown in Figue. Ou pototype system has two modes: demo mode and testing mode. Duing the demo mode, a use can bowse though the image collection and submit any image as the quey image, which is shown at the top-left cone. Fo each of the etuned image, thee is a degee-of-elevance (i.e., similaity π() slide to allow the use to povide his/he elevance feedback. Duing testing mode, the system uses the Coel high-level categoy infomation as the gound tuth to obtain elevance feedback. his is a vey challenging task and we next epot detailed expeimental esults Results and obsevations Fowad leaning vs. backwad leaning able shows the fowad/backwad leaning esults fo LMS when the top 0, 00, and 80 most simila images ae etuned. o bette compae the two leaning odes, in Figue 3, we also plot the pecision-ecall cuve fo LMS able. Compaison between fowad leaning (LMS_F) and backwad leaning (LMS_B) fo LMS. Pecision (pecentage) 0 iteation iteation iteations Retun top LMS_F LMS_B Retun top LMS_F LMS_B Retun top LMS_F LMS_B able. Compaison between fowad leaning (RLS_F) and backwad leaning (RLS_B) fo RLS. Pecision (pecentage) 0 iteation iteation iteations Retun top RLS_F RLS_B Retun top RLS_F RLS_B Retun top RLS_F RLS_B afte two iteations of feedback. Similaly, able shows the fowad/backwad leaning esults fo RLS, and Figue 4 shows thei pecision-ecall cuve. Regading the fowad leaning and backwad leaning, following obsevations can be made based on the above tables and figues: Because the backwad leaning ode simulates the coase-to-fine leaning pocess, it benefits the Figue 3. Compaison between fowad leaning and backwad leaning fo LMS. Solid cuve is fo fowad leaning and dashed cuve is fo backwad leaning. Figue 4. Compaison between fowad leaning and backwad leaning fo RLS. Solid cuve is fo fowad leaning and dashed cuve is fo backwad leaning.

7 able 3. Compaison between OPL, LMS with backwad leaning and RLS with backwad leaning. Pecision (pecentage) 0 iteation iteation iteations Retun top OPL LMS_B Retun top 00 Retun top 80 RLS_B OPL LMS_B RLS_B OPL LMS_B RLS_B adaptive filteing algoithms to fine-tune the weights at the last stage. Fo both LMS and RLS, the backwad leaning outpefoms the fowad leaning. he odeing effect is moe noticeable in LMS than in RLS. When the numbe of feedback examples is elatively lage (e.g., the bottom-ight potion of Figue 4), the fowad and backwad leaning ae almost the same fo RLS. his is because RLS continuously leans all the examples, while LMS quickly adapts to the new example, fogetting the olde ones LMS vs. RLS vs. OPL able 3 shows the compaison between OPL, LMS with backwad leaning and RLS with backwad leaning when the top 0, 00, and 80 most simila images ae etuned, and Figue 5 shows thei pecision-ecall cuve afte two iteations of elevance feedback. Based on the table and figue, the following obsevations can be made: With moe feedback iteations, the etieval pefomance inceases in all the thee algoithms. When the numbe of etuned images is small (e.g., etun top 0 images only), the pefomance of LMS and RLS is significantly bette than that of OPL. his is because OPL switches to a diagonal matix, esulting in sub-optimal solution (see Section 3). When the numbe of etuned images is sufficiently lage (e.g., 80 images), the pefomance of all the thee algoithms is compaable. his is because, all the thee algoithms ae close to the optimal solution conditioned Figue 5. Compaison between OPL, LMS, and RLS. he Solid cuve epesents OPL, dashed cuve epesents LMS and dotted cuve epesents RLS. on the feedback examples. he LMS with backwad leaning seems to be the winning appoach. Not only its etieval pefomance is the best, its computation complexity is significantly cheape than the othe two appoaches (see Section 4.5). 6. Conclusions Motivated by the fact that human s vision system can be simulated by a signal filte, in this pape, we developed optimal adaptive filte based elevance feedback techniques fo CBIR. Both LMS and RLS ae effective in leaning and efficient in computation. Futhemoe, they both lean incementally, i.e., they suppot on-line leaning. his is paticula useful in that when a new example aives, the algoithms can lean without stating fom scatch. Given the chaacteistics in CBIR, we futhe studied two leaning odes (fowad and backwad) fo the adaptive filtes. he backwad leaning ode is supeio because it simulates a coase-to-fine leaning pocess. o evaluate the pefomance of the poposed appoaches, we conducted extensive tests on a lage-scale heteogeneous image collection and compaed them against a state-of-the-at appoach. Of equal impotance, ou study used high-level concepts, instead of low-level featues, as the gound tuth, which ealistically meets aveage uses image etieval needs [4]. Expeimental esults show that LMS with backwad leaning is the winning technique that is both accuate and efficient. 7. Acknowledgement he Coel image set was obtained fom Coel and used in accodance with thei copyight statement. he fist autho is suppoted in pat by China NSF gant Appendix A: RLS Algoithm fo CBIR Fo a finite time seial signal X (,wehave n R ( = X ( i) X ( i) i= n (A) = X ( i) i) i= Hence we have the following ecusion fo updating the covaiance matix and the coss-coelation vecto R ( = R ( + X ( X ( (A) = + X ( A majo obstacle we want to avoid is the matix invesion of R when solving fo the optimal Wiene solution (see Equation (5)). he matix invesion lemma helps us to ovecome this difficulty. Let A, B, andf all be positive definite matices, the matix invesion lemma says [9], if A = B + CF C (A3) then A = B BC( F + C BC) C B (A4) Because both R( and R(n-) ae positive definite, let A = R(, B = R (, C = X (, F = (A5)

8 Fo convenience, let s futhe define = R (, = (A6) = σ ln( π ( ) W( X ( By substituting Equations (A5) and (A6) into (A4), we have = X ( (A7) whee X ( K ( = (A8) [ + X ( X ( ] is called the gain vecto. Reaanging the tems in Equation (A8), we have K( = X ( X ( X ( = ( X ( ) X ( (A9) = X ( o summaize, the ecusive W( can be calculated as follows: W ( = R ( = = [ + X ( ] = n ) + X ( = X ( + X ( = W ( + K( ( X ( W ( ) = W ( + K( Refeences. Begen, J. and Adelson, E.H. Hieachical, Computationally Efficient Motion Estimation Algoithm. J. of the Optical Society of Ameica A, 4:35, 987. Communication of the ACM, Special issue on Digital Libay, Guest editos: Edwad Fox and Gay Machionini, May 00, Vol. 44, No Cox, I.J.; Mille, M.L.; Minka,.P.; Papathomas,.V.; Yianilos, P.N., he Bayesian image etieval system, Pichunte: theoy, implementation, and psychophysical expeiments. IEEE ans. on Image Pocessing, Vol.9(3), Mach 000, pp: Cox, I.J., Mille, M.L, Omohundo, S.M., and Yianilos, P.N., PicHunte: Bayesian Relevance Feedback fo Image Retieval," Poc. Int. Conf. on Patten Recognition, Vienna, Austia, C:36-369, August Daubechies, I., en Lectues on Wavelets, CBMS-NSF Lectue Notes n. 6, SIAM, IEEE Compute Magazine, Special Issue on Content-based image etieval systems, Guest editos: Venkat N. Gudivada and Jijay V. Raghavan, 995, Vol. 8, No Ishikawa, Y.; Subamanya, R. and Faloutsos, C., Mindeade: Quey databases though multiple examples. Poc. of the 4 th VLDB Confeence (New Yo, Lee, C., Ma, W. Y., and Zhang, H.J., Infomation embedding based on use s elevance feedback fo image etieval, Poc. of Multimedia Stoage and Achiving Systems IV, Boston, Septembe Lee, H.K.; Yoo, S.I., A neual netwok-based image etieval using nonlinea combination of heteogeneous featues. Poc. of the 000 Congess on Evolutionay Computation, Vol., 000, pp: Ma, W.; Zhang, H.J., Benchmaking of image featues fo content-based etieval. Poc. Of the hity-second Asiloma Confeence on Signals, Systems & Computes, Vol., 998, pp: MacAthu, S.D.; Bodley, C.E.; Shyu, C., Relevance feedback decision tees in content-based image etieval, Poc. IEEE Wokshop on Content-based Access of Image and Video Libaies, 000, pp: Minka,., and Picad, R., Inteactive leaning using a society of models, Patten Recognition, Special issue on Image Databases, 30(4), Peng, J., Adaptive multi-class metic content-based image etieval, Poc. of the 4th Int. Confeence on Visual Infomation Systems, Lyon, Fance, Novembe Rodden, K., Basalaj, W., Sinclai, D., and Wood, K., Does oganization by similaity assist image bowsing, Poc. ACM Compute-Human Inteaction (CHI), 000. pp Rui, Y.; Huang,. and Chang, S. F., Image etieval: cuent techniques, pomising diections and open issues, Jounal of Visual Communication and Image Repesentation,Vol.0, 39-6, Mach, Rui, Y.; Huang,. and Mehota, S., Content-based image etieval with elevance feedback in MARS. Poc. IEEE Int. Conf. on Image Pocessing, Rui, Y.; Huang,. Optimizing leaning in image etieval. Poc. IEEE Int. Conf. on Compute Vision and Patten Recognition, Vol., 000, pp: Salton, G., and McGill, M. J., Intoduction to moden infomation etieval, McGaw-Hill Book Company, New Yok, Simon Haykin, Adaptive filte theoy, 3d ed., Uppe Saddle Rive, N.J: Pentice Hall, 996, Pentice Hall infomation and system sciences seies. 0. Smeuldes, A. W. M.; Woing, M.; Santini, S.; et al. Content-based image etieval at the end of the ealy yeas, IEEE ans. PAMI, Vol. (), Dec. 000 pp: Swets, D.L.; Weng, J.J. Using disciminant eigen-featues fo image etieval, IEEE ans PAMI, Vol. 8(8),Aug. 996,pp: ian Q. Hong, P., Huang,. S., Update elevant image weights fo content-based image etieval using suppot vecto machines, Poc. IEEE Int. Conf. On Multimedia and Expo, Vol., pp: 99-0, ieu, K.; Viola, P. Boosting image etieval. Poc. IEEE CVPR, Vol., 000, pp: Vasconcelos, N.; Lippman, A. A pobabilistic achitectue fo content-based image etieval, Poc. IEEE CVPR, Vol., 000 pp: Widow, B., and Hoff, M. E., Adaptive switching cicuits, IRE WESCON Conv. Rec., pp Wu, Y.; ian, Q.; Huang,.S., Disciminant-EM algoithm with application to image etieval. Poc. IEEE CVPR,Vol., 000, pp: Zhou, X., Rui, Y., and Huang,.S., Wate-Filling Algoithm: A Novel Way fo Image Featue Extaction based on Edge Maps, Poc. of IEEE ICIP 999, Kobe, Japan.