Semi-Supervised Biased Maximum Margin Analysis for Interactive Image Retrieval

Transcription

1 IP R3 Sem-Suervsed Based Maxmum Margn Analyss for Interactve Image Retreval Lnng Zhang,, Student Member, IEEE, Lo Wang, Senor Member, IEEE and Wes Ln 3, Senor Member, IEEE School of Electrcal and Electronc Engneerng, Nanyang echnologcal Unversty, Sngaore, Insttute for Meda Innovaton, Nanyang echnologcal Unversty, Sngaore, School of Comuter Engneerng, Nanyang echnologcal Unversty, Sngaore, Abstract Wth many otental ractcal alcatons, Content-Based Image Retreval (CBIR) has attracted substantal attenton durng the ast few years. A varety of Relevance Feedbac (RF) schemes have been develoed as a owerful tool to brdge the semantc ga between low-level vsual features and hgh-level semantc concets and thus to mrove the erformance of CBIR systems. Among varous RF aroaches, Suort Vector Machne () based RF s one of the most oular technques n CBIR. Deste the success, drectly usng as a RF scheme has two man drawbacs. Frst, t treats the ostve and negatve feedbacs equally, whch s not arorate snce the two grous of tranng feedbacs have dstnct roertes. Second, most of the based RF technques do not tae nto account the unlabelled samles although they are very helful n constructng a good classfer. o exlore solutons to overcome these two drawbacs, n ths wor, we roose a Based Maxmum Margn Analyss (BMMA) and a Sem-Suervsed Based Maxmum Analyss (SemBMMA), for ntegratng the dstnct roertes of feedbacs and utlzng the nformaton of unlabeled samles for based RF schemes. he BMMA dfferentates ostve feedbacs from negatve ones based on local analyss, whle the SemBMMA can effectvely ntegrate nformaton of unlabelled samles by ntroducng a Lalacan regularzer to the BMMA. We formulate the roblem nto a general subsace learnng tas and then roose an automatc aroach of determnng the dmensonalty of the embedded subsace for RF. Extensve exerments on a large real world mage database demonstrate that the roosed scheme combned wth the RF can sgnfcantly mrove the erformance of CBIR systems. Index erms Suort Vector Machne, Relevance Feedbac, Grah Embeddng, Content-Based Image Retreval D I. INRODUCION urng the ast few years, Content-Based Image Retreval (CBIR) has ganed more attenton for ts otental alcaton n multmeda management [, ]. It s motvated by the exlosve growth of mage records and onlne accessblty of remotely stored mages. An effectve search scheme s urgently requred to manage the huge mage database. Dfferent from the tradtonal search engne, n CBIR, an mage query s descrbed usng one or more examle mages and low-level vsual features (e.g., color [3-5], texture [5-7], shae [8-0], etc.) are automatcally extracted to reresent the mages n the database. However, the low-level features catured from the mages may not accurately characterze the hgh-level semantc concets [, ]. o narrow down the so called semantc ga, Relevance Feedbac (RF) was ntroduced as a owerful tool to enhance the erformance of CBIR [, ]. Huang et al ntroduced both the query movement and re-weghtng technques [3, 4]. Self Organzng Ma was used to construct the RF algorthms [5]. In [6], one-class Suort Vector Machne () estmated the densty of ostve feedbac samles. Derved from one-class, a based nherted the merts of one-class but ncororated the negatve feedbac samles [7]. Consderng the geometry structure of mage low-level vsual features, [8, 9] roosed manfold learnng based aroaches to fnd ntrnsc structure of mages and mrove the retreval erformance. Wth the observaton that all ostve examles are ale; each negatve examle s negatve n ts own way, RF was formulated as a based subsace learnng roblem, n whch there s an unnown number of classes, but the user s only concerned about the ostve class [0,,]. However, all of these methods have some lmtatons. For examle, the method n [3, 4] s heurstcally based, the densty estmaton method n [6] gnores any nformaton contaned n the negatve feedbac samles, and the dscrmnant subsace learnng technques n [0, ] often suffer from the so-called Small Samle Sze roblem. Regardng the ostve and negatve feedbacs as two dfferent grous, classfcaton-based RFs [3, 4, 5] have become a oular technque n the CBIR communty. However, RF s very dfferent from the tradtonal classfcaton roblem because the feedbacs rovded by the user are often lmted n real-world mage retreval systems. herefore, small samle learnng methods are most romsng for RF. wo-class s one of the oular small samle learnng methods wdely used n recent years and obtans the state of the art erformance n classfcaton for ts good generalzaton ablty [4-8]. he can acheve a mnmal structural rs by mnmzng the Van-Chervonens dmensons [7]. Guo et al develoed a constraned smlarty measure for mage

2 IP R3 Fg. A tycal set of ostve and negatve feedbac samles n a relevance feedbac teraton retreval [6], whch learns a boundary that dvdes the mages nto two grous and samles nsde the boundary are raned by ther Eucldean dstance to the query mage. he actve learnng method selects samles close to the boundary as the most nformatve samles for the user to label [8]. Random samlng technques were aled to allevate unstable, based and overfttng roblems n RF [5]. L et al roosed a multtranng method by adatng a co-tranng technque and a random samlng method [9]. Nevertheless, most of the RF aroaches gnore the basc dfference between the two dstnct grous of feedbacs, that s, all ostve feedbacs share a smlar concet whle each negatve feedbac usually vares wth dfferent concets. For nstance, a tycal set of feedbac samles n RF teraton are shown n Fg.. All the samles labeled as ostve feedbacs share a common concet (.e., elehant), whle each samle labeled as negatve feedbac vares wth dverse concets (.e., flower, horse, banquet, hll, etc.). radtonal RF technques treat ostve and negatve feedbacs equally [4, 5, 6, 8, 9]. Drectly usng the as a RF scheme s otentally damagng to the erformance of CBIR systems. One roblem stems from the fact that dfferent semantc concets lve n dfferent subsaces and each mage can lve n many dfferent subsaces, and t s the goal of RF schemes to fgure out whch one [0]. However, t wll be a burden for tradtonal based RF schemes to tune the nternal arameters to adat to the changes of the subsace. Such dffcultes have severely degraded the effectveness of tradtonal RF aroaches for CBIR. Addtonally, t s roblematc to ncororate the nformaton of unlabelled samles nto tradtonal based RF schemes for CBIR, although unlabelled samles are very helful n constructng the otmal classfer, allevatng nose and enhancng the erformance of the system. o exlore solutons to these two aforementoned roblems n the current technology, we roose a Based Maxmum Margn Analyss (BMMA) and a Sem-Suervsed Based Maxmum Margn Analyss (SemBMMA) for the tradtonal RF schemes, based on the grah embeddng framewor [30]. he roosed scheme s manly based on (a) the effectveness of treatng ostve examles and negatve examles unequally [0,, ]; (b) the sgnfcance of the otmal subsace or feature subset n nteractve CBIR; (c) the success of grah embeddng n characterzng ntrnsc geometrc roertes of the data set n hgh-dmensonal sace [30, 3, 3]; and (d) the convenence of the grah embeddng framewor n constructng sem-suervsed learnng technques. Wth the ncororaton of BMMA, labeled ostve feedbacs are maed as close as ossble, whle labeled negatve feedbacs are searated from labeled ostve feedbacs by a maxmum margn n the reduced subsace. he tradtonal combned wth BMMA can better model the relevance feedbac rocess and reduce the erformance degradaton caused by dstnct roertes of the two grous of feedbacs. he SemBMMA can ncororate the nformaton of unlabelled samles nto the relevance feedbac and effectvely allevate the over fttng roblem caused by the small sze of labeled tranng samles. o show the effectveness of the roosed scheme combned wth the RF, we wll comare t wth the tradtonal RF and some other relevant exstng technques for RF on a real world mage collecton. Exermental results demonstrate that the roosed scheme can sgnfcantly mrove the erformance of the RF for mage retreval. he rest of ths aer s organzed as follows: n Secton II, the related revous wor,.e., the rncle of RF for CBIR and the grah embeddng framewor, are brefly revewed; n Secton III, we ntroduce the BMMA and the SemBMMA for RF; an mage retreval system s gven n Secton IV; a large number of exerments whch valdate the effectveness of the roosed scheme are gven n Secton V; concluson and future wor are resented n Secton VI. II. RELAED PREVIOUS WORK A. he rncle of RF for CBIR In ths secton, we brefly ntroduce the rncle of the tradtonal based RF for CBIR. he mlements the structure rs mnmzaton by mnmzng Van-Chervonens dmensons [7]. Consder a lnearly searable bnary classfcaton roblem as follows: {( x, y ),,( x, y )} and y {, } () N N,, N where x denotes a h-dmensonal vector, N s the number of tranng samles and y s the label of the class that the vector belongs to. he obectve functon of ams to fnd an otmal hyerlane to searate the two classes,.e., w x b 0 () where x s an nut vector, w s a weght vector and b s a bas. he attemts to fnd the two arameters w and b for the otmal hyerlane by maxmzng the geometrc margn / w, subect to: y( w x b) (3) he soluton of the obectve functon can be found through a Wolf dual roblem wth the Lagrangan multled by : N N Q( ) y y ( x x ) / (4) subect to 0 and, N 0 y.

3 IP R3 3 In general, n the dual roblem data onts aear only n the nner roduct, whch can often be relaced wth a ostve defnte ernel functon for better erformance. x x ( x ) ( x ) K( x, x ) (5) where K () s a ernel functon. he ernel verson of the Wolfe dual roblem s N N Q( ) y y K( x x ) / (6), hus, for a gven ernel functon, the classfer s gven by F( x) sgn( f ( x)) (7) s where f ( x) y K( x, x) b s the outut hyerlane decson functon of and s s the number of suort vectors. Generally, the outut of (.e., f( x )), s usually used to measure the smlarty between a gven attern and the query mage n the tradtonal RF for CBIR. he erformance of a classfer deends manly on the number of suort vectors. Orthogonal Comlement Comonent Analyss (OCCA) decreases the number of suort vectors by fndng a subsace, n whch all the ostve feedbacs are merged [33]. However, t stll totally gnores the nformaton contaned n negatve feedbacs, whch s very helful n fndng a homogeneous subsace. Intutvely, good searaton s acheved by the hyerlane that has the largest dstance to the nearest tranng samles, snce n general, the larger the margn, the lower the generalzaton error of the classfer. B. Grah embeddng framewor In order to descrbe our roosed aroach clearly, we frstly revew the grah embeddng framewor ntroduced n [30]. Generally, for a classfcaton roblem, the samle set can be h n reresented as a matrx X [ x, x,, xn ] R, where n ndcates the total number of the samles and h s the feature dmenson. Let G { X, W} be an undrected smlarty grah, whch s called an ntrnsc grah, wth vertces set X and n* n smlarty matrx W R. he smlarty matrx W s real and symmetrc, and measures the smlarty between a ar of vertces; W can be formed usng varous smlarty crtera. he corresondng dagonal matrx D and the Lalacan matrx L of the grah G can be defned as follows: L D W, D W,,, n (8) Grah embeddng of the grah G s defned as an algorthm to determne the low-dmensonal vector reresentatons l n Y [ y, y, y ] R of the vertex set X, where l s lower n than h for dmensonalty. he column vector y s the embeddng vector for the vertex x, whch reserves the smlartes between ars of vertces n the orgnal hgh-dmensonal sace. hen n order to characterze the dfference between ars of vertces n the orgnal hgh-dmensonal sace, a enalty grah G { X, W } s also defned, where the vertces X are the same as those of G, but the edge weght matrx W corresonds to the smlarty characterstcs that are to be suressed n the low-dmensonal feature sace. For a dmensonalty reducton roblem, drect grah embeddng requres an ntrnsc grah G, whle a enalty grah G s not a necessary nut. hen the smlartes among vertex ars can be mantaned accordng to the grah reservng crteron as follows: y arg mn y y W arg mn tr( YLY ) (9) tr( YBY ) c tr( YBY ) c where tr() s the trace of an arbtrary square matrx; c s a constant; B s the constrant matrx. B may tycally be a dagonal matrx for scale normalzaton or exress more general constrants among vertces n a enalty grah G, and t descrbes the smlartes between vertces that should be avoded; B or L s the Lalacan matrx of G, smlarly to Equaton (8), whch can also be defned as follows: L D W, D W,,, n (0) where W s the smlarty matrx of enalty grah G to measure the dfference between a ar of vertces n G. he grah embeddng framewor reserves the ntrnsc roerty of the samles n two ways: For larger smlarty between samles x and x, the dstance between y and y should be smaller to mnmze the obectve functon. Conversely, smaller smlarty between x and x should lead to larger dstance between y and y. Hence, through the P ntrnsc grah G and enalty grah G, the smlartes and dfferences among vertex ars n a grah G can be reserved n the embeddng. In [30], based on the grah embeddng framewor, Equaton (9) can be resolved by convertng t nto the followng trace rato formulaton: arg mn tr( YLY ) Y () ( Y tr YBY ) Generally, f the constrant matrx reresents only scale normalzaton, then ths rato formulaton can be drectly solved by egenvalue decomoston. However, for a more general constrant matrx, t can be aroxmately solved wth Generalzed Egenvalue Decomoston by transformng the obectve functon nto a more tractable aroxmate form arg mn r(( YBY ) ( YLY )). Y Wth the assumton that the low-dmensonal vector reresentatons of the vertces can be obtaned from a lnear roecton,.e., y x, where s the roecton matrx, then the obectve functon () can be changed to tr( XLX ) arg mn () ( tr XBX ) Durng the ast few years, a number of manfold learnng based feature extracton methods have been roosed to cature the ntrnsc geometry roerty [3, 3, 34,35,36,37], In [30], Yan et al. clamed that all of the mentoned manfold learnng algorthms can be mathematcally unfed wthn the grah embeddng framewor descrbed n ths subsecton. hey also

4 IP R3 4 roosed Margnal Fsher Analyss whch taes both the manfold geometry and the class nformaton nto consderaton. However, Margnal Fsher Analyss stll suffers from the Small Samle Sze roblem when the tranng samles are nsuffcent, whch s always the case n mage retreval. III. BMMA AND SEMIBMMA FOR RF IN CBIR Wth the observaton that all ostve examles are ale; each negatve examle s negatve n ts own way, the two grous of feedbacs have dstnct roertes for CBIR [0]. However, the tradtonal RF treats the ostve and negatve feedbacs equally. o allevate the erformance degradaton when usng the as a RF scheme for CBIR, we exlore solutons based on the argument that dfferent semantc concets le n dfferent subsaces and each mage can le n many dfferent concet subsaces [0]. We formally formulate ths roblem nto a general subsace learnng roblem and roose a BMMA for the RF scheme. In the reduced subsace, the negatve feedbacs, whch dffer n dverse concets wth the query samle, are searated by a maxmum margn from the ostve feedbacs, whch share a smlar concet wth the query samle. herefore, we can easly ma the ostve and negatve feedbacs onto a semantc subsace n accordance wth human erceton of the mage contents. o utlze the nformaton of unlabelled samles n the database, we ntroduced a Lalacan regularzer to the BMMA, whch wll lead to SemBMMA for the RF. he resultant Lalacan regularzer s largely based on the noton of local consstency whch was nsred by the recently emergng manfold learnng communty and can effectvely dect the wea smlarty relatonsh between unlabeled samles ars. hen, the remanng mages n the database are roected onto ths resultant semantc subsace and a smlarty measure s aled to sort the mages based on the new reresentatons. For the based RFs, the dstance to the hyerlane of the classfer s the crteron to dscrmnate the query relevant samles from the query rrelevant samles. After the roecton ste, all ostve feedbacs are clustered together whle negatve feedbacs are well searated from ostve feedbacs by a maxmum margn. herefore, the resultant classfer hyerlane n ths subsace wll be much smler and better than n the orgnal hgh dmensonal feature sace. Dfferent from the classcal subsace learnng methods, e.g., PCA and LDA, whch can only see the lnear global Eucldean structure of samles, BMMA ams to learn a roecton matrx such that n the roected sace, the ostve samles have hgh local wthn-class smlarty, but the samles wth dfferent labels have hgh local between-class searablty. o descrbe the algorthm clearly, frst we ntroduce some notatons of ths aroach. In each round of feedbac teraton, there are n samles h X { x, x,, x } R. For smlcty, we assume that the n frst n samles are ostve feedbacs x ( n ), the next n samles are negatve feedbacs x ( n n n ), and all the others are unlabelled samles x ( n n n). Let lx ( ) be the class label of samle x, we denote lx ( ) for ostve feedbacs, lx ( ) for negatve feedbacs and lx ( ) 0 for unlabelled samles. o better show the relatonsh between the roosed aroaches and the grah embeddng framewor, we use the smlar notatons and equatons n the orgnal grah embeddng framewor, whch rovdes us a general latform to develo varous new aroaches for dmensonalty reducton. Frstly, two dfferent grahs are formed: ) the ntrnsc grah G, whch characterzes the local smlarty of the feedbac samles; ) the enalty grah G, whch characterzes the local dscrmnant structure of the feedbac samles. For all the ostve feedbacs, we frst comute the ar-wse dstance between each ar of ostve feedbacs. hen for each ostve feedbac x, we fnd ts nearest neghborhood ostve feedbacs, whch can be reresented as a samle s set N, and ut an edge between x and ts neghborhood ostve feedbacs. hen the ntrnsc grah s characterzed as follows: S I x x * W s s : or (3) tr[ X ( D W ) X ] W /, f l( ) and l( ), or 0, else s s s (4) where D s a dagonal matrx whose dagonal elements are calculated by D W ; s denotes the total number of nearest neghborhood ostve samle ars for each ostve feedbac. Bascally, the ntrnsc grah measures the total s average dstance of the nearest neghborhood samle ars, and s used to characterze the local wthn-class comactness for all the ostve feedbacs. For the enalty grah G, ts smlarty matrx W reresents geometrc or statstcal roertes to be avoded and s used as a constrant matrx n the grah embeddng framewor. In the BMMA, the enalty grah G s constructed to reresent the local searablty between the ostve class and the negatve class. More strctly seang, we exect that the total average margn between the samle ars wth dfferent labels should be as large as ossble. For each feedbac samle, we fnd ts neghbor feedbacs wth dfferent labels and ut edges between corresondng ars of feedbac samles wth weghtsw. hen, the enalty grah can be formed as follows: S x x W W : or tr[ X ( D W ) X ] /, f l( ) and l( ), or 0, else (5) (6)

5 IP R3 5 (a) (b) (c) Fg. (a) red dots are ostve samles and blue dots are negatve samles n the orgnal sace (b) the BMMA aroach (c) the ostve samles and negatve samles n the maxmum margn subsace where D s a dagonal matrx whose dagonal elements are calculated by D W ; denotes the total number of neghborhood samle ars wth dfferent labels. Smlarly, the enalty grah measures the total average dstance of the nearest neghbor samle ars n dfferent class, and s used to characterze the local between-class searablty. In the followng, we descrbe how to utlze the grah embeddng framewor to develo algorthms based on the desgned ntrnsc and enalty grahs. Dfferent from the orgnal formulaton of the grah embeddng framewor n [30], the BMMA algorthm otmzes the obectve functon n a trace dfference form nstead,.e., arg max tr[ X ( D W ) X ] tr[ X ( D W ) X ] arg max tr[ X ( D W ) X ] tr[ X ( D W ) X ] arg max tr( XBX ) tr( XLX ) arg max tr[ X ( B L) X ] (7) As gven n Equaton (7), we can notce that the obectve functon wors n two ways, whch tres to maxmze tr( XBX ) and at the same tme mnmze tr( XLX ). Intutvely, we can analyze the meanng of the obectve functon n (7) geometrcally. By formulatng the obectve functon as a trace dfference form, we can regard t as the total average local margn between ostve samles and negatve samles. herefore, Equaton (7) can be used as a crteron to dscrmnate the dfferent classes. In [38], the maxmum margn crteron (MMC) was resented as an obectve functon wth a smlar dfference form. he dfferences between BMMA and MMC are the defntons of the nterclass searablty and ntraclass comactness. In MMC, both of the nterclass searablty and ntraclass comactness are defned as the same n LDA, whch treats the two dfferent classes equally, and MMC can only see the lnear global Eucldean structure. In BMMA, the ntraclass comactness s constructed by only consderng one class (e.g., ostve feedbacs) and characterzed by a sum of the dstances between each ostve samle and ts nearest neghbors n the same class. he nterclass searablty s defned by resortng to nterclass margnal samles nstead of the mean vectors of dfferent classes as n MMC. Wthout ror nformaton on data dstrbutons, BMMA can fnd more relable low-dmensonal reresentatons of the data comared to the MMC [38] and also follow the orgnal assumton n BDA [0] (.e., all ostve examles are ale; each negatve examle s negatve n ts own way). It should be noted that revous methods [39, 40] that followed MMC cannot be drectly used for the RF n mage retreval because these methods treat samles n dfferent classes equally. In order to remove an arbtrary scalng factor n the roecton, we addtonally requre that s consttuted by the unt vectors,.e.,,,, l. hs means that we need to solve the followng constrant otmzaton. max tr( X ( B L) X ) l = X ( B L) X (8) s. t. 0,,,, l Note that we may also use other constrants nstead. For examle, we may requre tr( XBX ) and then mnmze tr( XLX ). It s easy to chec that the above maxmum margn aroach wth such a constrant n fact results n the tradtonal Margnal Fsher Analyss (MFA) [30]. he only dfference s that Equaton (8) nvolves a constraned otmzaton roblem, whereas the tradtonal MFA solves an unconstraned otmzaton roblem. he motvaton for usng the constrant,,,, l s to avod calculatng the nverse of XBX, whch leads to the otental Small Samle Sze roblem. In order to solve the above constrant otmzaton roblem, we ntroduce a Lagrangan l L(, ) X ( B L) X ( ) (9) wth the multlers. he Lagrangan L should be maxmzed wth resect to both and. he condton that at the statonary ont, the dervatves of L resect to must vansh,.e., L(, ) ( X ( B L) X I ) 0,,,, l (0) and therefore,

6 6 X ( B L) X,,,, l () whch means that the ' s are the egenvalues of X ( B L) X and ' s are the corresondng egenvectors. hus, we have l l l () J( ) X ( B L) X herefore, the obectve functon s maxmzed when s comosed of the largest egenvectors of X ( B L) X. Here, by mosng constrant,,,, l, we need not calculate the nverse of XBX, and ths allows us to avod the Small Samle Sze roblem easly. he BMMA can be llustrated n Fg.. In the revous subsecton, we have formulated the BMMA algorthm and shown that the otmal roecton matrx can be obtaned by Generalzed Egenvalue Decomoston on a matrx. hen the roblem s how to determne an otmal dmensonalty for RF,.e., the roected subsace. o acheve such a goal, we gve the detal of determnng the otmal dmensonalty. In general, max tr( X ( B L) X ) (3) l ' where are the assocated egenvalues and we have s (4) d 0 d l o maxmze the margn between the ostve samles and negatve samles, we should reserve all the egenvectors assocated wth the ostve egenvalues. However, as ndcated n [33], for mage retreval the orthogonal comlement comonents are essental to cature the same concet shared by all ostve samles. Based on ths observaton, we should also reserve the comonents assocated wth zero egenvalues although they do not contrbute to maxmze the margn. hs technque can effectvely reserve more geometry roertes of the feedbacs n the orgnal hgh-dmensonal feature sace. herefore, the otmal dmensonalty of the roected subsace ust corresonds to the number of nonnegatve egenvalues of the matrx. herefore, comared to the orgnal formulaton of the grah embeddng framewor n [30], the new formulaton (8) can easly avod the ntrnsc Small Samle Sze roblem and also rovde us wth a smle way to determne the otmal dmensonalty for ths subsace learnng roblem. Based Dscrmnant Analyss (BDA) and ts ernel verson BasMa [0] were frst roosed to address the asymmetry between the ostve and negatve samles n nteractve mage retreval. However, to use BDA, the Small Samle Sze roblem and Gaussan assumton for ostve feedbacs are two maor challenges. Whle the ernel method BasMa cannot exert ts normal caablty snce the feature dmensons are much hgher than the number of tranng samles. Addtonally, t s stll roblematc to determne the otmal dmensonalty of BDA and BasMa for CBIR. Dfferent from the orgnal BDA, our BMMA algorthm s a local dscrmnant analyss aroach, whch does not mae any assumton on the dstrbuton of the samles. Based Fg.3 An llustraton of the hyerlance comarson between BMMA and SemBMMA for two classes of feedbacs towards the ostve samles, maxmzng the obectve functon n the roected sace can ush the nearby negatve samles away from the ostve samles whle ullng the nearby ostve samles towards the ostve samles. herefore, the defnton n Equaton (8) can maxmze the overall average margn between the ostve samles and negatve samles. In such a way, each samle n the orgnal sace s maed onto a low-dmensonal local maxmum margn subsace n accordance wth human erceton of the mage contents. Snce the grah embeddng technque s an effectve way to cature the ntrnsc geometry structure n the orgnal feature sace, we roose a way to ncororate the unlabelled samles based on the ntrnsc grah, whch s helful n caturng the manfold structure of samles and allevatng the over fttng roblem. In the followng, we desgn a regularzaton term based on ntrnsc grah for the unlabelled samles n the mage database. For each unlabelled samle x ( n n n), we exect that the nearby unlabelled samles are lely to have the smlar low-dmensonal reresentatons. Secfcally, for each unlabelled samle, we fnd ts nearest neghborhood unlabelled samles, whch can be reresented as a samle set u N, and ut an edge between the unlabelled x and ts neghborhood unlabelled samles. hen the ntrnsc grah for the unlabelled samles s characterzed as follows: W u S x x W u U * s s : or u u tr[ X ( D W ) X ] tr[ XUX ] 0, else u u n ex( x / ), ( ) ( ) 0, x f l l or (5) (6) u whch reflects the affnty of the samle ars; D s a dagonal matrx whose dagonal elements are calculated by u u D W ; u D denotes the total number of nearest neghborhood unlabelled samle ars for each unlabelled u u u samle. L D W can be nown as a Lalacan matrx.

7 7 User Retreval Results User Labels System Labels System Selects Postve Negatve Unlabelled Fnal Results Query Image YES Relevance Feedbac Model NO Vsual Features Smlarty Metrc Image Database Fg.4 he flowchart of the Content-Based Image Retreval system Hence, we call ths term as a Lalacan regularzer. he motvaton for ntroducng ths term s nsred by the regularzaton rncle, whch s the ey to enhancng the generalzaton and robust erformance of the aroach n ractcal alcatons. here are a lot of ossble ways to choose a regularzer for the roosed BMMA. In ths wor we chose the Lalacan regularzer, whch s largely nsred by the recently emergng manfold learnng communty. Actually, ths scheme can reserve wea (robably correct) smlartes between all unlabeled samle ars and thus effectvely ntegrate the smlarty nformaton of unlabeled samles nto the BMMA. By ntegratng the Lalacan regularzer nto the suervsed BMMA, we can easly obtan the SemBMMA for the RF,.e., * arg max [ tr X ( B L U ) X ] (7) where s used to trade off the contrbutons of the labeled samles and unlabelled samles. Smlarly, the soluton of (7) s obtaned by conductng the Generalzed Egenvalue Decomoston and s calculated as a set of egenvectors. he dfference between BMMA and SemBMMA for two classes of feedbacs can be shown n Fg.3. he SemBMMA algorthm s llustrated n able. able Sem-Suervsed Based Maxmum Margn Analyss h Inut: X { x, x, xn} R stand for all the feedbac samles and unlabelled samles, whch nclude the ostve samle set X, the negatve samle set X, and unlabelled samles. ) Construct the suervsed ntrnsc grah G, accordng to the formulaton (4) and calculate the matrx value XLX. ) Construct the suervsed enalty grah G, accordng to the formulaton (6) and calculate the matrx value XBX. 3) Construct the Lalacan regularzer accordng to the formulaton (6) and calculate the matrx value XUX. 4) Calculate the roecton matrx * accordng to Generalzed Egenvalue Decomoston on the matrx X ( B L U ) X. 5) Calculate the new reresentatons: roect all ostve, negatve and remanng samles n the database onto the reduced subsace resectvely,.e., * * Y X, Y X. Outut: Postve and negatve samles, Y and Y, n ths reduced subsace. hen all the unlabeled samles n the database are roected onto ths subsace. After the roecton, the tradtonal RF s executed on the new reresentatons. Fnally, smlar to the tradtonal RF, we can measure the degree of relevance through the outut of,.e., f ( x ). IV. HE CONEN-BASED IMAGE RERIEVAL SYSEM In exerments, we use a subset of the Corel Photo Gallery as the test data to evaluate the erformance of the roosed scheme. he orgnal Corel Photo Gallery ncludes lenty of semantc categores, each of whch contans 00 or more mages. However, some of the categores are not sutable for mage retreval, snce some mages wth dfferent concets are n the same category and many mages wth the same concet are n dfferent categores. herefore, the exstng categores n the orgnal Corel Photo Gallery are gnored and reorganzed nto 80 concetual classes based on the ground truth, such as lon, castle, avaton, tran, dog, autumn, cloud, tger, etc. Fnally, the test database comrses totally 0,763 real-world mages. Gven a query mage by the user, the CBIR system s exected to feed bac more semantcally relevant mages after each feedbac teraton []. However, durng RF, the number of the relevant mages s usually very small because of the semantc ga. At the same tme, the user would not le to label a large number of samles. he user also exects to obtan more relevant mages wth only a few rounds of RF teratons. Keeng the sze of labeled relevant mages small and the relevance feedbac teratons few are two ey ssues n desgnng the mage retreval system. herefore, we devse the followng CBIR framewor accordngly to evaluate the RF algorthms. From the flowchart n Fg.4, we can notce that when a query mage s rovded by the user, the mage retreval system frst extracts the low-level features. hen all the mages n the database are sorted based on a smlarty metrc,.e. Eucldean dstance. If the user s satsfed wth the results, the retreval rocess s ended, and the results are resented to the user. However, because of the semantc ga, most of the tme, the user s not satsfed wth the frst retreval results. hen she/he

8 8 wll label the most semantcally relevant mages as ostve feedbacs n to retreval results. All of the remanng mages n to results are automatcally labeled by the system as the negatve feedbacs. Based on the small sze ostve and negatve feedbacs, the RF model can be traned based on varous exstng technques. hen all the mages n the database are resorted based on a new smlarty metrc. After each round of retreval, the user wll chec whether the results are satsfed. If the user s satsfed wth the results, then the rocess s ended; otherwse, the feedbac rocess reeats untl the user s satsfed wth the retreval results. Generally, the mage reresentaton s a crucal roblem n CBIR. he mages are usually reresented by low level features, such as color [3-5], texture [5-7] and shae [8-0], each of whch can cature the content of an mage to some extent. For color, we extracted three moments: color mean, color varance, and color sewness n each color channel (L, U, V) resectvely. hus, a 9-dmensonal color moment s emloyed as the color features n our exerments to reresent the color nformaton. hen a 56-dmensonal (8*8*4) HSV color hstogram s calculated. Both hue and saturaton are quantzed nto 8 bns and the values are quantzed nto 4 bns. hese two nds of vsual features are formed as color features. Comarng wth the classcal global texture descrtors (e.g., Gabor features, wavelet features), the local dense features show good erformance n descrbng the content of an mage. he Webber Local Descrtors (WLD) [4] are adoted as feature descrtors whch are manly based on the mechansm of the human erceton of a attern. he WLD local descrtor results n a feature vector of 40 values. We emloy the edge drectonal hstogram [8] from the Y comonent n YCrCb sace to cature the satal dstrbuton of edges. he edge drecton hstogram s quantzed nto fve categores ncludng horzontal, 45 dagonal, vertcal, 35 dagonal and sotroc drectons to reresent the edge features. Generally, these features are combned nto a feature vector, whch results n a vector wth 50 values (.e., =50). hen all feature comonents are normalzed to normal dstrbutons wth zero mean and one standard devaton to reresent the mages. V. EXPERIMENAL RESULS ON A REAL WORLD IMAGE DAABASE A he ntrnsc roblems n the tradtonal RF An mage s usually reresented as a hgh-dmensonal feature vector n CBIR. However, one ey ssue n RF s that whch subset of features can reflect the basc roertes of dfferent grous of feedbac samles and beneft the constructon of the otmal classfer. hs roblem can be llustrated from some real-world data n relevance feedbac. here are fve ostve samles and fve negatve feedbac samles. We randomly select two features to construct the otmal hyerlane for three tmes. As shown n Fg.5, (a) (b) (c) Fg.5 he hyer lane s dverse for dfferent combnatons of features. we can see that the resultant classfers are dverse wth dfferent combnatons of features. It s essental to obtan a satsfactory classfer when the number of avalable feedbac samles s small, whch s always the case n RF, esecally n the frst few rounds of feedbacs. herefore, we frst show a smle examle to smulate the unstable roblem of when dealng wth a small number of tranng samles. he oen crcles n Fg.6 ndcate the ostve feedbac samles and the lus onts ndcate the negatve samles n relevance feedbac. he Fg.6 (a) shows an otmal hyerlane, whch s traned by the orgnal tranng samles. Fg.6 (b) and (c) show a dfferent otmal hyerlane, whch are traned by the orgnal tranng set wth only one and two ncremental ostve samle resectvely. From Fg.6, we can see that the hyerlane of the classfer changes sharly when a new ncremental samle s ntegrated nto the orgnal tranng set. Addtonally, we can also note that the otmal hyer lanes of are much comlex when the feedbacs have a comlcated dstrbuton. (a) (b) (c) Fg. 6 he hyer lane s unstable and comlex when dealng wth small sze of tranng set Note that the smlar results have been ndcated n the revous research [5]. However, n ths secton, we have shown slghtly dfferent roblems n the tradtonal RF, that s, dstnct roerty of feedbac samles n RF and unstable and comlex hyer lanes of the tradtonal n the frst few rounds of feedbacs. B Features extracton based on dfferent methods Sx exerments are conducted for comarng the BMMA wth the tradtonal LDA, BDA method and a Grah embeddng aroach MFA, n fndng the most dscrmnatve drectons. We lot the drectons whch corresond to the largest egenvalue of the decomosed matrces for LDA, BDA, MFA and BMMA resectvely. From these examles, we can clearly notce that LDA can fnd the best dscrmnatve drecton when the data from each class are dstrbuted as Gaussan wth smlar covarance matrces, as shown n Fg.7 (a) and (d), but t may confuse when the data dstrbuton s more comlcated, as gven n Fg.7(b), (c), (e) and (f). Based towards the ostve samles, BDA can fnd the drecton that

9 9 5 Postve Negatve 4 LDA 3 BDA MFA BMMA (a) 5 Postve Negatve 4 LDA BDA 3 MFA BMMA (c) 5 Postve 4 Negatve LDA 3 BDA MFA BMMA Postve Negatve 4 LDA 3 BDA MFA BMMA (b) 5 Postve Negatve 4 LDA BDA 3 MFA BMMA (d) 5 Postve 4 Negatve LDA 3 BDA MFA BMMA secfed by the number N of to-raned mages resented to the user. he recson s the maor evaluaton crteron, whch evaluates the effectveness of the algorthms. he recson-scoe curve descrbes the recson wth varous scoes and can gve the overall erformance evaluaton of the aroaches. Precson rate s the rato of the number of relevant mages retreved to the to N retreved mages, whch emhaszes the recson at a artcular value of scoe. Standard devaton descrbes the stablty of dfferent algorthms. herefore, the recson evaluates the effectveness of a gven algorthm and the corresondng standard devaton evaluates the robustness of the algorthm. We emrcally select the arameters, 4 accordng to manfold learnng aroaches. Consderng the comutable effcency, we randomly select 300 unlabeled samles n each round of feedbac teraton. For the trade off arameter between labeled samles and unlabeled samles, we smly set. For all the -based algorthms, we choose the Gaussan ernel: xy (8) K( x, y) e, 0.00 Note that, the ernel arameters and ernel tye can sgnfcantly affect the erformance of retreval. For dfferent mage database, we should tune the ernel arameters and ernel tye carefully. In our exerments, we determne the ernel arameters from a seres of values accordng to the erformance. Moreover, much better erformance can be acheved by tunng the ernel arameters further for dfferent queres (e) (f) Fg.7 Four feature extracton methods (.e., LDA, BDA, MFA and BMMA) for two classes of samles (.e., ostve samles and negatve samles) wth dfferent dstrbuton ) Exerments on a small sze mage database the ostve samles are well searated wth the negatve samles when the ostve samles have Gaussan dstrbuton, but t may also confuse when the dstrbuton of the ostve samles s more comlcated. For nstance, n Fg.7 (b), BDA can fnd the drecton for dstngushng ostve samles from negatve ones. However, n Fg.7 (c), (e) and (f), when the ostve samles have more comlcated dstrbuton, the BDA algorthm obvously fals. MFA can also fnd the dscrmnatve drecton when the dstrbuton of negatve feedbacs s smle, as shown n Fg.7 (a), (b), (c). But when the negatve samles ose a more comlcated dstrbuton, MFA wll fals as n Fg.7 (d), (e), (f). Based towards ostve samles, the BMMA method can fnd the most dscrmnatve drecton for all the 6 exerments based on local analyss, snce t doesn t mae any assumtons on the dstrbutons of the ostve and negatve samles. It should be noted that BMMA s a lnear method and therefore, we only gave the comarson results of lnear methods above. C Statstcal exermental results In ths secton, we evaluate the erformance of the roosed scheme on a real world mage database. We use recson-scoe curve, recson rate and standard devaton to evaluate the effectveness of the mage retreval algorthms. he scoe s Fg.8 Examle categores used n the small sze mage database In order to show how effcent the roosed BMMA combned wth n dealng wth the asymmetry roertes of feedbac samles, the frst evaluaton exerment s executed on a small sze database, whch ncludes 3899 mages wth 30 dfferent categores. We use all 3899 the mages n 30 categores as queres. Some examle categores used n exerments are shown n Fg.8. o avod the otental roblem caused by the asymmetry amount of ostve and negatve feedbacs [5], we selected equal number of ostve and negatve feedbacs n ths subsecton. In ractce, the frst 5 query relevant mages and frst 5 rrelevant mages n the to 0 retreved mages n the revous teratons were automatcally selected as ostve and negatve feedbacs resectvely. In [33], OCCA was roosed to only analyze the ostve feedbacs for RF n a retreval tas. Hence, we comared the RF erformance of BMMA combned wth (BMMA ), OCCA combned wth (OCCA ) and the tradtonal n ths subsecton.

10 Average Precson Average Precson Average Precson n o 0 Results BaseLne Category No Fg.9 Average recsons n the to 0 results of, OCCA and BMMA after rounds of feedbac Baselne Baselne Scoe Scoe (a) (b) Fg.0 he recson-scoe curves after the st feedbac and nd feedbac for, OCCA and BMMA In real world, t s not ractcal to requre the user to label many samles. herefore, small sze of tranng samles wll cause the severe unstable roblem n RF (as shown n Secton V. A). Fg.9 shows the recsons n the to 0 after the nd round of feedbac teraton for all the 30 categores. he baselne curve descrbes the ntal retreval results wthout any feedbac nformaton. Secally, at the begnnng of retreval, the Eucldean dstances n the orgnal hgh dmensonal sace are used to ran the mages n the database. After the user rovdes relevance feedbacs, the tradtonal, BMMA and OCCA algorthms are then aled to sort the mages n the database. As can be seen, the retreval erformance of these algorthms vares wth dfferent categores. For some easy categores, all the algorthms can erform well (for Categores, 4 even the baselne can acheve over 90% for recson). For some hard categores, all the algorthms erform oorly (e.g., Categores 8, 0, 4). After two rounds of feedbacs, all the algorthms are sgnfcantly better than the baselne, and ths ndcates that the relevance feedbacs rovded by the user are very helful n mrovng the retreval erformance. Fg.0 shows the average recson-scoe curves of the algorthms for the st and nd teratons. We can notce that both the BMMA and the OCCA can erform much better the tradtonal on the entre scoe, esecally the st round of feedbac. he man reason s that n the frst round of feedbac teraton, the number of tranng samles s esecally small (usually 8-0 tranng samles totally), and ths wll mae erform extremely oorly. he BMMA algorthm and the OCCA algorthm can sgnfcantly mrove the erformance of by treatng the ostve and negatve feedbacs unequally. herefore, we can conclude that the technque, whch asymmetrcally treats the feedbac samles (.e., based towards the ostve feedbacs), can sgnfcantly mrove the erformance of RF whch treats the feedbac samles equvalently. As shown n Fg.0 (b), wth the number of feedbacs ncreasng, the erformance dfference between the enhanced algorthms and the tradtonal gets small. Generally, by teratvely addng the user s feedbacs, more samles wll be fed bac as tranng samles, and wll mae the erformance of much more stable. Meanwhle, the dmenson of the BMMA and OCCA decreases wth the ncreasng of the ostve feedbacs. Consequently, the erformance of BMMA and OCCA wll be degraded by over fttng. herefore, the erformance dfference between the enhanced algorthms and the tradtonal gets small. However, the erformance of the frst a few rounds of feedbacs s usually most mortant, snce the user would not le to rovde more rounds of feedbac teraton. In the frst a few rounds of feedbacs, the classfer traned based on few labeled tranng samles s not relable, but ts erformance can

11 Standard Devaton Standard Devaton Standard Devaton Standard Devaton Standard Devaton Standard Devaton Average Precson Average Precson Average Precson Average Precson Average Precson Average Precson Average Precson n o 0 Results Average Precson n o 0 Results 0.9 Average Precson n o 30 Results One Number of Iteratons Sem BDA Sem BDA One Number of Iteratons Number of Iteratons (a) (b) (c) Average Precson n o 40 Results 5 Average Precson n o 50 Results 5 Sem BDA One Average Precson n o 60 Results Sem BDA One Number of Iteratons 5 Sem BDA One Number of Iteratons 5 Sem BDA One Number of Iteratons (d) (e) (f) Fg. he average recsons n to 0- to 60 results of the sx aroaches from the fvefold cross valdaton. Standard Devaton n o 0 Results Sem BDA One 4 8 Standard Devaton n o 0 Results Sem BDA One 9 8 Standard Devaton n o 30 Results Sem BDA One Number of Iteratons Standard Devaton n o 40 Results Number of Iteratons Number of Iteratons (a) (b) (c) Standard Devaton n o 50 Results Standard Devaton n o 60 Results Sem BDA One Number of Iteratons 4 Sem BDA One Number of Iteratons Sem BDA One Number of Iteratons (d) (e) (f) Fg. he standard devatons n to 0- to 60 results of the sx aroaches from the fvefold cross valdaton. be mroved when more mages are labeled by the user n the subsequent feedbac teratons. Both BMMA and OCCA can sgnfcantly mrove the erformance of the tradtonal n the frst two teratons. herefore, we can conclude that BMMA can effectvely ntegrate the dstnct roertes of two grous of feedbac samles nto the retreval rocess and thus enhance the erformance. ) Exerments on a large scale mage database We desgned a slghtly dfferent feedbac scheme to model the real world retreval rocess. In a real mage retreval system, a query mage s usually not n the mage database. o smulate such an envronment, we use fvefold cross valdaton to evaluate the algorthms. More recsely, we dvde the whole mage database nto fve subsets of equal sze. hus, there are 0 ercent mages er category n each subset. At each run of cross valdaton, one subset s selected as the query set, and the other four subsets are used as the database for retreval. hen 400 query samles are randomly selected from the query subset and the relevance feedbac s automatcally mlemented by the system. For each query mage, the system retreves and rans the mages n the database and 9 RF teratons are automatcally executed. At each teraton of the relevance feedbac rocess, to 0 mages are ced from the database and labeled as relevant

12 able Average recsons n to N results of the sx algorthms after the 9 th feedbac teraton (mean± standard devaton) Method Sem BDA One o ± ± ± ± ±0 ±795 o ± ± ±6 0.76± ±38 48±49 o ± ± ± ±7 996±34 878±797 o ±49 645±64 35±8 465±7 90± ±588 o50 396±54 98± ±76 89±63 706±993 33±404 o60 854±548 43± ±660 39±497 6±847 47±55 o70 46±53 964±549 69±544 95±36 806± ±5 o80 05± ±430 46± ±9 46± ±000 o90 865±387 4±98 94±87 30±0 66± ±0.890 and rrelevant feedbacs. Generally, n real world retreval systems, the negatve samles usually largely outnumber the ostve ones. o smulate such a case n the retreval system, the frst 3 relevant mages are labeled as ostve feedbacs and all the other rrelevant mages n to 0 results are automatcally mared as negatve feedbacs. Note that, the mages whch have been selected at the revous teratons are excluded from later selectons. he exermental results are shown n Fg. and Fg.. he average recson and standard devaton are comuted from the fvefold cross valdaton. o demonstrate the effectveness of the roosed scheme, we comare them wth the tradtonal, the OCCA, BDA and one-class (One). he tradtonal regards RF as a strct two-class classfcaton roblem, wth equal treatments on both ostve and negatve samles. he OCCA tres to fnd a subsace, n whch all ostve samles are merged, and then the tradtonal are mlemented to retreve the relevant mages to the query mage. For BDA, we select all the egenvectors wth the egenvalues larger than one ercent of the maxmum evenvalues and then the tradtonal are used to classfy the relevant and rrelevant mages, whch s the common way to select the dmenson of the subsace. One assumes RF as a one-class classfcaton roblem and estmates the dstrbuton of the target mages n the feature sace. Fg. and Fg. show the average recson and the standard devaton curves of dfferent algorthms resectvely. SemBMMA outerforms all the other algorthms on the entre scoe. Both BMMA and OCCA can mrove the erformance of the tradtonal RF, as shown n Fg. (a), (b), (c) and (d). Comarng wth OCCA, the BMMA erforms much better for all the to results, snce BMMA taes both the ostve and negatve feedbacs nto consderaton. However, both BMMA and OCCA wll encounter the over fttng roblem,.e., both of them combned wth wll degrade the erformance of after a few rounds of feedbacs although they can mrove the erformance of n the frst a few rounds of feedbac. As can be seen n Fg. (d) (e) (f), wth the ncrease of rounds of feedbacs, the OCCA erforms oorly n comarson wth the tradtonal. At the same tme, the erformance dfference between BMMA and the tradtonal gets smaller. he SemBMMA combned wth can sgnfcantly mrove the erformance of the tradtonal, snce t can effectvely utlze the basc roerty of the dfferent grous of the feedbac samles and ntegrate the nformaton of the unlabelled samles nto the constructon of the classfer. For to 0 results, the BDA can acheve better result than the tradtonal RF. However, the BDA algorthm stll dscards much nformaton contaned n the orthogonal comlement comonents of the ostve samles. herefore, BDA combned wth tradtonal erforms much worse than the tradtonal RF, as shown n Fg. (b), (c), (d), (e), (f). Although One tres to estmate the dstrbuton of the target mages n the feature sace, t cannot wor well wthout the hel of negatve samles. Consderng the stablty of the algorthms, we can also notce that SemBMMA and BMMA erform best among all the algorthms for to0, to 0 and to 30 results. Although One shows good stablty for to 40, to 50 and to 60 results, ts average recson for retreval s too low. We should ndcate that the erformance dfference of algorthms between exerments n Subsecton V.C ) and exerments n ths subsecton s manly caused by the dfferent exermental settng. Because the number of ostve and negatve feedbacs s equal n Subsecton V.C ), whle negatve feedbacs largely outnumber ostve feedbacs n subsecton ). Addtonally, the erformance of SemBMMA does not erform better comarng wth BMMA n the frst two rounds of feedbac teratons for most of the results. hs s manly because the maxmum margn between dfferent classes s essentally mortant when the number of tranng samles s extremely small. he detaled results of all the algorthms after the 9 th feedbac are shown n able.as can be seen, SemBMMA combned wth ntegrates all the avalable nformaton nto relevance feedbac teraton and acheves much better erformance comarng wth other aroaches for all the to results. he BMMA stll obtans satsfactory erformance comarng wth the tradtonal and OCCA. herefore, we can conclude that the roosed BMMA and SemBMMA combned wth the RF have shown much better erformance than the tradtonal RF (.e., drectly usng the as a RF scheme) for CBIR. 3) Vsualzaton of the retreval results In the revous subsectons, we have resented some statstc quanttatve results of the roosed scheme. In ths

13 3 Fg.3 o 0 results for 4 dfferent query mages based on ntal results and SemBMMA after 4 rounds of feedbac teraton. Incorrect results are hghlghted by green boxes. subsecton, we show the vsualzaton of retreval results. In exerments, we randomly select some mages (e.g., bobsled, cloud, cat and car) as the queres and erform the relevance feedbac rocess based on the ground truth. For each query mage, we do 4 RF teratons. For each RF teraton, we randomly select some relevant and rrelevant mages as ostve and negatve feedbacs from the frst screen, whch contan 0 mages n total. he number of selected ostve and negatve feedbacs s about 4 resectvely. We choose them accordng to the ground truth of the mages,.e., whether they share the same concet wth the query mage or not. Fg.3 shows the exermental results. he query mages are gven as the frst mage of each row. We show the to to to 0 mages of ntal results wthout feedbac and SemBMMA after 4 feedbac teratons resectvely. And ncorrect results are hghlghted by green boxes. From the results, we can notce that our roosed scheme can sgnfcantly mrove the erformance of the system. For the st nd and 4th query mages, our system roduce 0 relevant mages out of the to 0 retreved mages. For the 3rd query mage, our system roduces 9 relevant mages out of the to 0 retreved mages. herefore, SemBMMA can effectvely detect the homogeneous concet shared by the ostve samles and hence mrove the erformance of the retreval system. VI. CONCLUSION AND FUURE WORK Suort Vector Machne () based Relevance Feedbac (RF) has been wdely used to brdge the semantc ga and enhance the erformance of CBIR systems. However, drectly usng the as a RF scheme has two man drawbacs. Frst, t treats the ostve and negatve feedbacs equally although ths assumton s not arorate snce all ostve feedbacs share a common concet whle each negatve feedbac dffers n dverse concets. Second, t does not tae nto account the unlabelled samles although they are very helful n constructng a good classfer. In ths aer, we have exlored solutons based on the argument that dfferent semantc concets lve n dfferent subsaces and each mage can lve n many dfferent subsaces. We have desgned a Based Maxmum Margn Analyss and a Sem-Suervsed Based Maxmum Margn Analyss to allevate the two drawbacs n the tradtonal RF. he novel aroaches can dstngush the ostve feedbacs and negatve feedbacs by maxmzng the local margn and ntegrty the nformaton of unlabeled samle by ntroducng a Lalacan regularzer. Extensve exerments on a large real world Corel mage database have shown that the roosed scheme combned wth the tradtonal RF can sgnfcantly mrove the erformance of CBIR systems. Deste the romsng results, several questons reman to be nvestgated n our future wor: Frst, ths aroach nvolves dense matrces egen decomoston whch can be comutatonally exensve both n tme and memory. herefore, an effectve technque for comutaton s requred to allevate the drawbac. Second, theoretc questons need to be nvestgated regardng how the roosed scheme affects the generalzaton error of classfcaton models. More secfcally, we exect to get a better tradeoff between the ntegraton of the dstnct roertes of feedbacs and the generalzaton error of the classfer. ACKNOWLEDGEMEN he authors would le to than Prof. James Z. Wang (wth the College of Informaton Scences and echnology n Pennsylvana State Unversty) for hs ndly rovdng the Corel Image Gallery. he authors would le to than the

14 4 handlng Assocate Edtor and three anonymous revewers for ther constructve comments on ths manuscrt n the three rounds of revew. REFERENCES [] A. W. M. Smeulders, M. Worrng, S. Santn, A. Guta, and R. Jan, Content-Based Image Retreval at the End of the Early Years, IEEE rans. Pattern Anal. Mach. Intell., Vol., No., , Jan.000. [] R. Datta, D. Josh, Ja L, J.Z.Wang, Image Retreval: Ideas, Influences, and rends of the New Age. ACM Comutng Surveys, Vol. 40, No.,. 60, Ar [3] M. J. Swan and D. H. Ballard, Color Indexng, Internatonal. Journal of Comututer. Vson., Vol.7, No.,. 3, 99. [4] G. Pass, R. Zabh, and J. Mller, Comarng Images Usng Color Coherence Vectors, In Proc. ACM Multmeda, [5] Y. Rubner, J. Puzcha, C. omas, J. M. Buhmann, Emrcal Evaluaton of Dssmlarty Measures for Color and exture, Comuter Vson and Image Understandng, Vol.84, [6] H. amura, S. Mor, and.yamawa, exture Features Corresondng to Vsual Perceton, IEEE rans. Syst., Man, Cybern., Vol. 8, No.6, , Jun [7] J. Mao and A. Jan, exture Classfcaton and Segmentaton usng Multresoluton Smultaneous Autoregressve Models, Pattern Recognt.,Vol. 5, No., , 99. [8] A. Jan and A.Valaya, Image Retreval Usng Color and Shae, Pattern Recognt., Vol. 9, No. 8, , Aug [9] W. Nblac, R. Barber, W. Equtz, M. Flcner, E. Glasman, D. Petovc, P. Yaner, C. Faloutsos, and G. aubno, he QBICroect: queryng mages by content usng color, texture, and shae, In Proc. SPIE, Storage and Retreval for Images and Vdeo Databases, Feb [0] A. Jan and A. Valaya, Shae-Based Retreval: a Case Study wth rademar Image Databases, Pattern Recognt., Vol. 3, No. 9, , 998. [] Y. Ru,. Huang, M. Ortega, and S. Mehrotra, Relevance Feedbac: A Power ool n Interactve Content-Based Image Retreval, IEEE rans. Crcuts Syst. Vdeo echnol., Vol. 8, No. 5, , Se [] X. Zhou and. Huang, Relevance Feedbac for Image Retreval: A Comrehensve Revew, ACM Multmeda Systems J., Vol. 8, , 003. [3] Y. Ru,. S. Huang, and S. Mehrotra, Content-Based Image Retreval wth Relevance Feedbac n MARS, n Proc. IEEE Int. Conf. Image Process., 997, Vol., [4]. S. Huang and X. S. Zhou. Image Retreval by Relevance Feedbac: from Heurstc Weght Adustment to Otmal Learnng Methods. In Proc. IEEE Internatonal Conference on Image Processng (ICIP'0), hessalon,greece, Oct. 00. [5] J. Laasonen, M. Kosela, and E. Oa. Pcsom: Self-Organzng Mas for Content-Based Image Retreval. In Proceedngs of Internatonal Jont Conference on Neural Networs (IJCNN'99), Washngton, DC, USA, 999. [6] Y. Chen, X.-S. Zhou, and.-s. Huang, One-class for Learnng n Image Retreval, In Proceedngs of IEEE Int. Conf. Image Processng (ICIP'0), , 00. [7] C. Ho, C. Chan, K. Huang, M.R. Lyu and I.Kng, Based Suort Vector Machne for Relevance Feedbac n Image Retreval. In Proceedngs of Internatonal Jont Conference on Neural Networs (IJCNN 04), Budaest, Hungary, , 5-9 July, 004. [8] X. He, D. Ca, and J. Han Learnng a Maxmum Margn Subsace for Image Retreval. IEEE rans. on Knowledge and Data Engneerng, Vol. 0, No., Feb [9] W. Ban, D. ao: Based Dscrmnant Eucldean Embeddng for Content-Based Image Retreval. IEEE rans. on Image Processng Vol.9,No., : , Feb. 00 [0] X. Zhou and. Huang, Small Samle Learnng durng Multmeda Retreval Usng Basma, In Proc. IEEE Int l Conf. Comuter Vson and Pattern Recognton,Vol.,.-7,00. [] D. ao, X. ang, X. L, and Y. Ru, Kernel Drect Based Dscrmnant Analyss: A New Content-Based Image Retreval Relevance Feedbac Algorthm, IEEE rans. Multmeda, Vol.8, No.4,.76-74, August, 006 [] D. Xu, S. Yan, D. ao, S. Ln, H. Zhang, Margnal Fsher Analyss and Its Varants for Human Gat Recognton and Content-Based Image Retreval, IEEE rans. On Image Processng Vol.6, No,,.8-8, Nov [3] D. Ca, X. He, and J. Han. Regularzed Regresson on Image Manfold for Retreval, In Proceedngs of 9th ACM SIGMM Internatonal Worsho on Multmeda Informaton Retreval (MIR 07), Augsburg, Germany, Se [4] P. Hong, Q. an, and. S. Huang. Incororate suort vector machnes to content-based mage retreval wth relevant feedbac. In Proc. IEEE Internatonal Conference on Image Processng (ICIP'00), Vancouver,BC, Canada, 000. [5] D. ao, X. ang, X. L, and X. Wu, Asymmetrc Baggng and Random Subsace for Suort Vector Machnes-Based Relevance Feedbac n Image Retreval, IEEE rans. Pattern Anal. Mach. Intell., Vol. 8, No. 7, , Jul [6] G. Guo, A.K. Jan, W. Ma, and H. Zhang, Learnng Smlarty Measure for Natural Image Retreval wth Relevance Feedbac, IEEE rans. Neural Networs, Vol., No. 4,. 8-80, 00. [7] V. Van, he Nature of Statstcal Learnng heory. New Yor: Srnger-Verlag, 995. [8] S. ong and E. Chang, Suort Vector Machne Actve Learnng for Image Retreval, In Proc. ACM Multmeda, 00, [9] J. L, N. Allnson, D. ao and X. L, Multtranng Suort Vector Machne for Image Retreval. IEEE rans. On Image Processng. Vol.5 No., Nov.006. [30] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, and S. Ln, Grah embeddng and extensons: A general framewor for dmensonalty reducton, IEEE rans. Pattern Anal. Mach. Intell., Vol. 9, No.,.40 5, Jan [3] S.. Rowes, and L. K. Saul, Nonlnear Dmensonalty Reducton by Locally Lnear Embeddng, Scence, Vol 90, No., , Dec [3] J. enenbaum, V. Slva, and J. Langford, A Global Geometrc Framewor for Nonlnear Dmensonalty Reducton, Scence, Vol. 90, No., , Dec [33] D. ao, X. ang and X. L, Whch Comonents Are Imortant for Interactve Image Searchng? IEEE rans. Crcuts System Vdeo echnology, Vol. 8, No., Jan [34] M. Beln and P. Nyog. Lalacan Egenmas and Sectral echnques for Embeddng and Clusterng. In Advances n Neural Informaton Processng Systems (NIPS 00), Vol.4, 00. [35] X. He, D. Ca, S. Yan, and H.Zhang, Neghborhood Preservng Embeddng, Proc. IEEE Int l Conf.Comuter Vson (ICCV'05),. 08-3, 005. [36] X. He and P. Nyog, Localty Preservng Proectons, In Advances n Neural Informaton Processng Systems (NIPS 004),, Vol. 6, 004. [37] D. Ca, X. He, and J. Han Isometrc Proecton. In Proceedngs of nd Conference on Artfcal Intellgence (AAAI 07), Vancouver, Canada, July 007. [38]H. L,. Jang and K. Zhang, Effcent and Robust Feature Extracton by Maxmum Margn Crteron, Advances n Neural Informaton Processng Systems (NIPS 004),.57-65, 004.

15 5 [39]W. Yang, C. Sun, H. S. Du, J. Yang: Feature Extracton Usng Lalacan Maxmum Margn Crteron. Neural Processng Letters Vol. 33, No.,. 99-0, 0 [40] F. Wang, C. Zhang, Feature Extracton by Maxmzng the Average Neghborhood Margn, In Proc. IEEE Int l Conf. Comuter Vson and Pattern Recognton (CVPR 07),.-8, 007. [4] J. Chen, S. Shan, G. Zhao, X. Chen, W. Gao, Matt Petanen, WLD: A Robust Descrtor based on Weber s Law, IEEE rans. Pattern Anal. Mach. Intell., Vol. 3, No.9, , Se. 00. Lnng Zhang (S ) receved the B.Eng. degree and the M. Eng. degree n electronc engneerng from Xdan Unversty, X an, Chna, n 006 and 009, resectvely. He s currently worng towards the Ph.D. degree at the Nanyang echnologcal Unversty, Sngaore. Hs research nterests nclude comuter vson, machne learnng, multmeda nformaton retreval, data mnng and comutatonal ntellgence. He s a student member of the IEEE. Lo Wang (M 97-SM 98) receved the B.S. degree from the Natonal Unversty of Defense echnology, Changsha, Chna, n 983, and the Ph.D. degree from Lousana State Unversty, Baton Rouge, n 988. He s currently wth the School of Electrcal and Electronc Engneerng, Nanyang echnologcal Unversty, Sngaore. Hs research nterest s comutatonal ntellgence wth alcatons to bonformatcs, data mnng, otmzaton, and mage rocessng. He s (co-)author of over 00 aers (of whch 80+ are n ournals). He holds a U.S. atent n neural networs. He has co-authored monograhs and (co-)edted 5 boos. He was/wll be eynote/anel seaer for several nternatonal conferences. He s/was Assocate Edtor/Edtoral Board Member of 0 nternatonal ournals, ncludng IEEE ransactons on Neural Networs, IEEE ransactons on Knowledge and Data Engneerng, and IEEE ransactons on Evolutonary Comutaton. He s an elected member of the AdCom (Board of Governors, 00-0) of the IEEE Comutatonal Intellgence Socety (CIS) and served as IEEE CIS Vce Presdent for echncal Actvtes ( ) and Char of Emergent echnologes echncal Commttee ( ). He s an elected member of the Board of Governors of the Internatonal Neural Networ Socety (0-03) and a CIS Reresentatve to the AdCom of the IEEE Bometrcs Councl. He was Presdent of the Asa-Pacfc Neural Networ Assembly (APNNA) n 00/003 and receved the 007 APNNA Excellent Servce Award. He was Foundng Char of both the IEEE Engneerng n Medcne and Bology Sngaore Chater and IEEE Comutatonal Intellgence Sngaore Chater. He serves/served as IEEE EMBC 0 & 00 heme Co-Char, IJCNN 00 echncal Co-Char, CEC 007 Program Co-Char, IJCNN 006 Program Char, as well as on the steerng/advsory/organzng/rogram commttees of over 80 nternatonal conferences. Wes Ln (M 9 SM 98) receved the B.Sc. degree n electroncs and the M.Sc. degree n dgtal sgnal rocessng from Zhongshan Unversty, Guangzhou, Chna, n 98 and 985, resectvely, and the Ph.D. degree n comuter vson from Kng s College, London Unversty, London, U.K., n 99. He taught and conducted research at Zhongshan Unversty, Shantou Unversty (Chna), Bath Unversty (U.K.), the Natonal Unversty of Sngaore, the Insttute of Mcroelectroncs (Sngaore), and the Insttute for Infocomm Research (Sngaore). He has been the Proect Leader of 3 maor successfully-delvered roects n dgtal multmeda technology develoment. He also served as the Lab Head, Vsual Processng, and the Actng Deartment Manager, Meda Processng, for the Insttute for Infocomm Research. Currently, he s an Assocate Professor n the School of Comuter Engneerng, Nanyang echnologcal Unversty, Sngaore. Hs areas of exertse nclude mage rocessng, ercetual modelng, vdeo comresson, multmeda communcaton and comuter vson. He holds ten atents, edted one boo, authored one boo and fve boo chaters, and has ublshed over 80 refereed aers n nternatonal ournals and conferences. Dr. Ln s a fellow of Insttuton of Engneerng echnology. He s also a Chartered Engneer (U.K.). He organzed secal sessons n IEEE Internatonal Conference on Multmeda and Exo (ICME 006), IEEE Internatonal Worsho on Multmeda Analyss and Processng (007), IEEE Internatonal Symosum on Crcuts and Systems (ISCAS 00), Pacfc-Rm Conference on Multmeda (PCM 009), SPIE Vsual Communcatons and Image Processng (VCIP 00), Asa Pacfc Sgnal and Informaton Processng Assocaton (APSIPA 0), MobMeda 0 and ICME 0. He gave nvted/eynote/anelst tals n Internatonal Worsho on Vdeo Processng and Qualty Metrcs (006), IEEE Internatonal Conference on Comuter Communcatons and Networs (007), SPIE VCIP 00, and IEEE Multmeda Communcaton echncal Commttee (MMC) Interest Grou of Qualty of Exerence for Multmeda Communcatons (0), and tutorals n PCM 007, PCM 009, IEEE ISCAS 008, IEEE ICME 009, APSIPA 00, and IEEE Internatonal Conference on Image Processng (00). He s currently on the edtoral boards of IEEE rans. on Multmeda, IEEE SIGNAL PROCESSING LEERS and Journal of Vsual Communcaton and Image Reresentaton, and four IEEE echncal Commttees. He co-chars the IEEE MMC Secal Interest Grou on Qualty of Exerence. He has been on echncal Program Commttees and/or Organzng Commttees of a number of nternatonal conferences.