Person re-identification with content and context re-ranking

Transcription

1 DOI.7/s Person re-dentfcaton wth content and context re-rankng Qngmng Leng Rumn Hu Chao Lang Ymn Wang Jun Chen Sprnger Scence+Busness Meda New York 24 Abstract Ths paper proposes a novel and effcent re-rankng technque to solve the person re-dentfcaton problem n the survellance applcaton. Prevous methods treat person redentfcaton as a specal object retreval problem, and compute the retreval result purely based on a undrectonal matchng between the probe and all gallery mages. However, the correct matchng may be not ncluded n the top-k rankng result due to appearance changes caused by varatons n llumnaton, pose, vewpont and occluson. To obtan more accurate re-dentfcaton results, we propose to reversely query every gallery person mage n a new gallery composed of the orgnal probe person mage and other gallery person mages, and revse the ntal query result accordng to bdrectonal rankng lsts. The behnd phlosophy of our method s that mages of the same person should not only have smlar vsual content, refer to content smlarty, butalsopossesssmlark-nearestneghbors, refer tocontext smlarty. Furthermore, the proposed bdrectonal re-rankng method can be dvded nto offlne and onlne parts, where the majorty of computaton load s accomplshed by the offlne part and the onlne computaton complexty s only proportonal to the sze of the gallery data set, whch s especally suted to the real-tme requred vdeo nvestgaton task. Extensve experments conducted on a seres of standard data sets have valdated the effectveness and effcency of our proposed method. Keywords Person re-dentfcaton Content and context smlartes Bdrectonal rankng Re-rankng Q. Leng R. Hu ( ) C. Lang Y. Wang J. Chen Natonal Engneerng Research Center for Multmeda Software, School of Computer, Wuhan Unversty, Wuhan 4372, People s Republc of Chna e-mal: hurm964@gmal.com Q. Leng e-mal: lengqm@whu.edu.cn

2 Introducton Survellance vdeo has become one of the most mportant means for crmnal nvestgaton n recent years. In ths partcular applcaton, nvestgators often needs to detect and track a specfc suspect n a large-scale spatal regon covered by multple non-overlappng cameras []. Underthscondton, gvenahugeamountofsurvellancevdeo, purelymanual browsng s nether labor-savng nor tme-effcent to meet the requrement of effcent crmnal nvestgaton. To solve ths problem, matchng people across dsjont cameras, a.k.a. person re-dentfcaton, attracts ncreasng research nterests n the computer vson communty. Gven a query person mage (usually called probe), person re-dentfcaton tres to dentfy the correct person object n a huge amount of canddate persons mages (usually called gallery) captured by dfferent cameras, whch can be regarded as a specal mage retreval task. The general mage retreval work usually ncludes three key steps [2, 6, 8, 2] as shown n Fg. : Frst, varous global and/or local vsual features are extracted from both probe and gallery mages; Second, feature dstances between every probe and gallery mages are computed and an ntal rankng lst s generated; Thrd, the ntal rankng lst s further re-ranked for obtanng the better result. As above dscusson, person re-dentfcaton can be regarded as a target-specfc mage retreval problem. And recent methods [3, 7, 8,, 3, 22, 24] manly focus on the frst two steps of general mage retreval, where t can be generally categorzed nto two classes: feature-based and metrc-based methods. Feature-based methods am to fnd a set of dscrmnatve vsual descrptors that are robust among dfferent cameras. For example, Ghessar et al. [8] used a spatal-temporal segmentaton algorthm to generate salent edges and obtaned an nvarant dentty sgnature by combnng normalzed color and salent edge hstograms. Farenzena et al. [7] tred to combne multple features to descrbe the appearance mage, whch was dvded nto regons by explotng symmetry and asymmetry perceptual prncples. Xang et al. [22] used space and color nformaton characterzed by fuzzy quantzaton. Dfferent from feature-based methods whch pay more attenton to feature extracton, metrc-based methods are used to learn a proper dstance metrc functon. For example, transformng the orgnal feature nto a new feature space through Mahalanobs dstance learnng, where the feature dstance of the same person s smaller than that of dfferent persons. Because of consderng the dstrbuton characterstcs of tranng data, the performance of metrc-based methods s usually better than that of unsupervsed feature-based methods [24]. For nstance, Zheng et al. [24] proposed a relatve dstance comparson formulaton and used logstc functon for soft measure. Large maxmum-margn nearest Fg. Illustraton of general mage retreval process. Three key steps are ndcated by dotted boxes. In partcular, the re-rankng step ndcated by the red dotted box s a relatvely seldom touched topc n exstng person re-dentfcaton methods

3 Table Matchng rates (%) of forward and backward queryng at dfferent ranks on VIPeR data set Methods Forward queryng Backwardqueryng neghbor (LMNN) [2], whch s a popular metrc learnng method for k-nearest neghbor classfcaton, was revsed by explotng a fxed bound of neghborhood for person re-dentfcaton [3]. Kostnger et al. [3] proposed a smple but effcent metrc functon based on Gaussan dstrbuton of parwse samples wthout teratve procedures. Although both feature representaton and dstance measure have been well studed n person re-dentfcaton methods, research on the result re-rankng, the thrd step of general mage retreval procedure (ref. Fg. ), s a relatvely seldom touched topc. In prevous methods, the fnal rankng lst s purely computed based on a undrectonal smlarty measure, where a gallery person s placed at the top of rankng lst f t s most close to the probe, regardless of whether the probe s nearest to ths gallery person. Consderng the nfluence of varatons n llumnatons, poses, vewponts and occlusons, the correct match may be excluded from the top-k postons generated by the unrelable undrectonal matchng approaches. In ths paper, person re-dentfcaton problem s addressed by a newly proposed bdrectonal re-rankng technque wth reference to both content and context smlartes. Our dea s motvated by two basc prncples of evaluatng the best frendshp between ndvduals n socal network: () the frendshp of two persons are closest only when they treat each other as the best frend; (2) good frends are always trend to have more common frends [5]. We fnd some consstent characterstcs for the person re-dentfcaton problem. For example, on the publcly avalable VIPeR data set wth the sze 36 for test, we frstly query the probe person mage n the gallery, whch can be regarded as forward queryng; Second, every gallery person mage s quered n a new gallery composed of orgnal probe and other gallery person mages, then the poston of the probe person mage s used to rank the gallery, whch can be regarded as backward queryng (ref. Secton 2.2). Fnally, the performances of above two processes are calculated respectvely based on CMC matchng rates [9] as shown n Table. The phenomenon shows that dfferent mages of the same person may be not nsde the top of rankng lst, but they are stll postoned n the front of the bdrectonal rankng lsts. Moreover, when we further compare the k-nearest neghborhoods of two same and dfferent person mages, we fnd that mages of the same person have more common k-nearest neghbors than that of dfferent persons as shown n Table 2. Based on the above two observatons, t s vald to assume that mages comng from the same person should not only have smlar vsual content but also neghborng context. Therefore, we propose a novel bdrectonal person re-dentfcaton method to mprove the orgnal rankng result by referencng both content and context smlartes. Specfcally, we reversely query every gallery person n a new gallery composed of the orgnal probe and other gallery mages after obtanng an ntal rankng lst for the probe. Then we record the The bdrectonal ranks of two dfferent person mages may be not symmetrc, whch means even f person B s the top- nearest neghbor of person A, there can have another person C closest to B but far away from A.

4 Table 2 Percentage rate (%) of common k-nearest neghbors of pared mages counted from 36 persons and 9954 pars of dfferent persons on VIPeR data set Comparsons rank@5 2 5 Same person 8.35± ± ±6.22 Dfferent persons 2.35± ± ±2.2 poston of the probe mage n every backward rankng lst and count the matchng rates of common k-nearest neghbors between every two bdrectonal rankng lsts. The former reflects the content smlarty between the probe and gallery mage, whle the latter ndcates the context smlarty between two mages. Fnally, both content and context smlartes are adopted to revse the ntal person re-dentfcaton result. Moreover, t needs to declare that although both content and context smlartes come from the dstance between person mages, they have dfferent physcal meanngs and computatonal process. Content smlarty denotes the vsual smlarty between a probe mage p and a gallery mage g (see (3)), whle context smlarty ndcates the degree of approxmaton between the k-nearest neghborhood of p and g (see (4)). Ths paper s a an extent verson of the conference paper [5], besdes the obvous ncrease of paper length, the major mprovements of ths journal paper compared to the orgnal conference paper nclude: () Re-rankng process s expanded from mage-based to vdeo-based through cluster weghtng,.e. our bdrectonal re-rankng s ndependent to forms of object representaton; (2) More analyss and dscussons are conducted on the motvaton and dfference wth other re-rankng methods targeted for the general mage retreval; (3) More expermental results are processed on two new data sets, 3DPES [2]and PRID2 mult-shot [], and the performance and applcablty of our method s also evaluated and dscussed.. Related work The basc dea of our technque s smlar to Random Sample Consensus (RANSAC), whch generates an accurate model through mutual authentcaton between subsets [4]. But t s worthy notng that RANSAC algorthm only matches the mutual -nearest neghbor par ponts, and there s no context nformaton because ts bdrectonal matchng s conducted on two dsjont element sets. Besde RANSAC, there are some other smlar methods,.e. relevance feedback and re-rankng, resemble ours n the way that the mprovng ntal result n a new round computaton. Relevance feedback Relevance feedback technque s usually used to update the retreval model based on a set of relevant results through human feedback n mage retreval. Ru et al. [7] assgned hgher weghts to those features that are frequently occurrng n postve examples pcked up by the users. You et al. [23] learnt both user s postve and negatve examples. And Al et al.s [] frstly ntroduced relevance feedback nto person re-dentfcaton through learnng a dstance metrc teratvely. However, due to the nfluence of envronment change and ndvdual movement, the sngle person mage s not usually n the top of the rankng lst as shown n Table.

5 So Human-Computer Interacton (HCI) s a both labor and tme consumng work especally when there s a large-scale data set. In contrast, the proposed bdrectonal re-rankng technque can automatcally mprove the the ntal dentfcaton result, whch s both labor-effcent and tme-savng. Unsupervsed re-rankng It s unsupervsed process and refne the ntal rankng lst automatcally. For example, Wu et al. [2] constructed reference mages set to refne the canddate mages, however after generatng ntal rankng result, new feature extracton process was needed for re-rankng. Pedronette et al. [6] constructed complcated parwse feature structure for re-rankng based on analyzng contextual relatons of k-nearest neghborhood, whch brought expensve computaton complexty. Huang et al. [2] presented that user may pay more attenton to salent vsual object, so they re-ranked the searchng results based on both vsual consstency and salency, whle the characterstc of salency s not sutable for person re-dentfcaton. Shen et al. [8] re-ranked ntal search mages outsde k-nearest neghborhood depended on ther postons n the re-queryng rank of k- nearest neghborhood mages, and ts latent assumpton that many mages nsde k-nearest neghborhood are the groundtruth. Zhu et al. [4, 25] construct an elaborate context dstance for re-rankng whch gnores content dstance. However exstng person re-dentfcaton methods stll possess good performances n some publc data sets, t means that content dstance s also avalable to some extent, so dscardng the content dstance maybe brngs more naccuracy. Most of above re-rankng methods are from general mage/object retreval task. And n the data sets for general object retreval, there are usually several mages from one object. And a natural assumpton of the re-rankng methods from general object retreval s the majorty of these k-nearest neghborscontan the same objectas n thequery mage (Can be found n paragraph of page 5 on reference [8], so a canddate mage should be smlar as both the query mage and ts k-nearest neghborsas shown n Fg. 2a[6, 8, 2]. However, for person re-dentfcaton problem, there s only SINGLE correct person n the gallery for most publcly person re-dentfcaton data sets, e.g. VIPeR [9] and PRID2 []. (a) Tradtonal method (b) Our bdrectonal method Fg. 2 Illustraton of tradtonal re-rankng method and our bdrectonal method. a Tradtonal re-rankng method compares the gallery mage wth both probe mage and ts k-nearest neghbors; b Our method not only compare the probe mage wth every gallery mage, but also compare ther k-nearest neghborhoods. Crcle denotes the probe mage, damond presents the canddate mage,and k-nearest neghbors of both probe mage and canddate mage are descrbed by rectangles

6 Therefore the above assumpton used n the general mage retreval, s not reasonable for person re-dentfcaton task. In contrast, the proposed bdrectonal re-rankng technque refnes the ntal rankng lst based on both content and context smlartes, whch do not rely on above premses as shown n Fg. 2b, can mprove the effectveness of person redentfcaton sgnfcantly..2 Contrbuton Ths paper proposes a novel person re-dentfcaton technque based on content and context re-rankng, ts major contrbutons are as follows: Re-rankng s newly ntroduced nto person re-dentfcaton problem, and a novel and unsupervsed bdrectonal re-rankng technque wth content and context smlartes s used to refne the ntal rankng lst automatcally. The proposed method can be decomposed nto two parts, ts onlne computaton complexty of dstance measure s only proportonal to the sze of the gallery data set, whch s especally suted to those real-tme-requrng applcatons. Our re-rankng s ndependent to forms of object representaton, regardless of magebased or vdeo-based. The rest of ths paper s organzed as follows: we descrbe our bdrectonal rankng method n Secton 2, and present the mplementaton detals n Secton 3. Extensve expermental results are shown n Secton 4 and future work and conclusons are gven n Secton 5. 2 Bdrectonal rankng Content and context re-rankng processes are presented after generated a orgnal query result n ths secton as shown n Fg. 3. Forward content smlarty s calculated based on ntal rankng lst, and then backward content smlarty and context smlartes are computed through conversely queryng every gallery person mage n a new gallery, whch composes of orgnal probe and other gallery person mages. Fnally, forward and backward content smlartes, content smlarty are combned to re-rank the ntal rankng lst. Moreover, because the form of pedestrans n practcal survellance vdeo can be a key frame or mage sequence, we dscuss both sngle-shot and multple-shot patterns. For the convenence of Fg. 3 Illustraton of our bdrectonal person re-dentfcaton technque. Red dotted box denotes the processes of computng the content and context smlartes

7 followng dscusson, we frst focus our presentaton to the sngle-shot case, and further extend our method to the multple-shot case n the Secton Forward content smlarty Ths subsecton descrbes how to compute the forward content smlarty based on ntal rankng lst, whch s obtaned through feature extracton and dstance measure. Gven the mages of both probe and gallery, they are frstly dvded nto overlappng blocks of sze 6 6 and strde of 8 8. Then, HSV and RGB hstograms are used to descrbe color, and each wth 24 bns per channel. Texture nformaton s captured by the LBP feature. These vectors from all regons are concatenated and projected by the PCA dmenson reducton operaton. Wth the obtaned descrptor, Mahalanobs dstance s used to compute the smlartes between par of mages. Gven mage features from two person p and q,the mahalanobs dstance can be defned as: d 2 M (p, q ) = (p q ) T M (p q ) () where M s a postve semdefnte matrx. Ths paper used three dfferent methods to generate the ntal rankng result: mahalanobs dstance learnng, LMNN [2] and KISSME [3]. And then the ntal rankng lsts s calculated through dstance measure, where t s compared wth the fnal results after the content and context re-rankng operatons. Gven a probe person mage p and a gallery set G = {g =,..., n},wherens the sze of the gallery. Through computng ther dstances d(p,g ), an ntal forward query result can be obtaned as R p (G) = { g,g 2,..., n} g where g represents -th mage n the rankng lst, and the smlartes between p and g satsfy S(p,g ) )>S(p,g 2 )> >S(p,g n ). s used to defne the forward content smlarty between p and Here, the smlarty S ( p, g g. For clarty, we rewrte Sg p = S ( p, g ) to hghlght that p n the subscrpt denotng the Fg. 4 Illustraton of the backward content smlarty computaton. The parallelogram P ndcates probe person mage, I s a gallery person mage, and g s -th gallery person mage n the ntal rankng lst. The hollow arrows denotes forward queryng, vs-a-vs, soldarrows denotes backward queryng. Then, backward content smlarty between the probe person mage and a gallery person mage s defned by the rank poston of the probe n the gallery s reverse rankng lst

8 probe and g n the superscrpt denotng one mage n the orgnal gallery. And S g p can be defned as: S g p = Id(g ) (2) Id(g ) s the rank ndex of g n the generated rankng lst gven the probe p. 2.2 Backward content smlarty We further compute the backward content smlarty. Specfcally, every ntal gallery mage g s treated as a new probe, and reversely quered n ts new correspondng gallery set G g = { p, G\{g }}, whch composed of the orgnal probe mage and other gallery mages as shown n Fg. 4. Refer to the process of computng forward content smlarty, backward content smlarty S g p rankng lst R g (G g ). s defned by Id(p), whch s the rank poston of p n the reverse ( ) Then the complete bdrectonal content smlarty S cn g between p and g, can be defned as: ) S cn (p, g = S g p S p = g Id(g ) Id(p) (3) S g p s the forward content smlarty, and S p s backward content smlarty, they are g multpled to compute the whole content smlarty. Fg. 5 Illustratonof the context smlartycomputaton. The parallelogramp ndcates probe person mage, I s a gallery person mage, and g s -th gallery person mage n the ntal rankng lst. The hollow arrows denotes forward queryng, vs-a-vs, sold arrows denotes backward queryng. We use the number of common k-nearest neghbors to defne the context smlarty

9 2.3 Context smlarty Context smlarty s dscussed n ths subsecton. Denotng that k-nearest neghbors of p n the forward query lst s n k (p), and that of g ( ) n ts backward rankng lst s n k g as shown n Fg. 5. And then we can use the number of common k-nearest neghbors of p and g to present the context smlarty, k was set to percents of the sze of test ( ) gallery data set emprcally n ths work. Therefore, the context smlarty, S cx g, can be defned as: ) ) S cx (p, g = n k (p) n k (g (4) where n k (p) n k ( g ) represents the number of common k-nearest neghbors of p and g. 2.4 Re-rankng Fnally, both content and context smlartes are normalzed by sgmod functon, and combned to revse orgnal undrectonal rankng n accordance wth new smlarty S ( g ) defned as: S (p, g ) ( ) ( ) ( ) = S cn p, g S cx p, g nk (p) n k g = Id(g ) Id(p) (5) g ( ) can be re-ranked based on S g ultmately. Obvously, the smaller Id(g ) Id(p) s, the more smlar p and g are. On the contrary, the smaller ( ) n k (p) n k g s, the more dssmlar p and g are. 2.5 Multple-shot re-rankng As descrptons of sngle-shot re-rankng n above subsectons, content and context smlartes are computed through queryng the probe person and conversely queryng every gallery person. However n the multple-shot re-rankng, the representaton of a person s a seres of consecutve frames whch s provdng mult-vew appearance nformaton about a specfc target. Obvously, multple vdeo frames provde more appearance nformaton about a person object, but meanwhle, such multple frame representaton also brngs hgh computatonal complexty n the computaton of smlarty measure. To balance the doublesde requrements of accuracy and complexty, we frstly cluster every person sequence to several key frames [6], where the number of key frames do not need to be pre-defned. And then every probe key frame s quered n the gallery, where every gallery s formed as a set of key frames as shown n Fg. 6. Frstly, t must be clarfed that the consecutve sequence frames of a person usually contan herself sngle object n most standard multshot person re-dentfcaton data sets. Snce the proposed method s tested and compared on these data sets, t surely follows the above mentoned assumpton. Then consderng that the representaton of multple-shot probe person s p = { p (),..., p (a),..., p (A)}, where ts

10 Fg. 6 Illustraton of sequence queryng. Every key frame from probe sequence s compared wth the key frames set of every gallery sequence, all queryng results are combned to generate ntal rankng lst. And t the same way for conversely queryng every gallery person sequence. p (a) s a-th key frame of probe sequence, G ndcates the key frames set of a gallery sequence sze s{ A, andp (a) s a-th key } frame. Smlarly, -th gallery person can be descrbed by g = g (),..., g (b),...g (B ) wth the sze B. The dstance between a-th key frame of p and g s computed by: g (b) ) d (p (a),g = B b= ( d p (a),g (b) ) B (6) s the b-th key frame of -th gallery person. Then every gallery person s ranked based on d ( p (a) ),g for p (a), and the forward smlarty between p (a) and g s defned by the poston of g. Smlarly, we can rank the gallery for every key frame of the probe person. The entre forward smlarly S g p between p and g s the mean value of all forward smlartes between p s key frames and g. Then every gallery person s conversely quered n a new gallery composed of the orgnal probe person and other gallery persons through above process of dstance measure, and compute the backward content smlarty Sg p (ref. Secton 2.2). After Obtanng the forward and backward rankng lst, we total common k-nearest neghbors between the rankng lst of every key frame of probe person and that of every gallery person, whch s normalzed to calculate context smlarty (ref. Secton 2.3). Fnally, the ntal rankng lst s re-ranked based on smlar content and context smlartes as (5). 3 Implementaton detals Ths secton manly descrbes some mplementaton detals of the proposed method. For the convenence of dscusson, t manly focus on sngle-shot re-rankng, whch can be smlarly extended to multple-shot case. As can be seen from Fg. 3, the majorty of the computaton s spent on dervng the reverse rankng lsts, whch s consst of mass par-wse smlarty computaton between gallery mages. A tradtonal way of ths step s drectly calculatng the dstance of every par of mages and generatng the rankng lst based on above mage dstances. It assumes that the gallery contans n mages, the computaton complexty s O(n 2 ) for dstance measure and O ( n 2 log n ) for the rankng operatons. However, we fnd that gallery mages can be obtaned before queryng the probe mage n the

11 practcal applcaton, e.g. survellance vdeo nvestgaton. Therefore, our method s dvded nto two seperate phrases as shown n Algorthm. In the offlne part, all gallery mages are mutually compared for obtanng a rankng lst for every gallery person wth the computaton complexty O( n(n ) 2 ) and a rankng complexty of O ( n 2 log n ). In the onlne porton, the probe s nserted n every gallery s rankng lst comng from the offlne part, and then t only needs to compute the dstance between the probe and every gallery mage whose computaton complexty s O(n) and rankng complexty s O (n log n). Wth the above two-phrase mplementaton manner, the whole algorthm s complexty can be greatly reduced and ts onlne part s only proportonal to the sze of the gallery, whch s especally suted to those real-tme applcatons. 4 Expermental results In ths secton, extensve experments are presented to evaluate the usefulness of the proposed person re-dentfcaton technque. The experments are desgned to answer the followng four questons: How to compare proposed technque wth baselnes on the publcly person redentfcaton data sets? Are re-rankng processes based on the content smlarty alone, context smlarty alone and ther combnaton all useful to mprove the ntal person re-dentfcaton results? Does our re-rankng technque better than state-of-the-art of re-rankng methods from general mage/object retreval work? Are bdrectonal re-rankng technque effcent for multple-shot person? Does t reduce the computatonal complexty of onlne part after dvdng the framework nto offlne and onlne?

12 4. Experments settng Four publcly data sets are used for evaluaton: three sngle-shot data sets ncludng VIPeR [9], PRID2 [], 3DPES [2]; and PRID2 multple-shot data set []. Fgure 7 shows some examples of above data sets. The reasons for choosng above data sets are as follows: () The scenaro of these data sets are closed to practcal survellance vdeo, many challenges must be faced, e.g. vewpont changes, pose changes, llumnaton changes, camera chromatc aberratons and low mage resolutons; (2) they provde par person mages or sequences n two non-overlappng cameras, and are wdely used for evaluatng person re-dentfcaton methods; (3) the scenaros and szes of these data sets stll have some dfferences whch s sutable for estmatng the applcable vdeo survellance scene. The processes of experments manly contan three parts: () The effectveness of backward content and context smlartes are frstly tested. Because the re-rankng process should be based on ntal rankng lst, whch s used to compute forward content smlarty. We respectvely combne forward and backward content smlartes (called content-based), forward content smlarty and context smlarty (called context-based). And then above smlartes are used to re-rank ntal rankng lst on VIPeR; (2) The combnaton of bdrectonal content smlartes and context smlarty s used for re-rankng on VIPeR, PRID2 sngle-shot and 3DPES data sets; (3) The runtme of offlne and onlne processes are computed on VIPeR and PRID2 sngle-shot data sets fnally. Evaluaton procedure n every data set s looped tmes, cumulatve Matchng Characterstc (CMC) curves [9] s used to calculate the average performance, t descrbes the expectaton of fndng the true match wthn the frst r ranks, the formulaton below s used to calculate the CMC value when t returns the top r ranks. CMC@r = N f(p,r) = % (7) N (a) VIPeR (b) 3DPES (c) PRID2 Fg. 7 Examples of appearance changes caused by varant vewponts, llumnatons and camera chromatc aberratons from above three publc data sets. Each column represents a same-person par

13 p s -th query person, where the sze of probe s N. f(p,r) s an ndcatve functon whch s equal to f the true match s n the top r ranks,otherwse,ts.moreover,a relatve CMC gan s also appled to descrbe the mprovement ntutvely. The selecton of parameter k s mportant yet nsenstve n our method. We process our re-rankng method based on an ntal rankng lst that s produced by Mahal, and k s unequal-space valued from 5 to 2 on VIPeR data set (randomly selectng 36 persons for tranng, the other 36 for test) shown Fg. 8. As can be seen, the maxmum of the performance les on k = 3, where the value s close to percent of the sze of test data on VIPeR. For a smaller k, thecmc values notgoodenough.whenk >3, the performance s slghtly declned to a stable value. Therefore, 3 s assgned to k on VIPeR data set. And smlar phenomenon about k selecton s also found on some other data sets that the best k approxmately les on percent of the sze of test data. Therefore, k s set to the value of percent sze of test data on every data set. 4.2 VIPeR data set The VIPeR data set contans 632 person mage pars recorded from two dfferent statc camera vews. Appearance of the same person s changed by varaton of vewponts, llumnatons and poses between two cameras. For generatng the ntal rankng lst, the set of 632 mage pars were randomly splt nto two sets of 36 mage pars each, one for tranng and another for testng. The mage pars n test set were randomly apponted to a probe and a gallery set. The feature extracton processgot after thedescrptonn Secton 2., and ntal rankng lsts were acqured by three representatve person re-dentfcaton methods respectvely: mahalanobs dstance learnng based on stochastc gradent descent, LMNN [2] thatsa popular metrc learnng method for k-nearest neghbor classfcaton, and KISSME [3] learnng metrc functon based on Gaussan dstrbuton of parwse samples, whch s one of best person re-dentfcaton methods especally on the VIPeR. And the fnal results were.8 cmc@2 VIPeR CMC@ Mahal K Fg. 8 The CMC value when our method returns frst 2 result wth dfferent value of k

14 Fg. 9 Comparatve results between baselnes and our bdrectonal re-rankng usng content-based smlarty, contextbased smlarty and combned smlarty on VIPeR data set cmc VIPeR.2 Mahal Re rankng usng Context based smlarty. Re rankng usng Content based smlarty Re rankng usng combned smlarty (a) Comparson wth Mahalanobs cmc VIPeR.2 LMNN Re rankng usng Context based smlarty. Re rankng usng Content based smlarty Re rankng usng combned smlarty (b) Comparson wth LMNN [9] cmc VIPeR KISSME Re rankng usng Context based smlarty. Re rankng usng Content based smlarty Re rankng usng combned smlarty (c) Comparson wth KISSME [7]

15 gven after content and context re-rankng usng content-based smlarty, context-based smlarty and combned smlarty as shown n Fg. 9. Table 3 compares the performance of our technque usng combned smlarty to state-of-the-art n the range of the frst 5 ranks. In Fg. 9, green lne denotes the CMC curve of baselnes, blue and black lnes descrbe the results of context-based and content-based, and red lne show the results of combnaton smlarty. All our re-rankng results are better than baselnes, and re-rankng usng combnaton smlarty obtaned best performance. Table 3 shows that our technque based on combnaton smlarty outperforms all exstng methods over the whole range of ranks, especally there are more than % average relatve gans for Mahalanobs. Moreover, some examples descrbe the mprovement after re-rankng ntutonstcally as shown n Fg PRID2 sngle-shot data set The sngle-shot verson of the PRID2 data set whch conssts of person mage pars obtaned from two dfferent survellance cameras s consdered n ths subsecton. There are many challenges n PRID2 such as changes of vewponts, poses, llumnatons, and addtonal varant camera characterstcs dfferent from VIPeR data set. Partcularly, 385 persons mages are from camera vew A and that of camera vew A are 749 persons mages, wth 2 common mages n both vews. For generatng the ntal rankng lst, above 2 same person mage pars were randomly splt nto two equatonal subsets, where one for tranng and the other for test. Vew A was treated as the probe set and Vew B was used for the gallery set. The test persons were searched n all persons from Vew B (except the persons used for tranng), whch means the sze of testng gallery set s 649. The feature extracton and ntal rankng lsts were produced followng the way on VIPeR data set. And the ntal results were re-ranked usng combned smlarty. The CMC curves of baselnes (green lne) and our bdrectonal re-rankng usng combnaton smlarty (red lne) are presented n Fg.. They also prove the effcences of our method. In Table 4, we descrbe the CMC value n the range of the frst ranks comparng our technque based on combnaton smlarty wth state-of-the-art. Our technque exceeds baselnes over the range of ranks entrely, and there are more than 5 % mprovement at most of ranks. We also llustratng some specfc re-dentfcaton results n Fg. 2. Table 3 Matchng rates (%) at dfferent ranks on VIPeR data set Methods rank@ 25 5 Mahalanobs Mahalanobs+re-rankng Gan 4.3 % 7.8 % 4.5 % 3.6 % LMNN [2] LMNN+re-rankng Gan 5.9 % 8.5 % 2.3 % 6.9 % KISSME [3] KISSME+re-rankng Gan %.3 % 7.4 % 3.3 %

16 Fg. Three examples of comparatve results between KISSME and our approach on the VIPeR data set. Leftcolumnare the probes and top gallery rank results are lsted rght. The hollow arrows denotes forward queryng, vs-a-vs, sold arrows denotes backward queryng. The correct matchng s hghlghted by the red box. As we see, the correct matchngs are pull nto -nearest neghborhood after our re-rankng process 4.4 3DPES data set 3DPES project contans numerous vdeo sequences taken from a real survellance setup, composed of 8 dfferent survellance cameras. However, the number of persons are less than VIPeR and PRID2 sngle-shot data sets. For obtanng the ntal rankng lst, par mages of 92 persons from two camera vews were selected from the orgnal data set,

17 Fg. Comparatve results between baselnes and our bdrectonal re-rankng usng combned smlarty on PRID2 sngle-shot data set cmc PRID sngle shot Mahal Re rankng usng combned smlarty (a) Comparson wth Mahalanobs cmc PRID sngle shot LMNN Re rankng usng combned smlarty (b) Comparson wth LMNN [9] cmc PRID sngle shot KISSME Re rankng usng combned smlarty (c) Comparson wth KISSME [7]

18 Table 4 Matchng rates (%) at dfferent ranks on PRID2 sngle-shot data set Methods rank@5 2 5 Mahalanobs Mahalanobs+re-rankng Gan 2.8 % 23.3 % 2. % 2 % 4.3 % LMNN [2] LMNN+re-rankng Gan 26.3 % 2.4 % 8.9 % 7.3 % 7.4 % KISSME [3] KISSME+re-rankng Gan 8.8 % 2 % 3.9 % 7.5 % 7.3 % Fg. 2 Three examples of comparatve results between Mahalanobs and our approach on the PRID2 sngle-shot data set. Left column are the probes and top gallery rank results are lsted rght. The hollow arrows denotes forward queryng, vs-a-vs, sold arrows denotes backward queryng. The correct matchng s hghlghted by the red box. As we see, the correct matchngs are pull nto -nearest neghborhood after our re-rankng process

19 Fg. 3 Comparatve results between baselnes and our bdrectonal re-rankng usng combned smlarty on 3DPES data set cmc 3DPES.2. Mahal Re rankng usng combned smlarty (a) Comparson wth Mahalanobs cmc 3DPES LMNN Re rankng usng combned smlarty (b) Comparson wth LMNN[9] cmc 3DPES KISSME Re rankng usng combned smlarty (c) Comparson wth KISSME[7]

20 Fg. 4 Comparatve results between baselne and our bdrectonal re-rankng usng combnaton smlarty on PRID2 multple-shot data set cmc PRID mult shot.2. Mahal Re rankng usng combned smlarty (a) Comparson wth Mahalanobs cmc PRID mult shot LMNN Re rankng usng combned smlarty (b) Comparson wth LMNN[9] cmc PRID mult shot KISSME Re rankng usng combned smlarty (c) Comparson wth KISSME[7]

21 cmc VIPeR KISSME[Kostnger 22] Re rankng usng [Shen 22]. Re rankng usng [Wu 2] Re rankng usng Our Method cmc PRID sngle shot KISSME[Kostnger 22] Re rankng usng [Shen 22]. Re rankng usng [Wu 2] Re rankng usng Our Method Fg. 5 Comparatve results between other re-rankng methods and our bdrectonal re-rankng on the VIPeR and PRID2 sngle-shot data set and the sze of probe and gallery are both 96. The feature extracton and ntal rankng lsts were produced followng the way on VIPeR data set, and the ntal results were reranked usng combned smlarty. The averaged CMC curves are shown n Fg. 3. As can be seen, the performances after our re-rankng are mproved yet mnor, whch s due to neffectve context smlarty. Specfcally, context smlarty s avalable based on the statstcal characterstc same person have much more common nearest neghbors than that of dfferent person. And the statstcal characterstc s more feasble on a bggsh data set, whle the szes of 3DPES and PRID2 multple-shot data sets are very small, such as only 96 persons can be used for test on 3DPES far less than that of 36 persons on VIPeR.

22 Table 5 Runtme comparatve results (s) of offlne and onlne parts on VIPeR and PRID2 sngle-shot data sets Data sets Offlne Onlne VIPeR 9.9±.49.43±.2 PRID2 sngle-shot.82±.59.6±. 4.5 PRID2 multple-shot data set Persons n the PRID2 multple-shot data sets are the same as the sngle-shot verson, but they are mages sequence nstead of one key frame of sngle-shot person. For producng the ntal rankng lst, 2 common mages n two vews were used, and half person sequences were randomly selected for tranng and the other for test. The feature extracton and ntal rankng lsts were produced followng the way on VIPeR data set. And Multple-shot re-rankng process followed the descrpton n Secton 2.5 based on combned smlarty. Fgure 4 shows the averaged CMC curves of baselne and our technque. Smlar to the result on 3DPES, the sze of PRID2 multple-shot data sets s too small to dsable the context smlarty, leadng that the performances after re-rankng are mproved yet mnor. 4.6 Comparson wth other re-rankng methods Moreover, our technque s also compared wth two representatve state-of-the-art rerankng methods [8, 2] from the general mage retreval on two publcally person re-dentfcaton data sets VIPeR and PRID2 sngle-shot, the ntal rankng lst s produced by KISSME [3]. As can be seen from Fg. 5, the performances of baselnes are even worse than ntal result on both two data sets. The reason of the contrast of above expermental results s that these two re-rankng methods are effectve based on the assumpton the majorty of these k-nearest neghbors contan the same object as n the query mage (see paragraph of page 5 on reference [8]). However there s only SINGLE correct person n the gallery for person re-dentfcaton task, e.g. VIPeR [9], PRID2 [] and 3DPES [2], leadng above assumpton s not reasonable for person re-dentfcaton task. Whle n contrast, our method does not rely on the rgorous assumpton and hence can obvously mprove the orgnal result. 4.7 Runtme Ths subsecton tested the runtme of offlne and onlne processes. The expermental envronment was as follows: the CPU of our computer ncluded double core 2.8 GHz CPU and 2 GB RAM; mutl-features of person representaton contaned RGB, HSV and LBP, dstance measure was based on KISSME [3], and t was tested on the VIPeR and PRID2 sngleshot data sets. The sze of VIPeR was 36 n both the probe and the gallery, the probe of PRID had person mages and ts gallery had 649 person mages; at last, the runtme was a mean value by teratons. Table 5 descrbed that onlne part had less tme-consumng than that of offlne part, and sutable for real-tme-requrng vdeo survellance applcatons. 5Concluson In ths work, we present a bdrectonal person re-dentfcaton technque usng content and context re-rankng. The core dea s: f two mages come from the same person, should

23 possess smlar vsual content as well as neghbors compared to those mages from dfferent persons. Specfcally, probe s frstly quered n orgnal gallery after feature extracton and metrc learnng, and then conversely queryng every ntal gallery mage n a new data set composed of the orgnal probe mage and other lsted mages. Fnally, we use the content smlarty that s descrbed by the poston of the orgnal probe, and context smlarty whch s the number of common k-nearest neghborhood quered by the orgnal gallery mage to re-rank the ntal searched result. Moreover, both sngle-shot and multple-shot person are consdered n the re-rankng process. Extensve experments compared wth several typcal methods on three sngle-shot and one mult-shot publcly data sets have proved the effectveness of our proposed technque, even f dfferent cameras have obvous vewpont changes, pose changes, llumnaton changes, camera chromatc aberratons and low mage resolutons etc. In partcularly, our technque are more useful for large-sze data set whch s sutable for practcal vdeo nvestgaton applcatons, and the onlne part of our technque has a low computatonal complexty of dstance measure through offlne processng, where offlne can be carred out before start a query. Future work wll ntroduces more thoughts of socal nfluence, and further reduces the computatonal complexty through clusterng especally when there s a massve data set. e.g. clusterng the ntal rankng lst, and the clusterng centers are conversely quered for generatng reverse rankng lst. Then above results can be used to compute content and context smlartes refer to our bdrectonal rankng method. Moreover, n the practcal survellance applcaton, there exst massve occlusons among persons. In these condtons, more powerful person detecton and trackng algorthms are needed to obtan the clear person sequence contanng a sngle object, and those person mages that the overlap rates of foreground mask are less than a certan threshold or the smlartes wth ther contguous frames are more than a certan threshold can be used for our re-rankng. And above dscussons wll be studed n our future works. Acknowledgments Ths work was supported by the Natonal Nature Scence Foundaton of Chna (6235, 67273, 6334), the Major Scence and Technology Innovaton Plan of Hube Provnce (23AAA2), the Guangdong-Hongkong Key Doman Breakthrough Project of Chna (22A927), the Chna Postdoctoral Scence Foundaton funded project (23M5335), the Specalzed Research Fund for the Doctoral Program of Hgher Educaton (234224), the Key Technology R&D Program of Wuhan ( ) and the Presdent Fund of UCAS, and the Open Project Program of the Natonal Laboratory of Pattern Recognton (NLPR). References. Al S, Javed O, Haerng N, Kanade T (2) Interactve retreval of targets for wde area survellance. In: Proceedngs of the nternatonal conference on multmeda, pp Balter D, Vezzan R, Cucchara R (2) 3dpes: 3D people dataset for survellance and forenscs. In: Proceedngs of the st nternatonal ACM workshop on multmeda access to 3D human objects 3. Dkmen M, Akbas E, Huang TS, Ahuja N (2) Pedestran recognton wth a learned metrc.in: ACCV 4. Fschler MA, Bolles RC (98) Random sample consensus: a paradgm for model fttng wth applcatons to mage analyss and automated cartography. Commun ACM 24(6): Fredkn NE (998) A structural theory of socal nfluence. Cambrdge Unversty Press 6. Frey BJ, Dueck D (27) Clusterng by passng messages between data ponts. Scence 35(584): Farenzena M, Bazzan L, Perna A, Murno V, Crstan M (2) Person re-dentfcaton by symmetrydrven accumulaton of local features. In: CVPR, pp

24 8. Ghessar N, Sebastan TB, Tu PH, Rttscher J, Hartley R (26) Person re-dentfcaton usng spatotemporal appearance. In: CVPR, vol 2, pp Gray D, Brennan S, Tao H (27) Evaluatng appearance models for recognton, reacquston, and trackng. In: PETS. Gong SG, Loy CC, Xang T (2) Securty and survellance. In: VAH, part 4, pp Hrzer M, Belezna C, Roth PM, Bschof H (2) Person re-dentfcaton by descrptve and dscrmnatve classfcaton. In: SCIA, LNCS, vol 6688, pp Huang J, Yang X, Fang X, Ln W, Zhang R (2) Integratng vsual salency and consstency for rerankng mage search results. IEEE Trans Multmeda 3(4): Kostnger M, Hrzer M, Wohlhart P, Roth P, Bschof H (22) Large scale metrc learnng from equvalence constrants. In: CVPR, pp L W, Wu Y, Mukunok M et al. (22) Common-near-neghbor analyss for person re-dentfcaton. In: IEEE nternatonal conference on mage processng (ICIP), pp Leng Q, Hu R, Lang C, Wang Y, Chen J (23) Bdrectonal rankng for person re-dentfcaton In: Proceedngs of IEEE computer socety conference on multmeda and exposure (ICME) 6. Pedronette D, Gumares C, da S Torres R (2) Explotng contextual spaces for mage re-rankng and rank aggregaton. In: Proceedngs of the st ACM nternatonal conference on multmeda retreval 7. Ru Y, Huang TS, Ortega M, Mehrotra S (998) Relevance feedback: a power tool for nteractve contentbased mage retreval. IEEE Trans Crcuts Syst Vdeo Technol 8(5): Shen X, Ln Z, Brandt J, Avdan S, Wu Y (22) Object retreval and localzaton wth spatallyconstraned smlarty measure and k-nn re-rankng. In: IEEE conference on computer vson and pattern recognton (CVPR), pp Wang X, Doretto G, Sebastan T, Rttscher J, Tu P (27) Shape and appearance context modelng. In: ICCV, pp 8 2. Wenberger KQ, Saul LK (28) Fast solvers and effcent mplementatons for dstance metrc learnng. In: ICML 2. Wu Z, Ke Q, Sun J, Shum HY (2) Scalable face mage retreval wth dentty-based quantzatonand multreference rerankng. IEEE Trans Pattern Anal Machne Intell 33(): Xang ZJ, Chen Q, Lu Y (22) Person re-dentfcaton by fuzzy space color hstogram. Multmed Tools Appl. do:.7/s You H, Chang E, L B (2) NNEW: nearest neghbor expanson by weghtng n mage database retreval. In: Proceedngs of IEEE nternatonal conference multmeda and exposure, pp Zheng WS, Gong GS, Tao X (2) Person re-dentfcaton by probablstc relatve dstance comparson. In: CVPR, pp Zhu C, Wen F, Sun J (2) A rank-order dstance based clusterng algorthm for face taggng. In: IEEE conference on computer vson and pattern recognton (CVPR), pp Qngmng Leng receved the B.S degree n lfe scence from Nanchang Unversty, Nanchang, Chna, and n 27, M.S degree n Internatonal School of Software from Wuhan Unversty, Wuhan, Chna, n 29. He s currently pursung hs Ph.D. degree n Natonal Engneerng Research Center for Multmeda Software, School of Computer, Wuhan Unversty, Wuhan, Chna. Hs research nterests nclude computer vson, machne learnng and person re-dentfcaton.

25 Rumn Hu receved the B.S and M.S degrees from Nanjng. Unversty of Posts and Telecommuncatons, Nanjng Chna, n 984 and n 99 respectvely. and Ph.D degree n Communcaton and Electronc System from Huazhong Unversty of Scence and Technology, Wuhan, Chna n 994. Dr. Hu s the drector of Natonal Engneerng Research Center For Multmeda Software, Wuhan Unversty and Key Laboratory of Multmeda Network Communcaton Engneerng n Hube provnce. He s Executve Charman of the Audo Vdeo codng Standard (AVS) workgroup of Chna n Audo Secton. He has publshed two books and over scentfc papers. Hs research nterests nclude audo/vdeo codng and decodng, vdeo survellance and multmeda data processng. Chao Lang recevedthe B.S degree n Automaton from Huazhong Unversty of Scence and Technology, Wuhan, Chna, n 26 and the Ph.D degree n Pattern Recognton and Intellgent System from Insttute of Automaton, Chnese Academy of Scences, Bejng, Chna, n 22. He s currently workng as a postdoc teacher at Natonal Engneerng Research Center for Multmeda Software, Wuhan, Chna. Hs research nterests nclude multmeda content analyss, machne learnng, computer vson and pattern recognton.

26 Ymn Wang receved the B.S degree from School of Computer of Wuhan Unversty, Wuhan, Chna, n 28. He s currently pursung hs Ph.D. degree n Natonal Engneerng Research Center for Multmeda Software, School of Computer, Wuhan Unversty, Wuhan, Chna. Hs research nterests nclude computer vson, and machne learnng. Jun Chen receved M.S degree n Instrumentaton from Huazhong Unversty of Scence & Technology, Wuhan, Chna, n 997, and Ph.D degree n photogrammetry and remote sensng from Wuhan Unversty, Wuhan, Chna, n 27. Dr. Chen s the deputy drector of Natonal Engneerng Research Center for Multmeda Software. Hs research nterests nclude multmeda communcatons and securty emergency nformaton processng.