Person Re-identification by Salience Matching

Transcription

1 Person Re-dentfcaton by Salence Matchng Ru Zhao Wanl Ouyang Xaogang Wang Department of Electronc Engneerng, the Chnese Unversty of Hong Kong {rzhao, wlouyang, Abstract Human salence s dstnctve and relable nformaton n matchng pedestrans across dsjont camera vews. In ths paper, we explot the parwse salence dstrbuton relatonshp between pedestran mages, and solve the person re-dentfcaton problem by proposng a salence matchng strategy. To handle the msalgnment problem n pedestran mages, patch matchng s adopted and patch salence s estmated. Matchng patches wth nconsstent salence brngs penalty. Images of the same person are recognzed by mnmzng the salence matchng cost. Furthermore, our salence matchng s tghtly ntegrated wth patch matchng n a unfed structural RankSVM learnng framework. The effectveness of our approach s valdated on the VIPeR dataset and the CUHK Campus dataset. It outperforms the state-of-the-art methods on both datasets. 1. Introducton Person re-dentfcaton s a task of matchng persons observed from non-overlappng camera vews based on mage appearance. It has mportant applcatons n vdeo survellance ncludng threat detecton, human retreval, human trackng, and actvty analyss. It saves a lot of human efforts on exhaustvely searchng for a person from large amounts of vdeo sequences. Nevertheless, person re-dentfcaton s a very challengng task. A person observed n dfferent camera vews often undergoes sgnfcant varatons on vewponts, poses, appearance and llumnaton, whch make ntra-personal varatons even larger than nter-personal varatons. Background clutters and occlusons cause addtonal dffcultes. Our work s manly motvated by the followng several aspects. Msalgnments are caused by varatons of vewponts and poses, whch are commonly exst n person redentfcaton. For example n Fgure 1, the shoulder of (b1) close to the left boundary becomes a backpack at the same locaton n (b2). Most exstng methods [19, 23, 12, 4, 18] match pedestran mages by drectly comparng msalgned features. In our approach, salence matchng s ntegrated wth patch matchng, and both show robustness to spatal (p1) (p2) (p3) (p4) probe n camera A gallery n camera B query correct match ncorrect match (a1) (a2) (a3) (a4) (a5) (a6) (b1) (b2) (b3) (b4) (b5) (b6) Fgure 1. Illustraton of human salence and salence matchng wth examples. In the frst row, some salent parts of pedestrans are hghlghted wth yellow dashed regons. The second row and the thrd row show examples of salence matchng. The salence map of each pedestran mage s shown. Best vewed n color. varaton and msalgnment. Some local patches are more dstnctve and relable when matchng two persons. Some examples are shown n the frst row of Fgure 1, person (p1) carres a red hand bag, (p2) has an orange cap and a yellow horzontal strpe on hs jacket, (p3) wears a whte dress, and (p4) s dressed n red sweater wth floral texture. Human eyes can easly pck up these persons from other canddates because of these dstnctve features. These features can be relably detected across camera vews. If a body part s salent n one camera vew, t usually remans salent n another vew. However, most exstng approaches only consder clothes and trousers as the most mportant regons for person re-dentfcaton. Some dstnct features (such as the red bag n (p1)) may be consdered as outlers to be removed, because they do not belong to body parts. Also, these features may only take up small regons n body parts. If global features are adopted by exstng approaches, those small regons have lttle effect on person matchng. In contrast, our approach can 1

2 well estmate dstnctveness of patches as salence. Patches wth hgh salence values gan large weghts n person redentfcaton, because such patches not only have good dscrmnatve power but also can be relably detected durng patch matchng across camera vews. We observe that mages of the same person captured from dfferent camera vews have some nvarance property on ther spatal dstrbutons on salence, lke par (a1,a2) n Fgure 1. Snce the person n mage (a1) shows salence n her dress whle others n (a3)-(a6) have salent blouses. They can be well dstngushed smply from the spatal dstrbutons of salence. Therefore, human salence dstrbutons provde useful nformaton n person re-dentfcaton. Such nformaton can be encoded durng patch matchng. If two patches from two mages of the same person are matched, they are expected to have the same salence value; otherwse such matchng brngs salence matchng penalty. In the second row n Fgure 1, the query mage (b1) shows a smlar salence dstrbuton as those of gallery mages. In ths case, vsual smlarty needs to be consdered. Ths motvates us to relate salence matchng penalty to the vsual smlarty of two matched patches. Based on above consderatons, a new person redentfcaton approach by salence matchng s proposed. Ths work has three major contrbutons. Frst, a probablstc dstrbuton of salence s relably estmated wth our approach. Dfferent from general salence detecton [6], our salence s especally desgned for person re-dentfcaton. The estmated human salence s robust across dsjont camera vews and s used as a meanngful representaton of human appearance n recognton. Second, we formulate person re-dentfcaton as a salence matchng problem. Dense correspondences between local patches are establshed based on vsual smlarty. Matchng patches wth nconsstent salence brngs cost. Images of the same person are recognzed by mnmzng the salence matchng cost, whch not only depends on the locatons of patches but also the vsual smlarty of matched patches. Thrd, salence matchng and patch matchng are tghtly ntegrated nto a unfed structural RankSVM learnng framework. Structural RankSVM has good tranng effcency gven a very large number of rank constrants n person redentfcaton. Moreover, our approach has transformed the orgnal hgh-dmensonal vsual feature space to a much lower dmensonal salence feature space (80 tmes lower n ths work) to further mprove the tranng effcency and also avod overfttng. The effectveness of our approach s valdated on the VIPeR dataset [7] and the CUHK Campus dataset [12]. It outperforms the state-of-the-art methods on both datasets. 2. Related Works Exstng methods on person re-dentfcaton generally fall nto two categores: unsupervsed and supervsed. Our proposed approach s supervsed. Unsupervsed Methods. Ths category manly focuses on feature desgn. Farenzena et al. [5] proposed the Symmetry-Drven Accumulaton of Local Features (SDALF) by explotng the symmetry property n pedestran mages to handle vew varaton. Ma et al. [16] developed the BCov descrptor based on the Gabor flters and the covarance descrptor to handle llumnaton change and background varatons. Cheng et al.[3] utlzed the Pctoral Structures to estmate human body confguraton and also computed vsual features based on dfferent body parts to cope wth pose varatons. Lu et al. [15] employed assembly of part templates to handle the artculaton of human body. Zhao et al. [22] proposed an unsupervsed salence learnng method to explot dscrmnatve features, but they dd not consder salence tself as an mportant feature for patch matchng and person re-dentfcaton. Supervsed Methods. Dstance metrc learnng has been wdely used n person re-dentfcaton [23, 4, 12, 13, 18, 24]. They learn metrcs by mnmzng the ntra-class dstances whle maxmzng the nter-class dstances. Ther performance s lmted by the fact that metrc s based on the subtracton of msalgned feature vectors, whch causes sgnfcant nformaton loss and errors. L and Wang [11] learned a mxture of cross-vew transforms and projected features nto a common space for algnment. In contrast, our approach handles the problem of feature msalgnment through patch matchng. Lu et al. [14] allowed user feedback n the learnng procedure, whch acheved sgnfcant mprovement over metrc learnng methods. Some other models have also been employed to extract dscrmnatve features. Gray et al. [8] used boostng to select a subset of optmal features for matchng pedestran mages. Prosser et al. [19] formulated person re-dentfcaton as a rankng problem, and learned global feature weghts based on an ensemble of RankSVM. RankSVM optmzes over the parwse dfferences. In ths paper, we employ structural RankSVM [10], whch consders the rankng dfference rather than parwse dfference. General mage salence has been extensvely studed [6]. In the context of person re-dentfcaton, human salence s dfferent than general mage salence n the way of drawng vsual attentons. 3. Human Salence We compute the salence probablty map based on dense correspondence wth a K-nearest neghbors (KNN) method.

3 Algorthm 1 Compute human salence. Input: mage x A,u and a reference mage set R = {x B,v,v=1,...,N r } Output: salence probablty map P (lm,n A,u =1 x A,u m,n) 1: for each patch x A,u m,n X do 2: compute X NN (x A,u m,n) wth Eq. (1) 3: compute score(x A,u m,n) wth Eq. (2) 4: compute P (lm,n A,u =1 x A,u 5: end for 3.1. Dense Correspondence m,n) wth Eq. (3) Dense Features. Before buldng dense correspondence, local patches on a dense grd are extracted. The patch sze s and the grd step s 5 pxels. 32-bn color hstogram n each of LAB channels and 128-dmensonal SIFT features are then computed for each patch. To robustly capture the color nformaton, color hstograms are also computed on two other downsampled scales for each patch. The color hstograms and SIFT features are normalzed wth L2 norm, and are concatenated to form the fnal dense local features,.e. a 672-dmensonal ( ) feature vector for each local patch. Adjacency Constraned Search. Dense local features for an mage are denoted by x A,u = {x A,u m,n}, and x A,u m,n represents the feature of a local patch at the m-th row and n-th column n the u-th mage from camera vew A. When patch x A,u m,n searches for ts correspondng patch n the v- th mage from camera vew B,.e. x B,v = {x B,v,j }, the search set of x A,u m,n n x B,v s S(x A,u m,n, x B,v )={x B,v,j j = 1,...,N,=max(0,m l),...,mn(m,m + l)}, where l denotes the halft heght of adjacency search space, M s the number of rows, and N s the number of columns. If all pedestran mages are well algned and there s no vertcal pose varaton, l shall be set zero. However, msalgnment, camera vew change, and vertcal artculaton result n vertcal movement of the human body n the mage. Thus the relaxed adjacency search s necessary to handle spatal varatons. Smaller search space cannot tolerate large spatal varaton, whle larger search space wll ncreases the chance of msmatch. We choose l =2n our experment settng. Patch matchng s wdely used, and many off-theshelf methods [1] are avalable. We smply do a k- nearest-neghbor search for patch x A,u m,n n ts search set S(x A,u m,n, x B,v ). For each patch x A,u m,n, a nearest neghbor s sought from ts search set n every mage wthn a reference set. The adjacency constraned search s llustrated n Fgure 2. Fgure 2. Illustraton of adjacency constraned search. Green regon represents the adjacency constraned search set of the patch n yellow box. The patch n red box s the target match Unsupervsed Salence Learnng Human salence s computed based on prevously-bult dense correspondence. We utlze the KNN dstances to fnd patch samples n mnorty,.e. they are unque and specal. In the applcaton of person re-dentfcaton, we fnd salent patches that possess property of unqueness among a reference set R. Denote the number of mages n the reference set by N r. For an mage x A,u = {x A,u m,n}, a nearestneghbor (NN) set of sze N r s bult for every patch x A,u m,n, X NN (x A,u m,n) ={x argmn d(x A,u x B,v,j x B,v,j m,n,x B,v,j ), (1) S(x A,u m,n, x B,v ),v =1,...,N r }, where S(x A,u m,n, x B,v ) s the adjacency search set of patch x A,u m,n, and functon d( ) computes the Eucldean dstance between two patch features. Our goal of computng human salence s to dentfy patches wth specal appearance. We use the KNN dstances to defne the salence score: score(x A,u m,n) =d k (X NN (x A,u m,n)), (2) and the probablty of x A,u m,n beng a salent patch s P (l A,u m,n =1 x A,u m,n) =1 exp( score(x A,u m,n) 2 /σ 2 0), (3) where d k denotes the dstance of the k-th nearest neghbor, lm,n A,u s a bnary salence label and σ 0 s a bandwdth parameter. We set k = N r /2 n the salence learnng scheme wth an emprcal assumpton that a patch s consdered to have specal appearance such that more than half of the people n the reference set do not share smlar patch wth t. N r reference mages are randomly sampled from tranng set, and we set N r = 100 n our experments. Enlargng the reference dataset wll not deterorate salence detecton, because the salence s defned n the statstcal sense. It s robust as long as the dstrbuton of the reference dataset well reflects the test scenaro. Our human salence learnng method s summarzed n algorthm Supervsed Salence Matchng One of the man contrbutons of ths work s to match pedestran mages based on the salence probablty map. In

4 contrast wth most of the works on person re-dentfcaton, whch focus on feature selecton, feature weghtng, or dstance metrc learnng, we nstead explot the consstence property of human salence and ncorporate t n person matchng. Ths s based on our observaton that person n dfferent camera vews shows consstence n the salence probablty map, as shown n Fgure Matchng based on Salence Snce matchng s appled for arbtrary mage pars, we omt the mage ndex n notaton for clarty,.e. change x A,u to x A and x B,v to x B. Also, the patch notatons are changed accordngly,.e. x A,u m,n to x A p and x B,v,j to x B p, where p s the patch ndex n mage x A and p s the correspondng matched patch ndex n mage x B produced by prevously bult dense correspondence. To ncorporate the salence nto matchng, we ntroduce l A = {lp A lp A {0, 1}} and l B = {lp B l B p {0, 1}} as salence labels for all the patches n mage x A and x B respectvely. If all the salence labels are known, we can perform person matchng by computng salence matchng score as follows: f z (x A, x B, l A, l B ; p, z) = (4) { z p,1lp A lp B + z p,2lp A (1 lp B ) p + z p,3(1 l A p )l B p + z p,4(1 l A p )(1 l B p ) }, where p = {(p,p )} are dense correspondence patch ndex pars, and z = {z p,k} k=1,2,3,4 are the matchng scores for four dfferent salence matchng results at one local patch. z p,k s not a constant for all the patches. Instead, t depends on the spatal locaton p. For example, the score of matchng patches on the background should be dfferent than those on legs. z p,k also depends on the vsual smlarty between patch x A p and patch x B p, s(x A p,x B p )=exp ( d(xa p,x B 2 ), (5) 2σ0 2 where σ 0 s bandwdth of the Gaussan functon. Instead of drectly usng the Eucldean dstance d(x A p,x B, we convert t to smlarty to reduce the sde effect n summaton of very large dstances n ncorrect matchng, whch may be caused by msalgnment, occluson, or background clutters. Therefore, we defne the matchng score z p,k as a lnear functon of the smlarty as follows, z p,k = α p,k s(x A p,x B p )+β p,k, (6) where α p,k and β p,k are weghtng parameters. Thus Eq.(4) jontly consders salence matchng and vsual smlarty. Snce the salence label lp A and lp B n Eq.(4) are hdden varables, they can be margnalzed by computng the expectaton of the salence matchng score as f (x A, x B ; p, z) = l A,l B f z(x A, x B, l A, l B ; p, z)p(l A, l B x A, x B ) = p 4 k=1 [ ] α p,k s(x A p,x B +β p,k c p,k(x A p,x B, (7) where c p,k(x A p,x B p s the probablstc salence matchng ) cost dependng on salence probabltes P (lp A =1 x A p ) and P (lp B =1 x B p gven n Eq.(3), ) c p,k(x A p,x B (8) P (lp A =1 x A p )P (l B p =1 xb, k =1, P (lp A = =1 x A p )P (l B p =0 xb, k =2, P (lp A =0 x A p )P (l B p =1 xb, k =3, P (lp A =0 x A p )P (l B p =0 xb, k =4. To better formulate ths learnng problem, we extract out all the weghtng parameters n Eq.(7) asw, and have where f (x A, x B ; p, z) =w T Φ(x A, x B ; p) (9) = wp T φ(x A p,x B p ), p Φ(x A, x B ; p) =[φ(x A p 1,x B p 1 )T,...,φ(x A p N,x B p N )T ] T, (10) w =[w p1,...,w pn ] T, w p =[{α p,k} k=1,2,3,4, {β p,k} k=1,2,3,4 ]. Φ(x A, x B ; p) s the feature map descrbng the matchng between x A and x B. For each patch p, the matchng feature φ(x A p,x B s an eght dmensonal vector: φ(x A p,x B = (11) s(x A p,x B P (la p =1 x A p )P (l B p =1 xb s(x A p,x B P (la p =1 x A p )P (l B p =0 xb s(x A p,x B P (la p =0 x A p )P (l B p =1 xb s(x A p,x B P (la p =0 x A p )P (l B p =0 xb P (lp A =1 x A p )P (l B p =1 xb. P (lp A =1 x A p )P (l B p =0 xb P (lp A =0 x A p )P (l B p =1 xb P (lp A =0 x A p )P (l B p =0 xb As shown n Eq.(11), the parwse feature map Φ(x A, x B ; p) combnes the salence probablty map wth appearance matchng smlartes. For each query mage x A, the mages n the gallery are ranked accordng to the expectatons of salence matchng scores n Eq.(7). There

5 are three advantages of matchng wth human salence : (1) the human salence probablty dstrbuton s more nvarant than other features n dfferent camera vews; (2) because the salence probablty map s bult based on dense correspondence, so t nherts the property of toleratng spatal varaton; and (3) t can be weghted by vsual smlarty to mprove the performance of person re-dentfcaton. We wll present the detals n next secton by formulatng the person re-dentfcaton problem wth Φ(x A, x B ; p) n structural RankSVM framework, and the effectveness of salence matchng wll be shown n expermental results Rankng by Partal Order We cast person re-dentfcaton as a rankng problem for tranng. The rankng problem wll be solved by fndng an optmal partal order, whch wll be mathematcally defned n Eq.(12)(13)(16). Gven a dataset of pedestran mages, D A = {x A,u,d A,u } U u=1 from camera vew A and D B = {x B,v,d B,v } V v=1 from camera vew B, where x A,u s the u-th mage, y u s ts dentty label, and U s the total number of mages n D A. Smlar notatons apply for varables of camera vew B. Each mage x A,u has ts relevant mages (same dentty) and rrelevant mages (dfferent denttes) n dataset D B. Our goal s to learn the weght parameters w that order relevant gallery mages before rrelevant ones. For the mage x A,u, the orders n ts groundtruth rankng are not all known,.e., we rank the relevant mages before rrelevant ones, but no nformaton of the orders wthn relevant mages or rrelevant ones s provded n groundtruth. The partal order y A,u s denoted as, y A,u = {y A,u v,v }, y A,u v,v = { +1 x B,v x B,v, 1 x B,v x B,v, (12) where x B,v x B,v (x B,v x B,v ) represents that x B,v s ranked before (after) x B,v n partal order y A,u. The partal order feature [9, 17] s approprate for our goal and can well encode the dfference between relevant pars and rrelevant pars wth only partal orders. The partal order feature for mage x A,u s formulated as, Ψ po(x A,u, y A,u ; {x B,v } V v=1, {p u,v } V v=1) = y A,u Φ(x A,u, x B,v ; p u,v ) Φ(x A,u, x B,v ; p u,v ) v,v S + S, x A,u x A,u x B,v S + x A,u x B,v S x A,u (13) S + = {x B,v d B,v = d A,u }, x A,u (14) S = {x B,v d B,v d A,u }, x A,u (15) where {p u,v } V v=1 are the dense correspondences between mage x A,u and every gallery mage x B,v, S + x A,u s relevant mage set of x A,u, S s rrelevant mage set, Φ(x A,u, x B,v ; p u,v ) s the feature map defned x A,u n Eq.(10), and the dfference vector of two feature maps Φ(x A,u, x B,v ; p u,v ) Φ(x A,u, x B,v ; p u,v ) s added f x B,v x B,v and subtracted otherwse. A partal order may correspond to multple rankngs. Our task s to fnd a good rankng satsfyng the optmal partal order y A,u that maxmzes followng score functon, y A,u = argmax w T Ψ po(x A,u, y A,u ; {x B,v } V v=1, {p u,v } V v=1), y A,u Y A,u (16) where Y A,u s space consstng of all possble partal orders. As dscussed n [9, 21], the good rankng can be obtaned smply by sortng gallery mages by {w T Φ(x A,u, x B,v ; p u,v )} v n descendng order. The remanng problem s how to learn w Structural RankSVM In ths work, we employ structural SVM to learn the weghtng parameters w. Dfferent than many prevous SVM-based approaches optmzng over the parwse dfferences (e.g., [2, 19]), structural SVM optmzes over rankng dfferences and t can ncorporate non-lnear multvarate loss functons drectly nto global optmzaton n SVM tranng. Objectve functon. Our goal s to learn a lnear model and the tranng s based on n-slack structural SVM [10]. The objectve functon s as follows, mn w,ξ 1 2 w 2 + C U ξ u, (17) u=1 s.t. w T δψ po(x A,u, y A,u, ŷ A,u ; {x B,v } V v=1, {p u,v } V v=1) Δ(y A,u, ŷ A,u ) ξ u, ŷ A,u Y A,u y A,u,ξ u 0, foru=1,...,u, where δψ po s defned as δψ po(x A,u, y A,u, ŷ A,u ; {x B,v } V v=1, {p u,v } V v=1) =Ψ po(x A,u, y A,u ; {x B,v } V v=1, {p u,v } V v=1) Ψ po(x A,u, ŷ A,u ; {x B,v } V v=1, {p u,v } V v=1), (18) w s the weght vector, C s a parameter to balance between the margn and the tranng error, y A,u s a correct partal order that ranks all correct matches before ncorrect matches, and ŷ A,u s an ncorrect partal order that volates some of the parwse relatons, e.g. a correct match s ranked after an ncorrect match n ŷ A,u. The constrants n Eq. (17) force the dscrmnant score of correct partal order y A,u to be larger than that of ncorrect one ŷ A,u by a margn, whch s determned by a loss functon Δ(y A,u, ŷ A,u ) and a slack varable ξ u. AUC loss functon. Many loss functons can be appled n structural SVM. In the applcaton of person redentfcaton, we choose the ROC Area loss, whch s also

6 Fgure 3. We normalze the learnt weght vector w to a 2-dmensonal mportance map for dfferent spatal locaton. Eght mportance maps correspond to {α p,k} k=1,2,3,4 and {β p,k} k=1,2,3,4 n Eq. (7). known as Area Under Curve (AUC) loss. It s computed from the number of swapped pars, N swap = {(v, v ):x B,v x B,v and (19) w T Φ(x A,u, x B,v ; p u,v ) < w T Φ(x A,u, x B,v ; p u,v )},.e. the number of pars of samples that are not ranked n correct order. In the case of partal order rankng, the loss functon s Δ(y A,u, ŷ A,u )= N swap / S + S, (20) x A,u x A,u = (1 ŷ A,u v,v )/(2 S + S ). x A,u x A,u v,v We note that there are an exponental number of constrants n Eq.(17) due to the huge dmensonalty of Y A,u. [10] shows that the problem can be effcently solved by a cuttng plane algorthm. In our problem, the dscrmnatve model s learned by structural RankSVM algorthm, and the weght vector w n our model means how mportant t s for each term n Eq.(11). In Eq.(11), {α p,k} k=1,2,3,4 correspond to the frst four terms based on salence matchng wth vsual smlarty, and {β p,k} k=3,4 correspond to the last four terms only dependng on salence matchng. We vsualze the learnng result of w n Fgure 3, and fnd that the frst four terms n Eq.(11) are heavly weghted n the central part of human body whch mples the mportance of salence matchng based on vsual smlarty. {β p,k} k=1,2 are not relevant to vsual smlarty and they correspond to the two cases when lp A =1,.e. the patches on the query mages are salent. It s observed that ther weghtng maps are hghlghted on the upper body, whch matches to our observaton that salent patches usually appear on the upper body. {β p,k} k=3,4 are not relevant to vsual smlarty ether, but they correspond to the cases when lp A =0,.e. the patches on the query mages are not salent. We fnd that ther weghts are very low on the whole maps. It means that non-salent patches on query mages have lttle effect on person re-dentfcaton f the contrbuton of vsual smlarty s not consdered. 5. Expermental Results We evaluate our approach on two publc datasets,.e. the VIPeR dataset [7], and the CUHK Campus dataset [12]. The VIPeR dataset s the mostly used person redentfcaton dataset for evaluaton, and the recently publshed CUHK Campus dataset contans more mages than VIPeR (3884 vs specfcally). Both are very challengng datasets for person re-dentfcaton because they contan sgnfcant varatons on vewponts, poses, and llumnatons, and ther mages are n low resolutons, wth occlusons and background clutters. All the quanttatve results are reported n standard Cumulated Matchng Characterstcs (CMC) curves [20]. Evaluaton Protocol. Our experments on both datasets follow the evaluaton protocol n [8],.e. we randomly partton the dataset nto two even parts, 50% for tranng and 50% for testng, wthout overlap on person denttes. Images from camera A are used as probe and those from camera B as gallery. Each probe mage s matched wth every mage n gallery, and the rank of correct match s obtaned. Rank-k recognton rate s the expectaton of correct match at rank k, and the cumulated values of recognton rate at all ranks s recorded as one-tral CMC result. 10 trals of evaluaton are conducted to acheve stable statstcs, and the expectaton s reported. We denote our salence matchng approach by SalMatch. To valdate the usefulness of salence matchng, we repeat all the tranng and testng evaluaton on our approach, but wthout usng salence. Ths control experment s denoted by PatMatch. VIPeR Dataset [7]. The VIPeR dataset 1 contans mages from two cameras n outdoor academc envronment. It contans 632 pedestran pars, and each par contans two mages of the same person observed from dfferent camera vews. Most of the mage pars show vewpont change larger than 90 degrees. All mages are normalzed to for experments. On VIPeR dataset, comparng PatMatch and SalMatch wth several exstng unsupervsed methods,.e. SDALF [5], CPS [3], ebcov [16] and esdc [22], expermental results show sgnfcant mprovements n Fgure 5 (a). We also compare our approaches wth sx alternatve supervsed learnng methods, ncludng four benchmarkng dstance metrc learnng methods,.e. PRDC [23], LMNN- R[4], PCCA [18], and attrbute-based PRDC (aprdc) [13], a boostng approach (ELF) [8] and Rank SVM (RankSVM) [19]. As seen from the the comparson results n Fgure 5 (a), our approach SalMatch acheves 30.16% at rank one wth standard devaton 1.23%, and outperforms all these methods. The control experment PatMatch acheves 26.90%, whch shows the effectveness of ntegratng salence matchng nto patch matchng. For dstance metrc learnng methods, they gnore the doman knowledge of person re-dentfcaton that pedestran mages suf- 1 The VIPeR dataset s avalable to download at: soe.ucsc.edu/?q=node/178

7 (a) VIPeR dataset (b) CUHK Campus dataset Fgure 4. Some nterestng examples of salence matchng n our experments. Ths fgure shows four categores of salence probablty types: salence n upper body (n blue dashed box), salence of takng bags (n green dashed box), salence of lower body (n orange dashed box), and salence of strpes on human body (n red dashed box). Best vewed n color Matchng Rate (%) % SDALF 21.84% CPS % ebcov 12.00% ELF % PRSVM 15.66% PRDC 20.00% LMNN R % aprdc 19.27% PCCA % esdc 26.90% PatMatch 30.16% SalMatch Rank (a) VIPeR dataset Matchng Rate (%) % L2 norm 10.33% L1 norm 9.90% SDALF 13.45% LMNN 15.98% ITML 20.39% PatMatch 28.45% SalMatch Rank (b) CUHK Campus dataset Fgure 5. CMC statstcs on the VIPeR dataset and the CUHK Campus dataset. (a) On VIPeR dataset, our approach (PatMatch and SalMatch) s compared wth benchmarkng methods ncludng SDALF [5], CPS [3], ebcov [16], esdc [22], ELF [8], PRSVM [19], PRDC [23], LMNN-R [4], aprdc [13], and PCCA [18]; (b) On CUHK Campus dataset, our approach s compared wth L1-norm dstance, L2-norm dstance, SDALF, LMNN [12], and ITML [12]. All the rank-1 performances are marked n the front of method names. Unsupervsed methods are drawn n dashed lnes whle supervsed method n sold lnes. fer spatal varaton caused by msalgnment and pose varaton, as dscussed n Secton 2. Among these metrc learnng approaches, although the aprdc also tres to fnd the unque and nherent appearance property of pedestran mages, our approach s almost more than doubled on rank-1 accuracy aganst aprdc. aprdc weghts global features nstead local patches based on ther dstnctveness. It dd not consder the consstency of salence dstrbuton as a cue or matchng pedestran mages. ELF gans a lower performance snce t selects features n orgnal feature space n whch features of dfferent denttes are hghly correlated. The RankSVM also formulate person re-dentfcaton as a rankng problem, but ours shows much better performance because t adopts dscrmnatve salence matchng strategy for parwse matchng, and the structural SVM ncorporates rankng loss n global optmzaton. Ths mples the mpor-

8 tance of explotng human salence matchng and the effectveness of structural SVM tranng. CUHK Campus Dataset [12]. The CUHK Campus dataset 2 s also captured wth two camera vews n a campus envronment. Dfferent than the VIPeR dataset, mages n ths dataset are of hgher resoluton and are more sutable to show the effectveness of salence matchng. The CUHK Campus dataset contans 971 persons, and each person has two mages n each camera vew. Camera A captures the frontal vew or back vew of pedestrans, whle camera B captures the sde vews. All the mages are normalzed to for evaluatons. Snce no unsupervsed methods are publshed on the CUHK Campus dataset, so we compare wth L 1 -norm and L 2 -norm dstances of our dense features ntroduced n Secton 3.1. Features of all local patches are drectly concatenated regardless of spatal msalgnment problem (therefore, patch matchng s not used), and the parwse dstance s smply computed by L 1 -norm and L 2 -norm. Also, we compare wth the result of a benchmarkng unsupervsed method,.e. SDALF [5], whch s obtaned by runnng the orgnal mplementaton 3 on the CUHK Campus dataset. As shown n Fgure 5 (b), our approach greatly outperforms these unsupervsed methods. Our approach s also compared wth avalable results of dstance learnng methods ncludng LMNN [12], and ITML [12]. On the CUHK Campus dataset, SalMatch obtans a matchng rate of 28.45% at rank one wth standard devaton 1.02% whle PatMatch acheves 20.39%. Apparently, our salence matchng approach outperforms the others methods, and smlar conclusons as n the VIPeR dataset can be drawn from the comparsons. 6. Concluson In ths paper, we formulate person re-dentfcaton as a salence matchng problem. The dense correspondences of local patches are establshed by patch matchng. Salence probablty maps of pedestran mages are relably estmated to fnd the dstnctve local patches. Matchng patches wth nconsstent salence brngs penalty. Images of the same person are recognzed by mnmzng the salence matchng cost. We tghtly ntegrate patch matchng and salence matchng n the partal order feature and feed them nto a unfed structural RankSVM learnng framework. Expermental results show our salence matchng approach greatly mproved the performance of person redentfcaton. 2 The CUHK Campus s avalable to download at: cuhk.edu.hk/ xgwang/cuhk_dentfcaton.html 3 The mplementaton of SDALF method s provded by authors at the webste: sdalf-descrptor/ 7. Acknowledgement Ths work s supported by the General Research Fund sponsored by the Research Grants Councl of Hong Kong (Project No. CUHK , CUHK , CUHK ) and Natonal Natural Scence Foundaton of Chna (Project No ). References [1] C. Barnes, E. Shechtman, D. B. Goldman, and A. Fnkelsten. The generalzed patchmatch correspondence algorthm. In ECCV [2] B. Carterette and D. Petkova. Learnng a rankng from parwse preferences. In ACM SIGIR, [3] D. S. Cheng, M. Crstan, M. Stoppa, L. Bazzan, and V. Murno. Custom pctoral structures for re-dentfcaton. In BMVC, , 6, 7 [4] M. Dkmen, E. Akbas, T. S. Huang, and N. Ahuja. Pedestran recognton wth a learned metrc. In ACCV , 2, 6, 7 [5] M. Farenzena, L. Bazzan, A. Perna, V. Murno, and M. Crstan. Person re-dentfcaton by symmetry-drven accumulaton of local features. In CVPR, , 6, 7, 8 [6] S. Goferman, L. Zelnk-Manor, and A. Tal. Context-aware salency detecton. PAMI, [7] D. Gray, S. Brennan, and H. Tao. Evaluatng appearance models for recognton, reacquston, and trackng. In PETS, , 6 [8] D. Gray and H. Tao. Vewpont nvarant pedestran recognton wth an ensemble of localzed features. In ECCV , 6, 7 [9] T. Joachms. A support vector method for multvarate performance measures. In ICML, [10] T. Joachms, T. Fnley, and C.-N. J. Yu. Cuttng-plane tranng of structural svms. Machne Learnng, , 5, 6 [11] W. L and X. Wang. Locally algned feature transforms across vews. In CVPR, [12] W. L, R. Zhao, and X. Wang. Human redentfcaton wth transferred metrc learnng. In ACCV, , 2, 6, 7, 8 [13] C. Lu, S. Gong, C. C. Loy, and X. Ln. Person re-dentfcaton: what features are mportant? In ECCV, , 6, 7 [14] C. Lu, C. C. Loy, S. Gong, and G. Wang. Pop: Person redentfcaton post-rank optmsaton. In ICCV, [15] Y. Lu, L. Ln, and W.-S. Zheng. Human re-dentfcaton by matchng compostonal template wth cluster samplng. In ICCV, [16] B. Ma, Y. Su, and F. Jure. Bcov: a novel mage representaton for person re-dentfcaton and face verfcaton , 6, 7 [17] B. McFee and G. Lanckret. Metrc learnng to rank [18] A. Mgnon and F. Jure. Pcca: A new approach for dstance learnng from sparse parwse constrants. In CVPR, , 2, 6, 7 [19] B. Prosser, W.-S. Zheng, S. Gong, T. Xang, and Q. Mary. Person re-dentfcaton by support vector rankng. In BMVC, , 2, 5, 6, 7 [20] X. Wang, G. Doretto, T. Sebastan, J. Rttscher, and P. Tu. Shape and appearance context modelng. In ICCV, [21] Y. Yue, T. Fnley, F. Radlnsk, and T. Joachms. A support vector method for optmzng average precson. In ACM SIGIR, [22] R. Zhao, W. Ouyang, and X. Wang. Unsupervsed salence learnng for person re-dentfcaton. In CVPR, , 6, 7 [23] W.-S. Zheng, S. Gong, and T. Xang. Person re-dentfcaton by probablstc relatve dstance comparson. In CVPR, , 2, 6, 7 [24] W.-S. Zheng, S. Gong, and T. Xang. Re-dentfcaton by relatve dstance comparson. In PAMI,