Using Spatial Pyramids with Compacted VLAT for Image Categorization

Transcription

1 Usng Spatal Pyramds wth Compacted VLAT for Image Categorzaton Roman Negrel, Davd Pcard, Phlppe-Henr Gosseln To cte ths verson: Roman Negrel, Davd Pcard, Phlppe-Henr Gosseln. Usng Spatal Pyramds wth Compacted VLAT for Image Categorzaton. Internatonal Conference on Pattern Recognton, Nov 2012, Tsukuba, Japan <hal > HAL Id: hal Submtted on 17 Nov 2012 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.

2 Usng Spatal Pyramds wth Compacted VLAT for Image Categorzaton Roman Negrel, Davd Pcard and Phlppe-Henr Gosseln ETIS/ENSEA - Unversty of Cergy-Pontose - CNRS, UMR , avenue du Ponceau, BP44, F95014 Cergy-Pontose, France {roman.negrel,pcard,gosseln}@ensea.fr Abstract In ths paper, we propose a compact mage sgnature based on VLAT. Our method ntegrates spatal nformaton whle sgnfcantly reducng the sze of orgnal VLAT by usng two pojecton steps. we carry out experments showng our approach s compettve wth state of the art sgnatures. 1. Introducton Content Based Image Categorzaton s a research feld that has known a rapd development n the last years. The typcal scheme s composed of the followng steps: extracton of local mage descrptors, aggregaton of the descrptors n an mage sgnature, and learnng a classfer n sgnatures space. The most recent and successful methods are Codng/poolng [9], Fsher Vector [6], VLAD [4], and VLAT [7]. In ths paper we present a new method for aggregatng local descrptors based on VLAT [7]. Our method mproves sgnfcantly the classfcaton results over of VLAT, whle reducng ts sze. Ths paper s organzed as follows: the next secton descrbes recent related work on mage representaton and smlartes. Secton 3 ntroduces our novel approach. Fnally we present successful experments on the well known VOC2007 dataset [3]. 2. State of the art In ths secton we present the most recent and effcent mage sgnatures usng local descrptors Codng/Pollng Schemes Codng/Pollng methods proposed n [9] by J. Wang et al. are a generalzaton of BoW methods[8]. These methods are based on vsual words dctonary. The vsual words are computed usng a clusterng algorthm on a tranng set of descrptors. The Codng/Poolng schemes dvde the problem nto two steps. The frst step maps each descrptor of the mage on the codebook ( codng step). The second step aggregates the mapped descrptors nto a sngle sgnature ( poolng step). For the Codng step n [9], they compute a mapped vector α as the result of the followng reconstructon problem: α = argmn b Wα 2 + d α 2, (1) α wth b beng a descrptor, W the codebook matrx, α the projecton coeffcents and d a localty constrant. The output vector α s thus optmzed wth respect to the reconstructon error and constraned to the projecton on few nearby codewords. For the Poolng step, they propose 2 methods: Sum Poolng (sum of all codes) or Max Poolng (maxmum value for each codng coeffcent). Compared to BoW, the codng/poolng schemes gve good performance wth lnear smlartes, whle retanng a small sgnature sze. However ths method s more complex to mplement and requres a fne tunng of the parameters to obtan good results [2] Fsher Vector Recently, Perronnn et al.[6] presented a new method called Fsher Vectors. The authors hypothesze the descrptors can be modeled by a probablty densty functon denoted as u λ of parameters λ. To descrbe the mage, they use the dervatve of the log-lkelhood of all mage descrptors to the model. The key dea s to descrbe the contrbuton of the descrptors to the devaton of the model parameters. The model used s a GMM of parameters µ c and σ c. The elements of the Fsher Vector for each Gaussanccan be wrtten as: G B µ,c = 1 T ω c r γ c(b r ) G B σ,c = 1 T ω c r γ c(b r ) ( ) br µ c σ c [ (br µ c ) 2 σ 2 c, (2) ] 1, (3) Where b r are the descrptors of mage, (ω c,µ c,σ c ) are the weght, mean and standard devaton of Gaussan c, and γ c (b r ) the normalzed lkelhood of b r to

3 Gaussan c. The fnal descrptor s obtaned by concatenaton of Gµ,c B and Gσ,c B for all Gaussans. The Fsher Vector method acheve very good results [2]. However, Fsher Vectors are lmted to the smple model of mxtures of Gaussans wth dagonal matrces. Moreover, the GMM algorthm s computatonally very ntensve Vector of Locally Aggregated Tensors In [7], Pcard et al. proposed an extenson of Fsher vectors, called VLAT for Vector of Locally Aggregated Tensors. Ths method uses a codebook of vsual words, computed wth a clusterng algorthm (typcally: k- mean). They compute the 1st and 2nd order moments for each vsual word (cluster): T c = 1 c µ c = 1 c b rc, (4) r (b rc µ c )(b rc µ c ), (5) r wth c the number of descrptors n cluster c and b rc the descrptors of mage belongng to cluster c. For each mage, the dfference between mage 2nd order moment and the cluster 2nd order moment s computed for each clusterc: T,c = r (b rc µ c )(b rc µ c ) T c. (6) Each T,c s flattened nto a vector v,c. The VLAT sgnature v for mage conssts n the concatenaton of v,c for all clustersc: v = (v,1...v,c ). (7) It s advsable to perform a normalzaton step for best performance. j, v [j] = sgn(v [j]) v [j] α, (8) x = v v (9), wthα = 0.5 typcally. VLAT gves very good results n smlarty search and automatc ndexng of mages wth lnear metrc, but leads to large feature vectors. The sze of the VLAT sgnature sc D D, wthc the number of clusters andd the sze of descrptors Spatal pyramd bnnng The spatal pyramd method [5] allows the ntroducton of spatal nformaton n mage sgnatures. The process conssts n computng a set of p sgnatures for dfferent regons of the mage. To compute the smlarty between mages, the weghted sum of smlartes for all regons s performed. It can be wrtten as the followng kernel functon: K(X,X j ) = p m=1 α m k(x (m),x (m) j ), (10) wth k(, ) the mnor kernel functon and X (m) the sgnatures of mageandj for regon m. 3. Proposed method,x (m) j We propose a new VLAT based sgnature that sgnfcantly reduces the sze of the sgnature, whle ncreasng ther dscrmnatve power. Our scheme s as follows: 1. We perform a Prncpal Component Analyss (PCA) wthn each cluster (We project the descrptors n the space of the egenvectors of ther cluster). 2. We compute VLAT sgnatures for dfferent part of the spatal pyramd. 3. For some tranng set of mages, we then compute the Gram matrx usng Eq. (10) wth a lnear mnor kernel. 4. We then perform a low rank approxmaton of the gram matrx and compute the set of projecton vectors assocated wth the generated subspace. The fnal sgnatures are the projectons of normalzed pyramd VLAT usng the obtaned projecton vectors Pre-projecton: descrptors PCA by cluster As for standard VLAT, we compute the 1st and 2nd order moments (Eq. 4 and 5) of each cluster. We perform a Takag factorzaton oft c matrx: T c = V c D c V c, (11) where D c s a dagonal matrx formed from the egenvalues of T c and V c s a matrx of egenvectors of T c. We denote byd c,pc the matrx wth thep c largest egenvalues on the dagonal: D c,pc = dag(λ c,1...λ c,pc ), (12) wth λ 1,pc λ 2,pc λ c,pc and we denote by V c,pc the matrx of the frstp c egenvectors: V c,pc = [v c,1...v c,pc ]. (13) We perform an approxmaton of the mage descrptors by projecton on the subspace spanned by thep c largest egenvectors oft c : We can thus rewrte Eq. (6): b rc = V c,p c (b rc µ c ). (14) T,c = V c,p c T,c V c,pc. (15)

4 aeroplane bcycle brd boat bottle bus car cat char cow table dog horse bke person plant sheep sofa tran tv Fgure 1. Images from PASCAL Vsual Object Classes Challenge Lke n VLAT, we concatenate each cluster sgnature and do proper normalzaton. Due to the tensor product, the sze of the new sgnature depends on the square of the ( number of egenvector selected n each cluster C ) c=1 p2 c. We propose 2 strateges for selectng the egenvectors n each cluster. The fxed dmenson method s to arbtrarly set the number of egenvalue that s kept for each cluster : p 1 = = p C. The error threshold method s to set a threshold ε on the error ntroduced by the subspace projecton: p c = mn p p s.t Spatal pyramd p =1 1 λ c, D =1 λ ε. (16) c, To add spatal nformaton, we compute the sgnature of Eq.(15) for dfferent regon of spatal pyramd. The spatal regons are obtaned by dvdng mage usng 1 1, 3 1, and 2 2 grds, for a total of 8 regons lke n most mage categorzaton setups [3, 2]. We use a lnear metrc for classfcaton (e.g. the mnor kernel of Eq. (10) s the dot product). Hence, the global sgnature for the mage s the concatenaton of weghted sgnatures for each regon of the pyramd: x = ( α 1 x,1... α m x,m... α 8 x,8 ), (17) wth x,m s the sgnature of the mage computed n the areamof the pyramd Post-projecton: Low rank approxmaton Gven a tranng set S of N mages,we compute the Gram matrxgof sgnatures defned n Eq. (17): G,j = x x j, x S,x j S. (18) Then, we perform the Takag factorzaton ofg: G = ULU, (19) L = dag(λ 1...λ...λ N ), (20) U = (u 1...u...u N ), (21) wth λ 1 λ 2 λ N and u the egenvector assocated wth the egenvalue λ. We want to compute a low rank approxmaton of the Gram matrx. We denote by L t the matrx wth the t largest egenvalues on the dagonal: L t = dag(λ 1...λ t ), (22) and we denote by U t the matrx of the frst t egenvectors: U t = [u 1...u t ]. (23) Then, we compute the projectors n the approxmated subspace: P t = XU t L 1/2 t, (24) wthx = [x 1...x N ] s the matrx of sgnatures for the tranng set S. Ths method akn to Kernel-PCA usng the Spatal Pyramdal Kernel wth a lnear mnor kernel Projecton vectors For some mage, we compute the projected sgnature n the approxmated space: y = P t x, (25) y contans an approxmate and compressed verson of x (from eq.(17)). Ths method selects the most energy drectons of sgnature space. The sze of y s very small compared to the sze of x, as t drectly depends on t (varyng from 1 to N). The fnal step s a l 2 normalzaton ofy : 4. Experments z = y y. (26) In ths secton we descrbe our expermental setup and some results on the PASCAL-VOC 2007 dataset [3]. Ths dataset conssts n about 10,000 mages and 20 categores, and s dvded nto 3 parts : tran, val and test. We used an 1-vs-rest lnear SVM classfer traned on tran + val sets and tested on the test set. We used a fast stochastc gradent descent algorthm wth C =

5 map aeroplane bcycle brd boat bottle bus car cat char cow Our method FV LLC Sze table dog horse bke person plant sheep sofa tran tv Our method 10k FV 320k LLC 200k Table 1. Image classfcaton results on Pascal VOC 2007 dataset compared to [2] map(%) Error threshold method Fxed dmenson method VLAT Sgnature sze Fgure 2. Effect of pre-projecton 10 [1]. We used Mean Average Precson (map) over the 20 classes for performance measurement. We extracted HOG features wth 4 dfferent scales (4px, 6px, 8px and 10px cell sze), every 3 or 6 pxels, dependng on the test. We used a smple k-means clusterng algorthm n order to obtan the codebook. For the low-order approxmaton of the post-projecton step, we used the full dataset Gram matrx Pre-projecton effect In ths secton we study the acton of the preprojecton (secton 3.1). We used a 6 pxels step for extractng HOG features, a codebook of 64 vsual words, only the frst level of the spatal pyramd (1 1) and no post-projecton. To compare the two strateges for estmatng p c, we measured the classfcaton score aganst the sgnature sze. For the fxed dmenson strategy, we reduced the descrptor sze to 32, 48, 64, 80, 112 and 128 dmensons. For the error threshold strategy, we tested 30%, 25%, 20%, 15%, 10%, 5% and 0% of reconstructon error rate ε. As we can see n Fgure 2 the pre-projecton sgnfcantly ncreases the performance relatve to VLAT even f t greatly reduces the sgnatures sze. We can also see that the thresholdng strategy gves better results, whch s consstent wth the fact t balances the error among clusters Classfcaton results To compare our method to the state of the art, we used a smlar setup to [2]. We used a 3 pxels step for extractng HOG features, a codebook of 256 vsual words, and all levels of the spatal pyramd. We set the thresholdεto 10% of reconstructon error (about 74 dmensons). We used all projectors n post-projecton step (about 10k sgnature sze). We can see from Table 1, our sgnature gves comparatve results to the of state of the art methods. However, our sgnature s much smaller and we used HOG descrptors (smpler than SIFT used n [2]). 5. Concluson In ths paper, we proposed a new mage sgnature based on VLAT. Our representaton ncreases dscrmnatve power of VLAT by ntroducng a spatal nformaton whle reducng ts sze sgnfcantly. Tests on VOC 2007 dataset show that our sgnature gves results smlar to state of the art methods, wth smpler descrptors and smaller sgnature. The results are very promsng, and we are lookng forward to a full nvestgaton of the nfluence of two projecton steps. References [1] L. Bottou. Large-scale machne learnng wth stochastc gradent descent. In Y. Lechevaller and G. Saporta, edtors, ICCS, pages , Pars, France, August [2] K. Chatfeld, V. Lemptsky, A. Vedald, and A. Zsserman. The devl s n the detals: an evaluaton of recent feature encodng methods. In BMVC, volume 76, [3] M. Everngham, L. Van Gool, C. K. I. Wllams, J. Wnn, and A. Zsserman. The PASCAL Vsual Object Classes Challenge 2007 Results, [4] H. Jégou, F. Perronnn, M. Douze, J. Sánchez, P. Pérez, and C. Schmd. Aggregatng local mage descrptors nto compact codes. IEEE PAMI, QUAERO. [5] S. Lazebnk, C. Schmd, and J. Ponce. Beyond bags of features: Spatal pyramd matchng for recognzng natural scene categores. In IEEE CVPR, pages , Washngton, DC, USA, IEEE Computer Socety. [6] F. Perronnn, J. Sánchez, and T. Mensnk. Improvng the fsher kernel for large-scale mage classfcaton. In ECCV, pages , [7] D. Pcard and P. Gosseln. Improvng mage smlarty wth vectors of locally aggregated tensors. In ICIP, Brussels, Belgque, September [8] J. Svc and A. Zsserman. Vdeo google: A text retreval approach to object matchng n vdeos. In ICCV, [9] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong. Localty-constraned lnear codng for mage classfcaton. In IEEE CVPR, pages , 2010.