Learning Semantic Visual Dictionaries: A new Method For Local Feature Encoding

Transcription

1 Learnng Semantc Vsual Dctonares: A new Method For Local Feature Encodng Bng Shua, Zhen Zuo, Gang Wang School of Electrcal and Electronc Engneerng, Nanyang Technologcal Unversty, Sngapore. Emal: {bshua001, zzuo1, wanggang}@ntu.edu.sg Abstract In ths paper, we develop a new method to learn semantc vsual dctonares for local mage feature encodng. Conventonal methods usually learn dctonares from random local mage patches. Dfferent from them, we manually select a number of object classes whose vsual patterns can be seen at local mage patch level n complex mages (Fgure 1, and learn dctonares from ther thumbnal mages. The beneft s that these thumbnal mages have class labels, so we can cluster semantcally smlar mages together to generate meanngful cluster centers. Some other contrbutons of ths paper nclude developng an adaptaton method to adapt the learned dctonares to target datasets, and developng effcent algorthms to encode local patches wth our semantc vsual dctonares. Expermental results on three benchmark datasets demonstrate the effectveness of the proposed methods. Index Terms Semantc Dctonary Learnng, Greedy Group Sparse Codng, Scene Classfcaton. centpede blackboard eraser Bathtub cross ball Lght bulb grass pen I. INTRODUCTION Buldng vsual dctonares for local mage feature encodng s an mportant component for modern mage classfcaton ppelnes [1], [2], [3]. The dctonary atoms are expected to encode dstnctve and representatve vsual patterns, whch can act as vsual prmtves to represent complcated mage content. Prevous methods learn dctonares by clusterng features from random mage patches, whch have several lmtatons: (1 No precse category labels can be assgned to local patches. Therefore, clusterng methods may group patches whch are not semantcally smlar together. The resultng clusterng centers then cannot represent meanngful vsual patterns. (2 Most cluster centers may correspond to hghfrequent local patterns due to the lack of proper supervson n the clusterng procedure[4]. In many cases, the vsual patterns of low-frequent mage patches play a sgnfcant role n dscrmnatng dfferent classes, for example, class bocce and croquet n UIUC-Sports dataset[5] are almost the same except several cross and lne patterns appearng n croquet. To overcome these lmtatons, we propose a novel dea to buld semantc vsual dctonares. We observe that many smple object categores exhbt basc vsual patterns that can be seen at local patch level n complex-content mages. One example s shown n Fgure 1. We manually select a number of object categores, whch are expected to cover dverse vsual patterns useful for the classfcaton task. We resze the objectcentrc mages to produce thumbnals, whch are utlzed to learn dctonares. An nterestng property of ths method s Fg. 1. Image patches n complex mages can be represented by smple object categores, whch exhbt basc yet mportant elementary vsual patterns. For example, centpede category can approxmate curve vsual pattern, pen can be a substtute to lne pattern, and ball carres crcle pattern, etc. that the class labels of these thumbnals are known. Durng the dctonary learnng procedure, we only cluster thumbnals from the same object class together. As a result, the cluster centers correspond to vsual patterns wth semantc meanngs. Besdes, by carefully selectng object categores, we could also ensure that our dctonary encodes dverse patterns rather than domnant ones. Each sub-dctonary s learned from an auxlary dataset (ImageNet, and the concatenaton of them consttute our global dctonary. So t naturally has a group/block structure, n whch each group corresponds to an object semantcs. We explot ths group structure when encodng a local mage patch by selectng only a small quantty of groups. However, t s neffcent to solve ths problem through Group Lasso solver [6]. Therefore, two effcent greedy codng algorthms are proposed to generate the group sparse codes. We further develop a doman adapton approach to adapt the learned dctonares to ft the target database (e.g., Indoor scene 67 by learnng a lnear transformaton matrx. On three benchmark datasets, we demonstrate that the state-of-the-art codng method [2] can beneft from our adapted semantc dctonary. Meanwhle, the experments also show that our codng methods can acheve promsng results based on our semantc vsual dctonary /15/$ IEEE. 901

2 II. RELATED WORK Dctonary Learnng: Dctonary learnng has receved ncreasngly attenton recently, as the performance of mage classfcaton accuracy heavly depends on the qualty of dctonares. A number of papers have been publshed [7], [8], [9], [10], [11], [12], [13], [14], [15] n the past several years. All of these papers learn dctonares from random mage patches. Our work s novel because we use thumbnals of object mages to learn vsual dctonares, whch can represent vsual patterns wth semantc meanngs. The work that s closest to ours s to learn a dscrmnatve dctonary for classfcaton [7], [10], [11], [12]. They treat the labels of patches to be the same wth the labels of correspondng mages. However, the labels assgned to patches n ths way s qute nosy. In our work, the labels for the thumbnal mages are accurate and semantc meanngful, therefore more dscrmnatve dctonary are expected to learn from them. Usng auxlary object data for classfcaton: The dea of usng auxlary object data to help classfcaton or retreval can be seen n [16], [17], [18], [19]. Dfferent from our work, these papers buld the global representaton based on ther responses to the pre-traned object classfers/detectors. We leverage auxlary object data to learn semantc dctonary to mprove the qualty of local feature encodng based on the assumpton that local patches n mages can be approxmated by object-centrc mage thumbnal. Moreover, we emprcally demonstrate that our approach works much better than [17] on several scene classfcaton benchmarks. III. LEARNING SEMANTIC VISUAL DICTIONARIES We observe that many object categores exhbt basc yet mportant vsual patterns that can be seen at local patch level n complex content mages (Fgure 1. It motvates us to buld vsual semantc dctonares by clusterng the object-centrc mage thumbnals. Instead of clusterng random local patches, the dctonary generated from labeled thumbnals carry more dscrmnatve and semantc nformaton. Specfcally, ther class labels are known, whch allows the clusterng algorthms to group mages from the same category, thus avodng producng clusters whose atoms are not really semantcally smlar, as seen n the tradtonal dctonary learnng methods. We manually select 55 object categores based on the followng crtera: (1 the object categores exhbt smple and elementary vsual patterns; (2 the vsual appearance of dfferent object classes does not overlap and (3 some textureless background classes are also ncluded. The selected objects nclude pen, box, cross, grass, etc, whch are supposed to carry mportant and basc vsual patterns. We download ther mages from ImageNet dataset, and boundng boxes are used to generate clean foreground object regon. Next, each object regon mage s reszed to I I thumbnal (I = 40 n our case, so they can be consdered as local mage patches. For each object class, we learn a sub-dctonary D (, and the concatenaton of sub-dctonares yelds our fnal group-structured semantc dctonary D. IV. FEATURE ENCODING BASED ON SEMANTIC VISUAL DICTIONARY Our semantc dctonary s organzed n group structure ( D = [D (1...D (...D (M ], where D ( contans K cluster centers [D ( (1...D ( (K]. Prevous works [6], [20], [21] have shown that, n some cases, t s more robust to reconstruct the nput sgnal by mposng group/block structure to ts reconstructon coeffcent. Here we develop a local feature encodng approach to account for the group-structured dctonary: only a few semantc concepts (groups are actvated to reconstruct the mage patch. [21] Therefore, the group sparse code β for the nput y s generated va optmzng the functon: z =argmn y Dβ 2 2 β s.t. β 0 L (1... β ( 1,... 0 T, 1 M where β s the reconstructon coeffcent of nput y, β ( denotes the code for the -th group n the dctonary D; T and L are the maxmum number of selected groups and selected atoms respectvely; M s the number of groups n the groupstructured dctonary D and β 0 s the pseudo-norm whch counts the number of non-zero elements n β. Ths optmzaton problem s NP hard, whch requres T ( M L ( j j Nj evaluatons, n whch N j s the total number of atoms n the selected j(j T groups. We have to relax t to obtan a good suboptmal soluton. The frst relaxaton case s group Lasso [6], whch only consders group-level sparsty, and t mnmzes the l 1 /l 2 norm regularzaton problem [6]. A better relaxaton case s the sparse group Lasso [20], whch enforces the sparsty over the group and wthn-group level reconstructon coeffcents. Both of these two relaxed problems are convex to β, sotcan be solved usng convex optmzaton solvers. However, these two approxmatons are computatonally expensve [6], [20]. To speed up the encodng modules, we approach the orgnal NP hard problem (Equaton 1 from a dfferent perspectve. Our relaxed problem s cast as a Lasso problem nsde the selected few selected groups. It has the l 0 /l 1 norm form: z =argmn y Dβ λ β 1 β (2 s.t.... β ( 1,... 0 T, 1 M where λ controls the tradeoff between reconstructon error and wthn group sparsty. The group-structured reconstructon code β s generated through the followng way: the coeffcents are set to zero for the groups that are not selected, and the codes β T for the selected T groups are obtaned by solvng a small-scale Lasso problem: z =argmn β T y D T β T λ β T 1 (3 where D T s a smaller dctonary made up atoms from T selected groups. Ths relaxed optmzaton problem s also a NP combnatoral problem, whch requres T C ( M 902

3 Algorthm 1: Greedy Group Sparse Codng Input: D: Dctonary wth group structure [G 1,G 2,...,G M ] y: The nput sgnal T : Maxmum number of selected groups λ: sparse regularzaton term coeffcent ɛ: tolerance of reconstructon error Output: β (k : the group sparse coeffcents Intalzaton Normalze each atom n D to be unt vector β (0 = 0 T,g (0 =,D (0 O = for k = 1...T do Let j (k = argmax j DG T j (y Dβ (k 1 2 g (k = Unon(g (k 1, G j (k D (k O = Unon(D(k 1 O, D(:,G j (k α = argmn α y D (k O α λ α 1 β (k = 0 T, β (k (g (k =α f (y Dβ (k 2 ɛ then break end end evaluatons, n whch M s the number of groups and C s the cost of solvng Equaton 3. The computaton cost to solve our l 0 /l 1 problem s much less compared to group Lasso, whch ( Nj evaluatons to Equaton 1. Furthermore, two requres L effcent encodng methods are proposed to select T groups to solve t: Greedy Group Sparse Codng(GGSC: It works lke Orthogonal Matchng Pursut[22] (OMP that selects the dctonary atoms. At each teraton, t chooses the groups that best correlates to the sgnal resdue. Algorthm 1 present the detals. Local-constraned Lnear Group Codng(LLGC: The T groups that most correlate wth the nput sgnal y are chosen to be the vsual base for nput y. The correlaton between the nput sgnal (resdue y and dctonary group (sub-dctonary D ( s defned as the square of l 2 norm of cosne smlarty of y and every atom D ( j n D (. Our encodng methods are effcent as the soluton of the small-scale Lasso problem s very effcent. V. DICTIONARY ADAPTATION The semantc dctonary s learned n a dfferent doman wthout awareness of the characterstcs of mage patches n the target classfcaton dataset. Therefore, we ntroduce a smple doman adaptaton method to adapt the learned semantc dctonary to our classfcaton task, by learnng a transformaton matrx W. The adaptaton matrx W s learned by mnmzng the overall reconstructon error on mage patches from the target dataset, z =argmn W 1 2 N y WDβ λω(β (4 =1 Fg. 2. Comparson of dctonary atoms. The left column presents 6 group atoms of our semantc dctonary, whch exhbt semantcally smlar vsual patterns. The rght column shows 30 randomly sampled atoms from the dctonary generated through clusterng local mage patches. where N s the number of patches; D s the off-learned semantc dctonary and Ω(β denotes the regularzaton term wth respect to dfferent encodng methods (LLC, GGSC. Snce the objectve functon s not jontly convex to W and β, we learn W usng an teratve method: frst, W s ntalzed to be an dentty matrx; next, β s obtaned va applyng correspondng encodng solver; fnally, gven the new β,we update the adaptaton matrx W by the gradent ΔW of the objectve functon (Equaton 4 to W : ΔW = N (y WDβ β T D T (5 =1 We terate the update procedure untl t converges. VI. EXPERIMENTS AND RESULTS Dataset: We evaluate our semantc dctonary and encodng methods on three scene classfcaton datasets, ncludng generc natural scene mages(15-scene, cluttered ndoor mages (MIT 67 Indoor Scenes, and complex event and actvty mages (UIUC-Sports. The classfcaton performance s the mean of mult-way classfcaton accuracy scores over all the classes n the dataset. Experment Setup: We densely sample mage patches followng [23], the sze of whch s and the sldng strde s 8 pxels. Meanwhle, only SIFT [24] feature s used to represent the mage patches 1. All mages are reszed to maxmum 300 pxels along the smaller axs. Fnally, The mage-level representaton s generated by performng max poolng [2], [1], [17] operaton to all the sparse codes of mage patches. Besdes, Spatal Pyramd Matchng (SPM [23] s also appled to enforce the spatal layout constrant. A. Superorty of Semantc Dctonary We verfy the effectveness of our semantc dctonary n three datasets, and the results are nsprng: the state-of-the-art codng method (LLC [2] can beneft from our semantc vsual dctonary. Table I demonstrates that LLC acheves better result on our semantc dctonary than on the dctonary generated 1 buldng rcher feature for mage patches and usng mult-scale patches can sgnfcantly mprove the performance [25], but that s not our man focus. 903

4 Methods 15-Scene UIUC-Sports Indoor GIST[27] % DPM[28] % Patches[29] % Object Bank[17] 80.9% 76.3% 37.6% Topc Model[30] 78% 82.5% - Kwtt[31] 82.3% 83% - LLC(RP+KSVD[32] 74.9% ± 0.48% 77.5% ± 1.73% 29.2% LLC(RP+Kmeans[2] 80.6% ± 0.72% 85.2% ± 1.53% 41.2% LLC(Semantc 81.0% ± 0.37% 85.1% ± 1.42% 42.0% LLC(ASemantc 81.1% ± 0.52% 85.8% ± 1.57% 43.2% LLC(RP+ASemantc 82.0% ± 0.49% 86.5% ± 1.26% 45.8% GGSC(Semantc 82.0% ± 0.66% 85.6% ± 2.07% 43.2% LLGC(Semantc 81.2% ± 0.42% 84.4% ± 2.50% 42.0% TABLE I CLASSIFICATION PERFORMANCE ON THREE SCENE CLASSIFICATION DATASETS. RP AND SEMANTIC DENOTE DICTIONARY GENERATED FROM RANDOM PATCHES AND SEMANTIC THUMBNAIL IMAGES RESPECTIVELY, ASEMANTIC REPRESENTS THE ADAPTED SEMANTIC DICTIONARY. through clusterng random mage patches. The largest performance gan s observed when these representaton codes are fused, demonstratng that our semantc and random patch (conventonal dctonares are complementary. We further nvestgate the performance of dfferent encodng methods. SLEP [26] toolbox s utlzed to solve the group Lasso (GLasso and sparse group Lasso (Sparse GLasso problem. For GGSC and LLGC, at most T =5groups are actvated to generate the sparse codes. The classfcaton accuracy and average elapsed tme for every mage are reported n Table II. We notce that both our two encodng alternatves acheve sgnfcant better performance than GLasso and Sparse GLasso, whle n the mean tme ours are much more effcent. Table I shows that GGSC also outperforms LLC, whch elucdates the advantage of consderng the group structure of our semantc dctonary. However, we fal to see obvous mprovement over GGSC on the adapted semantc dctonary, as the adaptaton may have broken the group structure of the semantc dctonary. B. Dscrmnatve power over sze of sub-dctonary We beleve that each object group has a dfferent optmal sze for correspondng sub-dctonary, as the ntra-class semantc varance and number of object examples n each object class vares dramatcally. For example, the object semantc ball ncludes basketball, soccer, golf ball, bowlng ball, tenns ball and poolng ball n our case, whch exhbt large ntra-class varance; whle the object class hook has a much smaller ntra-class semantc varance and very few tranng nstances. We smply fx the sze of all the sub-dctonares to be the same (K = D ( n our mplementaton. Then we analyze how K affects the dscrmnatve power of our semantc dctonary. On one hand, the sub-dctonary wth larger sze encodes fner-graned vsual patterns, whle on the other hand, the dmensonalty of the sparse codes grows lnearly wth the sze of sub-dctonary. As demonstrated n Fgure 3, K s fxed to be 20 n our experments as t s a good compromse between the dctonary sze and dscrmnatve power. Methods accuracy tme (Seconds/mage GLasso 82.60% ± 1.33% Sparse GLasso 83.62% ± 1.13% GGSC 85.60% ± 2.07% 1.99 LLGC 84.35% ± 2.50% 0.73 TABLE II CLASSIFICATION ACCURACY AND TIME CONSUMPTION COMPARISON ON UIUC-SPORTS. (a Fg. 3. (a classfcaton accuracy w.r.t sze of sub-dctonary. X-axs denotes the sze of sub-dctonary. (b classfcaton performance w.r.t number of object groups that consttute the semantc dctonary (each group s composed of 20 atoms, X-axs s the number of groups. 5 rounds of randomzed samplng s performed to choose the groups from 55 object groups. Statstcs are based on the experments on 8-UIUC-Sports. C. Proftablty over growng number of object groups We envsage that the larger-sze dctonary wth more object semantcs s gong to preserve rcher nformaton, therefore may serve as a better dctonary. In ths part, we verfy ths assumpton. To smulate what wll happen wth growng number of groups, we randomly sample P = 11(20%, 22(40%, 33(60%, 44(80% and 55(100% out of M =55 object classes for multple tmes. The subset dctonary s generated through concatenatng the correspondng P subdctonares, therefore D sub = Unon(D (1...D (P, note that D sub D. As shown n Fgure 3, the classfcaton performance of larger-sze dctonary contnuously ncreases as more object semantcs are encoded n the dctonary. Therefore, we conclude that the dscrmnatve power of our semantc dctonary could be further boosted by addng more complementary object semantcs. VII. CONCLUSION In ths paper, we propose a new perspectve to learn semantc dctonary from thumbnal mages. Next, we explot the group structure of the dctonary and develop a greedy group sparse codng algorthm to effcently solve the orgnal NP l 0 /l 1 optmzaton problem. Fnally, to make our semantc dctonary better ft the target classfcaton task, a domanaware adaptaton algorthm s developed. The promsng results on three benchmarks demonstrated the effectveness of our proposed methods. REFERENCES [1] J. Yang, K. Yu, Y. Gong, and T. Huang, Lnear spatal pyramd matchng usng sparse codng for mage classfcaton, n Computer Vson and Pattern Recognton, CVPR IEEE Conference on. IEEE, 2009, pp (b 904

5 [2] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, Localtyconstraned lnear codng for mage classfcaton, n Computer Vson and Pattern Recognton (CVPR, 2010 IEEE Conference on. IEEE, 2010, pp [3] J. Maral, F. Bach, J. Ponce, G. Sapro, and A. Zsserman, Dscrmnatve learned dctonares for local mage analyss, n Computer Vson and Pattern Recognton, CVPR IEEE Conference on. IEEE, 2008, pp [4] O. Boman, E. Shechtman, and M. Iran, In defense of nearest-neghbor based mage classfcaton, n Computer Vson and Pattern Recognton, CVPR IEEE Conference on. IEEE, 2008, pp [5] L.-J. L and L. Fe-Fe, What, where and who? classfyng events by scene and object recognton, n Computer Vson, ICCV IEEE 11th Internatonal Conference on. IEEE, 2007, pp [6] M. Yuan and Y. Ln, Model selecton and estmaton n regresson wth grouped varables, Journal of the Royal Statstcal Socety: Seres B (Statstcal Methodology, vol. 68, no. 1, pp , [7] S. Kong and D. Wang, A dctonary learnng approach for classfcaton: separatng the partcularty and the commonalty, n Computer Vson ECCV Sprnger, 2012, pp [8] J. Maral, F. Bach, J. Ponce, and G. Sapro, Onlne dctonary learnng for sparse codng, n Proceedngs of the 26th Annual Internatonal Conference on Machne Learnng. ACM, 2009, pp [9] F. Bach, J. Maral, J. Ponce, and G. Sapro, Sparse codng and dctonary learnng for mage analyss, n IEEE Conference on Computer Vson and Pattern Recognton (CVPR. tutoral, San Francsco, [10] Z. Jang, Z. Ln, and L. S. Davs, Learnng a dscrmnatve dctonary for sparse codng va label consstent k-svd, n Computer Vson and Pattern Recognton (CVPR, 2011 IEEE Conference on. IEEE, 2011, pp [11] Q. Zhang and B. L, Dscrmnatve k-svd for dctonary learnng n face recognton, n Computer Vson and Pattern Recognton (CVPR, 2010 IEEE Conference on. IEEE, 2010, pp [12] M. Yang, L. Zhang, X. Feng, and D. Zhang, Fsher dscrmnaton dctonary learnng for sparse representaton, n Computer Vson (ICCV, 2011 IEEE Internatonal Conference on. IEEE, 2011, pp [13] H. V. Nguyen, V. M. Patel, N. M. Nasrabad, and R. Chellappa, Kernel dctonary learnng, n Acoustcs, Speech and Sgnal Processng (ICASSP, 2012 IEEE Internatonal Conference on. IEEE, 2012, pp [14] Z. Szabó, B. Póczos, and A. Lorncz, Onlne group-structured dctonary learnng, n Computer Vson and Pattern Recognton (CVPR, 2011 IEEE Conference on. IEEE, 2011, pp [15] H. Wang, C. Yuan, W. Hu, and C. Sun, Supervsed class-specfc dctonary learnng for sparse modelng n acton recognton, Pattern Recognton, vol. 45, no. 11, pp , [16] G. Wang, D. Hoem, and D. Forsyth, Learnng mage smlarty from flckr groups usng fast kernel machnes, [17] L.-J. L, H. Su, L. Fe-Fe, and E. P. Xng, Object bank: A hghlevel mage representaton for scene classfcaton & semantc feature sparsfcaton, n Advances n neural nformaton processng systems, 2010, pp [18] L. Torresan, M. Szummer, and A. Ftzgbbon, Effcent object category recognton usng classemes, n Computer Vson ECCV Sprnger, 2010, pp [19] J. Vogel and B. Schele, Semantc modelng of natural scenes for content-based mage retreval, Internatonal Journal of Computer Vson, vol. 72, no. 2, pp , [20] J. Fredman, T. Haste, and R. Tbshran, A note on the group lasso and a sparse group lasso, arxv preprnt arxv: , [21] E. Elhamfar and R. Vdal, Robust classfcaton usng structured sparse representaton, n Computer Vson and Pattern Recognton (CVPR, 2011 IEEE Conference on. IEEE, 2011, pp [22] G. Swrszcz, N. Abe, and A. C. Lozano, Grouped orthogonal matchng pursut for varable selecton and predcton, n Advances n Neural Informaton Processng Systems, 2009, pp [23] S. Lazebnk, C. Schmd, and J. Ponce, Beyond bags of features: Spatal pyramd matchng for recognzng natural scene categores, n Computer Vson and Pattern Recognton, 2006 IEEE Computer Socety Conference on, vol. 2. IEEE, 2006, pp [24] D. G. Lowe, Dstnctve mage features from scale-nvarant keyponts, Internatonal journal of computer vson, vol. 60, no. 2, pp , [25] M. Juneja, A. Vedald, C. Jawahar, and A. Zsserman, Blocks that shout: Dstnctve parts for scene classfcaton, n Proceedngs of the IEEE Conf. on Computer Vson and Pattern Recognton (CVPR, vol. 1, [26] J. Lu, S. J, and J. Ye, SLEP: Sparse Learnng wth Effcent Projectons, Arzona State Unversty, [Onlne]. Avalable: jye02/software/slep [27] A. Quatton and A. Torralba, Recognzng ndoor scenes, n Computer Vson and Pattern Recognton, 2009 IEEE Computer Socety Conference on, [28] M. Pandey and S. Lazebnk, Scene recognton and weakly supervsed object localzaton wth deformable part-based models, n Computer Vson (ICCV, 2011 IEEE Internatonal Conference on. IEEE, 2011, pp [29] S. Sngh, A. Gupta, and A. A. Efros, Unsupervsed dscovery of md-level dscrmnatve patches, n Computer Vson ECCV Sprnger, 2012, pp [30] Z. Nu, G. Hua, X. Gao, and Q. Tan, Context aware topc model for scene recognton, n Computer Vson and Pattern Recognton (CVPR, 2012 IEEE Conference on. IEEE, 2012, pp [31] R. Kwtt, N. Vasconcelos, and N. Raswasa, Scene recognton on the semantc manfold, n Computer Vson ECCV Sprnger, 2012, pp [32] M. Aharon, M. Elad, and A. Brucksten, K-svd: An algorthm for desgnng overcomplete dctonares for sparse representaton, Sgnal Processng, IEEE Transactons on, vol. 54, no. 11, pp ,