Fusion of Deep Features and Weighted VLAD Vectors based on Multiple Features for Image Retrieval

Transcription

1 MATEC Web of Conferences, 0500 (07) DTS-07 DO: 005/matecconf/ Fuson of Deep Features and Weghted VLAD Vectors based on Multple Features for mage Retreval Yanhong Wang,, Ygang Cen,, Lequan Lang,*, Lnna Zhang 4, Vacheslav Voronn 5, Vladmr Mladenovc 6 School of Computer and nformaton Technology, Bejng Jaotong Unversty, 0008, Bejng, Chna Key Laboratory of Advanced nformaton Scence and Network Technology of Bejng, 0008, Bejng, Chna nsttute of Electronc Commerce, Guangdong Unversty of Fnance & Economcs, 500, Guangzhou, Chna 4 College of Mechancal Engneerng, Guzhou Unversty, 55005, Guyang, Chna 5 Department of Rado-electronc systems, Don State Techncal Unversty, 46500, Rostov-on-Don, Russa 6 Faculty of Techncal Scences Unversty of Kragujevac, 000, Cacak, Serba Abstract n tradtonal vector of locally aggregated descrptors (VLAD) method, the fnal VLAD vector s reshaped by summng up the resduals between each descrptor and ts correspondng vsual word The norm of the resduals vares sgnfcantly, and t can make vsual burst Ths s caused by a fact that the contrbuton of each descrptor to VLAD vector s not the same To address ths problem, we add a dfferent weght to each resdual such that the contrbuton of each descrptor to the VLAD vector becomes even to a certan degree Also, tradtonal VLAD method only uses the local gradent features of mages Thus t has a low dscrmnaton n ths paper, local color features are extracted and used to the VLAD method Moreover, we fuse deep features and the multple VLAD vectors based on local gradent and color nformaton Also, n order to reduce runnng tme and mprove retreval accuracy, PCA and whtenng operatons are used for VLAD vectors Our proposed method s evaluated on three benchmark datasets, e, Holdays, Ukbench and Oxford5k Expermental results show that our proposed method acheves good performance ntroducton n ths paper we consder the task of large-scale mage retreval n the past few years, Bag-of-Vsual-Words (BOW) [] [] method has acheved great effect n mage retreval area Generally, n order to ensure retreval recall, a relatvely large vocabulary wll be requred Thus, t wll lead to a low effcency of retreval tme and hgh memory consumpton Recently, Jégou et al [] proposed vector of locally aggregated descrptors (VLAD) model, whch aggregates descrptors based on a localty crteron n feature space n fact, VLAD s a knd of representaton of Fsher vector wthout probablty ts mplementaton s very smlar to the BOW model Also, VLAD s very cheap n consumptons of tme and memory n tradtonal VLAD method, the fnal VLAD vector s reshaped by summng up the resduals between each descrptor and ts correspondng vsual word The norm of the resduals vares sgnfcantly, thus t can make vsual burst [4] To address ths problem, we add a weght to each resdual such that the contrbuton of each descrptor to the VLAD vector becomes even to a certan degree Orgnally, the SFT [5] descrptors are adopted n VLAD method, and has shown good performance As we all known, the descrptor [6] s faster than the SFT descrptor Moreover, the performance of and SFT s comparatve n most cases [7] verfed that the descrptor was not only more effcent but also leadng to hgher accuracy than SFT and rootsft descrptors However, both the SFT and descrptors represent only local gradent nformaton, whch mss mportant color nformaton n order to solve ths problem, many works combne the gradent and color nformaton For examples, n [7] C feature was proposed, whch are -based color nformaton; n [8], the author fused the CLOG [9] features and the features at the stage of smlarty measurement; n [0] the author proposed color- descrptors whch combned wth the approxmate color local kernel hstograms n ths paper, Color names () [] and features are used n VLAD method n recent years, deep features are popular for mage processng, such as mage classfcaton [], object detecton [] and speech recognton [4] etc n ths paper, we adopt the pre-traned networks to obtan the deep features of mages Also, n order to mprove retreval accuracy, multple VLAD vectors and mage representaton based on deep features are fused * Correspondng author: langlq@gdufeeducn The Authors, publshed by EDP Scences Ths s an open access artcle dstrbuted under the terms of the Creatve Commons Attrbuton Lcense 40 (

2 MATEC Web of Conferences, 0500 (07) DTS-07 DO: 005/matecconf/ Methodology Framework of our proposed method The framework of our proposed method s shown n Fg Vocabulary based on local features ( or ) are traned from an ndependent tranng dataset For an mage, and features are extracted and quantzed on correspondng vocabulary respectvely Here, we mprove the tradtonal VLAD method by addng a weght for each resdual, called as weghted VLAD The and features are adopted to weghted VLAD method, respectvely Then, weghted VLAD vectors based on the two features are obtaned Moreover, the deep features are extracted from the mage, and mage representaton based on deep features s computed Then the vectors are fused nto a vector to represent the mage Fnally, smlarty scores of the query mage and dataset mages are measured, and the retreval results are returned Fg The framework of our proposed retreval method Weghted VLAD For an mage, the generaton process of VLAD vector s as follows: () Detect nterest regons of the mage and extract local descrptors, denoted as d n x x, x, x n () The local descrptors are quantzed on the vocabulary C c, c, ck wth K vsual words ( L K), denoted as NN( x ) arg mn x c () The resduals of a vsual k k word and the descrptors that are quantzed to ths word are computed, formulated asvk x ck Then, : NN () k the resduals are summed A vector of length L K* d s obtaned, whch s called as the VLAD vector (4) Power-law normalzaton s adopted for the vector obtaned at the step () t contans two steps: frstly, there s the square root wth symbol, formula as V V sgn( V),0 ; secondly, the vector s normalzed by the L norm, denoted as V V / V Accordng to above step (), t may cause vsual burst phenomenon because contrbuton of each descrptor for the VLAD vector s not same, e, the closer from center of the cluster, the greater the contrbuton s, and vce versa n the smlarty measurement stage, ths resdual wll be reflected n the contrbuton snce the Eucldean dstance s used To address the problem, we add a dfferent weght for each resdual Here, the weght s set to be the normalzed dstance of the descrptor and ts nearest vsual word, denoted as Eq () and Eq (), e, the smaller (greater) s the dstance between the descrptor and ts nearest vsual word, the smaller (greater) s the weght The valdty s verfed n the expermental secton (Secton ) t should be noted that a same weght s added to the resduals correspondng to each vsual word n [4] But n our algorthm, dfferent weghts are added to the resduals respectvely Ths makes our algorthm become more flexble and adaptve for mage retreval () V x c k k : NN () k d( x, ck) x c : NN () k where s a weght coeffcent Also, d( x, c k) represents the dstance of the descrptor x and the vsual word c k - For a gven mage, descrptors are extracted, denoted as x x, x, x n For the based vocabulary C c, c, c K, SRUF accordng to Secton, a -based VLAD vector can be obtaned, denoted as V ts length s L K* d n addton, each descrptor s extracted from an mage patch of sze p p, denoted as x Specfcally, for each pxel of a patch, an -D descrptor s extracted Then, the average of all descrptors n the patch s regarded as the color descrptor of the patch Thus, a set of descrptors for the mage are obtaned, denoted x x, x, x n For the -based as k ()

3 MATEC Web of Conferences, 0500 (07) DTS-07 vocabulary C c, c,, c K, smlarly, the - based VLAD vector s computed, denoted asv Also, the length s L K* d n order to mprove retreval accuracy, deep features are extracted by usng pretraned deep convolutonal neural networks (N) models, and the length s L Feature fuson and smlarty measurement For an mage, the three mage representatons are fused to a vector, denoted as: V [ V V V ] N where are the weght parameters, and The best values fttng are selected by loop teraton Eucldean dstance s used to compute smlartes between the query mage and the dataset mages To reduce runnng tme, we adopt PCA and whtenng method whch suppressed the co-occurrence problem wth the dmensonalty reducton [5] Our proposed algorthm s summarzed as follows: Algorthm Fuson of deep features and weghted VLAD vectors based on multple features ) Off-lne Tran vocabulares C, C on the tranng dataset Extract the dense and descrptors from each mage of the dataset Compute weghted VLAD vectors V, V N and extract deep feature V for each mage n the dataset N Fuse V, V and V by Eq (), denoted asv Reduce dmensonalty of V, and the result s denoted asv R ) On-lne Extract the dense and descrptors from a query Compute weghted VLAD vectors V, V N and extract deep feature V for the query mage N Fuse V, V and V by Eq (), denoted asv Reduce dmensonalty ofv, and the result s denoted asv R Compute smlarty score Return mages of the dataset wth the hgh smlarty scores () Expermental results DO: 005/matecconf/ n ths secton we verfed our proposed method on three benchmark datasets, e, Holdays [6], Ukbench [7] and oxford5k [8] n addton, Pars60k [9] s used to tran vocabulares for Oxford5k Vocabulares are traned from Mrflckr5k [0] for other datasets All experments are mplemented on a computer wth 8GB memory and GHz CPU (ntel(r) Core(TM) ) Selecton of parameters n our experments, the dense descrptors are extracted for each mage Moreover, each descrptor s obtaned n an mage patch of sze 4 4 Also, the based and -based vocabulares of sze 64 are used Moreover, N features of mages are obtaned by the VGG-f model [] Here, a N-based representaton s obtaned from the second fully-connected layer of convolutonal networks for each mage, t s a 4096-D vector s a power of the absolute value of VLAD vectors matrx However, we fnd that the best value of s between 0 and 06 The accuraces wth dfferent by usng weghted VLAD based on dfferent features on three datasets are shown n Fg (a)holdays (b) UKbench (C) Oxford5k Fg The comparson wth dfferent values of by usng N-based vectors and VLAD vectors based on features and features n (a) and (c), the maps are

4 MATEC Web of Conferences, 0500 (07) DTS-07 DO: 005/matecconf/ shown on the Holdays and Oxford5k dataset n (b), the N-S scores on the Ukbench dataset are shown Here, weghted VLAD vectors based on and features are denoted as,, respectvely On Holdays, we select =09 for =05 for, and =0 for N-based vectors On UKbench datasets s set to be 09, 05, 05 for, and N-based vectors, respectvely Moreover, s set to be 0, 05, 0 on Oxford5k, respectvely n Eq () are the weght parameters of -based and -based VLAD vectors and Nbased vectors for feature fuson On Holdays they are respectvely set to be 0, 0, 04 Also, they are set to be 0, 05, 045 and 06, 005, 05 on the UKbench and Oxford5k Effectveness of weghted VLAD n ths subsecton, we verfed the effectveness of proposed weghted VLAD model n Fg, two examples on UKbench dataset are shown t can be seen that the results of are better than tradtonal VLAD- n addton, we compare our weghted VLAD wth [4] (VLAD-LCR-RN) n Table On Holdays, t can be seen that the results are the same, but the length of our vector s only about a half of VLAD-LCR-RN On Oxford5k when vectors are reduced to 8D, acheves a better result Fg Two examples by usng the VLAD vectors based on features on the UKbench dataset The left mages are queres, and the frst row represents the results obtaned by the tradtonal VLAD based on descrptors, where those wth green boxes are the correct results Also, the second row s the results obtaned by usng our proposed weghted VLAD based on the Table Comparson between VLAD-LCR-RN and weghted VLAD- Methods VLAD-LCR-RN[4] Holdays 0658/ /4096 Oxford5k 057/ /4096 Oxford5k 0/ /8 Fuson of multple features n experments, and are fused, denoted as + The multple weghted VLAD vectors and deep features are fused nto a vector, denoted as ++N n Table, the retreval accuraces on dfferent datasets are lsted, where L 8 denotes the length of VLAD vectors obtaned by PCA and whtenng operatons When L 8, on Holdays, t can be seen that the map obtaned by + ncreased by nearly % compared to and respectvely Also, the map acheves 0806 for + +N Also, On UKbench and Oxford5k, the N-S score and the maps reach respectvely 696 and 04 for ++N Thus, feature fuson make retreval accuraces obvously mproved Table Comparson of results obtaned by usng dfferent features and by reducton on Holdays, Oxford5k and UKbench datasets Methods N[] + + +N L Holdays Ukbench (N-S score) Oxford 5k Table Comparson between varous methods and our proposed method Methods VLAD-C [6] Trangulaton embeddng [] + L Holdays Oxford 5k Ukbench (N-S score) Deep fully conneted+vlad [] OUR OUR

5 MATEC Web of Conferences, 0500 (07) DTS-07 DO: 005/matecconf/ We compare our method wth varous methods n Table Specally, n reference [], the authors consttuted hand-crafted features lke SFT whch s called trangular embeddng t can be seen that the map of + s hgher than the maps of other methods on Holdays We fuse VLAD vectors and deep features n reference [], they proposed the mult-scale orderless poolng (MOP-N) scheme whch combned the deep features and VLAD The results compared wth [] are lsted n Table t can be seen that our method acheves the better results on three datasets 4 Concluson Snce the contrbuton of each descrptor to the VLAD vector s not the same n tradtonal VLAD method, t wll result n vsual burst phenomenon To address the problem, we added a dfferent weght for every resdual to balance the contrbuton of each descrptor for the VLAD vector The features descrbe local gradent nformaton of an mage, whle the features represent local color nformaton Thus, to mprove the mage retreval accuracy, we proposed a smple and effectve method that fused our proposed weghted VLAD vectors based on local texture features and local color features Moreover, n order to mprove the accuracy further, deep features are extracted and fused wth the multple weghted VLAD vectors n order to reduce runnng tme, the PCA and whtenng operatons were adopted n ths paper Fnally, our experments obtan the better results when compared wth other methods Ths work was supported by Natonal Natural Scence Foundaton of Chna (657067); nternatonal (Regonal) Project Cooperaton and Exchanges of Natonal Nature Scence Foundaton of Chna (665070); Bejng Muncpal Natural Scence Foundaton (46050); The Natural Scence Foundaton of Guangdong Provnce (06A00708) and the Fundamental Research Funds for the Central Unverstes (07JBZ08) References J Svc, A Zsserman, Proceedngs Nnth EEE nternatonal Conference on Computer Vson,, (00) J Phlbn, M sard, EEE Conference on Computer Vson and Pattern Recognton, CVPR, -8 (007) H Jégou, M Douze, C Schmd, EEE n Proc CVPR, 04-(00) 4 J Delhumeau, P H Gosseln, ACM nternatonal Conference on Multmeda, (0) 5 D Lowe, nternatonal Journal of Computer Vson, JCV, no, 9-0 (004) 6 H Bay, A Ess, T Tuytelaars, LV Gool, Computer vson and mage understandng, no, (008) 7 E Spyromtros-Xoufs, S Papadopoulos, Y Kompatsars, EEE Transactons on Multmeda, no6, 7-78(04) 8 S Chen, Y Y Dng, H L, J Wang, X Deng, EEE nternatonal Conference on System, Man, and Cybernetcs, 9-96(04) 9 D A R Vgo, F S Khan, J V D Wejer, 0th nternatonal Conference on Pattern Recognton, (00) 0 P Fan, A Men, M Chen, EEE nternatonal Conference on Network nfrastructure and Dgtal Content, 76-70(009) J V D Wejer, C Schmd, J Verbeek, EEE Transactons on mage Processng a Publcaton of the EEE Sgnal Processng Socety, no7, 5- (009) A Krzhevsky, Sutskever, G E Hnton, Advances n Neural nformaton Processng Systems, 5, (0) M D Zeler, R Fergus, n Proc ECCV, 8689, 88-8(04) 4 AS Razavan, H Azzpour, J Sullvan, S Carlsson, n Proc CVPR Workshops, (04) 5 H Jégou, O Chum, n Proc ECCV, (0) 6 H Jégou, M Douze, C Schmd, n Proc ECCV, 04-7 (008) 7 D Nster, H Stewenus, n Proc CVPR'06, 6-68 (006) 8 J Phlbn, O Chum, M sard, n Proc CVPR (007) 9 J Phlbn, O Chum, M sard, J Svc, A Zsserman, n Proc CVPR(008) 0 MJ Huskes, MS Lew Proceedng MR '08 Proceedngs of the st ACM nternatonal conference on Multmeda nformaton retreval, 9-4 (008) K Chatfeld, K Smonyan, A Vedald, A Zsserman, n proc BMVC(04) H Jégou and A Zsserman, n Proc CVPR, 0 7 (04) Y Gong, L Wang, R Guo, and S, Lazebnk n Proc ECCV, 9 407(04) 5