Hierarchical Image Retrieval by Multi-Feature Fusion

Transcription

1 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v Artcle Herarchcal Image Retreval by Mult- Fuson Xaojun Lu, Jaojuan Wang,Yngq Hou, Me Yang, Q Wang* and Xangde Zhang * College of Scences, Northeastern Unversty, Shenyang 089, Chna; luxaojun@mal.neu.edu.cn(x.l.), 60023@stu.neu.edu.cn(J.W.), @63.com (Y.H.), @qq.com (M.Y.), * Correspondence: wangqmath@mal.neu.edu.cn (Q.W.), zhangxangde@mal.neu.edu.cn (X.Z.); Tel.: Abstract: Amng at the problems that are poor generalzaton performance, low retreval accuracy and large tme consumpton of exstng content-based mage retreval system, the herarchcal mage retreval method based on mult feature fuson s proposed n ths paper. The retreval accuracy rates on Corel5, Ubeach and Holdays are 68.23(Top ), 3.73(N-S) and 88.20(mAp), respectvely. The expermental results show that the method proposed n ths paper can effectvely mprove the defcency of sngle feature retreval and save tme sgnfcantly n the premse of a small amount of loss of accuracy. eywords: Herarchcal search; Image retreval; Mult-feature fuson. Introducton Large-scale mage retreval based on vsual nformaton s a key technology n many researches. In order to acheve the purpose of retrevng mages, we need to descrbe the content of the mage at frst. There are a lot of methods to represent the mage content, such as color, texture, and so on [] [2]. From the perspectve of mage representaton, the features of mage content can be dvded nto two categores, local feature [3] and global feature [4]. For the occluded mage retreval task, the local feature has a better performance. However, t may return the mage whch s rrelevant to the query due to local feature only focus on part of the mage. The global feature depcts the overall feature dstrbuton of the mage. But t has a large amount of redundant nformaton, and retreval tasks tend to return mages that look smlarly but do not correlate. Therefore, gven a specfc query, the retreval system usng a sngle feature s often dffcult to meet the needs of users. To solve ths problem, the retreval system ntegratng multple features has attracted more and more attenton due to the characterstcs of multple complementary features. Mult-feature fuson can effectvely mprove the performance of the retreval system has been confrmed [5], and there are many researches on mult-feature fuson has been done[6] [7]. At present, the popular method can be dvded nto two categores: one s fuson on the feature level; the other s rank aggregaton. The former merges a varety of features nto a new feature for retreval whle the latter fuses retreval results of dfferent features n the rank stage. The retreval system based on two fuson methods can mprove the retreval precson. Ronald Fagn et al. [8] used a number of ndependent "voters" to sort the database mages based on ther smlarty to the query, and then combned the rankngs wth some effcent fuson algorthms. Orol Ramos et al. [9] used a non-bayesan probablstc framework to solve the problem of classfer combnaton, and got two lnear combnaton rules to mnmze the msclassfcaton rate under certan constrants. Shaotng Zhang et al. [0] proposed a graph-based query fuson method, whch re-ordered multple retreval sets by chan analyss on the fuson graph. On the one hand, compared wth the sngle feature retreval, mage retreval based on 207 by the author(s). Dstrbuted under a Creatve Commons CC BY lcense.

2 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 2 of mult-feature fuson ncreases the retreval tme. On the other hand, many researches on mage retreval have been carred out on large-scale datasets, whch may contan up to several mllon pctures, It s very tme-consumng to search for the mages we need from the massve mages. On ths ssue, Ja Deng et al. [] used a herarchcal relatonshp, whch s most sutable for large-scale problems. At the same tme, a new hash scheme s proposed, whch reduces the computatonal cost effcently. evn Ln [2] utlzed the hdden layer learnng bnary code to represent the potental concept of the mage, and used the herarchcal deep search, whch mproved the retreval performance to a great extent. Inspred by the above work, we present a herarchcal mage retreval system based on mult-feature fuson. Gven a query, frstly we extract the CNN feature, and pre-retreve wth the bnary CNN feature. Then, the Color, Lbp and GIST features are fused for second retreval to obtan the search results. Further, we apply our approach to several datasets, and demonstrate ts effectveness and scalablty. 2. Materals and Methods 2.. Datasets Corel-5[3] dataset contans 5,000 mages that are dvded nto 50 categores, such as beach, brd, jewelry, sunset, etc. Each category has 00 mages. Each mage s taken as query n turn, and the other 4999 mages are served as retreval lbrary. The precson of top-r mages, namely the rato of smlar mages n the returned mages, s used as the evaluaton standard of the retreval system. Ubench[4] dataset s released wth 2550 objects, and each object has 4 pctures taken from dfferent vsual angles. All the 0,200 mages were served as query mages. We use N-S score, the average number of smlar mages n Top-4, as a measurement of retreval performance. Holdays[5] dataset ncludes 49 personal holday pctures that are composed of 500 categores. The frst mage n each category s used as the query, and the remanng mages are relevant mages. map s used to evaluate the retreval performance s Color. For each mage, we compute 2,000-dm HSV hstogram. (H, S, V are 20, 0, 0). LBP. For each mage, we dvde the detecton wndow nto 6 * 6 small cells and extract 256-dmensonal lbp descrptors. GIST. Each mage s reszed to 28 * 28. We use 4 scales wth the number of orentatons (8, 8, 8, 8), and extract 52-dmensonal GIST descrptors. CNN. Here we use VGG network pre-traned on Imagenet, whch conssts of 3 convoluton layers, 5 poolng layers and 3 fully connected layers. For all the mages n the dataset, the fc6 layer (the frst fully connected layer) feature s obtaned by forward operaton. The output of ths layer contans a wealth of mage nformaton, whose dmenson s 4096 dmensons Our Method In the mage retreval based on mult-feature fuson, we manly focus on two aspects: one s how to determne the weght of each feature to mprove the retreval accuracy; the other s how to mprove the retreval effcency. Tradtonally, the weght of feature has two ways to be determned, the global weght and the

3 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 3 of adaptve weght. The former s an average value or s decded by experenced experts, whch leads the retreval system to have poor generalzaton performance and low retreval performance for dfferent retreval mages. The latter s derved from retreval feedback based on ths feature, whch s better than the global weght. However, n the sum or product fuson, the dstncton between good features and bad features s not obvous. If the weght of the bad features n the retreval work s large, t wll also reduce the retreval performance to a certan extent. The tradtonal mage retreval system based on mult-feature fuson s a sngle-layer mage retreval system. The retreval process s shown n Fgure. That s, the system receves a query mage and extracts a pluralty of features of query and mages n the mage database. Then multple features are fused to get the comprehensve smlarty measure. Fnally, the system returns relevant mages to the user accordng to the comprehensve smlarty measure. At each retreval, the retreval system needs to extract the features of all mages and calculate the smlarty between query and all mages n the mage database, whch may lead to low retreval effcency and poor adaptablty to the retreval task on the large-scale mage database. In order to overcome the shortcomngs of the tradtonal global weght method, a herarchcal mage retreval system based on mult-feature s proposed. The basc framework of the retreval system s shown n Fgure 2. Next, the herarchcal retreval and mult-feature fuson wll be descrbed n detal Herarchcal retreval As shown n Fgure 2, n the herarchcal retreval system proposed n ths paper, pre-retreval s performed based on the bnary CNN feature. Then Color, Lbp, GIST and bnary CNN features are fused to perform second search wthn the pre-search results. Pre-retreval. In the pre-retreval stage, the CNN feature of query mage and mages n mage database are extracted respectvely. The CNN feature from the mddle layer has a hgh dmenson, so t wll take a long tme to be carred out the smlarty measure by usng t. User Return Smlar Images Retreval Result Images Query Image... (feature) Smlarty... (Smlarty) Image Database... (feature) Smlarty Comprehensve Smlarty Measures Fgure. Tradtonal mage retreval system framework based on mult-feature fuson

4 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 4 of User Return smlar mages Retreval Result Gst Images Image Database Query Image Color Color Smlarty Bnarzaton CNN Bnarzaton CNN Lbp Smlarty Gst CNN Smlarty Pre-search Results Color Lbp Lbp Smlarty Gst Smlarty Comprehensve Smlarty Measures Fgure 2. The proposed herarchcal retreval system framework. The red box represents the pre-retreval process and the blue box represents the secondary retreval process. Accordng to the characterstcs of Hammng dstance operaton, ths paper bnarzes 4096-dmensonal CNN feature as follows: For each bt of feature, we output bnary codes F by: j j F ave( F ) 0 0 F = j n j 0 F ave( F ) 0 0 {,2,..., } () Here, ave( F ) s the mean of feature F 0 0, n s the feature dmenson Then, n the lght of feature F, the smlarty score between query and mages n mage database s calculated. We output top-n mages as the result of pre-retreval accordng to the smlarty score. Secondary-retreval. In the secondary-retreval stage, we adjust the smlarty measure based on CNN accordng to the smlarty score vector and the retreval results n the pre-retreval stage. Next we extract lbp, color, GIST features of query mage and mages n the pre-retreval results, and compute smlarty measure based on each feature. Then we obtan retreval performance of each feature by searchng smlar mages. We determne the weght of each feature n accordance wth t, and calculate the comprehensve smlarty measure. Fnally, the retreval results are obtaned accordng to the comprehensve smlarty measure. The herarchcal retreval system proposed n ths paper mproves the retreval effcency n two aspects. Frstly, we bnarze the hgh-dmensonal CNN feature and calculate the Hammng dstance to mprove the retreval effcency. Secondly, we fuse multple features to make accurate search wthn the pre-retreval results, whch reduces the retreval range and mproves the retreval effcency to a certan extent Mult-feature fuson Specfcally, features are fused, q s query mage, p s a database mage. The proposed fuson method s as follows.

5 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 5 of We normalze the features: F( p) = ( p (), p (2),..., p ( n)) ( {,2,..., } ) n p () j j= F( q) = ( q (), q (2),..., q ( n)) ( {,2,..., } ) n q () j j= (2) (3) Here, p () j s the j-th component of the -th feature of the database mage p. q () j s the j-th component of the -th feature of the query mage q. n s the dmenson of feature. We calculate the dstance between q and p and normalze t: n j= { } d ( k) = d ( q, p ) = q ( j) p ( j) k,2,..., m, {,2,..., } (4) k k D ( q) = ( d (), d (2),..., d ( m)) ( {,2,..., } ) m d ( k) k= (5) Here, D ( q) s the smlarty vector between the query mage q and the all database mages, whch s calculated based on -th feature. m s the total number of mages n the mage database. We calculate the comprehensve measure by fusng multple features: ~ () () q q = = + sm( q) w D ( q)+ w D ( q ) (6) = Here w, () q () w, q are the weght of a good feature, the weght of a bad feature, and the number of good features respectvely, when the query mage s q. Good features and bad features. ac ( q) s retreval performance, whch s obtaned by D ( q ), Good features and bad features are defned as follows: ac ( q) = ac _ mean = f ac ( q) > ac _ mean else feature { good _ feature} feature { bad _ feature} (7) Weght Determnaton. In order to make better use of good feature n gettng the comprehensve measure, after gettng the weghts, the paper carres on a power calculaton to the good weght, whch can ncrease the dfference of the good feature and the bad feature. The weghts of features are shown as follows:

6 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 6 of f feature { good _ feature} else ~ q W W q = exp( ac ( q) ) ac ( q) = = = ac ( q) ac ( q) (8) 3. Results 3.. The effectveness of feature fuson In ths paper, we conduct retreval experments based on maxmum fuson, multplcaton fuson [6] and sum fuson. And we experment wth adaptve weght and average global weght. The sum fuson s what we use n ths paper. The maxmum fuson and multplcaton fuson are shown as follows. The maxmum fuson: ~ sm( q) = arg max{max{ W =,2,..., },max{ W = +, + 2,..., }} (9) D ( q) The multplcaton fuson: q q ~ () () q q = = + sm( q) = w D ( q) w D ( q ) (0) The expermental results on the maxmum fuson, multplcaton fuson, and sum fuson are 86.5\87.74\88.20, whch are shown n Fgure 3(a). The expermental results on the adaptve weght and global average weght are shown n Fgure 3(b). From the Fgure 3(a) we can see that the sum fuson s much better than the other two. Furthermore, t can be seen from the Fgure 3(b) that the adaptve weght proposed n ths paper s of great performance. Table shows the performance comparson between the sngle-feature retreval and multple-feature retreval. As what can be seen from the table, the latter s performance s optmal. Expermental results show that the proposed adaptve mult-feature fuson method s effectve and has good performance. Fgure 3. (a) A comparson of the retreval performance based on the three fuson methods of maxmum fuson, product fuson and sum fuson. (b) The performance comparson between the adaptve weght proposed n ths paper and the global average weght.

7 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 7 of Table. On the Holdays dataset, the comparson of search results based on sngle features and fuson features s CNN color lbp GIST fuson map Retreval performance evaluaton On the Ubeach, Holdays, Corel-5 datasets,we carres out experments based on dfferent fuson features. The retreval results are shown n Fgure 4. From the fgure, we can see that on the Holdays dataset, fusng CNN, color, lbp, GIST features to search mage acheves the best performance. On the Ubeach dataset, optmal mage retreval performance can be obtaned by fusng CNN, color features. On the Corel-5 dataset, we get the best result by fusng CNN, color, lbp, GIST features. Fgure 4. On the Ubeach, Holdays, Corel-5 datasets, the experments based on dfferent fuson features are compared. Fgure (a) shows precson comparson retreved on the Corel5 dataset, here the number of smlar mages returned s 30. Fgure (b) shows the map comparson retreved on the Holdays dataset,. Fgure (c) shows the comparson of N-S values retreved on the Ubeach dataset. 3.3 The effectveness of herarchcal retreval On the Ubeach dataset, we conduct experments based on the drect retreval and herarchcal retreval respectvely. Fgure 5 shows that 00 mages are randomly selected as query mages. We compare the retreval tme based on bnary CNN feature and CNN feature, from what we can see that mage retreval based on the bnary CNN can acheve the effect of savng tme. Then we set the number of pre-retreved mages to 8 and randomly select 00 mages as query mage. Fgure 6 shows the comparson of the retreval tme and N-S score. Fgure 7 shows the query mages wth dfferent N_S value. Fgure 8 shows the retreval results obtaned by the herarchcal retreval and the drect retreval.

8 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 8 of Fgure 5. On the Ubeach dataset, 00 mages were randomly selected as query mages, the retreval tme of bnary CNN feature s compared to the retreval tme of CNN feature. Fgure 6. On Ubeach dataset, We randomly selected 00 mages as the retreval mage, and set the number of pre-retreved mages to 8. Comparson of drect and herarchcal retreval performance. (a)the comparson of retreval tme n each search. (b) The average tme of 00 searches comparson. (c) The N-S value of each search comparson. (d) The average N-S values of 00 searches comparson. Fgure 7. Query mages wth dfferent N-S value. (a) Query mages whose N-S value of the herarchcal retreval s greater than the drect retreval. (b) Query mages whose N-S value of the herarchcal retreval s less than the drect retreval.

9 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 9 of Fgure 8. The search results. The frst mage n the upper left corner s query mage, and the remanng mages are smlar mages. In accordance wth the smlarty from large to small, we arrange them from left to rght. The frst row s search results based on the drect retreval. The second row s search results based on the herarchcal retreval Comparson Table 2 shows the results of ths paper compared wth other papers. On the Corel-5 dataset, when the number of returned smlar mages s, the precson of our method s mproved by 4% compared wth the method n [3]. On the Ubeach dataset, the N-S value of our method s 3.73, whch s 0.04 lower than [3], but ncreased by about 0.8 compared wth [7] and [22], 0.3 compared wth [20], and 0.08 compared wth [9]. On the Holdays dataset, the map of mage retreval based on our method s 88.20%, whch s about 3.5% hgher than [3], [20] and [22]. Compared wth [8], [9] and [2], t ncreased by about 8%. Compared wth [7], ncreased by 8.9%. Table2. Comparson wth other methods wth post processng n mage retreval level. Model Corel5(top) Ubeach Holdays Ours Graph-densty [3] Global-CNN [7] MOP-CNN+PCA+Whtenng [8] Bag-8 + PCA [20] Dscusson In ths paper, we propose a herarchcal search method based on adaptve mult-feature fuson. Gven a query mage, frstly, we get the pre-retreval results accordng to bnary CNN feature. Then, we extract the color, lbp, GIST features of query mage and mages n the pre-retreval results and search mage based one every sngle feature respectvely. After gettng the accuracy of each retreval, we adjust ther weghts and gan the comprehensve measure. Fnally, the retreval results are obtaned accordng to the comprehensve measure. The proposed method has better effcency than tradtonal drect retreval, and mult-feature fuson can effectvely mprove the accuracy. In the comparson of the algorthms whch have acheved good performance n mage retreval, we fnd that our method has a good performance and generalzaton ablty for dfferent mage databases. In the future work, we wll consder usng our method n unsupervsed mage retreval work. Wth the ncrease of the number of network mages, there are many dffcultes n manual markng.

10 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v 0 of The tradtonal unsupervsed mage retreval system has low precson and weak generalzaton ablty. It s sgnfcant to consder the applcaton of our method n unsupervsed mage retreval. Acknowledgement:Ths research s partally supported by Natonal Natural Scence Foundaton of Chna (Grant No ). Author Contrbutons: X.Lu, Q.Wang, and X.Zhang conceved and desgned the experments; J.Wang and Y.Hou performed the experments; J.Wang, Y.Hou, and M.Yang analyzed the data; J.Wang and and Y.Hou wrote the paper. Conflcts of Interest: The authors declare no conflct of nterest. References. Hremath P S; Pujar J. Content Based Image Retreval Usng Color, Texture and Shape s. IEEE Computer Socety. 2007, [CrossRef] 2. Tamura B H; Mor S, Yamawak T. Texture features correspondng to vsual percepton. IEEE Trans. Syst. Man Cybern. 200, [CrossRef] 3. Lowe D G; Lowe D G. Dstnctve Image s from Scale-Invarant eyponts. Internatonal Journal of Computer Vson. 2004, 60(2):9-0.[ CrossRef] 4. Olva B A; Torralba. Modelng the shape of the scene: A holstc representaton of the spatal envelope. Int J Comput Vson. 200.,45 75.[CrossRef] 5. Zheng Y; Huang X; Feng S. An mage matchng algorthm based on combnaton of SIFT and the rotaton nvarant LBP. Journal of Computer-Aded Desgn & Computer Graphcs. 200, 22(2): [PubMed] 6. Yu J; Qn Z; Wan T; et al. ntegraton analyss of bag-of-features model for mage retreval. Neurocomputng. 203, 20(0): [CrossRef] 7. Wang X; Han T X; Yan S. An HOG-LBP human detector wth partal occluson handlng.computer Vson, 2009 IEEE 2th Internatonal Conference on. IEEE. 2009, [CrossRef] 8. Fagn; Ronald; umar; et al. Effcent smlarty search and classfcaton va rank aggregaton. Proceedngs of the 2003 ACM SIGMOD nternatonal conference on Management of data. 2003, [CrossRef] 9. Terrades O R; Valveny E; Tabbone S. Optmal Classfer Fuson n a Non-Bayesan Probablstc Framework. IEEE Transactons on Pattern Analyss & Machne Intellgence. 2008, 3(9): [CrossRef] 0. Zhang S; Yang M; Cour T; et al. Query specfc fuson for mage retreval. European Conference on Computer Vson. Sprnger-Verlag, 202, [CrossRef]. Ja D; Berg A C; L F F. Herarchcal semantc ndexng for large scale mage retreval. Computer Vson and Pattern Recognton (CVPR), 20 IEEE Conference on. 20, 32(4): [CrossRef] 2. Ln ; Yang H F; Hsao J H; et al. Deep learnng of bnary hash codes for fast mage retreval. IEEE Conference on Computer Vson and Pattern Recognton Workshops. 205, [CrossRef] 3. Zhang S; Yang M; Cour T; et al. Query specfc fuson for mage retreval. Computer Vson ECCV 202. Sprnger Berln Hedelberg. 202:, [CrossRef] 4. Nster D; Stewenus H. Scalable recognton wth a vocabulary tree IEEE Computer Socety Conference on Computer Vson and Pattern Recognton (CVPR'06). IEEE. 2006, 2: [CrossRef] 5. Jégou H; Douze M; Schmd C. Hammng embeddng and weak geometry consstency for large scale mage search-extended verson. 2008, HAL Id : nra , verson. [PubMed] 6. Zheng L; Wang S; Tan L; et al. Query-adaptve late fuson for mage search and person re-dentfcaton. Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton. 205, [CrossRef] 7. Babenko A; Slesarev A; Chgorn A; et al. Neural codes for mage retreval. European Conference on Computer Vson. 204, [CrossRef] 8. Gong Y; Wang L; Guo R; et al. Mult-scale orderless poolng of deep convolutonal actvaton features. European Conference on Computer Vson. 204, [CrossRef] 9. Babenko A; Lemptsky V. Aggregatng local deep features for mage retreval. Proceedngs of the IEEE Internatonal Conference on Computer Vson. 205, [CrossRef] 20. Perronnn F; Larlus D. Fsher vectors meet neural networks: A hybrd classfcaton archtecture. Proceedngs of the IEEE conference on computer vson and pattern recognton. 205, [CrossRef] 2. Zhang S; Yang M; Wang X; et al. Semantc-aware co-ndexng for mage retreval. Proceedngs of the IEEE Internatonal Conference on Computer Vson. 203, [CrossRef]

11 Preprnts ( NOT PEER-REVIEWED Posted: 26 Aprl 207 do: /preprnts v of 22. Jégou H; Douze M; Schmd C. Improvng bag-of-features for large scale mage search. Internatonal Journal of Computer Vson. 200, 87(3): [CrossRef]