Aggregating Local Descriptors for Epigraphs Recognition

Transcription

1 Aggegating Local Descitos fo Eigahs Recognition Giusee Amato, Fabizio Falchi, Fausto Rabitti, Lucia Vadicamo ISTI-CNR, via G. Mouzzi, 5624 Pisa (Italy) s: Abstact. In this ae, we conside the task of ecognizing eigahs in images such as hotos taken using mobile devices. Given a set of 7,55 hotos elated to 4,56 eigahs, we used a k-neaestneighbo aoach in ode to efom the ecognition. The contibution of this wok is in evaluating state-ofthe-at visual object ecognition techniques in this secific context. The exeimental esults conducted show that Vecto of Locally Aggegated Descitos obtained aggegating SIFT descitos is the best choice fo this task. Keywods: Eigahs Recognition, Object Recognition, Content-Base Image Retieval, Bag-of-Featues, VLAD. Intoduction Because of lage availability of digital cameas, esecially embedded in smathones and tablets, thee is a gowing demand of infomation etieval systems able to seach by using images as quey. The basic idea is to allow the use to make a hoto of the object she is inteested on in ode to eceive elated infomation. To his goal, the objects in the hoto ae comae with the ones stoed in a eositoy in ode to ecognize it and give back elevant metadata, links, etc. We conducted the wok eoted in this ae in the context of the Euoeana netwok of Ancient Geek and Latin Eigahy (EAGLE) CIP-Best Pactice Netwok. In EAGLE, an object is essentially an ancient eigah. In this wok, we focus on seaching fo the most simila eigahs with esect to the one eesented in a hoto made by the use. The dataset we used consists of 7,55 hotos elated to 4,56 eigahs that wee made available, within the EAGLE oject, by Saienza Univesity of Rome. This functionality will be integated on the flagshi mobile alication, to enable touists to undestand inscitions they find on location. The alication will allow a visito of a site whee one of the stoed eigahy is visible (museum, steet, achaeological site, inted eoduction, etc.) to take a ictue with a mobile hone, send the ictue to the cental eositoy and eceive back the eniched infomation associated with that ictue. Moeove, cowd-soucing functionalities will be develoed in the alication to acceleate the adotion of the EAGLE and eigahic content. Fo achieving the task of identifying objects in an image o video sequence, usually efeed to as object ecognition, eseach conducted in both Comute Vision and Multimedia Infomation Retieval fields has focused on local featues that ovide a Digital Pesentation and Pesevation of Cultual and Scientific Heitage, Vol. 4, 24, ISSN: 34-46

2 eesentation that allows matching local stuctues between images. Fist, distinctive key oints ae selected in each image. Second, a descition of the selected egions is given. Diect local featues matching has been oved to be vey effective in ecognizing the same objects in two hotos. Howeve, to achieve scalability (i.e., to be able to seach in lage datasets) aggegation techniques ae necessay in ode to summaize the infomation eoted fo each key oint. Taditionally, object ecognition has been successfully alied to consume oducts, buildings, monuments and landmaks. Howeve, we did not find any secific exeiments conducted on ancient eigahs. Moeove, state-of-the-at techniques ae vey effective on small sets (tens) of objects while aoximate techniques ae alied when millions of objects have to be ecognized. The numbe of hotos of eigahs exected in the context of the EAGLE oject is in the middle of these two extemes. Thus, it is vey inteesting to undestand what is the best technique fo ecognizing eigahs. In the following, we eot the esults obtained testing vaious state-of-the at techniques in ode to effectively ecognizing eigahs in hotos given a medium-lage scale set (tens of thousands) of known eigahs. The est of this ae is stuctued as follows. In Section 2, we discuss elated wok. In Section 3, we give infomation about the tested aoaches. Then, we secify the exeimental settings and discuss the obtained esults in Section 4. Finally, in Section 5, we eot conclusions and discuss futue wok. 2 Related Wok In the last few yeas, eseach on object ecognition has focused on local featues [8], []. Following this aoach, an image is eesented by descibing the visual content of tyically thousands of egions of inteest automatic selected. To achieve best effectiveness, images ae comaed by matching thei local featues and seaching fo a geometic tansfomation that can associate the egions of both images. In the last few yeas, the oblem of ecognizing cultual heitage elated objects, in aticula landmaks, has eceived gowing attention by the eseach community. As an examle, Google esented its aoach to building a web-scale landmak ecognition engine [3]. The oblem of landmak ecognition is tyically addessed by leveaging on techniques of automatic classification, as fo instances knn Classification [4], alied to image featues. Between 27 and 2, the VISITO Tuscany, (VIsual Suot to Inteactive TOuism in Tuscany) oject, has focused on technologies able to offe an inteactive and customized advanced tou guide sevice to visit the cities of at in Tuscany. This oject has investigated cultual heitage object ecognition, (such as monuments, landmaks, etc.) develoing a mobile alication and elated eseach aes such as [2]. Howeve, eigahs ecognition was out of the scoe of the VISITO Tuscany oject. htt:// 5

3 3 Tested Aoaches In this Section, we eot infomation about the tested aoaches. Fist we discuss the SIFT that we selected as local featue. Then we biefly discuss the Bag-of- Featues (BoF) and Vecto of Locally Aggegated Descitos (VLAD) that make use of the local featues (SIFT in ou case) in ode to achieve high efficiency and effectiveness via aggegation of the infomation they contain. 3. SIFT The Scale Invaiant Featue Tansfomation (SIFT) [7] ae extacted fom key oints selected using diffeence of Gaussians alied in scale sace to a seies of smoothed esamled images. The descition of the egion aound this selected oints ely on histogam of gadients. SIFT ae not only the most imotant and cited local featues eve defined, but they ae still almost unbeaten in tems of effectiveness. Recently, binay local featues have been oosed in ode to imove efficiency of diect local featues matching. In ou exeiments we achieve scalability aggegating featues and thus we ae moe inteested in effective eesentation of the images than efficient comaison of the featues themselves. 3.2 Bag-of-Featues The Bag-of-Featues (BoF) was initially oosed in [] and has been studied in many othe aes. The goal of the BoF aoach is to substitute each local descito of an image with visual wods obtained fom a edefined vocabulay in ode to aly taditional text etieval techniques to CBIR. The fist ste is selecting some visual wods ceating a vocabulay. The visual vocabulay is tyically built clusteing, using k-means, local descitos of the dataset and selecting the centoids. The second ste assigns each local descito to the identifie of the neaest wod in the vocabulay. At the end of the ocess, each image is descibed as a set of visual wods. The etieval hase is then efomed using text etieval techniques consideing a quey image as disjunctive text-quey. Tyically, the cosine similaity measue in conjunction with a tem weighting scheme (e.g., TF- IDF [9]) is adoted fo evaluating the similaity between any two images. As mentioned in [2], a fundamental diffeence between an image quey (e.g. 5 visual tems) and a text quey (e.g. 3 tems) is lagely ignoed in existing index design. Efficiency and memoy constaints have been ecently addessed by aggegating local descitos into a fixed-size vecto eesentation that descibe the whole image. In aticula, Fishe Vecto (FV) and VLAD have shown bette efomance than BoF. In this wok we will focus on VLAD which has been oved to be a simlified non-obabilistic vesion of FV [6]. Desite its simlicity, VLAD efomance is comaable to that of FV. 5

4 3.3 Vecto of Locally Aggegated Desctios (VLAD) The VLAD eesentation was oosed in [5]. As fo BoF, a codebook {μ,, μ k } is fist leaned using a cluste algoithm (e.g., k-means). Each local descito x t in each image is then associated to its neaest visual wod NN(x t ) in the codebook. Fo each codewod, the diffeences x t μ i of the vectos x t assigned to μ i ae accumulated: v i = x t μ i x t : NN(x t )=μ i. () The VLAD eesentation is the concatenation of the accumulated vectos, i.e. V = [v T v K T ]. Powe-law and L 2 nomalization ae usually alied and fo comaing two VLAD descition L 2 Euclidean distance has been oved to be effective. VLAD descitions have a high dimensionality. Pincial Comonent Analysis has been oosed to have a moe comact eesentation. 4 Exeiments 4. Dataset Being atne of the EAGLE oject, we had the ootunity to access a dataset of 7,55 hotos elated to 4,56 eigahs made available to us by Saienza Univesity of Roma. Fo ou exeiments, we also needed a gound tuth, i.e., hotos in which we want to automatically ecognize the eigah togethe with the actual eigah eesented in the image. We constucted this gound tuth selecting 7 hotos fom the whole dataset and emoving them fom the knowledge base. In othe wods, we emoved these quey hotos fom the ones that ae given to the comute in ode to undestand the visual content of each eigah. This was only ossible fo the eigahs that had moe than one hoto. We also caefully selected queies that could eesent the vaious tyes of eigahs. In Fig. we eot 5 quey examles togethe with the othe images fo the same object available in the dataset. 52

5 Fig.. Examles of quey images and associated images of the same eigah 4.2 Quality Measues In ode to ecognize the actual object in a quey image, we basically efom a visual similaity seach between all the images in the dataset. Thus, the main goal is to have one image of the same eigah as fist esult. Wheneve this is not the case, it is inteesting to undestand at which osition in the esult list the most visually simila hoto of the same object aeas. In fact, while in this ae we ae focusing on techniques able to scale u to the size of the dataset, taditional comute vision techniques could be alied on the esults obtained in ode to achieve bette effectiveness. Given this consideations, we decided to eot the obability of finding an image of the same object between the fist esults. Fo =, also equals the accuacy of a classifie that ecognizes the quey eigah as the most simila that have been found. Fo each technique, we eot the obability of finding an image containing the same eigah given as quey between the fist esults vaying between and. Results ae eoted with the values on a logaithmic scale. A moe common measue of effectiveness is mean-aveage Pecision (map). In this case, not only the fist elevant image but all the images associated with the quey ae consideed. This measue eveal how good is the aoach in eoting the elated images in the to ositions of the esult list. 53

6 4.3 Exeimental Setu We extacted SIFT fom images using the OenCV libay 2. The BoF and VLAD aoaches have been imlemented by the NeMIS gou of the ISTI-CNR in Java as at of ou Visual Infomation Retieval libay ublically available on GitHub 3. Given an image, thousands of local featues ae extacted. In ou case, we obtained an aveage of 59 SIFT e image. Howeve, the fact that some of them efe to bigge egions than othes allows to select a subset of local featues that ae in incile moe elevant [2]. Thus, in the exeiments we also tied to educe the numbe of local featues selecting only the most imotant ones u to about 25 local featues e image. In the following, we efe to this second aoach as educed-keyoints. We tested all the aoaches both on the whole extacted local featues and on the ones obtained filteing by egion size. 4.4 Results Bag-of-Featues 4k 2k k 5k k 4k 2k k 5k k Fig. 2. BoF, cos TF-IDF Fig. 3. BoF, cos TF-IDF, educed-keyoints Results in Fig. 2 have been obtained using the BoF featue aoach using the cosine TF-IDF similaity measue, vaying the size of the vocabulay between k and 4k. As exected, the lage the vocabulay the bette the esults. Howeve, diffeences between 2k o 4k ae only maginal. Thus, we did not tested lage vocabulaies. In Fig. 3 we eot the same tye of esults consideing only the most imotant local featues. Results show that the SIFT eduction is useful. 2 htt://oencv.og/ 3 htts://github.com/ffalchi/it.cn.isti.vi 54

7 VLAD Fig. 4. VLAD Fig. 5. VLAD, educed-keyoints In Fig. 4 and Fig. 5, we eot the esults obtained by the VLAD aoach vaying the size k of the codebook between 32 and 256 using all the extacted SIFT and the educed ones esectively. It is inteesting to see that the SIFT selection is useful when k is highe and is smalle. In the othe cases it can even decease the quality of the esults. Consideing that we ae moe inteested on small, which means having elevant images on the vey fist ositions, the oveall best esults ae the ones obtained fo k = 256 and educing numbe of local featues. d=6384 (no PCA) d=248 d=52 d=64 d=6384 (no PCA) d=248 d=52 d= Fig. 6. VLAD-PCA, k=28 Fig. 7. VLAD-PCA, k=28, educed-k We also tied to aly PCA on the VLAD vectos, esecially fo tying to educe the comlexity of the comaison. We then selected k = 28 and alied PCA in ode to educe the dimensionality of the vecto (6,384). Results ae eoted fo VLAD vecto educed to 248, 52 and 64 dimensions. Unfotunately, the dimensionality eduction significantly educes the quality of the esults. 55

8 Comaison In Table, we summaize the esults obtained, odeed with esect to the map quality measue. Only the best aoaches ae shown. In the fist column, we eot a bief text about the aoach. In the second column, the aveage numbe of SIFT consideed is shown (i.e., 235 when local featues eduction was alied and,59 othewise). The thid column eots the numbe of wods used. While the wods have been selected both fo BoF and VLAD using k-means, thei use is vey diffeent. Thus, in the bytes column, we comuted the aveage size in bytes of the esulting eesentation. As quality measues, we used the obability of having at least one elated image between the fist esults fo =,, and the map. In case we use these aoaches to ecognize the quey image elying on the neaest image in the dataset, the best aoach is the VLAD fo a codebook size of 256 and selecting the 25 most elevant local featues. In this case, the accuacy obtained is.69. The moe taditional BoF-cos TF-IDF aoach obtained good esults when a lage codebook (i.e., 4k) was used (as exected). It is inteesting to note that this aoach outefoms VLAD fo =,. Given that ecent woks as [3] have shown that VLAD can be moe efficiently indexed than BoF, still VLAD is efeable. Table. Comaison of esults obtained by the oveall best aoaches odeed by map avg codebook Aoach Bytes SIFTs size = = = map VLAD , BoF / cos TF-IDF 235 4, VLAD , BoF / cos TF-IDF 235 2, VLAD , VLAD , BoF / cos TF-IDF 235, VLAD , VLAD , VLAD-PCA (d'=52) , Conclusions and Futue Wok In this wok, we tested state-of-the-at object ecognition techniques on an eigahs dataset consisting of 7,55 hotos. The best accuacy was obtained by using the VLAD aoach that has been ecently oosed fo efoming object ecognition on a lage scale. The obtained accuacy was of.69, which is good consideing the difficulties of the task and the few images available fo each eigah in the dataset. In fact, the dataset consists of 7,55 hotos elated to 4,56 eigahs. This esults in most of the ei- 56

9 gahs been eesented by only one o two hotos. Howeve, we lan to imove this esults efoming e-anking of the images obtained using these scalable techniques efoming diect local featues matching. To this goal, we also eoted the obability of having a elevant images between the etieve images. The esults show that it is ossible to have a elevant image between etieved ones with obability.9 using the VLAD aoach with a codebook of size 256 and filteing the SIFT. Thus, we lan to ty binay local featues and othe techniques in ode to imove the obtained.69 accuacy u to the.9 obtainable, in theoy, by e-anking the obtained using VLAD. Bibliogahy. G. Amato, P. Bolettiei, F. Falchi and C. Gennao, "Lage Scale Image Retieval Using Vecto of Locally Aggegated Descitos," in Similaity Seach and Alications, vol. 899, N. Bisaboa, O. Pedeia and P. Zezula, Eds., Singe Belin Heidelbeg, 23, G. Amato, F. Falchi and C. Gennao, "Geometic consistency checks fo knn based image classification elying on local featues," in SISAP ': Fouth Intenational Confeence on Similaity Seach and Alications, SISAP 2, Liai Island, Italy, June 3 - July, 2, G. Amato, F. Falchi and C. Gennao, "On Reducing the Numbe of VisualWods in the Bag-of-Featues Reesentation," in VISAPP 23 - Poceedings of the Intenational Confeence on Comute Vision Theoy and Alications, S. Dudani, "The Distance-Weighted K-Neaest-Neighbou Rule," IEEE Tansactions on Systems, Man and Cybenetics, Vols. SMC-6(4), , H. Jégou, M. Douze, C. Schmid and P. Péez, "Aggegating local descitos into a comact image eesentation," in Comute Vision and Patten Recognition (CVPR), 2 IEEE Confeence on, H. Jégou, F. Peonnin, M. Douze, J. Sànchez, P. Péez and C. Schmid, "Aggegating local image descitos into comact codes," IEEE Tansactions on Patten Analysis and Machine Intelligence, Se D. G. Lowe, "Distinctive Image Featues fom Scale-Invaiant Keyoints," Intenational Jounal of Comute Vision, vol. 6, no. 2,. 9-, K. Mikolajczyk and C. Schmid, "A efomance evaluation of local descitos," Patten Analysis and Machine Intelligence, IEEE Tansactions on, vol. 27, no., , oct G. Salton and M. J. McGill, Intoduction to Moden Infomation Retieval, New Yok, NY, USA: McGaw-Hill, Inc., J. Sivic and A. Zisseman, "Video Google: A Text Retieval Aoach to Object Matching in Videos," in Poceedings of the Ninth IEEE Intenational Confeence on Comute Vision - Volume 2, Washington, DC, USA, 23.. T. Tuytelaas and K. Mikolajczyk, "Local invaiant featue detectos: a suvey," Found. Tends. Comut. Gah. Vis., vol. 3, no. 3, , X. Zhang, Z. Li, L. Zhang, W. Y. Ma and H. Y. Shum, "Efficient indexing fo lage scale visual seach," in Comute Vision, 29 IEEE 2th Intenational Confeence on,

10 3. Y. Zheng, M. Zhao, Y. Song, H. Adam, U. Buddemeie, A. Bissacco, F. Buche, T. S. Chua and H. Neven, "Tou the wold: Building a web-scale landmak ecognition," in CVPR, 29. Acknowledgement This wok was atially suoted by the Euoeana netwok of Ancient Geek and Latin Eigahy (EAGLE, gant ageement numbe: 32522) co-funded by the Euoean Commission within the ICT Policy Suot Pogamme. 58