SIFT Match Verification by Geometric Coding for Large-Scale Partial-Duplicate Web Image Search

Transcription

1 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Web Image Search WENGANG ZHOU and HOUQIANG LI, Unversty of Scence and Technology of Chna YIJUAN LU, Texas State Unversty QI TIAN, Unversty of Texas at San Antono Most large-scale mage retreval systems are based on the bag-of-vsual-words model. However, the tradtonal bag-of-vsualwords model does not capture the geometrc context among local features n mages well, whch plays an mportant role n mage retreval. In order to fully explore geometrc context of all vsual words n mages, effcent global geometrc verfcaton methods have been attractng lots of attenton. Unfortunately, current exstng methods on global geometrc verfcaton are ether computatonally expensve to ensure real-tme response, or cannot handle rotaton well. To solve the precedng problems, n ths artcle, we propose a novel geometrc codng algorthm, to encode the spatal context among local features for largescale partal-duplcate Web mage retreval. Our geometrc codng conssts of geometrc square codng and geometrc fan codng, whch descrbe the spatal relatonshps of SIFT features nto three geo-maps for global verfcaton to remove geometrcally nconsstent SIFT matches. Our approach s not only computatonally effcent, but also effectve n detectng partal-duplcate mages wth rotaton, scale changes, partal-occluson, and background clutter. Experments n partal-duplcate Web mage search, usng two datasets wth one mllon Web mages as dstractors, reveal that our approach outperforms the baselne bag-of-vsual-words approach even followng a RANSAC verfcaton n mean average precson. Besdes, our approach acheves comparable performance to other state-of-the-art global geometrc verfcaton methods, for example, spatal codng scheme, but s more computatonally effcent. 4 Categores and Subect Descrptors: I.2.10 [Vson and Scene Understandng] VISION General Terms: Algorthms, Expermentaton, Verfcaton Addtonal Key Words and Phrases: Image retreval, partal duplcate, large scale, rotaton-nvarant, geometrc square codng, geometrc fan codng ACM Reference Format: Zhou, W., L, H., Lu, Y., and Tan, Q SIFT match verfcaton by geometrc codng for large-scale partal-duplcate Web mage search. ACM Trans. Multmeda Comput. Commun. Appl. 9, 1, Artcle 4 (February 2013), 18 pages. DOI = / Ths work s supported n part by the Fundamental Research Funds for the Central Unverstes of Chna (WK ) to H. L, n part by Research Enhancement Program (REP), start-up fundng from the Texas State Unversty and DoD HBCU/MI grant W911NF to Y. Lu, and n part by NSF IIS , Faculty Research Awards by Google FXPAL, NEC Laboratores of Amerca, and ARO grant W911BF to Q. Tan. Authors addresses: W. Zhou, H. L, Department of EEIS, Unversty of Scence and Technology of Chna, Hefe , P. R. Chna; Y. Lu, Department of Computer Scence, Texas State Unversty at San Marcos, TX 78666; Q. Tan (correspondng author), Department of Computer Scence, Unversty of Texas at San Antono, TX 78249; emal: qtan@cs.utsa.edu. Permsson to make dgtal or hard copes of part or all of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes show ths notce on the frst page or ntal screen of a dsplay along wth the full ctaton. Copyrghts for components of ths work owned by others than ACM must be honored. Abstractng wth credt s permtted. To copy otherwse, to republsh, to post on servers, to redstrbute to lsts, or to use any component of ths work n other works requres pror specfc permsson and/or a fee. Permssons may be requested from Publcatons Dept., ACM, Inc., 2 Penn Plaza, Sute 701, New York, NY USA, fax +1 (212) , or permssons@acm.org. c 2013 ACM /2013/02-ART4 $15.00 DOI /

2 4:2 W. Zhou et al. 1. INTRODUCTION As more and more people become users of Tneye [2008] and Google Smlar Image Search [2009], partal-duplcate mage search has been attractng more and more attenton n recent years. Partalduplcate mages are those mages, part of whch are usually cropped from the same orgnal mage, that are edted wth modfcaton n color, scale, rotaton, partal occluson, etc. Fgure 1 llustrates some nstances of partal-duplcate mages from the Web. From these examples, we can fnd that they are partal duplcates of the orgnal mage wth dfferent appearances but stll sharng some duplcated patches. The task of partal-duplcate Web mage search s to fnd all the partal-duplcate versons of a gven query from a large Web mage database. Partal-duplcate mage search can be wdely used n many applcatons, such as mage/vdeo copyrght volaton detecton, fndng out where an mage comes from or how t s beng used, duplcate mage annotaton, etc. Besdes, t can also facltate many multmeda applcatons, such as semantc concept nference [Tang 2009] and mage annotaton [Tang 2010]. Most of the state-of-the-art approaches n large-scale mage retreval rely on the bag-of-vsual-words model [Svc and Zsserman 2003]. It quantzes local features [Lowe 2004] extracted from mages to vsual words and ndexes mages va nverted fle structure. Although the bag-of-vsual-words model makes t possble to represent, ndex, and retreve mages lke documents, t suffers from vsual word ambguty and feature quantzaton error. Those unavodable problems greatly decrease retreval precson and recall, snce dfferent features may be quantzed to the same vsual word, causng many false local matches between mages. To tackle these problems, many geometrc verfcaton [Chum et al. 2009; Jegou et al. 2008; Phlbn et al. 2007; Svc and Zsserman 2003; Wu et al. 2009; Zhou et al. 2010] approaches have been proposed n recent few years to address the negatve nfluence of these false matches to mprove retreval performance. Many of them are local geometrc verfcaton approaches, such as spatally nearest neghbors [Svc and Zsserman 2003], geometrc mn-hashng [Chum et al. 2009], and bundled feature [Wu et al. 2009]. Snce these approaches only can verfy spatal consstency of features wthn some local areas n mages, they wll fal f there s geometry nconsstency among local areas. Therefore, global geometrc verfcaton methods are demanded. RANSAC [Fschler and Bolles 1981] s the most popular method for global geometrc verfcaton. It can fully estmate the transformaton model between mages and then detect spatally nconsstent pars. However, due to the hgh computatonal cost, usually t s only appled to some top-ranked mage results, whch s not suffcent for good recall n large-scale mage retreval. To address such a problem, spatal codng [Zhou et al. 2010] s proposed to effcently check spatal consstency globally. It uses spatal maps to record the spatal relatonshp of all matched feature pars. But spatal codng s very senstve to rotaton due to the ntrnsc lmtaton of the spatal map. Although t can weakly handle rotated mages by tryng a set of predefned angles on the query mage, much more tme cost wll be ntroduced to retreve freely rotated duplcate mages. In ths artcle, we propose a novel geometrc codng scheme for global spatal verfcaton of SIFT matches, whch s both effcent and effectve for partal-duplcate mage search. We select the SIFT feature [Lowe 2004] for mage representaton and make full use of SIFT propertes. Generally, a SIFT feature s characterzed wth several property values: a 128D descrptor, a 1D domnant orentaton (rangng for π to π), a 1D characterstc scale, and the (x, y) coordnates of the key pont. In our approach, a SIFT descrptor s used based on the bag-of-vsual-words model, whle the orentaton, scale, and key pont poston are all exploted to buld our geometrc codng maps. In mage search, local matches are frst dscovered through feature quantzaton. To verfy the SIFT matches of two mages, we use Geometrc Square Codng (GSC) and Geometrc Fan Codng (GFC) to encode the relatve

3 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:3 Fg. 1. Examples of partal-duplcate Web mages. Top: KFC logo; bottom: Beng 2008 Olympc logo. The partally duplcated patches n the top row are hghlghted wth red boundng box. spatal postons of local features n mages. Then through spatal verfcaton based on three geometrc maps, the false matches of SIFT features can be removed effectvely and effcently, resultng n better accuracy. The rest of the artcle s organzed as follows. Secton 2 revews the related work. Secton 3 dscusses our approach n detals. Expermental results are provded n Secton 4. Fnally, we draw the concluson n Secton RELATED WORK In recent years, large-scale mage retreval [Chum et al. 2007a, 2007b; Jegou et al. 2008; Nster and Stewenus 2006; Phlbn et al. 2007; Wu et al. 2009; Zhang et al. 2009, 2010, 2011] wth local features has been sgnfcantly advanced based on the bag-of-vsual-words model [Svc and Zsserman 2003]. The maor contrbuton of the bag-of-vsual-words model s that t can acheve scalablty for large-scale mage retreval by quantzng local features to vsual words. Popular local features nclude SIFT [Lowe 2004], SURF [Bay et al. 2006], MSER [Matas et al. 2002], and so on. Local feature quantzaton not only makes mage representaton compact, but also makes t possble to ndex mages wth nverted fle structure, whch greatly reduces the number of canddate mages for comparson. However, local feature quantzaton reduces the dscrmnatve power of local descrptors. Dfferent descrptors may be quantzed to the same vsual word and cannot be dstngushed from each other. On the other hand, wth vsual word ambguty, descrptors from local patches of dfferent semantcs may also be very smlar to each other. Such quantzaton error and vsual word ambguty wll cause many false matches of local features between mages and therefore decrease retreval precson and recall. To reduce the quantzaton error, two knds of approaches have been proposed recently. The frst one s to mprove the dscrmnatve power of local features. Soft quantzaton [Phlbn et al. 2008; Jegou et al. 2007] and Hammng embeddng [Jegou et al. 2008] are two representatve works. Soft quantzaton quantzes a SIFT descrptor to multple vsual words. Hammng embeddng enrches the vsual word wth compact nformaton from ts orgnal local descrptor wth Hammng codes. The second category of approaches focus on utlzng geometrc nformaton n mages to mprove retreval precson. These approaches can be summarzed nto preprocessng or postprocessng approaches. Inspred by shape context [Belonge et al. 2002; Olva and Torralba 2001], the motvaton of preprocessng approaches s to encode spatal context of local features nto mage representaton. In Hoang et al. [2010], a new mage content representaton scheme s proposed to descrbe the spatal layout wth trangular relatonshps of vsual enttes for scene retreval. In Gao et al. [2010], a spatal-bag-of-features scheme s used to encode geometrc nformaton of obects wthn an mage. It

4 4:4 W. Zhou et al. proects local features of an mage to dfferent drectons to generate an ordered bag-of-features for mage search. However, wth the large amount of local features n mages, t s hard for the preprocessng approaches to fully encode varous spatal relatonshps. It makes mage representaton very complex and ncreases mage matchng tme. In Zhang et al. [2011], a Geometry-preservng Vsual Phrase (GVP) s proposed to encode the spatal nformaton of local features, ncludng both co-occurrences and the local and long-range spatal layouts of vsual words. Wth lttle ncrease n memory usage and computatonal tme, retreval accuracy mprovement s wtnessed. However, GVP only captures the translaton nvarance. Although ts extenson to scale and rotaton nvarance can be acheved by ncreasng dmenson of the offset space, more memory usage and runtme wll be ncurred. In Wang et al. [2011], the statstcs n the local neghborhood of an nvarant feature s used as ts spatal context to enhance the dscrmnatve power of the vsual word. The postprocessng approaches try to avod these problems. They do not change the mage representaton and mage matchng scheme. Instead, after obtanng the local matches between mages, they use geometrc consstency to flter those false matches. Snce the number of matched features s much smaller than the number of features n the mage, postprocessng approaches can be very effcent. The locally spatal consstency of some spatally nearest neghbors s used n Svc and Zsserman [2003] to suppress false vsual-word matches. However, the nearest-neghbors spatal consstency only mposes very loose geometrc constrants and may be senstve to the mage nose from background clutter. In Jegou et al. [2008], Weak Geometrc Consstency (WGC) s used to flter false local matches. A constrant s mposed that correct local matches should exhbt smlar characterstc scale and rotaton changes. Therefore, hstograms of characterstc scale and domnant orentaton dfferences have obvous peaks, whch are used to dentfy false local matches wth orentaton or scale dfferences far from those peaks. In Zhao et al. [2010], WGC s enhanced by ncludng translaton nformaton. An addtonal assumpton s made that the correct matches follow consstent translaton transformaton. Bundled feature [Wu et al. 2009] assembles features n local MSER [Matas et al. 2002] regons to ncrease the dscrmnatve power of local features. The local geometrc consstency s measured by proectng feature postons along horzontal and vertcal drectons n local MSER regons. However, when an mage suffers from rotaton changes, such proecton wll yeld a dfferent local geometrc representaton and cause ncorrect geometrc measurement. Geometrc mn-hashng [Chum et al. 2009] constructs repeatable hash keys wth loosely local geometrc nformaton for more dscrmnatve-break descrpton. All of the precedng postprocessng approaches only verfy spatal consstency of features wthn local areas nstead of the entre mage plane. Although computatonally effcent, they cannot capture the spatal relatonshp between all features, whch makes t hard to detect all false matches and hence obtans lmted precson mprovement. To capture geometrc relatonshps of all features n the entre mage, a global geometrc verfcaton method such as RANSAC [Chum et al. 2004; Fschler and Bolles 1981; Phlbn et al. 2007] s often used for ths task. It randomly samples a subset of matchng feature pars many tmes to estmate an optmal transformaton model. RANSAC can greatly mprove retreval precson. However, t s computatonally expensve. In practce, t s usually appled on the subset of the top-ranked canddate mages, whch may not be suffcent to acheve good recall n large-scale mage retreval systems. In the content-based mage search system VsualSEEk [Smth and Chang 1996], 2D strngs [Chang et al. 1987] are adopted to represent mages wth multple color regons for comparson. 2D strngs represent an mage as a symbolc proecton along the x and y drectons. Then, the mage retreval problem s converted to a problem of 2D sequence matchng. The spatal codng approach [Zhou et al. 2010] s another global geometrc verfcaton method proposed to remove false matches based on spatal maps. The problem of spatal codng s that t requres

5 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:5 Vsual Codebook Database Index Geometrc Codng Query Image Features Extrac on Feature Quan za on Look up Index Geometrc Square Codng Geometrc Fan Codng Spa al Verfca on Search Results Fg. 2. Our mage search framework. that the duplcated patches n the query and the matched mage share the same or very smlar spatal confguraton and cannot handle rotaton very effcently. Although weak rotaton nvarance can be acheved by rotatng the query mage wth some predefned angles [Zhou et al. 2010], n practce, t wll be tme consumng to enumerate all possble rotaton angles to search freely rotated target mages. In ths artcle, our motvaton s to desgn an effcent global geometrc verfcaton scheme, whch can acheve both rotaton and scale nvarance, and s not senstve to background clutter. We propose two codng schemes, that s, geometrc square codng and geometrc fan codng, to strctly descrbe the geometrc context of local features for global spatal verfcaton. Our approach can effcently and effectvely address mages wth arbtrary rotaton changes. 3. OUR APPROACH Based on the bag-of-vsual-words model, the framework of our large-scale partal-duplcate mage search system s llustrated n Fgure 2. Our man contrbuton les n geometrc codng and spatal verfcaton, as hghlghted wth red boundng box. In our approach, we adopt SIFT feature [Lowe 2004] for mage representaton. In Secton 3.1, we apply the SIFT descrptor for vector quantzaton. In Secton 3.2, the key pont locaton, orentaton, and scale of the SIFT feature are exploted for geometrc codng maps generaton. In Secton 3.3, we explan how to perform spatal verfcaton wth those geometrc codng maps. More detals of the framework are dscussed n Secton Feature Quantzaton We ndex mages on a large scale wth an nverted ndex fle structure for retreval. Before ndexng, SIFT features are quantzed to vsual words based on the bag-of-vsual-words model [Svc and Zsserman 2003]. A quantzer s defned to map a SIFT descrptor to a vsual word. The quantzer can be generated by clusterng a sample set of SIFT descrptors and the resultng cluster centrods are regarded as vsual words. Durng the quantzaton stage, a novel feature wll be assgned to the ndex of the closest vsual words. In our mplementaton, we use the herarchcal vsual vocabulary tree approach [Nster and Stewenus 2006] for vsual vocabulary generaton and feature quantzaton. Wth feature quantzaton, any two features from two dfferent mages quantzed to the same vsual word wll be consdered as a local match across two mages. 3.2 Geometrc Codng The spatal context among local features of an mage s crtcal n dentfyng duplcate mage patches. After SIFT quantzaton, SIFT matches between two mages can be obtaned. However, due to quantzaton error and vsual word ambguty, the matchng results are usually polluted by some false matches. Generally, geometrc verfcaton can be adopted to refne the matchng results by dscoverng the transformaton and flterng false postves [Phlbn et al. 2007]. Snce full geometrc

6 4:6 W. Zhou et al (a) (b) (c) (d) (e) (f) Fg. 3. Illustraton of mage plane dvson wth the key pont of feature 2 as reference pont. (a) Fve SIFT features n mage; (b) key pont of feature 2 dsplayed as vector ndcatng scale, orentaton, and locaton (red arrow); (c) mage plane dvson wth lnes and square (green dashed lnes) wth the key pont of feature 2 as reference pont; (d) mage plane rotaton from (c); (e) and (f): mage subdvsons from (d) verfcaton wth RANSAC [Chum et al. 2004; Fschler and Bolles 1981] s computatonally expensve, t s only used as a postprocessng stage to process ntally top-ranked canddate mages. A more effcent scheme to encode the spatal relatonshps of vsual words s desred. Wth such motvaton, we propose the geometrc codng scheme. The key dea of geometrc codng s to encode the geometrc context of local SIFT features for spatal consstency verfcaton. Our geometrc codng s composed of two types of codng strateges, that s, geometrc square codng and geometrc fan codng. The dfference between the two strateges les n the way the mage plane s dvded accordng to an nvarant reference feature. Before encodng, the mage plane has to be dvded wth a certan crteron that can address both rotaton nvarance and scale nvarance. We desgn the crteron va the ntrnsc nvarance mert of the SIFT feature. Fgure 3 gves a toy example of mage plane dvson wth the key pont of feature 2 as reference pont. Fgure 3(b) llustrates an arrow orgnated from the key pont of feature 2, whch corresponds to a vector ndcatng the characterstc scale and domnant orentaton of the SIFT feature. Usng the key pont of feature 2 as orgn and drecton of the arrow as the maor drecton, two lnes horzontal and vertcal to the arrow of feature 2 can be drawn. Besdes, centered at the same key pont, a square s also drawn along these two lnes, as shown n Fgure 3(c). The sde length of the square s proportonal to the characterstc scale of feature 2. For comparson convenence, we rotate all features to algn the red arrow to be horzontal, as shown n Fgure 3(d). After that, the mage plane dvson wth two coordnate axal lnes and a square can be decomposed nto two knds of subdvsons, as shown n Fgure 3(e) and (f), whch wll be used for geometrc square codng and geometrc fan codng, respectvely. The detals are dscussed n the followng two subsectons Geometrc Square Codng. Geometrc Square Codng (GSC) encodes the geometrc context n the axal drecton of reference features. In GSC, wth each SIFT feature as reference orgn, the mage plane s dvded by squares. A square codng map, called S-map, s constructed by checkng whether other features are nsde or outsde of the square. To acheve rotaton-nvarant representaton, before checkng relatve poston, we adust the locaton of each SIFT feature accordng to the SIFT orentaton of the reference feature. For nstance, gven an mage I wth M features { f (x, y )}, ( = 1, 2,...,M), wth feature f (x, y ) as reference pont, the adusted poston f () (x (), y () )of f (x, y ) s formulated as ( ) x () y () = ( ) ( ) cos(φ ) sn(φ ) x, 1, M, (1) sn(φ ) cos(φ ) y where φ s a rotaton angle equal to the SIFT orentaton of the reference feature f.

7 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:7 S-map descrbes whether other features are nsde or outsde of a square defned by the reference feature. For mage I, ts S-map s defned as { ( ) x () 1 fmax Smap(, ) = x (), y () y () < s, (2) 0 otherwse where s s a half-sde-length proportonal to the SIFT scale of feature f : s = α scl, α s a constant. The mpact of α wll be studed n the experment n Secton To descrbe the relatve postons more strctly, we advance to general squared maps. For each feature, n concentrc squares are drawn, wth an equally ncremental step of the half-sde-length on the mage plane. Then, the mage plane s dvded nto (n+1) nonoverlappng regons. Correspondngly, accordng to the mage plane dvson, a generalzed geo-map should encode the relatve spatal postons of feature pars. The general S-map s defned as GS ( x () max x () y () y () GS(, ) =, (3) s where s sthesameasthatneq.(2). Intutvely, we can also select a rng or crcle for mage plane dvson. In such a case, there s no need to adust the coordnates of local features. We defne the correspondng geometrc map as GR GR(, ) = d, s, ), (4) where x denotes the nearest nteger less than or equal to x, d, = (x x ) 2 + (y y ) 2, s = α scl, scl s the scale parameter of SIFT feature v, α s a constant. From the prevous dscusson, t can be seen that GS and GR are two knds of geometrc codng maps based on dfferent strateges of mage plan dvson. Although smlar results can be expected wth GS and GR, square n GSC fts the mage shape (.e., rectangle) better than crcles or rngs. In our experments, we selected the GS defned n Eq. (3) nstead of GR for geometrc verfcaton Geometrc Fan Codng. Geometrc square codng only consders the relatve spatal poston n radal drecton, and gnores the constrants along horzontal and vertcal drecton. To overcome ths drawback, we propose a Geometrc Fan Codng (GFC) scheme. In geometrc fan codng, we take each SIFT feature as reference pont and dvde the mage plane nto some regular fan regons. Then two fan codng maps, that s, H-map and V-map, are constructed by checkng whch fan regon other features fall nto. Our Geometrc Fan Codng (GFC) s nspred by the spatal codng scheme [Zhou et al. 2010]. The key dfference s that, key pont locatons are frst adusted as n Fgure 3 before comparng the relatve spatal postons. Wth each SIFT feature as reference pont, other SIFT key ponts locatons are rotated counterclockwse by the SIFT orentaton angle of the reference feature. The motvaton s to acheve rotaton nvarance, wthout the strong constrants that the duplcated patches n two comparson mages share the same or very smlar spatal confguraton, as mposed n Zhou et al. [2010]. Geometrc fan codng encodes the relatve spatal postons between each par of features n an mage. Based on the adusted new postons of SIFT feature n Eq. (1), two bnary geometrc maps, called H-map and V-map, are generated. H-map and V-map descrbe the relatve spatal postons between each feature par along the horzontal and vertcal drectons, respectvely. They are formulated as follows. Hmap(, ) = { 0 f x () 1 f x () x () > x () (5)

8 4:8 W. Zhou et al. Vmap(, ) = { 0 f y y 1 f y > y (6) The geometrc maps can be nterpreted as follows. In row, feature f s selected as the reference pont, and the mage plane s decomposed nto four quadrants along horzontal and vertcal drectons. H-map and V-map then show whch quadrant other features fall nto. In fact, the representaton of geometrc context among local features wth H-map and V-map s stll too weak. We can put forward the geometrc fan codng to more general formulatons, so as to mpose strcter geometrc constrants. The mage plane can be dvded nto 4 r parts, wth each quadrant evenly dvded nto r fan regons. Accordngly, two general fan codng maps GH and GV are requred to encode the relatve spatal postons of all SIFT features n an mage. For a dvson of mage plane nto 4 r parts, we decompose the dvson nto r ndependent subdvsons, each unformly dvdng the mage plane nto four quadrants. Each subdvson s then encoded ndependently and ther combnaton leads to the fnal fan codng maps. In each subdvson, to encode the spatal context of all features by the left-rght and below-above comparson, we ust need to rotate all the feature coordnates and the dvson lnes counterclockwse, untl the two dvson lnes become horzontal and vertcal, respectvely. The general fan codng maps GH and GV are both 3D and defned as follows. Specally, wth feature f as reference, the locaton of feature f s rotated counterclockwse by θ (k) 0, 1,...,r 1) accordng to the mage orgn pont, yeldng the new locaton f (,k) ( ) ) x (,k) y (,k) = ( cos(θ (k) ) sn(θ (k) ) GH(,, k) = sn(θ (k) ) cos(θ (k) ) = k π (x (,k) 2 r + φ degree (k =, y (,k) )as, ( ) x. (7) y Here φ s the SIFT orentaton angle of f, as used n Eq. (1). Then GH and GV are formulated as { 0 f x (,k) x (,k) GV (,, k) = 1 f x (,k) > x (,k) { 0 f y (,k) y (,k) 1 f y (,k) > y (,k), (8). (9) In geometrc fan codng, the factor r controls the strctness of geometrc constrants and wll affect verfcaton performance. We wll study ts mpact n Secton From the precedng dscusson, t can be seen that both geometrc square codng and geometrc fan codng can be effcently performed. However, t wll take consderable memory to store the whole geometrc maps of all features n an mage. Fortunately, that s not necessary at all. Instead, we only need keep the orentaton, scale, and x- and y-coordnate of each SIFT feature, respectvely. When checkng the feature matchng of two mages, we ust need geometrc clues of these SIFT matches, whch wll be employed to generate geometrc maps for spatal verfcaton n real tme. Snce the SIFT matches are often only a small set of the whole feature set of an mage, the correspondng memory cost on these geometrc codng maps s relatvely low. The detals are dscussed n the next secton. 3.3 Spatal Verfcaton Snce the focused problem s partal-duplcate mage retreval, there s an underlyng assumpton that the target mage and the query mage share some duplcated patches, or n other words, share some local features wth consstent geometry. Due to the unavodable quantzaton error and vsual word

9 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:9 ambguty, there always exst some false SIFT matches, whch wll degrade mage smlarty measurement. To more accurately defne the smlarty between mages, spatal verfcaton wth geometrc codng can be used to remove such false matches. Denote that a query mage I q and a matched mage I m are found to share N matchng pars of local features. Then the geo-maps of these matched features for both I q and I m can be generated and denoted as (GS q, GH q, GV q )and(gs m, GH m, GV m ) by Eq. (3), Eq. (8), and Eq. (9), respectvely. After that, we can compare these geometrc maps to remove false matches as follows. Snce the general geometrc fan codng maps are bnary, for effcent comparson, we perform a logcal Exclusve-OR (XOR) operaton on GH q and GH m, GV q and GV m, respectvely. V H (,, k) = GH q (,, k) GH m (,, k) (10) V V (,, k) = GV q (,, k) GV m (,, k) (11) Ideally, f all N matched pars are true, V H and V V wll be zero for all ther entres. If some false matches exst, the entres of these false matches on GH q and GH m may be nconsstent, and so are those on GV q and GV m. Those nconsstences wll cause the correspondng exclusve-or result of V H and V V to be 1. We defne the nconsstency from geometrc fan codng as follows. F H (, ) = r k=1 V H (,, k) (12) F V (, ) = r V V (,, k) (13) k=1 The nconsstency from geometrc square codng s defned as F S (, ) = GSq (, ) GS m (, ). (14) Consequently, by checkng F H, F V,andF S, the false matches can be dentfed and removed. Denote { 1 f FS (, ) >τand F T (, ) = H (, ) + F V (, ) >β, (15) 0 otherwse where β and τ are constant ntegers. When τ or β s greater than zero, T n Eq. (15) can tolerate some drftng error of relatve postons of local features. The mpact of τ and β wll be studed n the later experments n Secton and Secton 4.2.3, respectvely. Ideally, f all matched pars are true postves, the entres n T wll be all zeroes. If false matches exst, the entres of those false matches on those codng maps may be nconsstent. Those nconsstences wll cause the correspondng entres n T to be 1. We can teratvely remove such match that causes the most nconsstency, untl all reman matches are consstent wth each other. When two mages contan multple partal-duplcated obects and each obect has dfferent changes n scale or orentaton, the precedng manpulaton wll only dscover the domnant duplcated obects wth the largest number of local matches. However, the extenson to dentfy multple partal-duplcate obects s straghtforward. To address ths ssue, we can frst fnd those matches correspondng to the domnant duplcated obect and then focus on the sub-geo-maps of the remanng matches. Those matches correspondng to the second domnant obect can be dentfed n a smlar way. Such an operaton can be performed teratvely, untl all partal-duplcate obects are dscovered. Fgure 4 shows two nstances of the spatal verfcaton wth geometrc codng on a relevant mage par and an rrelevant mage par. Intally, both mage pars have many matches of local features. For the upper Apollo example, after spatal verfcaton va geometrc codng, 9 false matches are dentfed and removed, whle 12 true matches are satsfactorly kept. For the second nstance, although

10 4:10 W. Zhou et al. Fg. 4. An llustraton of spatal verfcaton wth geometrc codng on a relevant par (frst row) and an rrelevant par (second row). (left column): Intal matches after quantzaton; (mddle column): False matches detected by spatal verfcaton; (rght column): True matches that pass the spatal verfcaton. (Best vewed n color PDF) they are rrelevant n content, 17 SIFT matches stll exst after quantzaton. However, by spatal verfcaton, only one par of matches s kept. Wth those false matches removed, the smlarty between mages can be more accurately defned and that wll beneft retreval accuracy. The phlosophy behnd the effectveness of our geometrc verfcaton approach s that the probablty of two rrelevant mages sharng many spatally consstent vsual words s very low. 4. EXPERIMENTS 4.1 Experment Setup Dataset. Our basc dataset contans one mllon mages crawled from the Web. Two ground-truth datasets, that s, DupImage dataset [2011] and Copydays dataset [2008], are used for evaluaton, respectvely. The descrptons of these two datasets are gven n the followng. (1) DupImage dataset. DupImage dataset contans 1104 manually labeled partal-duplcate Web mages of 33 groups collected from the Web. Images n each group are partal duplcates of each other. Some typcal examples are shown n Fgure 1, Fgure 4, and Fgure 11. These ground-truth mages are open to the publc wth the lnk: wzhou/data/dupgroundtruth Dataset.tgz. (2) Copydays dataset. There are 3369 mages n the Copydays dataset generated from 175 orgnal mages. Each orgnal mage s subected to three knds of artfcal attacks: JPEG, croppng and strong. The cropped mages suffer from 10% to 80% of the mage surface removed. In JPEG attacks, each mage s scaled to 1/16 (pxels) wth nne dfferent JPEG qualty factors. Images from the thrd attacks of strong are obtaned by prntng and scannng, blurrng, pantng, rotatng, and so on. In summary, for each orgnal mage, there are nne cropped mages, nne JPEG attached mages, and 2 to 6 mages wth strong attacks. Local features. We use the standard SIFT feature [Lowe 2004] for mage representaton. Key ponts are detected wth the Dfference-of-Gaussan (DoG) detector, and a 128-dmensonal orentaton hstogram (SIFT descrptor), together wth scale and domnant orentaton, s extracted to descrbe the local patch around the key ponts. Before feature extracton, large mages are scaled to no larger than Snce the ground-truth mages do not exhbt dverse rotaton changes, t wll be nsuffcent to demonstrate the rotaton-nvarant capablty of our geometrc codng approach. To address ths ssue,

11 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:11 Fg. 5. Inverted fle structure for ndex. nstead, for each query mage, we randomly generate an angle rangng from π to π, and rotate the query mage counterclockwse by the angle. Snce the descrptor and scale n the SIFT feature are nvarant to rotaton change, we ust need to modfy the orentaton, and x- and y-coordnates of each SIFT feature accordngly n each query mage for testng. Indexng. We use an nverted fle structure to ndex mages. As llustrated n Fgure 5, each vsual word s followed by a lst of ndexed features that are quantzed to the vsual word. Each ndexed feature records the ID of the mage where the vsual word appears. Besdes, as dscussed n Secton 3.2, for each ndexed feature, we also need keep ts SIFT orentaton, scale, and the x- and y- coordnate, whch wll be used for generatng geometrc codng maps for retreval. Evaluaton and retreval. To evaluate the performance wth respect to the sze of dataset, we buld several smaller datasets by samplng the basc one-mllon dataset. For the DupImage dataset, 100 representatve query mages are selected from the ground-truth dataset for evaluaton comparson. For the Copydays dataset, all orgnal and attacked mages are used as queres for evaluaton comparson. We adopt mean Average Precson (map) [Phlbn et al. 2007] to evaluate the performance of all approaches. In retreval, each vsual word n the query mage casts a vote for ts matched mages. Instead of selectng the tf-df weght [Svc and Zsserman 2003; Nster and Stewenus 2006] to dstngush dfferent matched features, we smply count the number of SIFT matches that pass our spatal verfcaton. Image smlarty s formulated as the number of true matches, wth a term to dstngush mages wth the same amount of true matches by ther feature amount. 4.2 Impact of Parameters The performance of our approach s related wth fve parameters: vsual codebook sze, α and τ n GSC, and r and β n GFC. In the followng, we wll study ther mpacts wth the DupImage dataset and select the optmal values Vsual Codebook Sze. Vsual codebook sze reflects the space partton degree of the SIFT descrptor. The larger the vsual codebook, the fner the hgh-dmensonal descrptor space s dvded. We test three dfferent szes of vsual codebooks on the 1M mage database. The result s shown n Table I.

12 4:12 W. Zhou et al. Table I. Mean Average Precson (map) and Average Tme Cost per Query to Each Vsual Codebook Codebook sze 130K 500K 1M map Tme cost (s) map α (a) τ = 0 τ = 1 τ = 2 τ = 3 τ = 4 τ = 5 average tme cost per query (second) α (b) τ = 0 τ = 1 τ = 2 τ = 3 τ = 4 τ = 5 Fg. 6. Performance comparson of geometrc fan codng wth dfferent α and τ on the one-mllon mage dataset. (Best vewed n color PDF) From Table I, t can be observed that when the sze of descrptor vsual codebook ncreases from 130K to 500K, the map drops a lttle whle the tme cost per query decreases sharply. When usng a smaller codebook, smlar features wll be more lkely to be quantzed to the same vsual word, whch helps to gan mprovement n accuracy. Although more false local matches wll be ncurred, they can be effectvely dscovered and removed by our geometrc verfcaton. To make a trade-off n map and tme cost, we select the 1M vsual codebook, whch s used n the later experments Impact of α and τ. In geometrc square codng, the factor α and τ work together to cast geometrc consstency constrants on the relatve spatal postons between each par of SIFT features. We also need evaluate ther ont mpact on retreval performance so as to select the optmal values. We test the performance of our geometrc square codng usng dfferent values of α and τ on the 1M dataset, wthout geometrc fan codng constrants. Intutvely, smaller α value defnes strcter spatal relatonshps. However, t suffers from the scale detecton error n the SIFT feature. Smlarly, the parameter τ also tunes the strctness of spatal constrants. Small values of τ wll help tolerate dgtal errors n estmatng the scale of features durng detecton. When τ ncreases, the mposed constrant wll become loose and more nosy matches wll be found. As shown n Fgure 6(a), wth the ncreasng of α, the map performance frst ncreases, and then gradually drops after reachng ts maxmum. Snce the computatonal complexty of the geometrc square codng map s ndependent of α and τ, the tme cost should be smlar wth dfferent α and τ, as demonstrated n Fgure 6(b). Consderng both map and average tme cost, we select α = 5andτ = Impact of r and β. In geometrc fan codng, the factor r determnes the dvson degree of the mage plane, whle the factor β tunes the strctness of geometrc consstency verfcaton. In fact,

13 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:13 map r = 1 r = 2 r = 3 r = 4 r = 5 average tme cost per query (second) r = 1 r = 2 r = 3 r = 4 r = β (a) β (b) Fg. 7. Performance comparson of geometrc fan codng wth dfferent r and β on the one-mllon mage dataset. (a) map; (b) average tme cost per query. The best result s acheved wth r = 4andβ = 2. (Best vewed n color PDF) r and β work nteractvely to mpose geometrc constrants on the relatve spatal postons among local matches. Therefore, we need to evaluate the mpact of them together on retreval performance. We test the performance of our geometrc fan codng usng dfferent values of r and β on the 1M dataset, wthout geometrc squarng codng constrants. The map performance and tme cost are llustrated n Fgure 7. Intutvely, larger values of r and β cast strcter geometrc constrant and better performance s expected. However, due to the unavodable detecton errors of the SIFT key pont poston and SIFT orentaton, a strong geometrc constrant wll be subected to the drftng error from relatvely spatal postons, resultng n low accuracy, as shown n Fgure 7(a). For tme cost, large r wll ntroduce more computaton n the fan codng maps, and ncrease the requred tme cost, as shown n Fgure 7(b). Consderng both map and tme cost, the best trade-off s made when r = 4andβ = Evaluaton on DupImage Dataset Three approaches are consdered for comparson. The parameters of those comparson approaches are tuned based on the suggeston n the correspondng papers. The frst one s the bag-of-vsualwords approach wth vsual vocabulary tree [Nster and Stewenus 2006], denoted as the baselne approach. A vsual vocabulary of 1M vsual words s adopted. In fact, for the baselne, dfferent szes of vsual codebooks have been tested and the 1M vsual codebook s found to generate the best overall performance. The second one s rerankng va geometrc verfcaton, whch s based on the estmaton of an affne transformaton by a varant of RANSAC [Chum et al. 2004] as used n Phlbn et al. [2007]. We call ths method RANSAC. In the experment, all canddate mages wth no less than three local matches are nvolved n the RANSAC-based rerankng. The thrd one s Spatal Codng (SC) [Zhou et al. 2010], whch generates spatal maps for spatal verfcaton. Snce spatal codng requres that the duplcated patches n dfferent mages share the same or very smlar spatal confguraton, t cannot drectly deal wth mages wth rotaton changes. As dscussed n Zhou et al. [2010], t can acheve rotaton nvarance by mergng the results of a set of new queres, whch are obtaned by evenly rotatng the query mage wth a few predefned angles k π, k = 0, 1,...,m 1, where m denotes the rotaton tmes. For example, f m = 8, the query mage m wll be rotated n 8 angles: 0 π 8, 1 π 7 π,..., and therefore generate 7 more new query mages for further 8 8

14 4:14 W. Zhou et al. 1.5 map rotaton tmes (a) average tme cost per query (second) rotaton tmes (b) Fg. 8. Performance of spatal codng wth dfferent rotaton tmes: (a) mean average precson; (b) average tme cost per query. duplcate mage search. We test spatal codng on the one-mllon mage dataset wth a 1M vsual codebook on dfferent values of m. As shown n Fgure 8, when the rotaton tmes ncrease, ts map frst ncreases sharply and then keeps stable, whle the tme cost ncreases proportonally. When m = 60, t reaches the peak MAP. Consderng both accuracy and tme cost of spatal codng, n the comparson, we select the rotaton tmes of 8, 30, and 60, and denote the correspondng spatal codng approach as SC(8), SC(30), and SC(60), respectvely. SC(8) has smlar tme cost to our Geometrc Codng (GC) approach whle SC(30) acheves smlar map performance to our approach. SC(60) acheves the best map performance wth the least rotaton tmes. It can be noted that there s a small gap n map between SC(60) and GC. SC(60) performs slghtly better than GC, whch s manly due to the fact that the orentaton detecton suffers from trval errors n SIFT extracton. In our geometrc codng approach, the orentaton detecton error wll be propagated when local features coordnates are adusted by the reference feature s orentaton value wth Eqs. (1) and (7). Nevertheless, n the spatal codng approach, ths error s attenuated through soft quantzaton n orentaton space [Zhou et al. 2010]. We perform the experments on a server wth 2.4 GHz CPU and 8GB memory. Fgure 9 llustrates the map performance of the comparson algorthms and our Geometrc Codng (GC) approach. Fgure 10 shows the average tme cost per query of all sx approaches. The tme cost of SIFT feature extracton s not ncluded n all algorthms. Compared wth the baselne, our approach s more tme consumng, snce t s nvolved wth geometrc codng and verfcaton. It takes the baselne second to perform one mage query on average, whle for our approach the average query tme cost s second, 0.06 second more than the baselne. However, our approach ncreases the map from 0.37 to 0.54, an 46% mprovement over the baselne. Our approach s more effcent than SC(8), SC(30), SC(60), and rerankng wth RANSAC. SC(8) takes comparable tme cost wth GC, but ts map s much worse than our approach and even worse than the baselne. Our approach also acheves comparable map to spatal codng wth 30 rotaton tmes (SC(30)), However, the average tme cost per query of SC(30) s 2.8 tmes that of our approach. Snce SC(30) s nvolved wth 29 more rotated new queres to gan smlar map to GC, more tme cost s ntroduced. When the rotaton tmes ncrease to 60, the map of SC(60) reaches 0.548, wth a slght mprovement over our GC approach (0.540) but wth much hgher tme cost. The computatonal tme of SC(60) s second per query, whch s fve tmes that of GC. The reason that GC acheves a lttle

15 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4: SC(60) SC(30) GC RANSAC baselne SC(8) 0.55 map K 200K 500K 1M database sze Fg. 9. Comparson of map for dfferent methods on the 1M database. (Best vewed n color PDF) averate tme cost per query (second) baselne RANSAC SC(60) SC(30) SC(8) GC Fg. 10. The average tme cost of the comparson methods and our Geometrc Codng (GC) approach. lower map than SC(60) s that the orentaton detecton error from SIFT extracton s propagated to the coordnate adustment n Eq. (1) and Eq. (7). RANSAC s the most tme-consumng approach, due to the affne estmaton from many random samplngs. It costs seconds on average per query, whch s 6.7 tmes more than our approach. However, t s notable that our approach acheves even better map performance than the RANSAC method. Ths s because that RANSAC only makes use of the coordnate nformaton, whle our approach fully explots geometrc clues ncludng the scale, orentaton, and spatal poston. Therefore, our approach s more powerful n dstngushng those geometrc nconsstent matches. Fgure 11 llustrates some sample results usng our geometrc codng approach on the one-mllon mage dataset. It can be observed that the retreved results are not only dverse but also contan large changes n color, scale, rotaton, sgnfcant occluson, etc. 4.4 Evaluaton on Copydays Dataset Wth the parameters selected n Secton 4.2, we also compare our approaches wth the other algorthms on the Copydays dataset wth four dfferent szes of dstractng mages, that s, 100K, 200K, 500K, and 1M. The map results are gven n Table II. Snce there are four categores each wth dfferent number of mages, we lst the map respectvely. As can be seen, ths dataset s relatvely easer than

16 4:16 W. Zhou et al. Fg. 11. Sample results on the one-mllon mage dataset. Query mages are shown on the left of the arrow n each row, and hghly ranked returned mages (selected from those before the frst false postve) are shown on the rght. Table II. Comparson of All Methods n Accuracy of Queres from Four Parts of The Copydays Dataset on Four Dfferent Szes of Database (100K, 200K, 500K, and 1M) Baselne RANSAC SC(60) SC(30) SC(8) GC Orgnal 100K K K M JPEG 100K K K M Croppng 100K K K M Strong 100K K K M Table III. Comparson of All Methods n Average Tme Cost on Copydays Images on an One-Mllon Image Dataset Approach Baselne RANSAC SC(60) SC(30) SC(8) GC Average tme cost (second) the aforesad DupImage dataset and the map values of all approaches are relatvely hgher. Among the four categores, the best results are from queres wth orgnal mage, whle the worst results are obtaned wth queres from strong attacked mages. The average tme cost per query of all approaches s shown n Table III. The tme cost of SIFT feature extracton s not ncluded. Compared wth the baselne, our approach s more tme consumng. It takes the baselne second to perform one mage query on average, whle for our approach the average query tme cost s second, about twce the tme cost of the baselne. However, for each category of the ground-truth mages, our approach acheves better map performance over the baselne.

17 SIFT Match Verfcaton by Geometrc Codng for Large-Scale Partal-Duplcate Search 4:17 Smlar to the results n Secton 4.3, our approach s more effcent than SC(8), SC(30), SC(60), and rerankng wth RANSAC. SC(8) takes even more tme cost than GC, but ts map s much worse than our approach and even worse than the baselne. Our approach also acheves comparable map to SC(30), However, the average tme cost per query of SC(30) s about twce of our approach. The map of SC(60) s hgher on all four categores of mages over our GC approach, but wth much hgher tme cost. RANSAC s the most tme-consumng approach. It costs seconds on average per query, whch s 22.6 tmes more than our approach. Also, our approach acheves better map performance than the RANSAC method. 5. CONCLUSION In ths artcle, we propose a novel geometrc codng scheme for SIFT match verfcaton n large-scale partal-duplcate mage search. The geometrc codng conssts of geometrc square codng and geometrc fan codng. It effcently encodes the global geometrc context of local features n an mage and effectvely dscovers false feature matches between mages. Our approach can effectvely detect duplcate mages wth rotaton, scale change, occluson, and background clutter wth low computatonal cost. Experments on two mllon-scale datasets reveal that our approach outperforms the baselne even followng a RANSAC verfcaton. Besdes, our approach also attans comparable performance wth the spatal codng scheme, but takes much less tme. Our geometrc fan codng s nspred by the spatal codng scheme [Zhou et al. 2010]. The key dfference s that, key pont locatons are frst adusted by SIFT orentaton and the generated codng maps are nvarant to rotaton changes. Our GFC can adaptvely acheve rotaton nvarance, whch cannot be well addressed by spatal codng. Besdes, the spatal verfcaton s performed n a soft manner to tolerate drftng error of relatve postons of local features. Our geometrc codng scheme ams to dscover Web mages sharng duplcated patches. Wth hgh accuracy and effcency, our approach s effectve for large-scale partal-duplcate mage retreval. However, when searchng general obects where dstnctve SIFT features are not repeatable, t may not perform well. REFERENCES BAY, H., TUYTELAARS, T., GOOL, L. V SURF: Speeded up robust features. In Proceedngs of the 9th European Conference on Computer Vson (ECCV 06) BELONGIE, S., MALIK, J., AND PUZICHA, J Shape matchng and obect recognton usng shape context. IEEE Trans. Pattern Anal. Mach. Intell. 24, 4, CHANG, S.-K., SHI, Q. Y., AND YAN, C. Y Iconc ndexng by 2-D strngs. IEEE Trans. Pattern Anal. Mach. Intell. 9, 3, CHUM, O., PHILBIN, J., SIVIC, J., ISARD, M., AND ZISSERMAN, A. 2007a. Total recall: Automatc query expanson wth a generatve feature model for obect retreval. In Proceedngs of the IEEE 11th Internatonal Conference on Computer Vson CHUM, O., PHILBIN, J., ISARD, M., AND ZISSERMAN, A. 2007b. Scalable near dentcal mage and shot detecton. In Proceedngs of the 6th ACM nternatonal Conference on Image and Vdeo Retreval. ACM, 1 8. CHUM, O., PERDOCH, M., AND MATAS, J Geometrc mnhashng: Fndng a (thck) needle n a haystack. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton CHUM, O., MATAS, J., AND OBDRZALEK, S Enhancng RANSAC by generalzed model optmzaton. In Proceedngs of the Asan Conference on Computer Vson COPYDAYS, egou/data.php DUPIMAGE, wzhou/data/dupgroundtruthdataset.tgz FISCHLER, M. A. AND BOLLES, R. C Random sample consensus: A paradgm for model fttng wth applcatons to mage analyss and automated cartography. Comm. ACM. 24, 6, GAO, Y., WANG, C., LI, Z., ZHANG, L., AND ZHANG, L Spatal-Bag-of-Features. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton