Augmented Distinctive Features for Efficient Image Matching

Transcription

1 Augmented Dstnctve Features for Effcent Image Matchng Quan Wang, We Guan and Suya You CGIT/IMSC USC Los Angeles, CA and Abstract Fndng correspondng mage ponts s a challengng computer vson problem, especally for confusng scenes wth surfaces of low textures or repeated patterns. Despte the well-known challenges of extractng conceptually meanngful hgh-level matchng prmtves, many recent works descrbe hgh-level mage features such as edge groups, lnes and regons, whch are more dstnctve than tradtonal local appearance based features, to tackle such dffcult scenes. In ths paper, we propose a dfferent and more general approach, whch treats the mage matchng problem as a recognton problem of spatally related mage patch sets. We construct augmented sem-global descrptors (ordnal codes) based on subsets of scale and orentaton nvarant local keypont descrptors. Ted rankng problem of ordnal codes s handled by ncreasngly keypont samplng around mage patch sets. Fnally, smlartes of augmented features are measured usng Spearman correlaton coeffcent. Our proposed method s compatble wth a large range of exstng local mage descrptors. Expermental results based on standard benchmark datasets and SURF descrptors have demonstrated ts dstnctveness and effectveness. 1. Introducton and related works Correspondence s one of the fundamental and ntensvely studed problems n computer vson. Approaches based on texture analyss around local nterest ponts [1, 2, 3, 4, 5, 6] have ganed much attenton recently due to ther robustness to dstorton and occluson, and avodance of dffcult problems such as edge detecton and segmentaton. However, when used alone, the localty property of such features frequently leads to nosy results. Geometrc constrants and consstency check generally need to be appled as addtonal layer to refne the result. Furthermore, although expermentally proven to be remarkably robust and nvarant to vewpont changes and dstortons usually at the cost of heavly computatonal demands, local features are also typcally not very dstnctve bascally because of the lack of bult-n geometry nformaton. Therefore, they tend to face great dffculty for scenes wthout hgh-textured surfaces or wth repeated patterns, easly confuse texture-based methods. To tackle the problem, many recent works n matchng and recognton domans try to drectly extract and descrbe hgh-level matchng prmtves such as constellaton of edges [7, 8] and regons [9, 10, 25] to enhance the descrptors dstnctveness. However, the accurate and clean acquston of those prmtves remans a challengng open problem n computer vson. The acqured curves and shapes are often nsuffcent stable between mages due to vewpont and lghtng changes. Input Images keypont detecton Sub- Images offset aggregaton keypont descrpton Relatve Features compute ranks Ordnal Codes handle ted Augmented Features Database Fgure 1: Overvew of our feature augmentaton process Ths paper presents our novel approach n a general framework to generate more dstnctve features based on nvarant local feature descrptors, wthout the need of hgh-level feature extracton. We propose to treat the classc mage matchng problem as a collecton of mage recognton problems, whch naturally ntegrates geometry nformaton nto our augmented features. Fg. 1 shows the maor steps nvolved when generatng augmented features. Frst, we detect stable nterest ponts n nput mages and extract correspondng mage patches based on postons, orentatons and scales of those ponts. Sub-mages, whch are bascally subsets of mage patches that are close to each other n mage space are constructed, consequently convertng the orgnal problem nto an mage recognton problem for those sub-mages. Next, the sem-global descrptor for each sub-mage s computed by frst accumulatng offsets between the sub-mage center and member patches descrptors, whch produces what we 1

2 called relatve features, contanng geometry nformaton of the neghborhood. The augmented features (ordnal codes) are generated by consderng the relatve rankngs of features components nstead of ther orgnal values, whch can be regarded as a feature normalzaton process. From certan pont of vew, our feature augmentaton process s analogous to the codng process n the general framework of mage recognton. One dfference s what we are codng here are sub-mages, not local patches (e.g. [19]) or the whole mage (e.g. [24]). Addtonally, we propose ncreasngly keypont samplng around submages to handle ted rankng problem of ordnal codes. Fnally, Spearman correlaton s used to measure the smlarty between augmented features and establsh the pont-to-pont correspondences for mage pars. Extensve expermental results based on SURF features have demonstrated the effectveness of our augmented features. We conduct experences usng benchmark datasets plus precson-recall analyss, together wth supplemental real-word challengng mage pars, both ndcatng hgher dstnctveness and lower outler level of our augmented features compared wth base features. The addtonal computatonal cost of the feature augmentaton process s nomnal, whch makes buldng nteractve or even real-tme computer vson applcatons possble when combned wth fast base features. Furthermore, our feature augmentaton framework s general enough to be compatble wth a wde range of exstng nvarant local features, provdng mportant performance gan to mage representaton methods or upgradng exstng mage feature databases at a mnmum cost. We also provde theoretcal analyss about compatblty ssues of our method workng wth dfferent knds of local features. Our proposed approach s related to the followng recent works: In [11], Boureau, et al. provded a systematc evaluaton about combnatons of varous codng and poolng technques for recognton, and concluded that large performance ncrease can be obtaned by merely representng neghborng descrptors ontly. Smlar dea was explored n [12] whch also proposed a novel way to optmze dmenson reducton and approxmated nearest neghbor searchng at the same tme. Concernng the ordnal descrpton method we used, the early work of ordnal measurement tracks back to M. Kendall [15]. Recent works applyng ordnal descrpton on mage correspondence and recognton problems nclude R. Zabh and J. Woodfll [16], J. Luo, et al. [17] and M. Toews and W. Wells [18], whch bult upon SIFT descrptors and reported superor results n terms of precson-recall when compared wth many wdely-used descrptors such as orgnal SIFT, PCA-SIFT and GLOH. Our sub-mage concept s also enlghtened by the followng classc technques: frst of all, the shape context [20] takes sampled contour ponts as nputs and constructs shape descrptors n log-polar hstograms usng relatve postons (drecton and dstance) between contour ponts. Two fundamental dfferences are: our sub-mages are descrbed by countng offsets n nvarant feature space, not n mage space. And the constructon of sub-mages s drectly based on stable nterest ponts, therefore avodng the need to acqure clean shapes and contour ponts from mages, whch s a challengng fundamental vson problem by tself. Second, our sub-mage descrpton s also related to Fsher kernel, whch provdes the offset drectons n parameter space nto whch the learnt dstrbuton should be modfed n order to better ft the newly observed data. Recent work of Perronnn et al. [21] uses fsher kernel to obtan compact and effcent mage representatons for mage classfcaton. Last, the famous bag-of-features approach [22] uses K-mean to generate code-books based on nvarant local features, assgns each query descrptor to one tem n the code-book, producng a hstogram of codes representng the whole mage, whle our approach proposes sub-mage as matchng prmtves and uses accumulated offsets nstead of the cluster centers n the feature space. Each base descrptor s converted nto exactly one augmented descrptor, facltatng dense pontto-pont correspondences. 2. Feature augmentaton process Ths secton presents the reasonng and detals of our feature augmentaton process. Based on keypont locatons and descrptons, we construct sub-mages representng nterested neghborhood n mage space, then compute relatve features for each sub-mage ntegratng geometry nformaton n feature space, whch are normalzed usng ordnal descrpton and produce fnal augmented features Sub-mages Many exstng mage matchng methods use local mage patches around nterested keyponts as matchng prmtves. The mage patches can be extracted ether from scale and rotaton-adaptve neghborhoods, where transformaton parameters are determned through searchng n scale space and orentaton hstograms (e.g. [2], [3], [4] and [5]), or from regular regons of fxed szes, whch acheve vewpont nvarance through separated multple-vew tranng process (e.g. [6], [19] and [23]). The produced local features can be very robust aganst vewpont changes and dstortons, but usually nsuffcent dstnctve due to the lack of global geometry nformaton. On the other hand, mage classfcaton and recognton works focus more on dstnctveness among dfferent mages rather than robustness, generally producng one unfed descrptor for each mage and can by no means provde robust pont-to-pont correspondences. We propose to use sub-mages as matchng prmtves, amng to fll the gap between the above two extremes. Sub-mages are sets of local mage patches that are close 2

3 to each other n mage space (Fg. 2). As relatve local structures, they are robust to factors such dstortons and occlusons. They also ntegrate sem-global geometry nformaton n order to mprove feature dstnctveness. Fgure 2: Illustratons of sub-mage concept (k = 5 cases). Red and blue dots are leaders and members of each sub-mage respectvely. Grey crosses are other nterest ponts. To construct sub-mages, we frst detect stable nterest ponts (represented by P ) and extract fxed or nvarantly adaptve mage patches around each nterest pont dependng on whch detector s used. After the poston of each P s stored and properly ndexed (herarchy tree structure preferred especally when the total number of keyponts, represented as num, s large), we construct one sub-mage structure for each P (represented by S(P )) wth P as ts leader and ts k nearest neghbors n mage space as members. S( P) = { P} { k NN( P)}, 1 num Each S(P ) s an abstract type of mage contanng nformaton for the neghborhood of P. Both leaders and members of sub-mages are centrally organzed by keyponts. As a result, the keypont detector should be suffcent stable so that the consttuton of the same submage wll reman consstent to certan degree between dfferent nput mages under varous vewng condtons. In our current experments, we use the effcent keypont detector n [3] based on ntegral mages, whch s very fast to apply and provdes us wth suffcent repeatablty. After the sub-mage constructon, fndng matchng ponts n the nput mage par s equvalent to classfyng a large set of sub-mages generated from both mages Relatve features Gven sub-mages, our next task s to form dstnctve descrptors, whch wll be used later to measure the smlartes between sub-mages effcently. Ths step s analogous to the codng process n mage recognton. Suppose we choose an nvarant local feature descrptor (wth descrptor dmensonalty represented by dm) and D(P ) s the feature vector of P and D (P ) represents ts - th vector component. Notce that there s no need to adopt the keypont detector and descrptor all belongng to the same mage matchng system, as long as the detector selected s stable and the descrptor nvarant, meetng general requrement of those components. Ths provdes extra flexblty for our augmented feature to work well wth varous combnatons of exstng keypont detecton and mage representaton technques. If we seek to combne the sub-mage s descrptors drectly, for example, smply concatenatng D (P ) for all the P belongng to the same sub-mage, at least two drawbacks wll mmedately follow. One s the rather hgh dmensonalty of the resultng descrptors (when k = 5 and standard SURF [3] descrptor s used, each sub-mage wll be assocated wth a 384-D vector), whch rases practcal obstacles to later ndexng and searchng steps due to dmensonalty curse. The other s such combnaton wll hghly lkely compromse the nvarant propertes of the base descrptors because of the dfferent orders when performng the concatenaton. Encouraged by successful classc technques such as Fsher kernel [21] and shape context [22], we beleve that generally speakng, features based on relatve and aggregated values (ether n mage space lke shape context or n feature space lke macro features n [11]) usually demonstrate more robust performance than features drectly based on raw values of base descrptors. Fgure 3: Sub-mages (red and blue crosses) of smlar physcal locatons n one mage matchng par, and ther correspondng relatve features vsualzed as 2D (8 by 8) hstogram patterns. Based on the above reasonng and analyss, we compute relatve features for S(P ) (denoted as R(P )) by accumulatng k offsets between leader and member descrptors of S(P ) for each dmenson. Formally, we can defne: Then the -th dmenson of the relatve feature s computed as: 3

4 R ( P ) = δ n [1, num], n D ( P ) D ( Pn ), 1 dm The computed relatve features have a constant low dmensonalty regardless of k, the number of members n each sub-mage. Moreover, assumng the bases descrptors are scale and rotaton-nvarant, and the keypont detector s stable, t can be easly proved that the relatve features produced wll also be nvarant, snce the sub-mage members are consdered n an order-ndependent way. Ths s a consderable advantage for mage matchng applcatons, acheved by computng relatve features n nvarant feature space and mantanng nvarance durng the computaton process. As a comparson, features computed by aggregatng offsets n mage space, such as shape context, are fundamentally not nvarant to orentatons and typcally obtan lmted degree of scale nvarance through normalzaton technques. In our current experments, we use SURF descrptor [3] wth dm = 64. We vsualze the generated relatve features by frst transformng ther components nto 0~255 ntensty range and then renderng each descrptor as a 2D (8 by 8) hstogram. Fg. 3 shows some of our relatve features generated usng a modfed verson of the famous testng mage par from [2] (extra lghtng changes and dstortons are added) Normalzaton After relatve features n each mage are computed, we can drectly measure ther smlartes usng standard dstance metrcs such as Eucldean dstance and establsh the ntal correspondences between two feature sets. For smple mage matchng pars, e.g. one mage s the n-plane rotated or unformly scaled verson of the other, accurate results could be expected. However, when t comes to challengng mage pars nvolvng complcated vewponts and lghtng changes (e.g. Fg. 3), we found such ntal matchng results are generally very nosy. The prmary reason les n the lnear relatonshp assumpton of today s wdely-used computatonal effcent metrcs such as Eucldean or ch-squared dstances. To handle the volaton of lnear assumpton (e.g. from llumnaton changes), orgnal feature vectors are usually normalzed to remove non-lnearty before enterng the matchng process. Many exstng methods use a set of experences- or experments-determned parameters to threshold, translate and rescale, fttng the orgnal data nto a fxed range, whch s naturally dffcult to generalze to other vewng condtons or combnatons of stuatons. As can be observed from fgure 3, n terms of raw values magntudes (the ntenstes of the 2D hstogram cells), sub-mages of smlar physcal locatons could produce rather dfferent relatve features due to vewng condton changes, whch s exactly the reason of nosy results when drectly measurng the smlartes of relatve features. However, t can also be observed that although those bns absolute values are not dentcal across dfferent vews, ther relatve orders are overall consstent. Based on the above analyss and observatons, we propose to use ordnal descrpton as a parameter-free and computatonal effcent approach to normalze our relatve features. Frst, for each relatve feature, we locally sort ts vector components and produce a sorted array. The augmented feature (represented by A(P )) s generated by replacng the orgnal values wth ts correspondng ranks n the sorted array for every dmenson. The overall computatonal complexty of normalzng each relatve feature s only dm*log(dm) due to sortng. A ( P ) = { R ( P ) R ( P ) R ( P )} 1 dm m m Our fnal augmented features consder the relatve rankng of features each dmenson nstead of ther orgnal values. As a result, each augmented feature s represented as an nteger vector of dm dmensons, more specfcally, a permutaton of the set {1, 2,, dm}. The ordnal descrpton we use generates normalzed mage representatons nvarant to monotonc deformatons and also robust aganst certan degree of challengng nonlnear and non-unform factors such as partal lghtngs. Snce each ordnal codes s a permutaton of nteger set {1, 2,, dm}, we treat those ntegers as ntensty values and smply vsualze each ordnal codes as a barcodes-lke 1D pattern. Fgure 4 shows the barcodes assocated wth the same sub-mages as n fg.3 n the two nput mages. It ndcates that after the ordnal descrpton, our feature smlarty s notably mproved. Under dfferent vewng condtons, features belongs to the same physcal locatons are more smlar whle others reman dstnctve. Fgure 4: Augmented features vsualzaton for smlar physcal locatons n the two nput mages. The left column s extracted from the left mage whle the rght column from the rght mage. One crucal ssue for ordnal descrpton s how to handle ted rankng. Gven a relatve feature vector, when ts two components wth dfferent ndces contan the same orgnal value, the produced ordnal codes wll contan ted dentcal rankngs for the two ndces, resultng n 4

5 unexpected outputs. The smplest way to handle ted rankng s to use the dfferent ndces as fal safe reference to break the ted stuaton. Ths method s equvalent to usng a vector of orgnal ndces as a reference. In [18], such reference vector s obtaned by averagng a large set of descrptors, e.g. the descrptors of the whole mage, offlne, whch s approprate for codng the whole mage for recognton purpose. However, we argue that for mage matchng, the reference vector used by sem-local ordnal descrpton should reflect the local orderng trend, whch s usually not consstent wth the global trend. Algorthm: Ted Rankng Breaker 1: Assume: R = R ; ; 2: Intalze: set S = {all keyponts}-s(p ); 3: do teraton t = 1, 2,, M; 4: P n = NN(P, S); 5: S = S P n ; 6: S(P ) = S(P ) + P n ; 7: compute new R(P, S(P )); 8: f (R R ) break; 9: end teraton; 10: f (R R ) use (R, R ) to handle ted rankng; 11: else use (, ) nstead; Therefore, we propose ncreasng keypont samplng around sub-mages to break ted rankng. The dea s to teratvely nclude addtonal nearest neghbors of the current sub-mage one by one and compute new relatve features durng each teraton untl the ted rankng s broken, or, for effcency consderaton, untl a maxmum number of teraton (denoted by M) s reached, n whch case, we use the orgnal ndces as reference vector nstead. Pseudo codes for ted rankng handlng are provded above. NN(P,S) s a functon returnng the nearest neghbor of P n S. 3. Establshng correspondences Although standard dstance metrcs e.g. Eucldean dstance can be used to measure the smlartes, better comparson results can be obtaned more effcently usng specal metrcs based on characterstcs of the ordnal codes, for example, gven dm, there are a fxed total number of unque codes (dm!) and are all of the same length. Ordnal descrpton and measurement have been studed for nearly a century and many specally desgned dstance metrcs were proposed. In ths paper, we studed three measurements compatble well wth our features and problem doman. Frst s the measurement of element-wse consstency countng. Assume A and A are two augmented feature vectors for two sub-mages, and A s the -th vector components. Ther smlarty can be effcently computed as: Dst( A, A' ) = { A A A' } 1 dm The second metrc s based on relatve orderng of vector component pars measured by sgn() functon, whch returns 1 f the nput parameters have the same sgn and -1 otherwse. Formally, the Kendall coeffcent [15] s defned as the followng: Dst ( A, A' ) = 2 [ 1,dm] m [ + 1, dm] sgn( A A m dm(dm 1), A' A' Last, based on the ordnal codes property of equal length, our modfed verson Spearman correlaton coeffcent [14] returns the smlarty of two ordnal codes n the range of 0~ ( A A' ) [1,dm] Dst ( A, A' ) = 1 2 dm(dm 1) Our own experments ndcate that the Spearman correlaton coeffcent generally provdes better matchng results than the other two. As addtonal but rather practcal consderatons, ts conventonal unt range also works well wth many exstng mage matchng components and frameworks, such as dstance rato (used n [2] and [5]) and consstent check methods lke RanSac. Therefore, we select the Spearman correlaton coeffcent as our dstance metrc n the next secton s evaluaton. 4. Expermental results Our proposed augmented feature has been ntensvely tested usng standard benchmark datasets [5] wth known ground truth. We prmarly focus on outdoor scenes accordng to our proect s nterest. Besdes the standard test data, we also conducted experments on other realworld mage pars of outdoor nature scenes wth manmade obects such as buldngs and sgns. Many of those addtonal testng mage pars contan nterest obects wth surfaces of low-textures or repeated patterns, partcularly challengng for varous exstng local mage features. We experment dfferent knds of nvarant mage descrpton methods and fnally select SURF (Speeded Up Robust Feature) [3] as our base descrptor, because t provdes the best tradeoff between matchng speed and feature robustness. SURF s bult upon other expermentsverfed and successful detectors and descrptors, such as famous SIFT [2], but smplfy those steps to the essental. It proposes the use of ntegral mages to drastcally reduce the number of operatons for smple box convolutons, ndependent of the chosen scale. The feature descrpton s based on sums of Haar wavelet components, whch can be constructed and matched more effcently compared wth other state-of-the-art methods. In our evaluaton, the wdely-used 64 dmenson of SURF descrptor s used. We set k, the only parameter of the whole feature m ) 5

6 augmentaton process, to 5 whch we expermentally fnd to provde best results. Overall, our expermental results demonstrate that compared wth base descrptors, the proposed augmented features acheve remarkably hgher level of dstnctveness wthout loss of robustness, at a nomnal addtonal computatonal cost. Specfcally, besdes outperformng base features n the standard precson-recall curves, the new augmented feature ntegratng geometry nformaton generally produces larger number of correspondences under the same dstance rato requrement and less number of nosy matchngs especally for challengng real-world scenes. The proposed method s ntegrated nto our mage matchng system, whch generates augmented features of one target mage offlne then matches the features of query mages onlne and frame by frame. The total processng tme each frame ncludng keypont detecton, descrpton, feature augmentaton process and fnally establshng correspondences takes only 100~150ms wth a peak memory occupaton of around 15M, makng the system an attractve soluton for mage matchng problems of nteractve/real-tme applcatons such as outdoor navgaton and augmented realty on small moble devces Standard tests The standard test sets we expermented consst of mage sets of varous scenes (K. Mkolaczyk and C. Schmd [5]), wth each contanng sx mages (fve pars when the frst mage s fxed as reference mage) of successve ncrements of one type of mage deformaton ncludng mage blur and compresson, lghtng and vewpont changes. One mage transformaton matrx s assocated wth each mage par, so that for any keypont poston n the reference mage, we are able to compute the ground truth correspondng poston n any of the other fve mages, verfyng matchngs returned by varous methods. Durng the evaluaton process, for each mage par, we construct base descrptors and two versons of augmented descrptors usng local (L) and global (G) reference vector respectvely, for the same set of nterest ponts and then use Eucldean dstance and Spearman correlaton coeffcent to measure ther smlartes and establsh ntal matchngs. Wthn one test, the same dstance rato s appled to both base and augmented tracks to obtan fnal reported matchngs. Dstance ratos are vared n dfferent tests to generate the fgure data ponts. Next, for each keypont poston n the reference mage, by comparng ts reported matchng postons wth ground truth matchng postons, we are able to dstngush true and false matchngs. Ths crucal nformaton, combned wth the total number of reported and possble correspondences, allow us to compute precson and recall for one mage par, under one dstance rato. Accordng to our proect nterests, we focus our experments on challengng outdoor scenes wth nonplanar obects and low-textured surfaces, handlng complcated vewpont changes. To generate fgure 5, we choose 13 mage pars meetng the above gudelnes from three mage sets (the boat set, the grafft set, and the bke set), whch also represents challengng factors n outdoor moble applcatons such as rotatons of the devce and blurred nput mages. Each mage par s tested by usng base and augmented features and ten dfferent dstance ratos. Fgure 5 shows the precson and recall results summary of over 300 mage matchng tests, demonstratng the superor performance of our proposed method. Recall Precson Aug. Feature (L) Aug. Feature (G) Base Feature Fgure 5: Recall-precson for our standard test results summary, evaluated on 64-D SURF as base descrptors. Results averaged over 13 mages pars of K. Mkolaczyk s testng data set Dense matchng from dstnctve features In ths subsecton, we demonstrate that our augmented features are more dstnctve than base features, resultng denser and cleaner correspondences. Towards ths goal, we compute base and augmented features usng the fve mage pars of boat mage set (wth ncreasng level of zoom and rotaton), apply the same dstance rato and record the number of reported matchngs for the two feature tracks respectvely. Snce dstance rato s defned as the rato of the best matchng s smlarty over the second best, we beleve that from some aspects of vew, the number of remanng matchngs after applyng a fxed dstance rato can also be used as a smple ndcator of feature dstnctveness. Assume one method generates smlar, clustered thus less dstnctve features whle the other produces features well scattered and dstrbuted n the feature space. After applyng the same dstance rato flter, the frst method wll have much less matchngs remanng because the dfference between ts dfferent matchng s smlartes s much lmted. 6

7 350 Total Number of Features # of Reported Matchngs (Aug.) # of Reported Matchngs (Base) representatons such as [13] and [23] couldn t beneft from the feature augmentaton process. Number of Features and Matchngs Test Index (Increasng Zoom and Rotaton) Fgure 6: Number of matchngs remanng after applyng the same dstance rato. Evaluaton based on the boat mage sets. Fgure 6 shows that our augmented features lead to sgnfcant larger number of reported matchngs than the base features, under the same dstance rato. Ths result can also be consdered ontly wth secton 4.1, whch ndcates better precson and recall tradeoff of the denser matchngs, together wth secton 4.3, whch demonstrates consstent results for mages outsde the standard datasets as well Matchng vsualzaton and analyss Ths subsecton provdes some lne by lne matchng vsualzaton results for both standard test mages and other real-world scenes. We apply the same dstance rato and for each of remanng fnal matchng, draw a blue lne from ts keypont n reference mage to the matched keypont n the current query mage. As can be observed from fgure 7 and consstent wth the quanttatve evaluaton results from the above two subsectons, through augmented features, we are able to obtan denser correspondences and less matchng outlers than usng base features. The prmary reason les n the fact that our augmented features enhance dstnctveness by ntegratng sem-global geometry nformaton based on sub-mages concepts and offset aggregaton n feature space, whle mantanng and even mprovng feature nvarance and robustness through ordnal descrpton. Last, regardng the compatblty ssue of our feature augmentaton, theoretcally the proposed method should work well wth a large range of exstng nvarant local features ncludng but not lmt to SIFT [2], SURF [3], and GLOH [5]. However, there are a few ssues worth consderng beforehand. Frst of all, t goes wthout sayng that the base feature needs to have descrptors n vector form. Otherwse the computaton of relatve features and ordnal descrpton can not be appled drectly. So specal approaches based on drect learnng on local mage patches wthout vector Fgure 7: Matchng vsualzaton and comparson of base (top) and augmented (bottom) features. Standard dataset: (a) Grafft; (b) Boat; Extra testng dataset: (c) buldng wth smlar or repeated patterns; (d) street scene wth large vewpont changes. Next, bag-of-features [22] and some mult-vew based mage matchng technques such as [6] and [19] use K- mean cluster centers nstead of the orgnal nterest ponts to represent mages or features. In such case, specal cautons are needed to make sure except stable keyponts, those cluster centers are also suffcent stable otherwse t s hard to guarantee that sub-mages, as the very foundaton of our feature augmentaton process, wll 7

8 reman repeated and consstent across dfferent vewng condtons. Fnally, concernng the normalzaton component n our method, ordnal descrpton generally works better when the feature vectors t apples to have a hgh dmensonalty, provdng a large pool of unque codes. Image matchng methods advocatng dmenson reducton (e.g. by usng PCA) of ther feature vectors, such as [4] and [19] are ll-suted for our normalzaton component. It s reported n [18] that drectly applyng ordnal descrpton onto PCA-SIFT [4] can even lead to nferor performance. 5. Concluson Ths paper presents our feature augmentaton process, producng more dstnctve features for effcently mage matchng. The proposed method s bult upon the concept of sub-mages, whch connects close small mage patches n mage space, convertng one mage matchng problem nto a collecton of mage recognton problems. Our relatve features aggregate descrptor offsets n nvarant feature space and wthn sub-mages, n order to ntegrate geometry nformaton and produce sem-global features. Based on vsualzaton and analyss of correspondng relatve features, we propose to use ordnal descrpton to normalze and generate augmented features, nvarant to monotonc deformatons and beyond. Fnally, smlartes are measured by Spearman correlaton coeffcent and correspondences are establshed. Expermental results usng standard and supplemental datasets verfed the augmented features contanng addtonal geometry nformaton are more dstnctve, leadng to denser and cleaner correspondences. References [1] C. Schmd and R. Mohr, Local grayvalue nvarants for mage retreval, IEEE Transactons on Pattern Analyss and Machne Intellgence, pp , May [2] D. Lowe. Dstnctve mage features from scale-nvarant keyponts. Internatonal Journal of Computer Vson, [3] H. Bay, T. Tuytelaars and L. V. Gool, SURF: Speeded Up Robust Features, Proceedngs of the nnth European Conference on Computer Vson, May [4] Y. Ke and R. Sukthankar. PCA-SIFT: A more dstnctve representaton for local mage descrptors. Computer Vson and Pattern Recogn-ton, pages , [5] K. Mkolaczyk, C. Schmd, A performance evaluaton of local descrptors. In PAMI 27(10): , [6] Q. Wang and S. You, Real-Tme Image Matchng Based on Multple Vew Kernel Proecton, IEEE Conference on Computer Vson and Pattern Recognton, June [7] O. Danelsson, S. Carlsson and J. Sullvan, Automatc learnng and extracton of mult-local features, 2009 IEEE 12th Internatonal Conference on Computer Vson, pp , Sept Oct [8] Z. S, H. Gong, Y. Wu and S. Zhu, Learnng mxed templates for obect recognton, Computer Vson and Pattern Recognton, pp , June [9] Q. Wang and S. You, A Vson-based 2D-3D Regstraton System, IEEE Workshop on Applcatons of Computer Vson (WACV), Snowbrd, Utah, December 7-8, 2009 [10] C. Gu, J.J. Lm, P. Arbelaez and J. Malk, Recognton usng regons, IEEE Conference on Computer Vson and Pattern Recognton, pp , June [11] Y.L. Boureau, F. Bach, Y. LeCun and J. Ponce, Learnng md-level features for recognton, 2010 IEEE Conference on Computer Vson and Pattern Recognton, pp , June [12] H. Jegou, M. Douze, C. Schmd and P. Perez, Aggregatng local descrptors nto a compact mage representaton, 2010 IEEE Conference on Computer Vson and Pattern Recognton, pp , June [13] M. Calonder, V. Lepett, P. Fua, K. Konolge, J. Bowmanand P. Mhelch, Compact sgnatures for hghspeed nterest pont descrpton and matchng, 2009 IEEE 12th Internatonal Conference on Computer Vson, pp , Sept Oct [14] C. E. Spearman. Proof and measurement of assocaton between two thngs. Amercan Journal of Psychology, 15:72 101, [15] M. Kendall. A new measure of rank correlaton. Bometrka, 30:81 89, [16] R. Zabh and J. Woodfll, Non-parametrc local transforms for computng vsual correspondance. European Conference on Computer Vson, pages , [17] J. Luo, S. P. Etz, and R. T. Gray. Normalzed Kemeny and Snell Dstance: A Novel Metrc for Quanttatve Evaluaton of Rank-Order Smlarty of Images. IEEE TPAMI, 24(8): , [18] M. Toews and W. Wells, SIFT-Rank: Ordnal descrpton for nvarant feature correspondence, 2009 IEEE Conference on Computer Vson and Pattern Recognton, pp , June [19] V. Lepett, J. Plet and P. Fua, Pont matchng as a classfcaton problem for fast and robust obect pose estmaton, IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, [20] S. Belonge, J. Malk and J. Puzcha, Shape matchng and obect recognton usng shape contexts, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol.24, no.4, pp , Apr [21] F. Perronnn and C. R. Dance. Fsher kernels on vsual vocabulares for mage categorzaton. IEEE Conference on Computer Vson and Pattern Recognton, June [22] J. Svc and A. Zsserman. Vdeo Google: A text retreval approach to obect matchng n vdeos. IEEE Internatonal Conference on Computer Vson, [23] V. Lepett and P. Fua, Keypont Recognton usng Randomzed Trees, In PAMI, Vol. 28, Nr. 9, pp , [24] S. Lazebnk, C. Schmd, and J. Ponce, Beyond Bags of Features: Spatal Pyramd Matchng for Recognzng Natural Scene Categores, IEEE Conference on Computer Vson and Pattern Recognton, [25] J. Km and K. Grauman, Asymmetrc Regon-to-Image Matchng for Comparng Images wth Generc Obect Categores, Computer Vson and Pattern Recognton, pp , June