Hierarchical Motion Consistency Constraint for Efficient Geometrical Verification in UAV Image Matching

Transcription

1 Herarchcal Moton Consstency Constrant for Effcent Geometrcal Verfcaton n UAV Image Matchng San Jang 1, Wanshou Jang 1,2, * 1 State Key Laboratory of Informaton Engneerng n Surveyng, Mappng and Remote Sensng, Wuhan Unversty, Wuhan, , Chna; jangsan870211@whu.edu.cn 2 Collaboratve Innovaton Center of Geospatal Technology, Wuhan Unversty, Wuhan, , Chna * Correspondence: jws@whu.edu.cn; Tel.: (ext. 8321) Abstract: Ths paper proposes a strategy for effcent geometrcal verfcaton n unmanned aeral vehcle (UAV) mage matchng. Frst, consderng the complex transformaton model between correspondence set n the mage-space, feature ponts of ntal canddate matches are projected onto an elevaton plane n the object-space, wth assstant of UAV flght control data and camera mountng angles. Spatal relatonshps are smplfed as a 2D-translaton n whch a moton establshes the relaton of two correspondence ponts. Second, a herarchcal moton consstency constrant, termed HMCC, s desgned to elmnate outlers from ntal canddate matches, whch ncludes three major steps, namely the global drecton consstency constrant, the local drecton-change consstency constrant and the global length consstency constrant. To cope wth scenaros wth hgh outler ratos, the HMCC s acheved by usng a votng scheme. Fnally, an effcent geometrcal verfcaton strategy s proposed by usng the HMCC as a pre-processng step to ncrease nler ratos before the consequent applcaton of the basc RANSAC algorthm. The performance of the proposed strategy s verfed through comprehensve comparson and analyss by usng real UAV datasets captured wth dfferent photogrammetrc systems. Expermental results demonstrate that the generated motons have notceable separaton ablty, and the HMCC-RANSAC algorthm can effcently elmnate outlers based on the moton consstency constrant, wth a speedup rato reachng to 6 for oblque UAV mages. Even though the completeness sacrfce of approxmately 7 percent of ponts s observed from mage orentaton tests, compettve orentaton accuracy s acheved from all used datasets. For geometrcal verfcaton of both nadr and oblque UAV mages, the proposed method can be a more effcent soluton. Keywords: unmanned aeral vehcle; geometrcal verfcaton; moton consstency constrant; mage matchng; spatal matchng 1. Introducton Unmanned aeral vehcle (UAV) mages have been extensvely used n many applcatons, e.g., agrcultural management (Habb et al., 2016), buldng model reconstructon (Acard et al., 2016) and transmsson lne nspecton (Jang et al., 2017), due to ther hgh spatal resoluton and flexble data acquston. Usually, for most market-avalable UAV platforms, conventonal navgaton devces, namely, the combned GNSS/IMU (Global Navgaton Satellte System / Inertal Measurement Unt) systems, cannot be used onboard for the drect geo-referencng (Turner et al., 2014), manly because of the payload lmtaton of UAV platforms and the hgh costs of these devces. Therefore, pror to above-mentoned applcatons, mage orentaton s requred to resume accurate camera poses for subsequent processng and nterpretaton, and relable mage matchng guarantees the success and precson of mage orentaton. 1

2 In the lterature, local feature based matchng has become the domnant paradgm, whch conssts of three major steps: (1) feature extracton for ndvdual mages; (2) feature matchng for mage pars; and (3) geometrcal verfcaton to elmnate outlers. Recent years have seen an exploson of actvty n the areas of feature extracton and matchng, whch can be observed from the earlest corner detectors (Harrs and Stephens, 1988) to the newly nvarant detectors (Mkolajczyk and Schmd, 2005) n the felds of dgtal photogrammetry and computer vson. Invarant detectors descrbe the local regons of nterest ponts wth feature vectors, namely, descrptors of feature ponts, whch smplfes feature matchng by searchng the nearest pont wth the smallest Eucldean dstance between two descrptor sets. Although feature descrptors facltate mage matchng wth smple vector comparsons, ntal canddate matches generated from the nearest-neghbor searchng technque are nevtably contamnated by false matches, due to the only usage of local appearances for feature descrptor and smlarty measurement. In addton, for oblquely captured mages (Jang and Jang, 2017b), occlusons and perspectve deformatons would further cause a majorty of false matches, even wth the ntegraton of the two well-known technques for outler elmnaton: cross-check (two features are the nearest neghbors of each other at the same tme) and rato-test (the rato between the shortest and the second shortest dstances s lower than a specfed threshold) (Lu et al., 2016). Therefore, geometrcal verfcaton plays a crucal role n the ppelne of local feature based matchng. Geometrcal verfcaton s the step of computng a transformaton from ntal matches, and classfyng them nto nlers and outlers based on whether or not a pont s geometrcally consstent wth the estmated transformaton model, where nlers and outlers represent true matches and false matches, respectvely. Generally, the strateges for geometrcal verfcaton can be categorzed nto two groups. For the frst group, a geometrcal transformaton, e.g., a smlarty or affne transformaton, s explctly estmated from ntal canddate matches. Due to ts ablty to tolerate a large fracton of outlers, the random sample consensus (RANSAC) (Fschler and Bolles, 1981) method s one of the most popular tools for robust model estmaton. The RANSAC algorthm operates n a hypothesze-and-verfy framework n whch subsets of nput data ponts are randomly selected and model hypotheses are estmated from the subsets; each model s then scored usng the entre data ponts, and the model wth the best score s the soluton. The RANSAC algorthm can provde a very accurate soluton even wth hgh levels of outler ratos reachng to 70 percent (Chum and Matas, 2008). However, the computatonal costs of the basc RANSAC method ncrease exponentally wth the percentage of outlers. To cope wth ths frequently arsng problem, many varants of the RANSAC method have been desgned and proposed to mprove ts effcency n terms of hypothess generaton and model verfcaton (Raguram et al., 2008), and promsng results have been reported. In the work of Chum et al. (2003), a locally optmzed RANSAC, namely, LO-RANSAC, was ntroduced to decrease the number of samples by ntegratng a local optmzaton stage nto the classcal RANSAC framework. Chum and Matas (2005) proposed a progressve sample consensus (PROSAC) for effcent hypothess generaton, whch exploted the lnear orderng structure of ntal correspondences to draw samples, nstead of the unform samplng strategy used n the RANSAC method. An overvew of recent research n the varants of the RANSAC method was comprehensvely revewed n Raguram et al. (2013). In contrast to the explct model estmaton, methods from the second group mplctly fnd a geometrcal transformaton, whch s estmated by usng the Hough transform (HT) (Hough, 1962). The core dea of HT s that feature locatons and transformaton parameters are nternally correlated; each feature locaton n the feature space votes for one bn or multple bns n the parameter space, and peaks of bns reveal promsng geometrcal transformatons between two mages. Compared wth RANSAC-based methods, the Hough votng scheme has two mportant advantages. Frst, t can tolerate hgher outler ratos based on the observaton that transformatons between false matches are not consstent, whch leads to random votes n the votng space; consstent transformatons between true matches result n centralzed votes, 2

3 whch forms salent peaks n the votng space. Second, the Hough votng scheme replaces the hypothesze-and-verfy framework of the RANSAC method wth a drect votng strategy, and hgh effcency can be acheved even for a degenerated confguraton wth a very hgh outler rato. Consequently, the Hough votng scheme has been reported n some research for the sake of effcent geometrcal verfcaton. In the work of Lowe (2004), the Hough transform that estmates a smlarty transformaton was utlzed to fnd all feature clusters wth at least three entres n a bn, and each cluster s then used n a geometrcal verfcaton procedure to refne the transformaton. For mage retreval n large-scale scenaros, L et al. (2015) ntroduced a strategy that uses parwse geometrc relatons derved from the rotaton and scalng relatons between correspondences for match verfcaton, and a reduced correspondence set was generated to accelerate geometrcal verfcaton, whch was acheved by usng the one-versusone matchng strategy and the Hough votng scheme. To reduce the tme costs of hypothess generaton, Schönberger et al. (2016) ntegrated a Hough votng scheme nto the tradtonal RANSAC method for the searchng of the most promsng transformatons. However, verfcaton approaches based on the Hough votng scheme are not as accurate as the RANSAC-based methods, manly because of ether the coarse votng space quantzaton or the weak geometrcal consstency, e.g., only the rotaton and scalng parameters used for the transformaton approxmaton between two mages. Therefore, other attempts that ncorporate spatal flters for outler elmnaton have been exploted, whch are used as a pre-processng or post-processng step before or after a RANSAC-based method. Usually, a spatal flter used as a pre-processng step ams to ncrease the rato of nlers and accelerate the convergence speed of the RANSAC method. In the work of Sattler et al. (2009), a spatal consstency check was ntroduced to generate a reduced correspondence set wth a sgnfcantly ncreased nler rato, whch was mplemented by takng nto account the matchng qualty n a feature s spatal neghborhood. Consderng mage matchng wth multple nearest neghbors for each feature pont, Lu et al. (2016) proposed a straghtforward and effectve method to flter false matches from ntal canddate matches usng a geometrcal consstency votng strategy. In the research, two transformaton parameters, namely, the rotaton and scale, were adopted to construct the votng space and to reduce the votng complexty, and promsng results were presented when the nler rato was below 10 percent. On the contrary, a spatal flter used as a post-processng step ntends to remove remanng false matches after geometrcal verfcaton, whch s usually mplemented by the analyss of local spatal relatonshps. Yao and Cham (2007) desgned a moton consstent functon to detect outlers based on the observaton that true matches n a small neghborhood tend to have consstent locaton changes. Consderng three local spatal relatonshp constrants, Hu et al. (2015) amended the standard feature matchng ppelne to ncrease ts relablty, rather than only usng the appearance nformaton. Compared wth the former strategy,.e., pre-processng, the post-processng strategy depends on a hypothess that a global transformaton has been establshed, and ts performance s prone to be affected by outlers because only local relatonshps are used for spatal consstency check. Consequently, the combnaton of the Hough votng scheme and the RANSAC algorthm can enhance both of ther strengths to acheve hgh effcency and hgh precson for geometrcal verfcaton. The Hough votng scheme s adopted to flter obvous outlers and ncrease nler ratos of ntal canddate matches; then, the RANSAC method s used to extract relable nlers that are geometrcally consstent wth an estmated transformaton. Ths combnaton has been reported n recent research, e.g., the work of Lu et al. (2016). However, as noted n Sattler et al. (2009), hgh computatonal costs can frequently arse due to parwse geometrcal comparsons, whch s commonly used to derve the rotaton and scale relatons of two mages. For hgh spatal-resoluton mages n the feld of photogrammetry, a large number of ntal canddate matches can be searched, especally for UAV mages; besdes, mage matchng wth multple nearest neghbors further ncreases the number of matches (Lu et al., 2016), whch notceably ncreases the complexty of the establshment of geometrcal relatons. Second, as observed 3

4 from the above-mentoned research, almost all verfcaton strateges are executed n the magespace, and a complex transformaton wth at least two parameters s mandatorly requred, whch leads to further ncreasng computatonal costs. Consderng dfferent characterstcs of mages captured n the feld of photogrammetry, some consderatons for geometrcal verfcaton have been documented. For UAV mage georegstraton, Zhuo et al. (2017) proposed a matchng ppelne wth pxel-dstances as a global geometrcal constrant, where a hstogram votng technque for locaton changes of matches was used to verfy ntal matches after the elmnaton of dfferences n the rotaton and scale. Tsa and Ln (2017) suggested checkng whether the dstances of feature ponts n horzontal and vertcal drectons are smlar to others, because the UAV orthomages have been coarsely algned. Then, a smple three-sgma rule was utlzed to elmnate false matches. These two technques can acheve hgh effcency wth 1D-votng because complex transformatons have been smplfed as a 2D-translaton, and parwse geometrcal comparsons for the estmaton of rotaton and scale parameters can also be avoded. Inspred by ths observaton, ths paper explots onboard GNSS/IMU data to acheve effcent geometrcal verfcaton n UAV mage matchng, whch s mplemented by smplfyng the complex transformaton and avodng the parwse geometrcal comparson for parameter estmaton. In our prevous studes, relevant research usng onboard GNSS/IMU data for match par selecton and geometrcal rectfcaton has been documented (Jang and Jang, 2017a, b). Ths paper proposes an effcent strategy for geometrcal verfcaton n UAV mage match. The basc dea s to smplfy the complex geometrcal transformaton of correspondences n the mage-space by ther projecton n the object-space. Frst, rough POS of each mage s calculated by usng on-board GNSS/IMU data, and feature ponts of canddate matches are projected onto a specfed elevaton plane. Second, two projected ponts of each ntal canddate match form a moton n the object-space. Then, a herarchcal moton consstency constrant (HMCC) s desgned and mplemented for obvous outler elmnaton. Fnally, comprehensve analyss and comparson of the proposed algorthm are conducted by usng real UAV mages. Ths paper s organzed as follows. Secton 2 detals the HMCC, whch s followed by the mplementaton of the HMCC-RANSAC algorthm n Secton 3. The proposed scheme for geometrcal verfcaton s comprehensvely compared and analyzed n Secton 4, and some aspects are dscussed n Secton 5. Fnally, Secton 6 presents the conclusons. 2. Herarchcal moton consstency constrant To acheve effcent geometrcal verfcaton n UAV mage matchng, ths paper ams to explot the use of onboard GNSS/IMU data for the smplfcaton of the transformaton model between correspondences. The workflow for UAV mage matchng s parttoned nto two parts, as shown n Fgure 1. The nput mages are two UAV mages. For the frst part labeled 1, feature ponts are extracted from ndvdual mages by usng the SIFT detector (Lowe, 2004), and a 128- dmensonal descrptor s assgned to one feature pont based on local appearances around the pont; feature matchng operatons are then conducted between two descrptor sets, whch s acheved by searchng the nearest feature pont wth the smallest Eucldan dstance. Two wellknown technques, namely cross-check and rato-test, are subsequently executed to detect false correspondences from ntal matches. The frst part represents a standard ppelne to obtan ntal canddate matches, as documented n the work of Lowe (2004). Therefore, the prmary contrbuton of ths paper s presented n the second part of the workflow, whch s desgned as a geometrcal verfcaton step for false match elmnaton. For the mprovement of effcency, ths procedure ncludes two steps: (1) the frst-stage outler elmnaton to ncrease nler ratos, and (2) the second-stage outler elmnaton wth the RANSAC method to refne fnal matches. In ths paper, the RANSAC method for the estmaton of a fundamental matrx s utlzed. Thus, the prmary work of ths paper s to mplement the herarchcal moton consstency constrant (HMCC) algorthm for the frst-stage outler elmnaton, whch s mplemented as follows. 4

5 Fgure 1. The overall workflow for UAV mage matchng. The work n the gray regon ndcates the prmary contrbuton of ths study From ntal canddate matches to motons The key dea of the HMCC algorthm s to convert the complex transformaton model of correspondences n the mage-space to a smple 2D-translaton n the object-space. Therefore, pror to the projecton transformaton between the mage-space and the object-space, camera poses should be calculated. For oblque UAV photogrammetry, onboard GNSS/IMU data provde the postons and orentatons of the platform, and camera poses can be parameterzed wth the platform poses and ther relatve poses (Sun et al., 2016). Suppose that the relatve pose between the platform and one camera s estmated and presented by a rotatonal matrx and a translatonal vector t ; for one exposure staton, the orentaton and poston of the platform are denoted by and, respectvely. Then, the actual camera pose can be calculated by Equaton (1) Rc rr T Tc R t T (1) where R and are the rotatonal matrx and translatonal vector of the camera pose at the r c T c R T current exposure staton, respectvely. For nadr UAV photogrammetry, the above formulas are also applcable except that the relatve rotaton s represented by an dentty matrx and the relatve translaton s ndcated by a zero vector. By usng the calculated camera poses, two projecton ponts of one correspondence can be computed by projectng feature ponts onto a specfed elevaton plane, whch construct a 5

6 drectonal vector that contans one startng vertex and one termnal vertex. In ths study, the drectonal vector defnes a prmtve, termed moton, to mplement the HMCC algorthm. The generaton of motons from an mage par s mplemented as follows: Gven an mage feature ponts are frst extracted and denoted by F I x j, y j, d j p j x j, y j and feature descrptor ; for an mage par 1 2 matches C f, f 1 2 f F I 1 and f F I 2. Wth the ad of camera poses, feature locatons one canddate match d j wth feature locaton I, I 1 2 I, ntal canddate are then establshed by matchng two descrptor sets, where c llustrated n Fgure 2(a). The projecton ponts of s and t s n C are projected onto a specfed elevaton plane 1 p and 2 p p, p 1 2 Z Z 0, of, as are represented by two ponts n the object-space, respectvely. These two projecton ponts form a moton t where s the startng vertex and s the termnal vertex. To facltate the analyss of moton consstency, a moton s defned as a drectonal vector wth two propertes, namely the length of the vector and the drecton of the vector wth respect to the X-axs of the object-space coordnate system, as shown n Fgure 2(b). Thus, ntal canddate matches of one mage par can generate a moton set, and the spatal relatonshps of correspondences n the mage-space are converted to that n the object-space. L st,, (a) (b) Fgure 2. The generaton of motons from ntal matches: (a) the projecton transformaton from mage-space to object-space; (b) the defnton of motons as drectonal vectors. After the projecton transformaton, spatal relatonshps of correspondences n the magespace are smplfed as a smple 2D-translaton n the object-space, as presented n Fgure 2(b). Consderng an deal data acquston wth no terran relef observed from felds and very precse camera poses are calculated, motons of true matches would degenerate to ponts when Z the heght of the projecton plane s dentcal to that of the feld. The degenerated motons are ntersecton ponts of correspondng rays (fred from projecton centers and goes through correspondng ponts), whch can be observed n Fgure 2(a). However, n ths study, camera poses are calculated from on-board GNSS/IMU data, whch leads to the rough approxmatons of real poses. Thus, motons of true matches n the object-space cannot degenerate to ponts. In addton, for one mage par, bas errors from camera poses can be modeled as the relatve orentaton between two mages, whch causes the consstent rotaton and translaton of correspondng rays. For true matches, consstent motons wth slght dfferences n length and drecton are generated; on the contrary, for false matches, generated motons would not be consstent n terms of length and drecton. Ths observaton s the most mportant to desgn and mplement the moton consstency constrant for outler elmnaton Herarchcal Moton Consstency Constrant - HMCC 6

7 Spatal relatonshps of correspondences are reduced to a smple 2D-translaton after the converson from the mage-space to the object-space. The 2D-translaton of one correspondence n the object-space s defned as a moton, whch s presented by a vector wth two attrbutons, namely, the drecton and length. Consstent motons wth lmted dfferences n drecton and length are generated from true matches; on the contrary, nconsstent motons are created from false matches. Ths s the foundaton for the moton consstency constrant. In ths study, for effcent and relable outler elmnaton, a herarchcal strategy s adopted to mplement the moton consstency constrant, namely HMCC, whch conssts of three major steps: (1) a global drecton consstency constrant for the frst-stage outler elmnaton; (2) a local drectonchange consstency constrant for the second-stage outler elmnaton; and (3) a global length consstency constrant for the thrd-stage outler elmnaton. Suppose one mage par wth ntal canddate matches, the motons generated from M m : 1,2,..., n ; each moton conssts ntal canddate matches are denoted by of a startng pont s x, y and a termnal pont t x, y s s n t t m ; the drecton of one moton s defned as the angle by whch the X-axs of the object-space coordnate system can be rotated counter-clockwse to the correspondng vector of the moton; the length of the moton s defned as the norm of the correspondng vector. By usng the startng pont termnal pont t Equatons (2) and (3), the drecton and length of the moton m yt y s drecton arctan xt xs 2 2 t s t s s and the can be respectvely calculated by length x x y y (3) where drecton ranges from 0 to 360, and length s a real value. Based on the defnton, the HMCC algorthm s mplemented as follows. For the frst step of the HMCC algorthm, outlers are elmnated by usng a global drecton consstency constrant. Ths constrant s acheved based on the observaton that drectons of motons for true matches vary n a lmted range; however, drectons of motons for false matches would be random. Thus, a statstcal analyss of drectons s capable of detectng the majorty of outlers n the frst-stage of the HMCC algorthm. For one moton m, the drecton s calculated by Equaton (2); then, a drecton lst dlst : 1,2,..., n n M can be obtaned from motons n, as shown n Fgure 3(a), where motons are represented by drectonal vectors, and drectons wth respect to the X-axs of the object-space coordnate system are computed and lsted at the bottom. Consderng a possble hgh outler rato of ntal canddate matches, a Hough votng scheme that s voted by the drecton lst s appled to remove outler wth random drectons, because of ts robustness to random noses (outlers). In Fgure 3(a), the moton n orange s consdered an outler, and the others are nlers wth consstent drectons. The algorthmc mplementaton of the global drecton consstency constrant s descrbed n detal n Secton 3. The second step s to elmnate the remanng outlers. A local consstency constrant s desgned based on drecton-changes of motons, whch s termed the local drecton-change dlst consstency constrant n ths study. For one moton (2) m, the k nearest neghborng motons M m : j 1,2,..., k, j are searched from M ; then, the drecton-change between nn the moton j m and one of ts neghbors subtracton of the drecton j from the drecton M nn s calculated by the absolute, as presented by Equaton (4) m j n 7

8 where dclst dcj j (4) denotes the operaton of absolute subtracton. Therefore, a drecton-change lst dcj between the current moton m and ts neghborng motons M nn s obtaned. As shown n Fgure 3(b), the red vector s the current motons, and ts neghborng motons are rendered n green and orange; drecton-changes of the current moton are sorted and lsted at the bottom. In order to relef the nfluence of outlers, the drecton-change of s defned as the medan of the lst dclst. Smlarly, a Hough votng scheme that s voted by each moton s drecton-changes s desgned for the further elmnaton of outlers. In Fgure 3(b), the moton n orange s consdered an outler. The algorthmc mplementaton of the local drecton-change consstency constrant s descrbed n detal n Secton 3. For the thrd step of the HMCC algorthm, motons wth very short or long lengths are consdered as outlers, because lengths of motons for true matches vary n a lmted range; on the contrary, lengths of motons for false matches would be random. In ths step, a global length consstency constrant s utlzed to elmnate outlers that are n nearly the same drectons of true matches. For one moton m, the length s calculated by Equaton (3); then, a length lst llst l : 1,2,..., n can be obtaned from l n motons n score test s used to remove outlers, whch s calculated by Equaton (5) where l zscore l l M m ; fnally, a smple z- (5) and are the average and the standard devaton of lengths n length of the current moton m llst ; l s the. In ths study, a match wth the z-score greater than three s consdered as an outler, whch s the commonly used three-sgma rule. Fgure 3. Illustraton of the HMCC algorthm for outler elmnaton: (a) the global drecton consstency constrant; (b) the local drecton-change consstency constrant. Motons are represented by drectonal vectors n the object-space. The motons correspondng to nlers and outlers are rendered n green and orange, respectvely. The current moton s n red. 3. Implementaton of the effcent geometrcal verfcaton The workflow of effcent geometrcal verfcaton conssts of the HMCC and RANSAC. The former ams to elmnate obvous outlers and ncrease the nler rato of ntal canddate matches; the latter s used to refne fnal matches wth a rgorous geometrcal model. Due to ts robustness to outlers, a Hough votng scheme s adopted n the frst and second steps of the HMCC, whch ncludes four man steps: (1) samplng of the votng space; (2) votng of the 8

9 accumulaton array; (3) peak determnaton; and (4) nler extracton. Suppose motons M m : 1,2,..., n generated from ntal canddate matches, the startng ponts of motons are extracted and denoted by S s : 1,2,..., n neghborng motons M nn m j : j 1,2,..., k, j ; then, for one moton n m, the can be searched from M, where the K-nearest neghbors algorthm (Cover and Hart, 1967) s ndexed by the startng ponts and used for neghbor searchng. The global drecton consstency constrant and the local drectonchange consstency constrant s mplemented as follows. The votng scheme for the global drecton consstency constrant (1) The varaton range of the drecton parameter s 0 to 360, and the samplng nterval of the votng space s set as 10. Thus, a one-dmensonal accumulaton array wth 36 bns s created and ntalzed to zero. (2) For each moton n M, the bn ndex calculated from ts drecton s A m dx recorded, and the votes of the bn wth the ndex dx S n the accumulaton array A are ncreased by one. Thus, n votes wll occur n the accumulaton array. (3) The peak bn wth most votes s determned from the accumulaton array, and the neghborng bns wthn 5 steps away from the peak bn are also selected, as long as ther votes are greater than 20 percent of the peak bn. (4) For one moton m, t s consdered as an nler f ts bn ndex ndex range of selected bns; otherwse, t s recorded as an outler. dx A A k s wthn the The votng scheme for the local drecton-change consstency constrant (1) The varaton range of the drecton-change parameter s 0 to 30, and the samplng nterval of the votng space s set as 3. Thus, a one-dmensonal accumulaton array (2) For one target moton A wth 10 bns s created and ntalzed to zero., M m : j 1,2,..., k, j m are searched, and a drecton-change tem one of ts neghborng motons drecton-change lst dclst dcj ndex dx j accumulaton array target moton k neghborng motons m j n dc j M nn nn between the target moton j m and s calculated, whch results n a ; then, for each tem dc j n s computed, and the votes of the bn wth the ndex A are ncreased by one. In addton, the bn ndex m s set as the medan of the drecton-change lst A dclst, ts bn dx j dx dclst. Thus, A n the of the votes wll occur n the accumulaton array. (3) The peak bn wth most votes s determned from the accumulaton array, and the neghborng bns wthn 3 steps away from the peak bn are also selected f ther votes are greater than 40 percent of the peak bn. (4) For one moton m, t s selected as an nler f ts bn ndex dx s wthn the ndex range of selected bns; otherwse, t s consdered as an outler. For neghbor searchng, k s set as 7 n ths study. After the applcaton of the HMCC algorthm, the nler rato of ntal canddate matches s ncreased, whch ensures the hgh convergent speed of the subsequent RANSAC method. In ths study, the classcal RANSAC method estmatng a fundamental matrx wth the seven-pont algorthm (Zhang, 1998) s utlzed to refne the fnal matches. To guarantee a hgh nler rato, the maxmum resdual error for the model estmaton s set as 1.0 pxel n the RANSAC method. In concluson, the above- kn 9

10 mentoned procedures mplement the effcent geometrcal verfcaton, namely HMCC- RANSAC, whch s lsted n Algorthm 1. Algorthm 1 Input: n Effcent Geometrcal Verfcaton (HMCC-RANSAC) ntal canddate matches Output: true matches C fn C, rough POS and projecton plane Z Z 0 1: procedure HMCC-Flter M m : 1,2,..., n from ntal canddate matches C 2: Generate motons 3: Vote wth the global drecton consstency constrant ( M d M 4: Vote wth the local drecton-change consstency constrant ( dc Md ) 5: Remove outler wth the global length consstency constrant ( Mred Mdc ) 6: Extract nlers accordng to the reduced motons ( Cred Mred ) 7: end procedure 1: procedure RANSAC-Estmaton 2: Set the maxmum nler error =1.0 3: RANSAC for rgorous geometrcal verfcaton 4: Extract nlers accordng to the result of the RANSAC ( Cfn Cred ) 5: end procedure M ) 4. Expermental results In the experments, a UAV dataset s used to evaluate the proposed algorthm. Image pars wth dfferent confguratons between oblque and nadr mages are desgned for the nsght analyss of the HMCC algorthm. Frst, motons are generated from ntal canddate matches. Second, the use of the three constrants, ncludng the global drecton consstency constrant, the local drecton-change consstency constrant and the global length consstency constrant, s analyzed n the procedure of outler elmnaton, whch results n a reduced correspondence set wth a sgnfcantly ncreased nler rato. Then, tests for ts robustness to outlers are devsed and conducted by usng the reduced correspondence set. Fnally, two methods that we refer to LO-RANSAC (Chum et al., 2003) and GC-RANSAC (Lu et al., 2016) are effcently mplemented, and comprehensve comparsons wth the HMCC-RANSAC are performed n tests of feature matchng and bundle adjustment (BA) Dataset Ths dataset s collected from an urban area wth the exstence of dense buldngs. A multrotors UAV platform equpped wth one Sony ILCE-7R camera, dmensons of 7360 by 4912 pxels, s adopted for the acquston campagn. For accurate photogrammetrc measurement, the used camera has been calbrated usng a calbraton model wth eght parameters: one for the focal length, two for the prncpal pont, and three and two for coeffcents of radal dstortons and tangent dstortons, respectvely. Because the magng system s equpped only one camera, two ndvdual campagns are conducted to obtan nadr (zero roll and ptch angles for camera nstallaton) and oblque mages (zero roll and 45 ptch angles for camera nstallaton), respectvely. The detals of data acquston can be found n our prevous work (Jang and Jang, 2017b). At the flght heght of 300 m wth respect to the locaton from whch the UAV takes off, a total number of 157 mages are captured, and the GSD of nadr mages s approxmately 4.20 cm. The detals of ths data acquston are presented n Table 1. 10

11 Item Name UAV type Table 1. Detaled nformaton for data acquston of the test ste. Value Flght heght (m) 300 mult-rotor Camera mode Sony ILCE-7R Sensor sze (mm mm) Focal length (mm) 35 Camera mount angle ( ) nadr: 0 oblque: 45/-45 Image sze (pxel pxel) Number of mages 157 Forward/sde overlap (%) 64/48 GSD (cm) 4.20 For system calbraton, the lever-arm offset s assumed to be zero because the dstance to the scene s much larger than the lever-arm offset. The bore-sght angles are much smaller than the camera nstallaton angles. Thus, only camera mountng angles are used to approxmate the relatve poses of camera sensors wth respect to UAV platforms. Image poses are calculated by usng on-board GNSS/IMU data of the UAV platform and camera mountng angles n the acquston campagn. Frst, GNSS/IMU data recorded n the navgaton system are converted to a photogrammetrc system. In ths study, a local tangent plane (LTP) coordnate system wth ts orgn located at the center of the test ste s used. Second, the rough pose of each mage can be calculated by usng camera mountng angles, because they are the approxmate respectve dsplacements between UAV platforms and oblque cameras. The detaled process for mage pose calculaton s presented n Jang and Jang (2017b). By usng the computed mage poses, ground coverage of mages for the dataset s llustrated n Fgure 4, whch are the projecton of mage corners on the average elevaton plane of the test ste. Fgure 4. Ground coverage of the dataset (Jang and Jang, 2017b). For performance evaluaton of the HMCC algorthm on feature matchng, four mage pars are confgured usng both nadr and oblque mages n the dataset, as lsted n Table 2. Images of the frst par come from the nadr acquston campagn, and a large number of features can be matched, whch s used to evaluate the performance of the moton consstency constrant on nadr mages. The second to the fourth mage pars wth ncreasng ntersecton angles contan 11

12 oblque mages, whch ams to assess the nfluence of varyng perspectve deformatons on the performance of the HMCC algorthm, as well as approvng ts adapton to oblque mages. Due to large perspectve deformatons, many fewer ntal canddate matches are extracted from the thrd and fourth mage pars. For the used dataset, drecton ndcators of the four mage pars are llustrated n Fgure 5. Fgure 5. Illustraton of a penta-vew oblque photogrammetrc system (Jang and Jang, 2017b). The acquston of the used dataset s desgned to smulate the ablty of ths system. Table 2. Image pars for performance evaluaton of feature matchng. No. Drecton Descrpton 1 V-V Two nadr mages 2 V-F One nadr mage and one front mage 3 B-F One back mage and one front mage 4 L-F One left mage and one front mage 4.2 Motons generated from ntal canddate matches The four mage pars lsted n Table 2 are used to evaluate the performance of the HMCC algorthm on feature matchng. The numbers of extracted feature ponts are 10,351 and 11,116 for the frst mage par; 10,893 and 8,193 for the second mage par; 8,575 and 8,808 for the thrd mage par; and 8,348 and 8,808 for the fourth mage par. By searchng the nearest feature pont between two descrptor sets, ntal canddate matches of each mage par are obtaned, where two technques, namely, cross-check and rato-test, are adopted to reject outlers. In ths study, the default parameters of the SIFT algorthm wth rato-test of 0.8 and max-dstance of 0.7 are used n the feature matchng stage. Intal canddate matches of the four mage pars are shown n Fgure 6, where the numbers of the ntal matches are 383, 134, 68 and 97 for the four mage pars, respectvely. For a better nterpretaton of the matchng results, the oblque mages are geometrcally transformed such that lnes of true matches are approxmately parallel to each other; however, lnes of false matches ntersect others. Observng the matchng results, we can see that even wth the two methods for outler elmnaton, a large proporton of false matches exst n ntal matches, especally for the last two mage pars, due to ther relatvely larger perspectve deformatons. By checkng the matchng results, nler ratos of the four mage pars are approxmately 52.5%, 32.8%, 26.5% and 27.8%, respectvely. 12

13 (a) (b) (c) (d) Fgure 6. Intal canddate matches of the four mage par: (a) ntal matches of the frst mage par; (b) ntal matches of the second mage par; (c) ntal matches of the thrd mage par; (d) ntal matches of the fourth mage par. 13

14 Pror to outler elmnaton, moton generaton from ntal canddate matches s desgned as the frst step of the HMCC algorthm, whch converts the complex relatonshp of matches n the mage-space nto a 2-dmensonal translaton n the object-space. Motons are generated by projectng the feature ponts of matches onto an elevaton plane of the test ste, as descrbed n Secton 2.1. In ths study, the average alttude of the test ste s near zero meters. In addton, to decrease the nfluence of terran relef (e.g., the heght of buldngs), a projecton plane wth an alttude of negatve one hundred meters s used for moton generaton. The results are shown n Fgure 7, where red dots and yellow dots represent startng ponts and termnal ponts of motons; and each moton s represented by a blue lne lnkng a start pont and a termnal pont. The motons reveal the geometrcal relatonshp of ntal canddate matches n the object-space, whch s modeled as a 2D-translaton nstead of the complex transformaton n the mage-space. By checkng the motons generated from the four mage pars, we can see that: (1) the motons of some matches are approxmately parallel to each other; however, for the other motons, both the drecton and the length are randomly assgned. Ths can be verfed by the motons of the frst mage par n whch a large number of motons have opposte drectons, as presented n the top-left part of Fgure 7(a); (2) the dscrmnatve ablty of the moton drecton s stronger than that of the moton length; thus, the HMCC algorthm depends on a herarchcal strategy from the moton drecton to the moton length for outler elmnaton. Ths observaton provdes a clue to remove outlers based on analyss of motons n the objectspace and forms the foundaton of the HMCC algorthm for outler elmnaton. (a) (b) (c) (d) Fgure 7. Motons generated from ntal canddate matches: (a), (b), (c) and (d) are motons of the frst, second, thrd and fourth mage par, respectvely. 14

15 4.3 Outler elmnaton based on the HMCC algorthm The HMCC algorthm conssts of three major steps. Frst, the global drecton consstency constrant s used to detect motons that are markedly dfferent n the moton drecton. A large proporton of ntal canddate matches would be contamnated by outlers, especally for pars of oblque mages, as shown n Fgure 7(c) and Fgure 7(d). Thus, the Hough votng scheme for the moton drecton s utlzed n the frst stage due to ts robustness to random noses. The results of the frst-stage outler elmnaton are shown n Fgure 8, where motons rendered n green ndcate nlers; motons rendered n red are consdered outlers. The results clearly show that the red-colored motons have randomly assgned drectons, whch are obvously dfferent from the motons n green; for the frst and second mage pars, almost all motons wth random drectons can be detected, manly because of the strong drecton consstency wthn nlers, as llustrated n Fgure 8(a) and Fgure 8(b). However, due to larger perspectve deformatons, the drecton consstency of the last two mage pars s weaker than that of the frst two mage pars, whch results n the partal detecton of the motons wth random drectons, as presented n Fgure 8(c) and Fgure 8(d). Although the weaker consstency observed from the motons of the last two mage pars, the varyng range of drectons s lmted, whch can be deduced from the votng results of the global drecton, as shown n Fgure 9 (Notce that, the accumulaton array s extended by copyng the frst nne elements to the end due to the cyclcty of the moton drecton). For each mage par, the number of salent peak bns does not greater than two. Consderng that the samplng nterval of the votng space s set as 20, the varyng range of drectons wth respect to the peak bns does not exceed 40. In addton, for the four mage pars, salent peaks can be easly determned from the accumulaton arrays. In ths stage, the numbers of detected outlers are 46, 21, 22 and 28 for the four mage pars, respectvely, whch are 12.0%, 15.7%, 32.4% and 28.9% of the correspondng total matches. The second step of the HMCC algorthm s to remove remanng outlers by usng the local drecton-change constrant based on the observaton that the drecton-change of one moton wth respect to ts neghborng motons s restrcted to a lmted range. Smlarly, to cope wth the nfluence of outlers, a votng scheme wth a strct parameter set s utlzed. The results are shown n Fgure 10, where motons n both green and red are generated from nlers of the global drecton consstency constrant. Thus, n ths stage, the estmaton precson of local drecton-changes can be ncreased by the hgh nlers ratos, when cooperatng wth the technque that the medan of drecton-changes s consdered as the expected value of the target moton. From the results of outler detecton, we can see that: (1) only a few outlers are detected for the frst and second mage pars, as shown n Fgure 10(a) and Fgure 10(b), because most of them have been fltered based on the drecton consstency constrant. (2) For the thrd and fourth mage pars, most of the remanng motons wth nconsstent drecton-changes are detected, whch can further ncrease nler ratos. Smlarly, the accumulaton arrays of the votng scheme for the drecton-change consstency constrant are presented n Fgure 11. For each accumulaton array, the peak bn can be notceably observed, and almost all drectonchanges of motons are lmted to the frst bn. We can conclude that a majorty of drectonchanges do not exceed 3, because the samplng nterval of the votng space n ths stage s confgured as 3. Thus, the nler ratos are dramatcally ncreased after the use of abovementoned votng schemes. For the thrd stage, z-score tests are conducted to elmnate outlers that have almost the same drectons as the other motons, but havng obvously dfferent moton lengths, whch are prone to be outlers. For the four mage pars, only one moton does not pass the z-score test, whch s labeled n Fgure 10(b). Usng the HMCC algorthm for outler elmnaton, a majorty of outlers can be detected from ntal canddate matches, and the reduced correspondences are presented n Fgure 12. For each mage par, almost all green lnes are parallel to each other, whch ndcates that no obvously nconsstent matches exst. Therefore, the HMCC algorthm can dramatcally ncrease nler ratos of the four mage pars. 15

16 (a) (b) (c) (d) Fgure 8. The global drecton consstency constrant for the frst-stage outler elmnaton: (a), (b), (c) and (d) are results of the frst, second, thrd and fourth mage par, respectvely. (a) (b) (c) (d) Fgure 9. The votng results for the global drecton consstency constrant: (a), (b), (c) and (d) correspond to the result of the frst, second, thrd and fourth mage par, respectvely. 16

17 (a) (b) (c) (d) Fgure 10. The local drecton-change consstency constrant for the second-stage outler elmnaton: (a), (b), (c) and (d) are results of the four mage pars, respectvely. (a) (b) (c) (d) Fgure 11. The votng results for the local drecton-change consstency constrant: (a), (b), (c) and (d) correspond to the result of the frst, second, thrd and fourth mage par, respectvely. 17

18 (a) (b) (c) (d) Fgure 12. Reduced correspondence set of the four mage par: (a), (b), (c) and (d) are the reduced correspondence sets of the four mage pars, respectvely. 18

19 4.4. Analyss of the nfluence of outlers on the HMCC algorthm The man purpose of the HMCC algorthm s to ncrease the nler rato of ntal canddate matches, whch tend to be domnated by outlers, especally for oblque mage pars wth large photometrc and geometrcal deformatons. Therefore, the robustness to outlers s an essental ndcator to evaluate ts performance. In ths secton, we desgn a strategy to generate matches wth a specfed outler rato and analyze the algorthm s robustness to outlers. To prepare expermental datasets, reduced correspondence sets of the four mage pars are frst refned by usng the RANSAC method wth the estmaton of a fundamental matrx, where the maxmum resdual error s confgured as 1.0 pxel to ensure hgh nler ratos. Then, outlers survvng from the eppolar constrant are further elmnated by manual nspecton, whch produces the total nlers for each mage par. Fnally, we can generate desred matches wth a specfed outler rato by addng random pont pars to correspondng nlers. For the four mage pars, the numbers of the total nlers are 204, 44, 15 and 20, respectvely. Outler ratos rangng from 0.1 to 0.9 are evenly sampled wth the nterval value of 0.1. To evaluate the robustness to outlers, the HMCC algorthm s executed on the artfcally contamnated matches for the frst-stage outler elmnaton, and the RANSAC method wth the same parameters as that for match refnement s subsequently appled to verfy the results of the HMCC. Two crtera, namely, precson and recall, are used for performance evaluaton. Precson s the rato of the numbers of nlers, whch are generated from the RANSAC method and the HMCC algorthm, respectvely. Smlarly, recall s defned as the rato of the number of nlers generated from the RANSAC method to the number of correspondng total nlers. For the four mage pars, statstcal results of precson and recall are respectvely shown n Fgure 13 and Fgure 14. It s clearly shown that hgh precson can be acheved when outler ratos are below 0.8 for the four mage pars; the precson of the frst and second mage pars s better than that of the last two mage pars even wth large outler ratos reachng to 0.9, due to ther relatve moderate deformatons caused by oblque angles. In addton, by observng the statstcal results of recall, as llustrated n Fgure 14, we can see that the metrc recall s scarcely affected by outler ratos, whch can be deduced from the slght changes of recall. For the frst and thrd mage pars, the mean of recall s greater than 90%; for the second mage par, the mean of recall s near 80%. However, the mean of recall s approxmately 65% for the fourth mage par, because large perspectve deformatons cause the weak consstency of motons. Ths can further lead to the elmnaton of matches n the analyss of the local drecton-changes. In concluson, the HMCC algorthm can robustly remove a majorty of outlers and notceably ncrease nler ratos when outler ratos are not greater than 0.8; besdes, ths algorthm can preserve a hgh percentage of nlers. Fgure 13. Statstcal results of precson for the four mage pars. 19

20 Fgure 14. Statstcal results of recall for the four mage pars Comparson wth varous geometrcal verfcaton methods In ths secton, two methods, namely, LO-RANSAC (Chum et al., 2003) and GC-RANSAC (Lu et al., 2016) are compared wth HMCC-RANSAC to assess ther performance n geometrcal verfcaton. Frst, the four mage pars, as descrbed n Secton 4.1, are used to evaluate ther performance n the flter and verfcaton stages of three methods. Second, an mage orentaton test s conducted to assess ther performance n terms of effcency, completeness, and accuracy. In ths study, the three methods are effcently mplemented by usng the C++ programmng language, and all experments are executed on an Intel Core PC on the Wndows platform wth a 3.4 GHz CPU and a 2.0 G GeForce GTX 770M graphcs card Comparson usng the four mage pars LO-RANSAC uses a local optmzaton stage to decrease the number of samples drawn. Smlar to our method, GC-RANSAC devses a canddate match flter to ncrease nler ratos based on the Hough votng scheme. Thus, performance comparson for the flter s frst conducted between GC-RANSAC and HMCC-RANSAC, and results are shown n Fgure 15. It s clearly shown that constant computaton costs are consumed by HMCC-RANSAC; however, for GC-RANSAC, tme consumpton s postvely proportonal to the number of matches. In addton, numbers of retaned matches from GC-RANSAC s greater than that from HMCC-RANSAC, manly because of the usage of more rgorous constrants n HMCC. The statstcal results of tme costs are lsted n Table 3. (a) (b) Fgure 15. Comparson of tme costs and numbers of nlers n the flter stage: (a) tme costs, and (b) numbers of nlers. 20

21 Fgure 16 shows the comparson results of tme costs and numbers of nlers for geometrcal verfcaton wth the estmaton of a fundamental matrx, and the statstcal results are lsted n Table 3. For both GC-RANSAC and HMCC-RANSAC, the classcal RANSAC method s used for the transformaton estmaton. The results show that the effcency can be dramatcally mproved by usng the flter for GC-RANSAC and HMCC-RANSAC, especally for the thrd and fourth mage pars wth lower nler ratos. In addton, comparatve effcency s acheved n GC-RANSAC and HMCC-RANSAC for the second mage pars; however, for the frst, thrd and fourth mage pars, HMCC-RANSAC acheves approxmately three tmes hgher effcency than that of GC-RANSAC, because more outlers are detected n HMCC- RANSAC based on the local drecton-change consstency constrant. (a) (b) Fgure 16. Comparson of tme costs and numbers of nlers n the geometrcal verfcaton stage: (a) tme costs, and (b) numbers of nlers. Table 3. Statstcal results of tme costs for the four mage pars (unt n seconds). No. LO-RANSAC GC-RANSAC HMCC-RANSAC Flter Verf Sum Flter Verf Sum Comparson usng bundle adjustment tests To assess the performance of the three methods for UAV mage matchng, the comparson s also conducted n an mage orentaton test. Image pars are frst selected by usng an overlap prncple, where one mage par would be preserved only f the dmenson of overlap exceeds half of the footprnt sze (Jang and Jang, 2017a). For the used dataset, a total number of 4130 pars are selected. Then, feature matchng s executed, and the comparson results of the three methods for geometrcal verfcaton s presented n Fgure 17, where the tme costs of the flter and verfcaton stages for GC-RANSAC are shown n Fgure 17(b) and Fgure 17(c), respectvely; Fgure 17(d) and Fgure 17(e) for HMCC-RANSAC. In addton, statstcal results of tme costs usng four metrcs, namely, Max, Average, Stddev and Sum, are lsted n Table 4. 21

22 (a) (b) (c) (d) (e) Fgure 17. Effcency comparson of the three methods: (a) tme costs of LO-RANSAC; (b) and (c) tme costs of GC-RANSAC n flter and verfcaton stages; (d) and (e) tme costs of HMCC- RANSAC n flter and verfcaton stages. 22

23 It s clearly shown that when drectly usng LO-RANSAC, the tme costs vary dramatcally; for some mage pars, more than 5 seconds are consumed n the verfcaton stage due to low nler ratos of ntal canddate matches, as llustrated n Fgure 17(a). On the contrary, by usng the pre-processng step, tme costs of the verfcaton wth RANSAC do not exceed one second n both GC-RANSAC and HMCC-RANSAC, as shown n Fgure 17(c) and Fgure 17(e). By comparng the results between GC-RANSAC and HMCC-RANSAC, we can conclude that: (1) constant and low tme costs are consumed n HMCC-RANSAC, as shown n Fgure 17(d); however, for GC-RANSAC, hgh tme costs can be observed from the flter step, as presented n Fgure 17(b); (2) by comparng the results shown n Fgure 17(c) and Fgure 17(e), HMCC- RANSAC acheves hgher effcency than GC-RANSAC for some mage pars, because the local drecton-change constrant can further ncrease nler ratos; (3) although effcent RANSAC can be acheved by usng the flter step n GC-RANSAC, the sum of tme costs s not notceably decreased because of the relatvely hgher tme complexty n the flter stage; however, for HMCC-RANSAC, hgh effcency can also be acheved n the flter stage wth a speedup rato greater than one hundred, whch s verfed by the mean and sum metrcs n Table 4. Table 4. Statstcal results of effcency comparson (unt n seconds). Item LO-RANSAC GC-RANSAC HMCC-RANSAC GC RANSAC HMCC RANSAC Max Mean Stddev Sum Table 5. Statstcal results of completeness and accuracy comparson (RMSE n pxels). Item LO-RANSAC GC-RANSAC HMCC-RANSAC No. ponts 86,110 85,491 82,472 RMSE Fgure 18. The reconstructed model by usng HMCC-RANSAC for geometrcal verfcaton. The Red rectangle represents the mage plane and the green lne lnks the projecton center and one of the corners of the mage plane. 23

24 After feature matchng of the selected mage pars, mage orentaton s subsequently conducted to resume accurate mage poses and reconstruct 3D ponts of the test ste. Table 5 presents the statstcal results of the experment n terms of the numbers of ponts and the RMSE n pxels. For lack of ground-truth data n ths test ste, only relatve orentaton s consdered for performance comparson. The results ndcate that mage orentaton can be successfully mplemented wth one of the three methods for geometrcal verfcaton; the RMSEs of mage orentaton wth GC-RANSAC and HMCC-RANSAC are comparable, whch s slghtly better than that wth LO-RANSAC. However, the largest number of ponts are reconstructed by LO- RANSAC, because a proporton of true matches are nevtably elmnated n the flter stage of both GC-RANSAC and HMCC-RANSAC. For the proposed algorthm n ths study, the number of lost ponts s approxmately 5 percent when compared wth the number of ponts recovered from LO-RANSAC. In other words, wth the completeness sacrfce of approxmately 0.05 tmes the total number of ponts, HMCC-RANSAC can mprove the verfcaton effcency wth a speedup rato near 5 for the dataset. The reconstructed model by usng the HMCC- RANSAC for geometrcal verfcaton s shown n Fgure 18. In order to assess the transferablty of the proposed algorthm, mage orentaton tests of addtonal three datasets are carred out. The three datasets cover dfferent landscapes: the frst dataset s collected from a suburban area covered by vegetaton and crossed by some ralroad tracks; the second test ste s a farmland wth domnant bare lands; the thrd test ste locates n an urban regon wth a central shoppng plaza surrounded by hgh resdual buldngs. For data acquston, a mult-rotor UAV equpped wth three dfferent oblque photogrammetrc systems s adopted, where an magng system equpped one camera wth 25 and -15 for ptch and roll angles s desgned for data acquston of the frst dataset; an magng system equpped two cameras for the second dataset; and a penta-vew magng system for the thrd dataset. The detaled nformaton for the three campagns s shown n Table 6. The numbers of collected mages are 320, 390 and 750 for the three datasets, respectvely. Based on the above-mentoned overlap prncple, 5,169, 5,842 and 18,283 mage pars are selected from the frst, second and thrd dataset, respectvely. Thus, mage orentaton tests are performed on the datasets after feature matchng of selected mage pars. Table 6. Specfcaton of mage orentaton tests for the addtonal three datasets. Item Dataset 1 Dataset 2 Dataset 3 (a) Data acquston UAV type mult-rotor mult-rotor mult-rotor No. cameras Camera mount angle ( ) front: 25, front: 25, -15 back: 0, -25 nadr: 0 oblque: 45/-45 Camera mode Sony RX1R Sony RX1R Sony NEX-7 Image sze (pxel pxel) No. mages (b) Image orentaton LO- RANSAC GC- RANSAC HMCC- RANSAC Tme (sec) No. ponts 173, , ,867 RMSE (pxel) Tme (sec) (1336.4/108.5) (2248.7/110.7) (990.6/256.6) No. ponts 171, , ,144 RMSE (pxel) Tme (sec) (9.4/118.7) (13.9/106.2) (17.0/420.2) No. ponts 162, , ,483 RMSE (pxel)

25 The statstcal results of mage orentaton are lsted n Table 6, and three metrcs, ncludng the tme nvolvng n geometrcal verfcaton, the number of reconstructed ponts and the RMSE estmated from bundle adjustment, are used for transferablty assessment. Smlarly, the three methods are compared n the mage orentaton tests of the three datasets. Notce, for GC-RANSAC and HMCC-RANSAC, the metrc tme consst of three parts, namely the total tme and the tme consumed respectvely n flter and verfcaton stages (values n the round bracket). We can conclude that: (1) for the three datasets, HMCC-RANSAC can acheve hgh effcency for geometrcal verfcaton wth lower tme consumpton of sec, sec and sec for the frst, second and thrd dataset, respectvely, and the average tme costs do not exceed sec n the flter stage; (2) comparable tme costs between LO-RANSAC and HMCC-RANSAC can be observed from the frst two datasets wth a speedup rato less than 2.0; however, for the thrd dataset, the rato of tme costs s approxmately 6.0, manly because larger oblque magng angles confgured n the thrd dataset cause a majorty of false matches and decrease the nler rato of ntal canddate matches; (3) domnant tme costs are consumed n the flter stage of GC-RANSAC, whch can be deduced from the three tests, especally for the frst and second datasets. The man reason s that many more ntal matches are searched from mage pars wth smaller oblque angles and consequently ncrease the tme costs n the flter stage of GC-RANSAC; (4) although approxmately 7 percent of ponts are lost n reconstructon models, dentcal orentaton accuracy can be observed by usng HMCC-RANSAC. The tme costs for the three datasets are llustrated n Fgure A1, Fgure A2 and Fgure A3, respectvely. In concluson, the propose algorthm n ths paper, namely HMCC-RANSAC, can acheve effcent geometrcal verfcaton n UAV mage matchng wth comparable accuracy. 5. Dscusson Ths paper proposes the HMCC-RANSAC algorthm for effcent geometrcal verfcaton n UAV mage matchng. By usng on-board GNSS/IMU data and camera mountng angles, the complex transformaton model n the mage-space can be smplfed as a smple 2D-translaton n the object-space, whch s acheved by the projecton of feature ponts. The expermental results from real datasets demonstrate that the tme consumpton of geometrcal verfcaton s obvously decreased for outler domnated matches. Compared wth exstng strateges, such as LO-RANSAC (Chum et al., 2003) and GC-RANSAC (Lu et al., 2016), the proposed algorthm has the followng advantages. Frst, the HMCC-RANSAC algorthm converts the transformaton model, e.g., a smlarty or affne transformaton, between correspondences to a smple translaton model. In the lterature, some other strateges have ether depended on prors, ncludng the orentaton and scale, from extracted features (Schönberger et al., 2016) or utlzed the parwse geometrcal relatonshps for parameter estmaton (L et al., 2015; Lu et al., 2016). The former requres that specal feature detectors that extract related parameters are adopted n the stage of feature extracton; the latter could consume a large proporton of tme costs n the verfcaton, whch can be deduced from the effcency analyss n Secton 4.5. By the further explotaton of onboard GNSS/IMU data, the complex transformaton model n the mage-space s smplfed, and a straghtforward model, termed moton n ths study, s defned and used to establsh the relatonshp between correspondence ponts. Second, based on the smplfed model, a moton consstent constrant s mplemented n terms of moton drecton and moton length, whch conssts of a global drecton consstency constrant, a local drecton-change consstency constrant and a global length consstency constrant. In contrast to a mult-dmensonal votng strategy, the herarchcal moton consstency constrant (HMCC) s mplemented n ths study by sequentally conductng the three constrants for outler elmnaton, whch s acheved by one-dmensonal votng. As verfed n Secton 4.3, the HMCC can detect a large proporton of outlers wth hgh precson and recall. Fnally, the HMCC-RANSAC algorthm does not rely on other data sources except for the on-board GNSS/IMU data from flght control systems and 25

26 camera mountng angles of oblque magng systems. These data sources can be easly accessed. Consequently, the proposed algorthm can be used wth wde range of applcaton n the feld of UAV photogrammetry. Through the effcency analyss presented n Secton 4.5.2, t s clearly shown that the proposed algorthm can accelerate geometrcal verfcaton wth a speedup rato reachng to 6 for oblque datasets; however, for datasets collected from bare land regons wth relatve small oblque magng angles, the HMCC-RANSAC algorthm cannot dramatcally ncrease the verfcaton effcency, because of hgh nler ratos of ntal canddate matches. Compared wth the parwse geometrcal verfcaton, very hgh effcency can also be notced from datasets captured by magng systems wth small oblque angles, because a computaton-effcent moton model s adopted n the HMCC-RANSAC algorthm. Thus, the proposed algorthm n ths paper s applcable for both nadr and oblque mages. Currently, all the used datasets for performance evaluaton are collected from stes wth moderate terran relef. Therefore, projecton planes are approxmated by average alttudes of test stes n moton generaton. To expand ts applcaton for scenaros wth dramatc mountan terran, auxlary data sources, such as SRTM (Shuttle Radar Topography Msson) (Rodrguez et al., 2006) can be used for accurate approxmatons of projecton planes wth a heght precson of 10 m n most regons. Based on the expermental results as presented n Secton 4.3, t would be nterestng to only utlzed the HMCC algorthm for geometrcal verfcaton wthout the use of RANSAC method, because almost all obvous outlers are elmnated. In addton, combned wth the work documented n (Jang and Jang, 2017a, b), a complete workflow from match par selecton to geometrcal verfcaton can be establshed to acheve effcent UAV mage matchng and orentaton. 6. Conclusons In ths paper, we propose the HMCC-RANSAC algorthm to acheve effcent geometrcal verfcaton n UAV mage matchng. Feature ponts of ntal matches are frst projected onto a plane and spatal relatonshps between correspondence set are smplfed as 2D-translaton, whch s modeled as motons wth two attrbutons, namely the moton drecton and length. Then, a herarchcal moton consstency constrant (HMCC) s desgned for outler elmnaton, whch s effcently mplemented by usng a two-stage votng scheme. Fnally, the proposed algorthm s evaluated by comprehensve comparson and analyss from the aspects of feature matchng and mage orentaton. As shown n the experments, strong separaton ablty can be notceably observed from motons generated from ntal canddate matches, and a majorty of outlers can be effcently elmnated by usng the HMCC before the consequent executon of the basc RANSAC algorthm. For UAV mage matchng, the algorthm proposed n ths paper can acheve hgh effcency n geometrcal verfcaton. Acknowledgment The authors would lke to thank authors who have made ther algorthms of SftGPU as a free and open-source software package, whch s really helpful to the research n ths paper. Meanwhle, heartfelt thanks to the anonymous revewers and the edtors, whose comments and advce mprove the qualty of the work. Appendx A See Fgure A1, Fgure A1, Fgure A3. 26

27 (a) (b) (c) (d) (e) Fgure A1. Effcency comparson of the dataset 1: (a) tme costs of LO-RANSAC; (b) and (c) tme costs of GC-RANSAC n flter and verfcaton stages; (d) and (e) tme costs of HMCC- RANSAC n flter and verfcaton stages. 27

28 (a) (b) (c) (d) (e) Fgure A2. Effcency comparson of the dataset 2: (a) tme costs of LO-RANSAC; (b) and (c) tme costs of GC-RANSAC n flter and verfcaton stages; (d) and (e) tme costs of HMCC- RANSAC n flter and verfcaton stages. 28

29 (a) (b) (c) (d) (e) Fgure A3. Effcency comparson of the dataset 3: (a) tme costs of LO-RANSAC; (b) and (c) tme costs of GC-RANSAC n flter and verfcaton stages; (d) and (e) tme costs of HMCC- RANSAC n flter and verfcaton stages. 29