Computer Vision and Image Understanding

Transcription

1 Computer Vision nd Imge Understnding 116 (2012) Contents lists ville t SciVerse ScienceDirect Computer Vision nd Imge Understnding journl homepge: A systemtic pproch for 2D-imge to 3D-rnge registrtion in urn environments q Lingyun Liu, Ionnis Stmos, Google, Mountin View, CA 94043, United Sttes Hunter College/CUNY, New York, NY 10065, United Sttes rticle info strct Article history: Received 26 My 2008 Accepted 20 July 2011 Aville online 17 August 2011 Keywords: 2D-to-3D Registrtion Photorelistic 3D modeling The photorelistic modeling of lrge-scle ojects, such s urn scenes, requires the comintion of rnge sensing technology nd digitl photogrphy. In this pper, we ttck the key prolem of cmer pose estimtion, in n utomtic nd efficient wy. First, the cmer orienttion is recovered y mtching vnishing points (extrcted from 2D imges) with 3D directions (derived from 3D rnge model). Then, hypothesis-nd-test lgorithm computes the cmer positions with respect to the 3D rnge model y mtching corresponding 2D nd 3D liner fetures. The cmer positions re further optimized y minimizing line-to-line distnce. The dvntge of our method over erlier work hs to do with the fct tht we do not need to rely on extrcted plnr fcdes, or other higher-order fetures; we re utilizing lowlevel liner fetures. Tht mkes this method more generl, roust, nd efficient. We hve lso developed user-interfce for llowing users to ccurtely texture-mp 2D imges onto 3D rnge models t interctive rtes. We hve tested our system in lrge vriety of urn scenes. Ó 2011 Elsevier Inc. All rights reserved. 1. Introduction The photorelistic modeling of lrge-scle scenes, such s urn structures, requires comintion of rnge sensing technology with trditionl digitl photogrphy. A systemtic wy for registering 3D rnge scns nd 2D imges is thus essentil. Applictions include virtul relity, Google-type mps, relistic sets for movies nd gmes, urn plnning, rchitecture, historicl preservtion nd rcheology, just to nme few. Recent commercil systems, such s Google Erth or Microsoft Virtul Erth, mke 2D-to-3D registrtion lgorithms even more relevnt. We elieve tht the ility to utomticlly register 2D imges cptured y freely moving cmers to 3D urn models, is of mjor importnce. This ility will llow the texture-mpping of vst 2D imge collections onto their corresponding models. This pper presents system tht enles the ccurte registrtion of individul 2D imges onto 3D model. Out work is prt of lrger frmework tht includes 3D-to- 3D registrtion nd multiview geometry [1,2]. Only liner fetures re utilized, mking our methods pplicle to models of ny type (i.e. 3D point clouds, 3D meshes, CAD, SketchUp models, etc.). Our system first extrcts 3D nd 2D liner fetures nd then groups them into mjor 3D directions nd mjor vnishing points. It finlly computes the rigid trnsformtion etween the 2D imges nd q Supported in prt y the following NSF grnts: IIS , CAREER IIS , MRI CNS nd MRI/RUI EIA Corresponding uthor. Fx: E-mil ddress: istmos@hunter.cuny.edu (I. Stmos). 3D rnge model y estimting mtches etween 2D nd 3D lines. We present results from experiments with exterior nd interior scenes of rel uildings. Despite the dvntges of feture-sed texture mpping solutions, most systems tht ttempt to recrete photorelistic models do so y requiring the mnul selection of fetures mong the 2D imges nd the 3D rnge scns, or y rigidly ttching cmer onto the rnge scnner nd therey fixing the reltive position nd orienttion of the two sensors with respect to ech other [3 8]. The fixed-reltive position pproch provides solution tht hs the following mjor limittions: () The cquisition of the imges nd rnge scns occur t the sme point in time nd from the sme loction in spce. This leds to lck of 2D sensing flexiility since the limittions of 3D rnge sensor positioning, such s stndoff distnce nd mximum distnce, will cuse constrints on the plcement of the cmer. Also, the imges my need to e cptured t different times, prticulrly if there were poor lighting conditions t the time tht the rnge scns were cquired. () The sttic rrngement of 3D nd 2D sensors prevents the cmer from eing dynmiclly djusted to the requirements of ech prticulr scene. As result, the focl length nd reltive position must remin fixed. (c) The fixed-reltive position pproch cnnot hndle the cse of mpping historicl photogrphs on the models or of mpping imges cptured t different instnces in time. These re cpilities tht our method chieves. In summry, fixing the reltive position etween the 3D rnge nd 2D imge sensors scrifices the flexiility of 2D imge cpture. Alterntively, methods tht require mnul interction for the /$ - see front mtter Ó 2011 Elsevier Inc. All rights reserved. doi: /j.cviu

2 26 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) selection of mtching fetures mong the 3D scns nd the 2D imges re error-prone, slow, nd not sclle to lrge dtsets. These limittions motivte the work descried in this pper, mking it essentil for producing photorelistic models of lrge-scle urn scenes. Formlly, the input consists of the pir (D(S),I(S)) of scene s S rnge scn D nd set of imges I. We ssume tht oth the cmer & rnge sensors view the sme prt of the rel scene, so tht the 3D nd 2D views hve significnt overlp (Fig. 1). The loctions of the cmers which produce the imges I is unknown nd must e utomticlly recovered. Thus the output is the pose P i ={R i,t i j Pp i,f i } which descries () the trnsformtion (rottion R i & trnsltion T i ) from the rnge-sensor to ech cmer-sensor s coordinte system nd () the mpping (internl cmer prmeters) from the 3D cmer frmes of reference to the 2D imge frmes of reference. We present novel system tht cn utomticlly register 2D imges with 3D rnge dt t interctive rtes (i.e. 10 s per 2D imge). New strtegies for feture extrction nd mtching re introduced. The contriutions of this work cn e summrized s follows: We hve developed working system tht is le to register 2D imges to 3D models t interctive rtes. This system requires miniml user interction. The whole spce of possile mtches etween 3D nd 2D liner fetures is explored efficiently (unlike proilistic RANSAC methods like [9]). Tht improves the possiility of convergence of our lgorithm. Our erlier systems ([9,10]) require the extrction of mjor fcdes, rectngles, or other higher-order structures from the 2D nd 3D dtsets. Our current method, on the other hnd, utilizes 3D nd 2D liner fetures for mtching without significnt grouping. This increses the generlity of our lgorithm since we mke fewer ssumptions out the 3D scene. Scenes with vrious lyers of plnr fcdes, or without cler mjor fcdes cn thus e hndled. This pper s method utilizes vnishing points nd mjor 3D directions, ut it does not require them to e orthogonl s most erlier methods ssume. Z Y 3 D depth mp of the scene Viewing Direction X Rnge Sensor s Coordinte System Viewing Direction Coordinte Trnsformtion R, T Y X 2 D Cmer Fig. 1. The pose estimtion prolem. The 3D model of the scene is represented in the coordinte system of the rnge sensor. The imge tken from the 2D cmer needs to e registered with the 3D model. Z The lgorithm consists of the following mjor steps: feture extrction (Section 3), internl clirtion nd rottion computtion vi vnishing points (Section 4), nd cmer position computtion vi feture mtching (Section 5). Results, evlution, nd conclusions re presented in Sections 6 8. We strt y discussing relted work. 2. Relted work There re mny pproches for the solution of the pose estimtion prolem from oth point correspondences [11 13] nd line correspondences [14 16], when set of mtched 3D nd 2D points or lines re known, respectively. In the erly work of [17], the proilistic RANSAC method ws introduced for utomticlly computing mtching 3D nd 2D points. RANSAC is roust method nd cn hndle lrge numer of outliers. The mjor drwck however hs to do with the inefficiency of the method when lrge percentge of outliers wrt inliers exist. Another drwck is tht it does not gurntee the finding of the solution. Our method on the other hnd explores the whole spce of possile solutions in n efficient mnner nd is not proilistic pproch. Solutions in utomted mtching of 3D with 2D fetures in the context of oject recognition nd locliztion include the following [18 23]. Recently numer of new methods were developed for ttcking the prolem of utomted lignment of imges with dense point clouds derived from rnge scnners. In the first ctegory of methods single 2D imge I is utomticlly registered with dense untextured 3D rnge model D(S). In the works presented in [9,10] orthogonlity constrints of urn scenes re used. Both methods utilize vnishing points in the 2D imge for computing the rottion. They differ in the individul fetures used for mtching for the finl trnsformtion computtion: 2D nd 3D rectngles in [9] nd 2D nd 3D prllelepipeds in [10]. In this pper we present method of this ctegory tht is sed on mtching 2D nd 3D liner segments. A preliminry version of this work ppered in [24 nd 2]. The work of [25], on the other hnd, presents n utomted 2D-to-3D registrtion method tht relies on mtching the reflectnce rnge imge (i.e. the 2D imge generted y the intensity components of the 3D rnge scn) with the regulr 2D imge. This lgorithm requires n initil estimte of the imge-to-rnge lignment in order to converge. In [26] registrtion method tht is sed on shdows computtion is presented. This lgorithm works well in outdoor scenes lighted y direct sunlight. In the second ctegory of methods single 2D imge I is utomticlly registered with dense textured 3D rnge model D(S). In tht cse the 3D rnge model hs een lredy texture-mpped y set of 2D imges I M. These imges hve een cptured y regulr 2D cmer pre-clirted with the rnge sensor. The single 2D imge I is cptured from seprte viewpoint. In the work of Yng et l. [27] SIFT [28] descriptors re computed on the 2D imge I nd lredy texture-mpped imges I M. The fetures cn e ck-projected from I M to the 3D model D(S), nd locl plnr frme cn e defined round them. Initiliztion is chieved through 2D-to-2D similrity estimtion method [29]. In the work of Schindler et l. [30] regulr ptterns on uilding fcdes cptured y 2D imge I re mtched with ptterns on textured low resolution tringulr model of the scene. In the third ctegory of methods set of 2D imges I is utomticlly registered with dense untextured 3D rnge model D(S). These methods [1,2,31] use informtion from set of imges or from video sequence nd provide comprehensive results y exploring 2D-to-2D, 2D-to-3D, nd 3D-to-3D mtching. In the work of Zho et l. [31], continuous video is ligned onto 3D point cloud otined from 3D sensor. First, n SFM/stereo lgorithm produces 3D point cloud from the video sequence. This point

3 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) cloud is then registered to the 3D point cloud cquired from the rnge scnner y pplying the ICP lgorithm [32]. In our other work [1,2] the 3D rnge scns nd the 2D photogrphs re respectively used to generte pir of 3D models of the scene. The first model consists of dense 3D point cloud cquired y the rnge scnner nd the second model consists of sprse 3D point cloud, produced y pplying multiview geometry (structure-from-motion) lgorithm directly on sequence of 2D photogrphs. A novel lgorithm for utomticlly recovering the similrity trnsformtion (rottion/scle/trnsltion) tht est ligns the sprse nd dense models is presented. This lignment is sed on ccurte registrtion of individul 2D imges (suset of the imges used to produce the sprse model) with the 3D model. This registrtion method is descried in the following sections. 3. Feture extrction In this section we descrie our lgorithms for extrcting fetures from 3D-rnge nd 2D-imge dt. These fetures re utilized for internl cmer clirtion nd cmer pose computtion. The fct tht our system requires low-level liner fetures, mkes our lgorithms generlly pplicle to most exterior nd interior urn scenes (see Section 6). Ech liner feture is lso ssocited with rdius r. In other words, 3D feture cn e considered s cylinder nd 2D feture s n oriented rectngle (Fig. 2). The vlue of the rdius is initilly defined y the user, nd is then dpted sed on the density of 3D nd 2D lines (see following sections) D Feture extrction The 3D line extrction step is sed on the segmenttion method of Stmos nd Allen [33], wheres the mjor directions clustering is sed on the work of Liu nd Stmos [10] (note tht if 3D informtion is provided in terms of CAD model, then the 3D line extrction step is trivil.) The result of this process is set of line clusters L 3D. Ech line in cluster hs similr orienttion s every other line in the sme cluster. The set of line clusters re then sorted sed on the numer of lines in ech cluster. We do not ssume knowledge of verticl or horizontl directions for the line clusters s in our previous method [10]. Ech 3D line is thus ssocited with cluster id, e.g. for the 3D lines in cluster L 3D i, their cluster id is i. In the next step, 3D fetures re extrcted. First, n l 3D feture merging ( l nd l merged into l c ) l l l c l c initil user-defined rdius (e.g. 0.1 m) is ssigned to ech 3D line. Then, line merging step genertes the finl 3D fetures. This reduces the numer of fetures, nd thus increses the efficiency of the mtching stge (Section 5). In this step, ech pir of 3D lines (l,l ) with the sme cluster id re merged into new line l c (Fig. 2) iff () the distnce etween them re smller thn the sum of their rdii, nd () their projections on l c overlp. The merging procedure is continued until there re no two remining 3D lines tht cn e merged. The finl result is set of 3D lines, ech of which is ssocited with cluster id nd rdius D Feture extrction The extrction of 2D fetures nd vnishing points is sed on well-known lgorithms (e.g. [9,30,34 36]). We cn thus extrct from ech imge set of lines tht generte vnishing points V 1, V 2,..., V n. Ech vnishing point defines cluster of 2D lines. The set of vnishing points re sorted sed on the numer of lines in the clusters. 1 Ech 2D line is then ssocited with cluster id (i.e. 2D lines of the cluster defined y V i hve id i). Lines tht re close to ech other re merged to generte the 2D fetures used for mtching. The pproch is similr to the 3D feture extrction s descried ove. Initilly, user defined rdius is ssocited with ech 2D line. In the merging step, if two lines, sy l nd l, hve sme cluster id, similr orienttions nd overlp with ech other, then they re merged into new 2D line l c (Fig. 2). The merging stge continues until no two remining 2D lines cn e merged. The finl result is set of 2D lines, ech of which is ssocited with cluster id nd rdius. 4. Internl cmer clirtion nd rottion computtion The internl cmer clirtion prmeters of ech 2D cmer (effective focl length nd principl point 2 ) cn e computed y the utiliztion of three orthogonl vnishing points (closed form solution) [36]. An itertive solution cn lso estimte the effective focl length nd principl point from two orthogonl vnishing points [10]. Finlly y mtching two orthogonl vnishing points with two orthogonl 3D directions (see Section 3) the rottion R etween the 2D cmer nd 3D model cn e computed. In this pper we present n dditionl method for the clcultion of the effective focl length f nd of the rottion R. We re using two vnishing points nd two mjor 3D directions. We, however, do not ssume tht these directions re orthogonl to ech other. Orthogonlity is prominent in urn scenes, ut is not lwys present. Our method strts with n initil estimte f init of the effective focl length, nd of the principl point P init. f init is included in the Exif met-dt, informtion tht is now provided y most digitl cmers. P init is estimted y the center of the imge. Bsed on these estimtes, n initil center of projection C init is determined. This is the origin of the cmer coordinte system (Fig. 3). Let us consider vnishing point V i extrcted y the 2D imges (see Section 3). The 3D coordintes of V i in the cmer coordinte system re [(V i ) x (P init ) x,(v i ) y (P init ) y, f init ] T. Thus, the normlized vector D 2D i ¼ uðc init V i Þ 3 represents the 3D direction tht genertes the vnishing point V i. This direction is expressed in the cmer coordinte system. Our gol is to mtch ech vnishing point with its corresponding 3D direction extrcted y the 3D rnge model 2D feture merging ( l nd l merged into l c ) Fig. 2. Exmple of 3D nd 2D fetures nd their merging steps. l 1 Note here tht oth 3D line clusters nd 2D line clusters re sorted sed on the numer of lines they contin. Assuming tht lrger 3D clusters mtch with lrger 2D clusters, this sort cn provide vlule hint for mtching etween 3D directions with 2D vnishing points. 2 Note tht we ssume tht rdil distortion is not significnt. 3 We use the nottion u(v) for descriing the unit vector derived from v.

4 28 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Imge Plne Z c V V Y c P f Y w C O w Z w X c X w Fig. 3. Rottion nd focl length computtion sed on two vnishing points nd their corresponding 3D directions (not shown in this imge). (see Section 3). This correspondence leds to the clcultion of the focl length nd of the rottion R. Let us represent ech 3D line cluster in L 3D (Section 3) y its 3D direction D 3D j ; j ¼ 1...n (where n is the numer of extrcted 3D clusters). The next step is to find the mtching pirs of directions hd 2D i ; D 3D j i. Consider for the moment tht we know the correspondence etween vnishing points (expressed in the cmer coordinte system) nd 3D directions (expressed in the world coordinte system). It is known tht with the principl point fixed t the center of the imge, two pirs ðhd 2D ; D3D i; hd2d ; D3D iþ of mtching vnishing point/3d directions re enough for the computtion of the focl length f. The focl length f (which is jcpj in Fig. 3) cn e computed vi the following equtions (tringles CV P, CV P nd CV V ) 4 : jcv j 2 ¼jPV j 2 þ f 2 jcv j 2 ¼jPV j 2 þ f 2 jv V j 2 ¼jCV j 2 þjcv j 2 2 jcv jjcv jcos where is the ngle etween D 3D nd D 3D. (Note tht the vnishing points V nd V hve een computed y using the initil estimtes f init nd P init. The ove computtion leds to the clcultion of focl length tht conforms to the 3D directions D 3D nd D 3D.) From the ove equtions, we cn get qurtic eqution: f 4 þ f 2 þ c ¼ 0 where ¼ sin 2 ; ¼ sin 2 jpv j 2 þjpv j 2 jv V j 2 ; c ¼ jvv j 2 jpvj 2 jpv j cos2 jpv j 2 jpv j 2. Solving this eqution, one qpffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi otins the refined focl length: f ¼ 2 4c. Since D 3D 2 D 3D ; sin will never e equl to 0. Finlly, the rottion R is computed sed on these two pirs of mtching directions [37]. Bsed on the ove nlysis, the tsk of our system is to find two mtching pirs of vnishing point/3d directions. Intuitively, pirs hd 2D ; D3D i; hd2d ; D3D i for which the ngle etween D 2D nd D 2D is not similr to the ngle etween D 3D nd D 3D cn e rejected. As result, we hve list of mtching cndidtes, ech of which contins two pirs of mtching vnishing points nd 3D 4 Plese note tht the coordintes of the center of projection C (in the rnge scnner s coordinte system) do not need to e known for the computtion of f. directions, refined focl length nd rottion. For ech one of these cndidtes we cn pply the lgorithm descried in the next section for clculting the cmer position, nd finlly keep the result tht provides the mximl lignment etween the 2D imge nd 3D model. In the worst cse scenrio though ll pirs of directions hve similr ngles (this scenrio is esily relizle in urn scenes where most ngles etween mjor directions is 90 ). In this cse n m there re 2 2 cndidte mtching pirs of directions (where n is the numer of 3D nd m the numer of vnishing points). Even though this is not lrge serch spce (n nd m re smll in most urn scenes), testing ll hypotheses involves the computtion of the trnsltion (see next section). This is computtionlly inefficient for the purposes of n interctive system, where response time of up to 10 s per imge is pproprite. For these resons we let the user to implicitly provide the correct pir of mtching directions, y rotting the 3D model to n orienttion tht produces rendering tht is similr (ut not exctly the sme) to the rel 2D imge. As shown in Fig. 7 nd Fig. 8, the rotted 3D view (left) is similr (ut not exctly the sme) to the 2D imge (right). This user-ssisted rottion cn pproximtely lign the corresponding 2D nd 3D directions. The forementioned user interction not only increses the computtionl efficiency of the whole system, ut lso mkes the registrtion prolem trctle. In generl, without constrining the possile loctions of 2D cmers wrt the 3D model, the 2Dto-3D registrtion prolem ecomes intrctle. This is due to the existence of possile lrge set of solutions. For exmple, photogrph of one of the columns of the 3D structure of Fig. 8 cn e mtched with ny of the symmetric 3D columns of the rel scene. By selecting synthetic view tht is similr, ut not exctly the sme s the 2D imge, the user cn provide n pproximte field of view to help the mtching lgorithm. In prticulr, only 3D fetures tht re viewle in the synthetic 3D view re used for mtching 2D imge fetures. Note here tht ll erlier pproches still require implicit user interction in order to ssist in tht direction. For exmple in [10] the user needs to explicitly provide the mtch etween vnishing points/3d directions. In tht system, the user lso needs to mtch fcdes etween the 2D imge nd 3D model. Our current pproch is more nturl nd leds to fster interction time.

5 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Imge Plne A l 3D S Y c l 2D B Z c T C Y w O w X c R X w Z w Fig. 4. Cmer position computtion sed on mtch etween 3D feture AB with imge feture ST. The finl result of this module is list of mtching cndidtes, ech of which contins two pirs of mtching vnishing points/ 3D directions, refined focl length nd rottion. The user cn cycle through them, nd cmer position is computed for ech mtching cndidte. Then, ech cndidte is quntittively evluted. The following section provides more detils. 5. Cmer position computtion A list of mtching cndidtes, nmed M, is otined s descried in the previous section. Ech element in M contins mtching pir of two vnishing points nd two 3D directions, refined focl length nd rottion. In this section, 2D cmer Fig. 5. Cmer position (trnsltion) computtion flowchrt. Through step 1 ll possile pirs of mtched 3D nd 2D lines (hl 3D ; l2d i nd hl3d ; l2d i) re selected (l3d nd l 3D re 3D lines extrcted from the 3D model, nd l 2D nd l 2D 2D lines extrcted from the 2D imge). Step 2 computes cmer position sed on hl 3D ; l2d i. The pir hl3d ; l2d i is used for the verifiction of this position. If the overlp etween l 2D nd the projection of l 3D on the imge is smller thn O th (20%) (i.e. the position is not verified) new pir is selected (step 1). Otherwise similr computtion is crried out for the pir hl 3D ; l2d i (step 3). If steps 2 nd 3 produce two verifile cmer positions, weighted verge is computed (step 4). This verge represents the position tht is generted y the hypothesis (hl 3D ; l2d i nd hl3d ; l2d i). All verified cmer positions re stored in list T. After ll pirs hve een explored, ech position in T is grded y projecting ll 3D lines on the 2D imge spce (step 5). Positions with high grde (greter thn G th numer of mtches) survive to the finl optimiztion step 6.

6 30 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) position will e computed for ech cndidte in M. Our method of finding the cmer position follows hypothesis-nd-test scheme y mtching the extrcted 3D nd 2D fetures sed on the frmework of Liu nd Stmos [10]. A numer of mjor differences with the forementioned method mke our lgorithm more generl nd more roust. In prticulr, our lgorithm does not require the extrction of plnr fcdes, nd does not require the grouping of low-level fetures in higher-order structures. Scenes tht do not contin cler mjor fcdes (such s the exmple of Fig. 8 nd, where vrious lyers of plnr fcdes exist) cn now e successfully hndled. Also since ll low-level fetures re used without significnt grouping, more roust results re chieved. Due to the fct tht we re utilizing the low-level liner fetures we hve developed new lgorithm for the computtion of cmer position (Step 2 of the following lgorithm). We now present detiled description of the lgorithm. First, cndidte from M i is selected, i.e. the mtching pir of vnishing points nd 3D directions re hv,v i nd hd 3D ; D3D i; the refined focl length is f i nd the rottion is R i. The cmer position (trnsltion) is then computed in the following six steps (Fig. 5): Step 1 A hypotheticl mtch etween two pirs of 3D nd 2D lines is selected (the lgorithm will go over ll possile such selections). Let us cll these pirs hl 3D ; l2d i nd hl 3D ; l2d i (l3d nd l 3D re 3D lines extrcted from the 3D model, nd l 2D nd l 2D 2D lines extrcted from the 2D imge). Step 2 [Computtion of cmer position in world coordinte system (trnsltion) sed on the mtch of l 3D with l 2D ]As shown in Fig. 4, A nd B re the endpoints of l 3D nd S nd T re the endpoints of l 2D. C is the center of projection. If l 3D mtches exctly with l 2D, then in the cmer coordinte system, C, S nd A should e colliner. The sme pplies for C, T nd B. We thus consider C s the intersection point of the following two lines: () one tht psses through A hving the orienttion of line CS nd () one tht psses through B hving the orienttion of line CT. To compute the world coordintes of C, we need to know the orienttions of CS nd CT in the world coordinte system. We know, however, the orienttions of CS nd CT in the cmer coordinte system, sy n nd n. We hve lso Fig. 6. Registrtion result of Building 2. Top row: Initil stte (efore registrtion). The 3D rnge model (left column) nd 2D imge (right column) re loded nd displyed in the interfce. Middle row: The stte of the system fter the feture extrction. The 3D viewer (left column) shows the clustered 3D lines while the 2D viewer (right column) shows the clustered 2D lines tht re drwn on the originl 2D imge. Different clusters re represented y different colors for clrity. Bottom row: The finl registrtion. The 2D imge is utomticlly registered with the 3D rnge dt. The 3D viewer (left) shows the texture mpped 3D rnge dt. The 2D viewer (right) shows the mtching 2D nd 3D line fetures (2D lines re displyed s red, while projected 3D lines re highlighted in green). Note tht ojects tht re not prt of the 3D model cnnot e texturempped (corner of other uilding shown in the 2D imge). ionnis/iccv07/ contins video of the process. (For interprettion of the references to color in this figure legend, the reder is referred to the we version of this rticle.)

7 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Fig. 7. Registrtion results from uilding 1. () For description see cption of Fig. 6. () (Top row): The 2D imge is in very different orienttion wrt the cquired 3D rnge model. (Middle row): The user rottes the 3D model so tht it is orientted similrly (note tht it does not hve to e exctly mtched) to the 2D imge. (Bottom row): The right imge shows the 2D imge long with the mtched 2D nd projected 3D fetures (see cption of Fig. 6). The left imge shows the texture-mpped 3D rnge model fter successful registrtion.

8 32 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Fig. 8. Registrtion results from the interior of uilding 3. () For description see cption of Fig. 6. () (Top row): The 2D imge is viewing smll prt of the 3D model. (Middle row): The user rottes the 3D model so tht it is orientted similrly (note tht it does not hve to e exctly mtched) to the 2D imge. (Bottom row): The right imge shows the 2D imge long with the mtched 2D nd projected 3D fetures (see cption of Fig. 6). The left imge shows the texture-mpped 3D rnge model fter successful registrtion. Note tht surfces tht re not prt of the 3D model cnnot e texture-mpped nd pper s lck holes. For exmple the floor is missing from our rnge model.

9 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Tle 1 Building 1 (13 imges). Ech row presents results from successful registrtion of different 2D imge with the 3D rnge model. The registrtion (mtching phse) of ech imge requires on verge 5 10 s (2 GHz Xeon Intel processor, 2GB of RAM). The first two columns show the numers of 3D nd 2D fetures used for mtching. Fi is the initil focl length extrcted from the Exif met-dt of the imge, while Fr is the refined focl length. M is the numer of mtched fetures of the est trnsformtion. Finlly, E is the verge line-to-line distnce (in pixels) etween the utomticlly mtched lines fter the optimiztion (Step 6). E2 is the verge line-to-line distnce etween the mnully selected lines (in pixels). E2 is used for evlution of ccurcy. F3D F2D Fi Fr M E E Tle 2 Building 2 (seven imges). See cption of Tle 1. F3D F2D Fi Fr M E E Tle 3 Building 3 (four imges). See cption of Tle 1. F3D F2D Fi Fr M E E computed the rottion R which rings the cmer nd world coordinte systems into lignment (see previous section). We cn thus compute the orienttions of CS nd CT in the world coordinte system s: R n nd R n. Then, the cmer position is otined y finding the intersection of two 3D lines: () one of which psses through A with the orienttion of R n nd () one which psses through B with the orienttion of R n. 5 Finlly, this computed center of projection is used to project l 3D onto the imge plne. If the projection of l 3D overlps with l 2D (within threshold of 80%), then the cmer position computed using ðl 3D ; l2d Þ is verified y the pir ðl3d ; l2d Þ. We therefore move to the next step. Otherwise, we return to step 1 (i.e. the mtch is discrded) to pick nother set of hypotheticl mtching lines. Step 3 Step 2 is repeted ssuming s hypothesis the mtch etween l 3D nd l 2D. The newly computed center of projection is used to compute the overlp etween l 2D nd the projection of l 3D. If this overlp is less thn threshold (i.e. the computed C is not verified y ðl 3D ; l2d Þ, we return to step 1 (i.e. the mtch is discrded). Otherwise, we proceed to the next step. Step 4 Step 2 hs thus computed cmer position C 1 y the hypothesis ðl 3D ; l2d Þ [verified y ðl3d ; l2d Þ], while step 3 hs computed cmer position C 2 y the hypothesis ðl 3D ; l2d Þ [verified y ðl3d ; l2d Þ]. In this step, the weighted verge (sed on the mount of overlp) of these two cmer positions is computed nd sved in list T. Step 5 Steps 1 4 re repeted for ll possile pirs of 3D nd 2D lines ðhl 3D ; l2d i; hl3d ; l2d iþ. All verified cmer positions (see Step 4) re stored in list T. Then, for ech position in T, ll 3D lines re projected onto the imge plne. For ech of the projected 3D lines, possile mtching 2D line is found y serching round its projection. This region is ounded y the rdius of the 3D nd 2D lines. The numer of found mtches grdes this cmer position. If the grde of cmer position is less thn threshold, it is removed from the list T. Step 6 The remining cmer positions in T re optimized y two steps. First, for ech cmer position C i refined position C ref is found. This is chieved y serching round smll neighorhood of C i in order to mximize the overlp etween the mtching 3D nd 2D lines. The overlp is mesured in the 2D spce y projecting the 3D lines on the imge plne nd then serching for 2D lines of mximum overlp. Then this refined position is further optimized y n itertive lgorithm tht uses the whole set of 3D nd 2D lines. In ech itertion, the current cmer position is used to generte list of mtching 2D nd 3D lines from the whole 2D nd 3D feture spce. Assuming tht the trnsltion is fixed, the rottion is optimized s follows. Ech 2D line long with the currently estimted est center of projection C ref define plne with norml N.IfR ref is the refined estimte of the rottion, then the corresponding 3D line should stisfy R ref N = 0 (i.e. the dot product of the rotted norml with the 3D direction should e zero). The set of corresponding 2D () nd 3D () lines thus generte liner system of equtions for the unknown nine prmeters of the rottion mtrix R ref. This system is solved in the lest-squres sense in order to compute R ref. 6 Assuming now tht the rottion is fixed to R ref we cn iterte nd refine further C ref y serching in smll neighorhood round it for mximizing overlp etween 2D nd projected 3D lines. This two step procedure converges when no significnt chnge in the rottion nd trnsltion occurs. The cmer position in T with the mximum grde is picked s the est one for the mtching cndidte M i. This is normlly correct, ut the list is still kept s well in cse tht the one with the mximum grde is not the est. Then, the user cn select other positions in the list. This mximum grde is lso used s the grde for M i. For ech mtching cndidte in M, list of cmer positions is computed y these 6 steps nd grde is ssigned. Then, the list M is sorted sed on the grde nd the one with the mximum grde is selected s the est one ut the user lso cn select other results in M. 5 A nd B re oth expressed in the world coordinte system. 6 We need to lso correct this estimte to ecome rottion mtrix using stndrd vision techniques [38].

10 34 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Fig. 9. Asin Society uilding, NYC. (Top row:) Texture-mpped imge on 3D model (left) nd 2D imge with mtched 2D fetures projected on 3D fetures (right). Correct trnsformtion. This is the est option provided to the user. Note tht lines from nery uildings hve een used for its computtion. (Bottom row:) Texture-mpped imge on 3D model (left) nd prt of 2D imge with mtched 2D fetures projected on 3D fetures (right). Incorrect trnsformtion. This is one of the wrong options generted. In tht exmple the rottion is correct, ut the trnsltion is not. The error is evident since imges of nery uildings re projected on the 3D model of the Asin Society uilding. Fig. 10. The evlution of the registrtion result on uilding 1. For this cse, 19 pirs of 3D nd 2D lines re selected nd the verge error is in pixels (see Section 7 nd Tle 1 for description of error mesurements). Top Left: The mnully mrked 3D lines; Top Right: Mrked 3D lines on 3D rnge point cloud; Bottom Left: The mnully mrked 2D lines; Bottom Right: The mrked 3D lines re projected onto the imge plne (green) fter utomted registrtion. The corresponding 2D lines re shown in red. (For interprettion of the references to color in this figure legend, the reder is referred to the we version of this rticle.)

11 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Fig. 11. The evlution of the registrtion result on uilding 2. For this cse, 96 pirs of 3D nd 2D lines re selected nd the verge error is in pixels (See Section 7 nd Tle 1 for description of error mesurements). Top Left: The mnully mrked 3D lines; Top Right: Mrked 3D lines on 3D rnge point cloud; Bottom Left: The mnully mrked 2D lines; Bottom Right: The mrked 3D lines re projected onto the imge plne (green) fter utomted registrtion. The corresponding 2D lines re shown in red. (For interprettion of the references to color in this figure legend, the reder is referred to the we version of this rticle.) Fig. 12. The evlution of the registrtion result sed on mnully selected line correspondences. For this cse, 21 pirs of 3D nd 2D lines re selected nd the verge error is in pixels (see Section 7 nd Tle 1 for description of error mesurements). The mrked 3D lines re projected onto the imge plne (green) fter utomted registrtion. The corresponding 2D lines re shown in red. (For interprettion of the references to color in this figure legend, the reder is referred to the we version of this rticle.) 6. Results We re presenting results from rel experiments in four urn settings tht we nme 1 (Fig. 7), 2 (Fig. 6), 3 (Fig. 8) nd Asi Society uilding in NYC (Fig. 9). Buildings 1 (Thoms Hunter uilding, NYC) nd 2 (uilding cross from Cooper Union, NYC) re the exteriors of regulr urn structures. Building 3 is the interior of Grnd Centrl Sttion, scene of rchitecturl complexity nd euty. First numer of 3D rnge scns of ech structure ws cquired using Leic HDS 2500 nd Leic ScnSttion2 time-of-flight lser rnge scnners [8]. This scnners provides solute 3D rnge mesurements up to distnce of 100 m, nd t n ccurcy of 6mm. Ech 3D point is ssocited with reflectnce informtion, tht corresponds to the mount of lser-intensity getting ck to the rnge sensor. 7 We then segment ech rnge scn, extrct liner 3D fetures, nd register the scns in common coordinte system. Figs. 6 8 provide individul registrtion results, s descried in our technicl sections. Note thn in the cse of Fig. 7 nd Fig. 8 the user needs to orient the 3D rnge model in position tht simultes the 2D color imge. As you cn see from these figures this simultion does not need to e exct. It is necessry for ssistnce in mtching vnishing points with 3D directions (Section 4) in order for our system to perform in interctive rtes (5 10 s for mtching per imge). Tles 1 3 present quntittive results for successful utomted registrtions. The lst two columns descrie error mesured s () verge distnce etween mtching lines used for utomted registrtion (nmed E ), nd () verge distnce etween mnully selected lines (nmed E2 ). In Fig. 9 we re showing two of the options tht re presented to the user y the interctive system. They include good (top) nd d (ottom) trnsformtion. Both options correspond to the sme rottion ut they include different trnsltions. Note tht these solutions re generted from different sets of correspondences etween 3D nd 2D lines. The good trnsformtion includes lines from vrious uilding in the re, wheres the d trnsformtion lines only from the Asin Society uilding (center of the imge). It seems tht support from lrge prt of the scene provides more roust results, something tht is true for cmer clirtion methods s well. In ll cses the first step (Section 4) never fils since the scenes contin t lest two vnishing points. The user my hve to select 7 Note tht the reflectnce depends on vrious prmeters (distnce, orienttion nd surfce mteril) nd is not the ctul color of the oject s cptured y 2D digitl cmer.

12 36 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) Fig. 13. Top: Texture mpping detils for Building 1 (left) nd Building 2 (right). The texture mpped urn structures (windows, street signs, AC units, etc.) demonstrte the high ccurcy of our registrtion system. Bottom: Texture mpping detils for Building 3 (notice ccurcy of registrtion of flg nd of light fixtures). the correct correspondence etween the possile mtches, ut this is smll set of possiilities. The second step however (Section 5) depends on the qulity of the extrcted low-level 2D nd 3D liner fetures. In cses tht we cnnot extrct fetures of high qulity (due to low contrst in 2D imges), this method will not e le to perform correctly. On the other hnd few correct 2D-to-3D registrtions cn e enhnced with multiview-geometry solutions to ring sequences in lignment with model (see [1]). 7. Evlution Tle 4 Quntittive results of filed registrtions. Building 1 (three imges) Building 2 (two imges) Building 3 (five imges). Ech row presents results from filed registrtion of different 2D imge with the 3D rnge model. For uilding 1 (top prt), three imges fil out of totl of 16 imges. For uilding 2 (middle prt), two imges fil out of totl of nine imges. For uilding 3, 5 imges fil out of totl of nine imges. The filure is due to the low qulity of the extrcted 3D/2D fetures (poor lighting conditions or cmer motion). The first two columns show the numers of 3D nd 2D fetures used for mtching. Fi is the initil focl length extrcted from the Exif met-dt of the imge, while Fr is the refined focl length. M is the numer of mtched fetures of the est trnsformtion. Finlly, E2 is the evlution error which is verge line-to-line distnce (in pixels) sed on the mnully selected line correspondences. F3D F2D Fi Fr M E In order to quntittively evlute the ccurcy of the registrtion results, we mnully selected prominent 3D nd 2D lines, nd clculted the distnce etween corresponding 2D nd projected 3D lines. Exmples of mrked 3D nd 2D lines re shown in Figs After utomted registrtion, the mrked 3D lines re projected onto the imge plne (ottom right imges in Figs ). The verge distnce (in pixels) etween the mrked 2D lines from the projections of their corresponding mrked 3D lines is our evlution metric. For ech pir of corresponding 2D nd 3D lines, the line-to-line distnce is the verge of the two distnces mesured from the two end points of the projected 3D line to the mtching 2D line. Tles 1 3 show the successful registrtion results. The lst column ( E2 ) contins the evlution results. Intuitively, smller errors correspond to etter registrtion results. An verge error round 1 2 pixels is t n cceptle rnge for good registrtion. This is lso verified visully y our high-qulity texture-mpping results. Exmples of texture-mpping detils re shown in Fig. 13. Tle 4 shows the results of filed registrtions. These filures re cused y extrcted 3D nd 2D fetures of low qulity due to poor lighting conditions or cmer motion. 8. Conclusion We hve presented systemtic wy for registering individul 2D imges with 3D rnge model. Our methods ssume the exis-

13 L. Liu, I. Stmos / Computer Vision nd Imge Understnding 116 (2012) tence of t lest two vnishing points in the scene (not necessrily orthogonl). No higher-order grouping of fetures is necessry. Our system llow us to register 2D imges with 3D model t interctive rtes. In our future work we would like to e le to hndle scenes of generl configurtion not contining ny mjor vnishing points. This would let the explortion of registrtion lgorithms in non-urn scenes. In summry, this new imge-to-rnge registrtion system requires miniml user interction nd cn register 2D imges with 3D rnge dt t interctive rtes. The user interction not only increses the computtionl efficiency of the whole system, ut lso mkes the registrtion prolem trctle. In ddition, the whole spce of possile mtches etween 3D nd 2D liner fetures is explored. The current system cn work well with scenes contining multiple lyers of plnr fcdes, or without mjor fcdes, s long s liner fetures exist. This increses the generlity of our lgorithm, since we mke few ssumptions out the 3D scene. Acknowledgments The uthors would like to thnk Hunter College s undergrdute student Thoms Flynn. Thoms mintined nd improved the softwre foundtion of the descried system. References [1] L. Liu, I. Stmos, G. Yu, G. Wolerg, S. Zoki, Multiview geometry for texture mpping 2D imges onto 3D rnge dt, in: IEEE Conference on Computer Vision nd Pttern Recognition, vol. II, New York City, 2006, pp [2] I. Stmos, L. Liu, C. Cho, G. Wolerg, G. Yu, S. Zoki, Integrting utomted rnge registrtion with multiview geometry for the photorelistic modeling of lrge-scle scenes, Interntionl Journl of Computer Vision 78 (2 3) (2008) Specil Issue on Modeling nd Representtion of Lrge-Scle 3D Scenes. [3] C. Früh, A. Zkhor, Constructing 3D city models y merging eril nd ground views, Computer Grphics nd Applictions 23 (6) (2003) [4] K. Pulli, H. Ai-Rched, T. Duchmp, L.G. Shpiro, W. Stuetzle, Acquisition nd visuliztion of colored 3D ojects, in: Interntionl Conference on Pttern Recognition, vol. 1, Austrli, 1998, p. 11. [5] V. Sequeir, J. Conclves, 3D relity modeling: photo-relistic 3D models of rel world scenes, in: Interntionl Symposium on 3D Dt Processing, Visuliztion nd Trnsmission, 2002, pp [6] Visul Informtion Technology Group, Cnd, < [7] H. Zho, R. Shiski, Reconstructing textured CAD model of n urn environment using vehicle-orne lser rnge scnners nd line cmers, Mchine Vision nd Applictions 14 (1) (2003) [8] Leic Geosystems, < [9] I. Stmos, P.K. Allen, Automtic registrtion of 3-D with 2-D imgery in urn environments, IEEE Interntionl Conference on Computer Vision (2001) [10] L. Liu, I. Stmos, Automtic 3D to 2D registrtion for the photorelistic rendering of urn scenes, in: IEEE Conference on Computer Vision nd Pttern Recognition, vol. II, Sn Diego, CA, 2005, pp [11] D. Oerkmpf, D. DeMenthon, L. Dvis, Itertive pose estimtion using coplnr feture points, Computer Vision Grphics nd Imge Processing 63 (3) (1996) [12] L. Qun, Z. Ln, Liner N-point cmer pose determintion, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence 21 (8) (1999) [13] M. Bujnk, Z. Kukelov, T. Pjdl, A generl solution to the P4P prolem for cmer with unknown focl length, in: IEEE Conference on Computer Vision nd Pttern Recognition, 2008, pp [14] S. Christy, R. Horud, Itertive pose computtion from line correspondences, Journl of Computer Vision nd Imge Understnding 73 (1) (1999) [15] R. Horud, F. Dornik, B. Lmiroy, S. Christy, Oject pose: the link etween wek perspective, prperspective, nd full perspective, Interntionl Journl of Computer Vision 22 (2) (1997) [16] R. Kumr, A.R. Hnson, Roust methods for estimting pose nd sensitivity nlysis, CVGIP 60 (3) (1994) [17] M.A. Fischler, R.C. Bolles, Rndom smple consensus: prdigm for model fitting with pplictions to imge nlysis nd utomted crtogrphy, Grphics nd Imge Processing 24 (6) (1981) [18] T. Css, Polynomil-time geometric mtching for oject recognition, Interntionl Journl of Computer Vision 21 (1 2) (1997) [19] G. Husler, D. Ritter, Feture-sed oject recognition nd locliztion in 3Dspce, using single video imge, Journl of Computer Vision nd Imge Understnding 73 (1) (1999) [20] D. Huttenlocher, S. Ullmn, Recognizing solid ojects y lignment with n imge, Interntionl Journl of Computer Vision 5 (7) (1990) [21] D.W. Jcos, Mtching 3 D models to 2 D imges, Interntionl Journl of Computer Vision 21 (1 2) (1997) [22] F. Jurie, Solution of the simultneous pose nd correspondence prolem using gussin error model, Journl of Computer Vision nd Imge Understnding 73 (3) (1999) [23] W. Wells, Sttisticl pproches to feture-sed oject recognition, Interntionl Journl of Computer Vision 21 (1 2) (1997) [24] L. Liu, I. Stmos, A systemtic pproch for 2D-imge to 3D-rnge registrtion in urn environments, in: VRML Workshop, 11th Interntionl Conference on Computer Vision, Rio de Jneiro, Brzil, [25] K. Ikeuchi, The gret uddh project, in: IEEE ISMAR03, Tokyo, Jpn, [26] A. Troccoli, P.K. Allen, A shdow sed method for imge to model registrtion, in: 2nd IEEE Workshop on Video nd Imge Registrtion, [27] G. Yng, J. Becker, C.V. Stewrt, Estimting the loction of cmer with respect to 3D model, in: The Sixth Interntionl Conference on 3-D Digitl Imging nd Modeling, Wshington, DC, 2007, pp [28] D.G. Lowe, Distinctive imge fetures from scle-invrint keypoint, Interntionl Journl of Computer Vision 60 (2) (2004) [29] C.V. Stewrt, C. ling Tsi, B. Roysm, The dul-ootstrp itertive closest point lgorithm with ppliction to retinl imge registrtion, IEEE Trnsctions in Medicl Imging 22 (11) (2003) [30] G. Schindler, P. Krishnmurthy, R. Lulinermn, Y. Liu, F. Dellert, Detecting nd mtching repeted ptterns for utomtic geo-tgging in urn environments, IEEE Conference on Computer Vision nd Pttern Recognition (2008) 1 7. [31] W. Zho, D. Nister, S. Hsu, Alignment of continuous video onto 3D point clouds, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence 27 (8) (2005) [32] P.J. Besl, N.D. McKy, A method for registrtion of 3D shpes, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence 14 (2) (1992) [33] I. Stmos, P.K. Allen, Geometry nd texture recovery of scenes of lrge scle, Journl of Computer Vision nd Imge Understnding 88 (2) (2002) [34] M. Antone, S. Teller, Sclle extrinsic clirtion of omni-directionl imge networks, Interntionl Journl of Computer Vision 49 (2 3) (2002) [35] S.C. Becker, Vision-ssisted modeling from model-sed video representtions., Ph.D. thesis, MIT, [36] B. Cprile, V. Torre, Using vnishing points for cmer clirtion, IEEE Interntionl Conference on Computer Vision 4 (1990) [37] O. Fugers, Three-Dimensionl Computer Vision, The MIT Press, [38] E. Trucco, A. Verri, Introductory Techniques for 3-D Computer Vision, Prentice Hll, 1998.