Improvng Intal Estmatons for Structure from Moton Methods Chrstopher Schwartz Renhard Klen Insttute for Computer Scence II, Unversty of Bonn Abstract In Computer Graphcs as well as n Computer Vson and Autonomous Navgaton, Structure from Moton s a common method to regster cameras. Usually several steps are nvolved wth bundle-adustment as the fnal one. A good ntal estmaton of camera postons s of crucal mportance for the success of the bundle-adustment and s the core component of any Structure from Moton system. Yet there are some lmtatons to current Structure from Moton tools regardng the qualty of the ntal estmaton. Wth our proposed method of mergng dfferent connected components resultng from a lack of good nput mages, we am to overcome the fact that at frst glance no global ntal estmaton could be found. We wll show that n many of these stuatons our method s applcable and may even be used to speed up the Structure from Moton process and lmt ts memory consumpton n general. Keywords: structure from moton, reconstructon, 3D scene analyss 1 Introducton Orgnally used for Computer Vson and Autonomous Navgaton, e.g. as a preprocessng step for dense reconstructon, Structure from Moton (SfM) holds great potental n the feld of Computer Graphcs as well. For nstance, Structure from Moton produces, besdes the estmaton of all camera parameters, a sparse pont cloud whch can be used to ft a proxy geometry to create nteractve walk-throughs [14] or mult-vew panoramc mages [1]. Thanks to Mcrosoft s Photosynth 1, SfM s used for ths purpose by a broad publc today. A crucal part of any Structure from Moton system s to compute good ntal estmates for a followng optmzaton process. Yet n most modern SfM systems these ntal estmatons are n many cases not as good as they could be. In envronments wth sparse features, e.g. ndoor envronments wth plan walls, there s a hgh rsk that the nput mages cannot be regstered globally and break up nto connected components, leadng to an ntal estmaton of only a part of the mages n one of the components. The other mages are slently gnored or processed n a subsequent, but separate SfM process. 1 http://lvelabs.com/photosynth/ Fgure 1: The result of mergng 7 dfferent connected components to regster about 1700 nput mages. Wthout splttng the nput nto components the regstraton faled, exhaustng 8GB of memory. As a result, nstead of one globally optmal regstraton we get a sngle ncomplete or several separate outputs, whch resde n dfferent local coordnate systems. In ths paper we wll show that ths can n many cases be avoded. Addtonally, the proposed method to avod separate components can be used to mprove speed and memory consumpton of any SfM method. 2 Prevous Work In photogrammetry as well as n computer vson there s a long tradton of automatc camera calbraton. Already n 1959 E. H. Thompson presented a relatonal algebrac soluton for the relatve orentaton of two mages [17]. Later work by D. Nstér [13] provded optmzed algorthms for the relatve orentaton of stereo-mages. However, n the recent decades the ncreasng performance of computers made t possble to process whole blocks of mages n reasonable tme. Trggs et al. presented n [19] a method called bundle adustment, a statstcal optmal soluton to the problem of orentng blocks of mages and homologous ponts at once. Ths method s
today regarded as a gold standard for performng optmal regstraton from correspondences [7]. In order to fnd such correspondences, powerful feature detecton and matchng technques are necessary. Frst wdely used, general feature detecton approaches were made n 1986 by W. Förstner [5] and 1988 by C. Harrs [6]. However, n contrast to Förstner and Harrs, modern feature detecton uses scale- or somewhat affnetransformaton-nvarant features [11]. A today wdely used scale nvarant feature detecton s SIFT, ntroduced by D. Lowe n 2004 [10]. In the same publcaton Lowe also provded a soluton for effcently matchng hs SIFT- Features. In the early 1990s effectve Structure from Moton technques were developed, whch are able to smultaneously reconstruct the unknown scene structure and camera calbratons from a set of feature correspondences [18], [16]. Then, n 2005, Brown and Lowe presented an automated Structure from Moton system [3] based upon Lowe s SIFT-Feature detecton and matchng. Ths system was later adapted by Snavely et al. n [15] and mplemented as the freely avalable program bundler. Both base upon fndng a global ntalzaton of camera- and pont-postons for a fnal bundle-adustment step. Contrary to the above mentoned global solvers there exst also some bottom up approaches lke the one of Frtzgbbon et al. n [4] or Nstér s n [12], whch frst fnd robust calbraton for trplets of mages and then put these together to larger sequences, applyng a bundleadustment step afterwards every tme. Ths s n some way smlar to the method we propose n ths paper, as they also provde an alternatve ntalzaton of the fnal bundle-adustment by onng dfferent smaller components. However, whle Frtzgbbon et al. and Nstér assume a lnear or loopy sequence of mages, our approach may also be used wth nput mages that are captured wth camera-postons showng any graph-lke structure. Also, wth the method of Frtzgbbon one mght encounter smlar problems to the ones descrbed wth bundler-lke approaches n Secton 4 when the nput set of mages s not a sngle sequence, as assumed by Frtzgbbon, and two subsequences lack the overlap to on them together. 3 Structure from Moton Overvew SfM takes a set of nput mages I = {I 1,...,I n } and estmates the ntrnsc and extrnsc camera parameters of the capturng cameras C = {C 1,...,C n } for mages = 1...n. Assumng an affne camera-model wthout lensdstorton we have as ntrnsc calbraton the matrx K = + m) c 0 s c(1 h x h y 0 0 1 wth 5 ntrnsc parameters c, s, m, h x and h y, all IR, where c s the camera-constant, s s the shear of the mage-coordnate-system, m s the scale factor between the mage-coordnate-axs and (h x,h y ) s the prncpal pont of the mage-plane. Addtonally, there are 6 extrnsc parameters, the orentaton and the poston of the th camera, descrbed n ( ) R t IR 3 4, where R IR 3 3 s a rotatonmatrx and t IR 3 s the poston of the camera, addng up to a total of 11 unknown parameters per camera, whch proecton may now be descrbed as the matrx C : C = K ( R t ) Every pont x k R 2, that s vsble (and measurable) n mage I s the proecton of a pont x k R 3, k [1,...,m], where m s the number of overall measured ponts. In homologous coordnates ths lead to the followng equaton: x k = C x k SfM uses the measured ponts x k, [1,...,n],k [1,...,m], and computes an optmal soluton for the unknown parameters n C and x k. As a fnal step any SfM system uses bundle-adustment, frst ntroduced by [19], to compute a statstcally optmal soluton. In order to work properly bundle-adustment needs good ntal estmates. Thus, we need to make an ntal guess how the cameras are postoned relatve to each other and how the ponts are dstrbuted n space. 3.1 Relatve Orentaton For every par of mages I,I wth overlappng magecontent t s possble to calculate the orentaton of camera relatve to camera f at least 5 ponts x k 1 are vsble n both mages (see [17]); thus, mplyng the proected ponts x k 1,x k 1. In ths case the frst camera s fxed n the orgn of the system wth t = 0 and R = I. We only have to fnd the orentaton and poston of the second camera. Ths adds up to 6 unknown extrnsc parameters. Unfortunately the stereo-model can only be computed up to an arbtrary scale factor (see Fgure 2). But ths also means that we only have to calculate the drecton from t to t and do not need to know the dstance, reducng the unknown parameters by 1 and makng t drectly solvable wth 10 measured ponts x k 1,x k 1 usng a lnear equaton system: C 1 x k l = C 1 x k l Please see [13], [9] and references theren for further detals. After the relatve orentaton step the poston of any pont x k l that s vsble n both mages can be drectly calculated by ntersectng the rays C 1 x k l and C 1 x k l. For numercal stable results the mages I and I should have a sgnfcant baselne b to ensure enough parallax, an essental factor for stereo vson.
z s=1 s=1.5 w x 1 x 2 f x 1 1 x 2 1 x 1 2 x 2 2 b 1.5 z 1.5 w x 1 x 2 f x 1 1 x 2 1 x 1 2 x 2 2 1.5 b Fgure 2: An example for the arbtrary scalng n relatve orented stereo-models. On the left hand sde we see the confguraton wth a scale s = 1. On the rght hand sde the same confguraton wth a dfferent scale s = 1.5 s shown. Please note that the focal length f and mage-plane wdth are constant. The dfference n scale has no nfluence on the poston of the proected ponts x1 1, x1 2, x2 1 and x2 2 on the mage plane. 3.2 Combnng the Stereo Models Although we can calculate stereo-models for pars of mages we cannot smply concatenate these stereo models due to the arbtrary scale. To be able to concatenate two stereo-models I,I and I,I l we need at least one pont x IR 3 that s vsble n all three mages, and l. The poston of such a pont x s then avalable n both stereomodels. Let x = C 1 stereo model and x = C 1 x k C 1 x k be the poston of x n the frst x k C 1 xl k be the poston of x n the second stereo model. Then the relatve scale s of the stereo models can be computed as 4 Problem s = x t x t Usually, n order to automate the SfM-process, correspondng ponts x k and x k are automatcally detected usng a feature detector, e.g. SIFT [10], and parwse compared to the features found n other mages. To ensure stablty outlers are removed usng RANSAC (see [3] for detals). Wth ths n mnd, the condton to have at least one pont x vsble n three mages I,I,I l n order to concatenate ther two stereo models appears even more restrctve. Now there has to be at least one pont x whose proectons x, x, and x l are automatcally detected as features and these three features then have to be matched as correspondent. To make thngs worse, most feature detectors react senstve to changes of the vewng angle. l Ths means we have to make sure that the mages have enough overlap and only small vew drecton changes to enable the feature-detecton to fnd such common ponts. Yet, n some cases, e.g. a corrdor, some regons of the surroundngs show so few features that t would take a tremendous amount of mages to ensure ths crteron (see Fgure 3). So t s lkely that durng the mage capturng process some areas are not captured wth the amount of detal necessary. The obvous soluton to ths problem s to revst the scene and take more pctures of the crtcal regons. Ths, however, can be expensve and tme consumng or n some cases even completely mpossble. Another way to solve ths concern would be to let the user manually dentfy correspondng ponts n trplets of mages. Apart from beng an extremely dull and lengthy actvty, ths soluton also holds the danger of beng errorprone and naccurate compared to automatc methods. That lack of connectng ponts x eventually leads to parttonng of the ntal estmates nto separate clouds of connected stereo models. Some tools, e.g. bundler [15], choose one ntal mage par and as a result only manage to fnd one of the connected components, leavng many mages unregstered. Other tools, e.g. Mcrosoft Photosynth or Brown s and Lowe s orgnal algorthm [3], are aware of ths problem and are able to fnd all connected components. 5 Component Mergng In the course of our research we dscovered that n many cases dfferent connected components have a common subset of mages. If, however, there are two connected components C 1 = {I f (1),...,I f (n1 )} and C 2 = {I g(1),...,i g(n2 )} wth a common subset of mages S = {I k k = f () = g( ), [1,...,n 1 ], [1,...,n 2 ]} and S 2 t s possble to combne these separate components nto C = C 1 C 2. Even wth S = 1 t would be possble to combne C 1 and C 2 wth respect to orentaton and poston of camera C S. The requrement S 2 s necessary, because every component once agan has an arbtrary scale as all added stereo-models were scaled relatve to the stereo-model of the ntal mage par of ths component. Ths scalng factor cannot be determned wth only one common camera C S. Wth S = 2 the scalng problem between the two connected components can be solved n a straght forward manner. Let t be the 3d poston of Camera C S n the connected component C. s = t1 1 t1 2 t 2 1 t2 2 For S > 2 the problem s overdetermned and can be solved n a least squares sense. The same apples for S
(a) Store 2 and translaton and rotaton of C2 n respect to C1. To further mprove the transformaton we mplemented an outler detecton based on RANSAC, dealng wth ncorrectly regstered cameras n one of the two components. As a SfM-system s output usually conssts of the camera parameters and only a sparse set of ponts, the transformaton has to be appled to only a small amount of vectors and an even smaller amount of orentatons. Thus, mergng two components s a matter of a few seconds or less. 5.1 (b) Hallway (c) Store Matches For any number of dfferent connected components an optmal regstraton can be found usng a graph G = (N, E) over the components. Ths s done by usng the components C as nodes N. If two components C and C have 2 ntersectng mages an edge (, ) s added to E. For each edge e E a rgd body transformaton can be computed as descrbed n secton 5. By fndng a mnmal spannng tree or makng use of possble loops n the graph the unavodable accumulaton of errors can be mnmzed. For an n-depth dscusson see [2] and references theren and [8], who use a smlar technque for the regstraton of laser range mages. [8] also proposes several heurstcs to deal wth outlers n the rgd body transformaton for edges e = (, ). Ths s of mportance for the proposed method, as t allows us to automatcally deal wth errors resultng from an naccurate regstraton of a camera C S n one of the components C or C. 6 (d) Hallway Matches Fgure 3: Comparson between rch and sparse feature envronments. (a) Three nput mages wth common overlap n a rch feature envronment. (b) Three nput mages wth common overlap n a sparse feature envronment. (c) matches between features n mage pars from nput set (a). (d) matches between features n mage pars from nput set (b). Although set (b) clearly shows a common regon n all three mages, no feature used for matches between (b left) and (b mddle) (see (d left)) s also present n the matches for (b mddle) and (b rght) (see (d mddle)). The only common features between (b left) and (b rght) are shown n (d rght). On the other hand the feature matchng found enough matches to buld a stereo model for (b left) and (b mddle) and probably for (b mddle) and (b rght). All matches were calculated wth Lowe s SIFT feature matcher [10]. Tree Optmzaton Results Please note that the proposed method s not necessarly able to merge all components nto only one model. In the worst case scenaro t s not possble to merge any components at all. In real world datasets, however, ths method worked qute well and wthout t t was smply not achevable to fnd a global regstraton for some of our mage sets wth exstng tools. Wth the proposed method of mergng connected components t s also possble to reduce runtme and memory consumpton of SfM methods by splttng an nput set of mages nto several subsets wth some mages overlap between these subsets. As such subsets can be sgnfcantly smaller and the memory consumpton and runtme of certan stages of SfM grows quadratc [3] ths s a real mprovement. Usng ths we were able to compute the camera regstraton and pont cloud seen n fgure 5 out of nearly 1700 nput mages. Ths amount of nput mages exhausted 8GB of memory when tryng to regster them all at once. Thus, we dvded the nput mages nto 7 subsets wth about 10 mages overlap between neghborng subsets, whch
(a) Hallway Component 1 (b) Hallway Component 2 (a) Subset 1 and 2 (c) Hallway Merged Result (b) Subset 3 and 4 Fgure 4: Mergng two connected components of a sparse feature hallway scene: (a) bundler decdes whch ntal par to use. (b) usng two prevously unregstered mages as ntal par. (c) Merged Result. then were easly processed wth the avalable memory and merged to the fnal output. In another scenaro seen n fgure 4 we wanted to reconstruct a hallway scene from about 400 nput mages. Unfortunately, the SfM could not regster them all at once, resultng n 2 connected components. We were able to regster all the mages usng our proposed method, as the components had 14 cameras overlap. 7 (c) Subset 5 and 6 Conclusons Gven the ncreasng popularty of tools lke Mcrosoft Photosynth and Phototoursm there wll be a hgh demand for effectve SfM algorthms n the near future. Although our method s not guaranteed to work wth every set of nput mages, t has proven tself n several scenaros we encountered durng our research. Addtonally, t are exactly those scenaros where our method was easly appled that are en vogue wth e.g. Mcrosoft Photosynth today. Especally those users could beneft from our approach as t mght be possble to regster all ther mages nto one component wthout the need of takng addtonal pctures. It mght be an nterestng topc for future research to analyze n whch scenaros our method can be appled and f t s possble to say f t works n advance. Furthermore, ntegratng addtonal knowledge about the nput mage set, e.g. f the mages are shot n sequences, nto our method, one mght manage to speed up the SfM process by automatcally dvdng the mages nto (d) Subset 7 (e) Store Merged Result Fgure 5: Mergng several components of a manually dvded set of nput mages: (a)-(d) resultng regstratons of the 7 subsets. (e) Merged Result. The 9 cameras overlap between the subsets n (a) are hghlghted n red.
subsets, e.g. by usng global mage descrptors lke the hstogram, and mergng them afterwards. References [1] Aseem Agarwala, Maneesh Agrawala, Mchael Cohen, Davd Salesn, and Rchard Szelsk. Photographng long scenes wth mult-vewpont panoramas. In SIGGRAPH 06: ACM SIGGRAPH 2006 Papers, pages 853 861, New York, NY, USA, 2006. ACM. [2] Gerhard H. Bendels, Patrck Degener, Roland Wahl, Marcel Körtgen, and Renhard Klen. Image-based regstraton of 3d-range data usng feature surface elements. In Y. Chrysanthou, K. Can, N. Slberman, and F. Nccolucc, edtors, The 5th Internatonal Symposum on Vrtual Realty, Archaeology and Cultural Hertage (VAST 2004), pages 115 124. Eurographcs, December 2004. [3] M. Brown and D. G. Lowe. Unsupervsed 3d obect recognton and reconstructon n unordered datasets. In 3DIM 05: Proceedngs of the Ffth Internatonal Conference on 3-D Dgtal Imagng and Modelng, pages 56 63, Washngton, DC, USA, 2005. IEEE Computer Socety. [4] Andrew W. Ftzgbbon and Andrew Zsserman. Automatc camera recovery for closed or open mage sequences. In ECCV 98: Proceedngs of the 5th European Conference on Computer Vson-Volume I, pages 311 326, London, UK, 1998. Sprnger-Verlag. [5] Wolfgang Förstner. A feature based correspondence algorthm for mage matchng. Internatonal Archves Photogrammetry and Remote Sensng, 3(26):160 166, 1986. [6] Chrs Harrs and Mke Stephens. A combned corner and edge detector. In The Fourth Alvey Vson Conference, pages 147 151, 1988. [11] K. Mkolaczyk, T. Tuytelaars, C. Schmd, A. Zsserman, J. Matas, F. Schaffaltzky, T. Kadr, and L. Van Gool. A comparson of affne regon detectors. Int. J. Comput. Vson, 65(1-2):43 72, 2005. [12] Davd Nstér. Reconstructon from uncalbrated sequences wth a herarchy of trfocal tensors. In ECCV 00: Proceedngs of the 6th European Conference on Computer Vson-Part I, pages 649 663, London, UK, 2000. Sprnger-Verlag. [13] Davd Nstér. An effcent soluton to the fve-pont relatve pose problem. IEEE Trans. Pattern Anal. Mach. Intell., 26(6):756 777, 2004. [14] Noah Snavely, Steven M. Setz, and Rchard Szelsk. Photo toursm: explorng photo collectons n 3d. In SIGGRAPH 06: ACM SIGGRAPH 2006 Papers, pages 835 846, New York, NY, USA, 2006. ACM. [15] Noah Snavely, Steven M. Setz, and Rchard Szelsk. Modelng the world from nternet photo collectons. Int. J. Comput. Vson, 80(2):189 210, 2008. [16] R. Szelsk and S.B. Kang. Recoverng 3d shape and moton from mage streams usng nonlnear least squares. Computer Vson and Pattern Recognton, 1993. Proceedngs CVPR 93., 1993 IEEE Computer Socety Conference on, pages 752 753, Jun 1993. [17] E. H. Thompson. A ratonal algebrac formulaton of the problem of relatve orentaton. Photogrammetrc Record, 3(14):152 159, 1959. [18] C. Tomas and T. Kanade. Shape and moton from mage streams under orthography: A factorzaton method. 9(2):137 154, November 1992. [19] Bll Trggs, P. McLauchlan, Rchard Hartley, and A. Ftzgbbon. Bundle adustment a modern synthess. In B. Trggs, A. Zsserman, and R. Szelsk, edtors, Vson Algorthms: Theory and Practce, volume 1883 of Lecture Notes n Computer Scence, pages 298 372. Sprnger-Verlag, 2000. [7] R. I. Hartley and A. Zsserman. Multple Vew Geometry n Computer Vson. Cambrdge Unversty Press, ISBN: 0521540518, second edton, 2004. [8] Marcel Körtgen. Robust automatc regstraton of range mages wth reflectance. Master s thess, Unversty of Bonn, 2006. [9] H. C. Longuet-Hggns. A computer algorthm for reconstructng a scene from two proectons. Nature, 293:133 135, Sep. 1981. [10] Davd G. Lowe. Dstnctve mage features from scale-nvarant keyponts. Int. J. Comput. Vson, 60(2):91 110, 2004.