A Scalable Projective Bundle Adjustment Algorithm using the L Norm

Transcription

1 Sxth Indan Conference on Computer Vson, Graphcs & Image Processng A Scalable Projectve Bundle Adjustment Algorthm usng the Norm Kaushk Mtra and Rama Chellappa Dept. of Electrcal and Computer Engneerng Unversty of Maryland, College Park, MD {kmtra, rama}@umacs.umd.edu Abstract The tradtonal bundle adjustment algorthm for structure from moton problem has a computatonal complexty of O((m + n) 3 ) per teraton and memory requrement of O(mn(m+n)),wherem s the number of cameras and n s the number of structure ponts. The sparse verson of bundle adjustment has a computatonal complexty of O(m 3 +mn) per teraton and memory requrement of O(mn). Herewe propose an algorthm that has a computatonal complexty of O(mn( m + n)) per teraton and memory requrement of O(max(m, n)). The proposed algorthm s based on mnmzng the norm of reprojecton error. It alternately estmates the camera and structure parameters, thus reducng the potentally large scale optmzaton problem to many small scale subproblems each of whch s a quasconvex optmzaton problem and hence can be solved globally. Experments usng synthetc and real data show that the proposed algorthm gves good performance n terms of mnmzng the reprojecton error and also has a good convergence rate. Introducton Bundle adjustment s the procedure for refnng a vsual reconstructon to produce optmal 3D structure and camera parameters. Explctly stated t s the problem of estmatng structure and camera parameters gven an ntal estmate and the observed mage data. An approprate cost functon to mnmze for ths problem s some norm of the reprojecton error [4]. Reprojecton error s the error between the reprojected mage ponts, obtaned from the current estmates of camera and structure parameters, and the observed mage ponts. The L norm of reprojecton error s the commonly used cost functon. A lot of work has been done on bundle ad- Ths work was prepared through collaboratve partcpaton n the Advanced Decson Archtectures Consortum sponsored by the U.S. Army Research Laboratory under the Collaboratve Technology Allance Program, Cooperatve Agreement DAAD justment based on the L norm. Trggs et al [4] provdes a good revew on ths approach. One reason for ths choce of the norm s that the cost functon becomes a dfferentable functon of parameters and ths allows the use of gradent and Hessan-based optmzaton methods. Second order methods lke Gauss-Newton and Levenberg- Marquardt(LM) algorthms are the preferred methods for bundle adjustment as they have some nce convergence propertes near local mnmum. The LM algorthm has a computatonal complexty of O((m+n) 3 ) per teraton and memory requrement of O(mn(m + n)), wherem s the number of cameras and n s the number of structure ponts. However, for the structure from moton problem there exsts a sparse verson of the LM algorthm that takes advantage of the fact that there are no drect nteractons among the cameras. They nteract only through the structure. The same s true for the structure ponts. Usng ths fact, a computatonal complexty of O(m 3 + mn) per teraton and memory requrement of O(mn) can be obtaned. Appendx 6 of [] provdes a good revew on ths. Computatonal complexty and memory requrement are very mportant ssues especally when solvng problems that nvolve hundreds of frames and ponts. Ths motvates us to look for an algorthm whch has lower computatonal complexty and memory requrement. Another problem n mnmzng the L norm of reprojecton error s that t s a hghly non-lnear and non-convex functon of the camera and structure parameters. Even the smpler problem of estmatng the structure parameters gven the camera parameters and the correspondng mage ponts, known as the trangulaton problem, s a nonlnear and non-convex optmzaton problem. The correspondng cost functon mght have multple mnma and fndng the global mnmum s a very dffcult problem. The same s true for the problem of estmatng the camera parameters gven the structure parameters and the correspondng mage ponts. Ths problem s known as the resecton problem. Another norm that s geometrcally and statstcally meanngful s the norm. Mnmzng the norm s For a dscusson on geometrcal sgnfcance, please see [5] /8 $5. 8 EE DOI.9/ICVGIP

2 the same as mnmax estmaton n statstcs. Apart from geometrcal and statstcal sgnfcance, the norm of reprojecton error has a nce analytcal form for some problems. Specfcally for the trangulaton and resecton problems, t s a quas-convex functon of the unknown parameters. A quas-convex functon has the property that any local mnmum s also a global mnmum. The global mnmum can be obtaned usng a bsecton algorthm ([3],[5]). Further, each step of the bsecton algorthm can be solved by checkng for the feasblty of a second order cone programmng problem(socp) [3],[5], for whch effcent software packages such as SeDuM [] are readly avalable. The avalablty of effcent means of fndng the global soluton for the trangulaton and resecton problems n the norm prompted us to look for bundle adjustment algorthms usng the same norm. Jont structure and camera parameter estmaton n the norm s not a quas-convex optmzaton problem because of the non-lnear couplng of structure and camera parameters. However f we fx one of the unknowns, say structure, and optmze over the camera parameters then we are back to the orgnal problem of resecton. The next step would be to fx the camera parameters and optmze over the structure. These two steps should be terated tll convergence. Ths algorthm s an nstance of alternaton algorthms and more specfcally of resectonntersecton algorthms n bundle adjustment [4]. In ths dscusson, ntersecton and trangulaton are synonymous. The proposed algorthm, usng the norm, has two advantages. One, fxng one of the unknown set of parameters, say the structure parameters, the camera parameters estmaton problem can be solved for each camera separately, whch effectvely reduces the hgh-dmensonal problem to many low-dmensonal subproblems. The same s true when the camera parameters are fxed and the structure parameters are estmated. Hence a hgh-dmensonal parameter estmaton problem transforms nto many low-dmensonal subproblems. The second advantage s that the subproblems that we have to solve are all quas-convex optmzaton problems whose global mnmum can be effcently found. Our algorthm s a projectve bundle adjustment algorthm and so the trangulaton and resecton subproblems have to be solved n the projectve space. In [3] and [5] the trangulaton problem s solved n the Eucldean/affne space where the optmzaton s done over the convex regon n front of all the cameras that the pont s vsble n. Ths regon s well defned n Eucldean/affne space but not n the projectve space. The search for ths regon results n an ncrease n the computatonal cost for the projectve trangulaton problem [4]. The same s true for the projectve resecton problem. We have avoded these computatons by ntalzng our algorthm usng a quas-affne reconstructon. Quas-affne reconstructon can be easly obtaned from an ntal projectve reconstructon by solvng a lnear programmng problem. There are resecton-ntersecton algorthms based on the L norm. The algorthms proposed by Chen et al [] and Mahamud et al [9] are some examples of ths. These algorthms have almost the same computatonal complexty as our algorthm but they are based on mnmzng algebrac errors, whch are approxmatons of the actual L reprojecton error. These approxmatons make them susceptble to non-physcal solutons [9]. Also n our experments we have found that though the algebrac cost based algorthms converge n the algebrac cost, they may not converge n the L reprojecton error cost. The organzaton of the paper s as follows: In secton., we provde some necessary background on solvng the trangulaton and resecton problems n the framework. In secton, we dscuss the proposed bundle adjustment algorthm. In secton 3, we compare the computatonal complexty and memory requrements of our algorthm wth the L bundle adjustment algorthm and the L based resecton-ntersecton algorthms. Secton 4, expermentally evaluates our algorthm for convergence, computatonal complexty, robustness to nose wth approprate comparson to other algorthms.. Background: geometrc reconstructon problems usng norm Here we gve a very bref overvew of the problem of mnmzng the norm for Eucldean/affne trangulaton and resecton problems. For more detals see Kahl [3] and Ke et. al [5]. We begn wth the defnton of a quas-convex functon snce the trangulaton and resecton cost functon reduces to ths form. Defnton. A functon f(x) s a quas-convex functon f ts sublevel sets are convex... Trangulaton Let P =(a T ; bt ; ct ),=,,..., M be the projecton matrces for M cameras and (u,v ),=,,..., M be the mages of the unknown 3D pont X n these M cameras. The problem s to estmate X gven P and (u,v ). Let X be the homogeneous coordnate of X.e, X =(X;). ( Then the reprojected ) mage pont, n camera, sgvenby a T X c T X, bt X c T X and the L norm of reprojecton error functon s gven by: ( f (X) = ( = a T c T a T ) X X u, bt X c T X v ) X u c T X, bt X v c T X X X c T c T () 8

3 In Eucldean trangulaton, the fact that X s n front of camera s expressed by: c T X > (also known as cheralty constrant). Wth ths constrant, we have: f (X) = a T = p (X) q (X) X u c T X,b T c T X X v c T X p (X) s a convex functon because t s a composton of a convex functon (norm) and an affne functon. q (X) s an affne functon. Functons of the form of f (X) are quas-convex (for proof, see [3],[5]). The norm of reprojecton error s F (X) =maxf (X) (3) whch s agan a quas-convex functon, as pont-wse maxmum of quas-convex functons s also quas-convex.(for proof, see [3],[5]). Mnmzaton of the quas-convex functon F can be done usng a bsecton algorthm n the range of F ([3], [5]). One step n the bsecton algorthm nvolves solvng the followng feasblty problem: fnd X s.t. X S α (4) where S α s the alpha sub-level set of F (X) wth the cheralty constrant. For trangulaton, S α = {X f (X) α, q (X) >, } (5) = {X p (X) αq (X),q (X) >, } S α s a convex set and hence we have to solve a convex feasblty problem. Moreover, snce p (X) s a L norm, ths problem s a second order cone programmng problem whch can be solved effcently usng software packages lke Sedum []... Resecton Here we are gven N 3D ponts X,=,,..., N and ther correspondng mage ponts (u,v ),=,,..., N. The problem s to estmate the camera projecton matrx P. Agan, the reprojecton error s a quas-convex functon of the unknown camera parameters and the global mnmum can be obtaned as n the trangulaton case [3],[5]. The projectve bundle adjustment algorthm For projectve bundle adjustment, we propose an teratve algorthm based on the prncple of resectonntersecton. We partton the unknown structure and camera parameters nto two separate sets and mnmze the () norm of reprojecton error over one set of parameters whle keepng the other set fxed. In the resecton step, the mnmzaton s done over the camera parameters whle keepng the structure parameters fxed and n the ntersecton step, the optmzaton s done over the structure parameters whle keepng the camera parameters fxed. These resecton-ntersecton steps are terated many tmes tll the algorthm converges. The resecton and ntersecton step of the proposed algorthm s stll a hgh-dmensonal optmzaton problem. In secton., we show that how these two steps can be further smplfed by solvng a large number of small optmzaton problems. In secton., we dscuss the correct way to ntalze our algorthm. Convergence s another mportant ssue for any teratve method. In secton.3, we show the convergence of our algorthm n the norm of reprojecton error.. Decouplng Consder the ntersecton step of the algorthm, where camera parameters are fxed and mnmzaton of the norm of reprojecton error s done over the structure parameters. Let P j,j =,,..., M be the gven projecton matrces of M cameras and X,=,,..., N be the N 3D ponts, whch are to be estmated. Let f j be the L norm of reprojecton error for the -th 3D pont maged n the j-th camera. The norm of reprojecton error s: F (X,X,..., X N )=maxf j,j (X,X,...,X N ) where, =max max f j j (X ) =maxf, (X ) (6) f, (X )=maxf j j (X ) (7) Theorem. The problem of mnmzng the norm of reprojecton error over the jont structure (X,X,...,X N ) can be solved by mnmzng, ndependently, the norm of reprojecton error correspondng to each 3D structure X,.e. suppose ( X, X,..., X N ) solves the jont structure problem and ( X, X,..., X N ) solves the ndependent problems of mnmzng f, (X ),f, (X ),...,f,n (X N ) respectvely, then F ( X, X,..., X N )=F ( X, X,..., X N ) (8) Proof. By defnton (see equaton 6), F ( X, X,..., X N )=maxf, ( X ) 8

4 F ( X, X,..., X N )=maxf, ( X ) Now, X solves mn X f, (X ) therefore, mples, f, ( X ) f, (X ) f, ( X ) f, ( X ) Ths s true for all, therefore.e., but snce max X f, ( X ) max f, ( X ) F ( X, X,..., X N ) F ( X, X,..., X N ) F ( X, X,..., X N )= therefore mn F (X,X,..., X N ) (X,...,X N ) F ( X, X,..., X N )=F ( X, X,..., X N ) It should be noted that the proof does not menton about the cheralty constrants. Ths s to keep the proof smple. The proof wth cheralty constrants wll follow essentally thesamearguments. In the resecton step, the structure s fxed and optmzaton s done over the camera parameters and we have the followng corollary: Corollary. The problem of mnmzng the norm of reprojecton error over the jont camera projecton matrces (P,P,...,P M ) can be solved by mnmzng, ndependently, the norm of reprojecton error correspondng to each projecton matrx P j,.e. suppose ( P, P,..., P M ) solves the jont camera matrces problem and ( P, P,..., P M ) solves the ndependent problem for P, P,...,P M respectvely, then F ( P, P,..., P M )=F ( P, P,..., P M ) (9) The proof s smlar to that of Theorem wth the structure parameters replaced by the camera parameters. Hence jont mnmzaton over all the structure/camera parameters can be solved by solvng for ndvdual structure/camera parameters.. Cheralty and quas-affne ntalzaton The usual way to ntalze a projectve bundle adjustment algorthm s a projectve reconstructon obtaned from the gven mages. Any of the methods mentoned n [] can be used for projectve reconstructon. However dong ths for the proposed algorthm wll ncrease ts computatonal complexty. To understand and get around ths problem, we make a small dgresson and precsely state the defnton of cheralty. Let X =(X, Y, Z, T) be a homogeneous representaton of a pont and P =(a T ; b T ; c T ) be the projecton matrx of a camera, then the maged pont x s gven by P X = ωˆx, where ˆx denotes the homogeneous representaton of x n whch the last coordnate s. The depth of the pont X wth respect to the camera s gven by: depth(x; P )= sgn(detm)ω () T m 3 where M s the left hand 3 3blockofP and m 3 s the thrd row of M []. A pont X s sad to be n front of the camera f and only f depth(x; P ) >. Defnton. The quantty sgn(depth(x; P )) s known as the cheralty of the pont X wth respect to the camera []. If detm > and T >, thenc T X > mples that the pont s n front of the camera(snce ω = c T X). In secton., we have seen that the trangulaton and resecton problems were solved whle satsfyng ths cheralty constrant. Cheralty s nvarant under Eucldean and affne transformatons (n fact, also under quas-affne transformatons) [] and so seekng a soluton wth the cheralty constrant s justfed when solvng for the Eucldean/affne trangulaton and resecton problems. However, cheralty s not nvarant under projectve transformatons. Hence when solvng the projectve trangulaton problem, we can t just restrct our search for X n the convex regon of {X : c jt X >, j}. If there are M cameras, then the M prncpal planes dvde the projectve space P 3 nto (M 3 +3M +8M)/6 regons [4]. The cost, wth respect to X, has to be mnmzed over each of these regons and the mnmum among them s the desred soluton of the projectve trangulaton problem[4]. To avod ths mnmzaton over all the regons durng trangulaton, a projectve reconstructon should be converted to an Eucldean/affne/quas-affne reconstructon. Once ths s done, we can mnmze the cost over the convex regon {X : c jt X >, j}. A smlar argument holds for the projectve resecton problem. A good choce would be to perform a converson from a projectve reconstructon to a quas-affne reconstructon. A quas-affne reconstructon les n between projectve and 8

5 Algorthm Bundle Adjustment usng norm mnmzaton Input: Set of mages Output: Projectve reconstructon. Do ntal projectve reconstructon from set of mages.. Convert to quas-affne reconstructon. 3. whle Reprojecton error >ɛ 4. do Get camera parameters for each camera by resecton. 5. Get 3D structure parameters for each pont by trangulaton..e. f, ( X ) f, (X ) hence, f, ( X ) f, ( ˆX ) Ths s true for all, therefore max X C f, ( X ) max f, ( ˆX ) () F ( X, X,..., X N ) F ( ˆX, ˆX,..., Xˆ N ) (3) F (I) F (R) (4) Fgure. BA Algorthm affne reconstructon. The only nformaton requred for ths converson s the fact that f a pont s maged by a camera, then t must be n front of the camera. The transformaton that takes a projectve reconstructon to a quas-affne reconstructon can be found by solvng a lnear programmng problem []. To summarze, we frst obtan an ntal projectve reconstructon from the mages and then convert ths to a quasaffne reconstructon by solvng a lnear programmng problem. Ths reconstructon s then used as an ntalzaton for our algorthm. After ths ntalzaton, we can use the trangulaton/resecton method of secton.. The summary of the algorthm s gven n Fgure..3 Convergence of the algorthm Here we gve a formal proof that at each step of resecton and ntersecton, the reprojecton error ether decreases or remans constant. Theorem. Consder the followng sequence of steps n the algorthm : Intal Structure Resecton (R) Intersecton (I). Then F (I) F (R), wheref (I) and F (R) are cost after ntersecton and resecton respectvely. Proof. Let the ntal structure ponts be ( ˆX, =,,...,N). In the resecton step, P j,j =,,...,M are estmated based on ˆX. Let the estmated P j be P ˆj = [ a ˆjT ; ˆ bjt ; ˆ cjt ].Thenc jt ˆ ˆX > for all and j (cheralty constrant mposed by the algorthm). Now consder the ntersecton step, where the structure s estmated from P ˆj s. Let C be the cheralty constrant set {X : ˆ cjt X>}. By decouplng (see secton.), the estmated structure X s gven by: X =arg mn X C f,(x ) () The ntuton of the proof s that when we look at the two steps, Intal Structure Resecton and Resecton Intersecton, resecton s the common step. So P j s wll be same n the two steps, wth the values of X s dfferng. In the ntersecton step, we are mnmzng over X s under the cheralty constrants and snce the orgnal structure also satsfes the same constrants, the structure after ntersecton wll contrbute a smaller cost than the orgnal structure. Corollary. Consder the followng step n the algorthm: Intal camera parameters Intersecton (I) Resecton (R). Then F (R) F (I), wheref (R) and F (I) are the cost after resecton and ntersecton respectvely. The proof s the same wth structure replaced by camera parameters and vce versa. Theorem and corollary together prove that the reprojecton error decreases or remans constant wth every step of resecton and ntersecton. Snce the reprojecton error s bounded from below by zero, by monotone convergence theorem, the algorthm converges n the reprojecton error. Ths proof only guarantees the convergence of the algorthm to a local mnmum of the cost functon. Convergence to global mnmum s dependent on good ntalzaton. We should note that the L based bundle adjustment also has smlar guarantee. 3 Computatonal complexty and memory requrement Ths secton frst descrbes the computatonal complexty and memory requrement of the proposed algorthm and then compares t wth that of L based bundle adjustment and L based resecton-ntersecton algorthms. As dscussed n secton., at any tme we are ether solvng the trangulaton problem for one structure pont or the resecton problem for one camera. We frst analyze the computatonal complexty and memory requrements for solvng one trangulaton problem. The trangulaton problem s solved by a bsecton algorthm [3],[5]. At each step of the bsecton, we solve a convex feasblty problem, 83

6 gven n (5) of secton.. If the pont that we are trangulatng s vsble n m cameras then we have to solve m second order cone feasblty problem. Ths problem has a computatonal complexty of O(m.5 ) and a memory requrement of O(m) [7]. By analogy, the resecton problem for one camera has an emprcal computatonal complexty O(n.5 ) and memory requrement O(n) where n ponts are vsble n that camera. Now consder one teraton of our algorthm. For the case where there are m cameras and n ponts and all the ponts are vsble n all the cameras, per teraton emprcal computatonal complexty s O(mn( m + n)) and the memory requrement s O(max(m, n)). Furthermore a parallel mplementaton of the algorthm s possble because durng each resecton/ntersecton step all the cameras/ponts can be estmated at the same tme. Ths s because of the decouplng dscussed n secton.. Such an mplementaton wll result n a reducton of computatonal complexty. The L norm bundle adjustment algorthm (L BA) s based on the Levenberg-Marquardt (LM) method. The central step nvolves solvng an equaton wth all the camera and structure parameters as unknowns. Hence ts computatonal complexty s O((m + n) 3 ) per teraton. For memory requrement, we can consder the Jacoban whch s O(mn(m + n)). For structure from moton problems, there exsts a sparse LM method whch uses the fact that the cameras are related to each other only through structure and vce-versa []. For sparse LM, computaton complexty s O(m 3 + mn) per teraton. The memory requrement s O(mn) []. The L based resecton-ntersecton algorthms [],[9] have computatonal complexty of O(mn) per teraton and same memory requrement as our algorthm,.e, O(max(m, n)) [9]. But snce these algorthms mnmze approxmate algebrac errors, they are not so relable as we found n our experments 4.. Uptll now we have dscussed only about the computatonal complexty per teraton. But there s also the queston of number of teratons requred for convergence. Theoretcally t s a dffcult queston to answer, but we would lke to pont out one fact whch favors our algorthm. In each step of resecton and ntersecton our algorthm fnds the global mnmum of the respectve cost functons. Ths s a bg advantage over L BA algorthm whch doesn t do an exact lne search over t s search drecton. Decdng how much to move along a search drecton s an mportant ssue for fast convergence and t goes by the name of step control [4]. The L based resecton-ntersecton algorthms [], [9] do fnd the global mnmum n each step but they do so of approxmate, algebrac cost functons. 4 Experments We have done expermental evaluaton of the proposed algorthm ( BA) for convergence, computatonal scalablty and robustness to nose. Comparsons wth L bundle adjustment based on LM algorthm (L BA) and L resecton-ntersecton algorthms are also gven. For L BA we have used a publcly avalable mplementaton of projectve bundle adjustment [8] based on the sparse LM method. The L based resecton-ntersecton algorthms that we have compared wth are the Weghted Iteratve Egen algorthm () proposed by Chen et al [] and a varaton of the same algorthm where we avod the reweghtng step, henceforth called the algorthm. In secton 4., whle studyng the convergence of the four algorthms we found that the performance of and are unrelable and hence n the rest of the sectons we have compared BA wth only L BA. For ntal reconstructon, we have used the projectve factorzaton method of Trggs et al [3] wth proper handlng of mssng data. Ths reconstructon was then converted to a quas-affne reconstructon. Any other projectve reconstructon followed by a converson to a quas-affne reconstructon wll also work fne. When comparng dfferent algorthms, all of them have been provded wth the same ntal reconstructon. 4. Convergence Here we study the convergence wth teraton of BA, L BA, and algorthms. Experments are done on one synthetc data set and three real data sets. The synthetc data set conssts of ponts dstrbuted unformly wthn a unt sphere and 5 cameras n a crcle around the sphere lookng straght at the sphere. Gaussan nose of standard devaton pxel s added to the feature ponts. In all the experments, reprojecton error for ths data set s the mean reprojecton error over ten trals. The real data sets used are the corrdor and dnosaur data set and the hotel data set 3. We have used a subset of 464 structure ponts from the dnosaur data set and a subset of 6 vews from the hotel data set. For the corrdor and the dnosaur data sets, feature ponts are already avalable and we have used them as they are. For the hotel data set we have used the KLT tracker to track feature ponts and then used Torr s Matlab SFM Toolbox [] to remove the outlers. We compare the convergence of the algorthms n the norm of reprojecton error and the Root Mean Squares reprojecton error (RMS) whch s a measure of the L norm. Fgure shows that the error decreases monotoncally for BA, as clamed n secton.3, but not so vgg/data/data-mvew.html

7 for the other algorthms. Fgure 3 shows RMS error decreases (almost) monotoncally for BA and L BA but not so for and. From Fgure 3, we can further conclude the followng. All the algorthms converge well for the sphere data set. For the corrdor data set, converges at a hgher value than others. For the hotel data set, both and frst dverge and then later converge. For the dnosaur data set, fals to converge. To further study the nature of and, we added Gaussan nose of standard devaton pxel to the real data sets and found that the algorthms fal to converge at many trals. However, each tme these algorthms have converged n the algebrac cost that they mnmze. Ths study tells us that algebrac cost based algorthms may not be very relable. For all of the above data sets, our algorthm converges wthn ten teratons wth smlar RMS reprojecton error as L BA. Fgure 4 shows the fnal 3-D reconstructon by our algorthm for the datasets, sphere and corrdor. RMS reprojecton error n pxels RMS reprojecton error n pxels RMS reprojecton error Vs Iteratons for Sphere Iteratons RMS reprojecton error Vs Iteratons for Hotel 3 4 Iteratons RMS reprojecton error n pxels RMS reprojecton error n pxels.5.5 RMS reprojecton error Vs Iteratons for Corrdor 3 4 Iteratons RMS reprojecton error Vs Iteratons for Dnosaur Iteratons Fgure 3. RMS reprojecton error versus teraton : RMS error decreases monotoncally for BA and L BA but not so for and. fals to converge for the dnosaur data set. reprojecton error n pxels reprojecton error n pxels L reprojecton error Vs Iteratons for Sphere Iteratons 5 reprojecton error Vs Iteratons for Hotel Iteratons reprojecton error n pxels reprojecton error n pxels 5 5 reprojecton error Vs Iteratons for Corrdor Iteratons reprojecton error Vs Iteratons for Dnosaur Iteratons Note that we have to estmate the camera parameters correspondng to each frame of the vdeo. Thus our algorthm s sutable for solvng reconstructon problems for vdeo data where the number of frames can be large. Recently, there has been some work on faster computatons of trangulaton and resecton problems [] and ncorporatng ths wll reduce the convergence tme of our algorthm. Further reducton n convergence tme s possble by a parallel mplementaton, whch we have not done here. 4.3 Behavor wth nose Fgure. reprojecton error versus teraton for the four algorthms on the data sets: sphere, corrdor, hotel and dnosaur. error decreases monotoncally for BA but not so for the other algorthms. 4. Computatonal scalablty We dd experment on the synthetc sphere data set to compare the total convergence tme for L BA and BA as the number of cameras s vared wth the number of ponts fxed at 5, Fgure 5. To ensure farness, both the algorthms were mplemented n Matlab wth the computatonally ntensve routnes as mex fles. L BA converges at about teratons and BA at about teratons. Fgure 5 clearly shows that our algorthm has the advantage n terms of tme from 5 cameras onwards. For a vdeo wth 3 frames per second ths s approxmately 8 sec of data. Gaussan nose of dfferent standard devatons are added to the feature ponts. Fgure 6 shows the RMS reprojecton error n pxels wth nose for the synthetc data set, sphere. Generally the norm has the reputaton of beng very senstve to nose, but here we see a graceful degradaton wth nose. Further to handle nose wth strong drectonal dependence, we can ncorporate the drectonal uncertanty model of Ke et. al. [6] nto the resecton and trangulaton steps of our algorthm, though we have not done t here. We have not consdered outlers here, as bundle adjustment s consdered to be the last step n the reconstructon process and outler detecton s generally done n the earler stages of the reconstructon. In fact as mentoned earler n secton 4., we have done outler removal for the hotel data set before the ntal reconstructon step. 5 Concluson We have proposed a scalable projectve bundle adjustment algorthm usng the norm. It s a resecton- 85

8 Intal reconstructon: Sphere.5.5 Intal reconstructon: Corrdor 5 Fnal reconstructon: Sphere.5.5 Fnal reconstructon: Corrdor 5 Groundtruth: Sphere.5.5 Groundtruth: Corrdor RMS reprojecton error n pxels Reprojecton error Vs feature nose for Sphere Standard devaton of Gaussan nose Fgure 4. 3-D reconstructon result for the datasets, Sphere and Corrdor. The Red * represents the camera center and Blue o represents the structure pont. The frst column shows the ntalzaton, second column shows the fnal reconstructon and the thrd column shows the groundtruth. Total convergence tme n mns Total convergence tme Vs Number of cameras Number of cameras Fgure 5. Total convergence tme of L BA and BA as the number of cameras s vared wth number of ponts fxed at 5. ntersecton algorthm whch converts the large scale optmzaton problem to many small scaled ones. There are scalable resecton-ntersecton algorthms based on the L norm but they are not so relable as we have found out n our experments. Our algorthm gves smlar performance as L BA n terms of mnmzng the reprojecton error. It has a good convergence rate and degrades gracefully wth nose. In concluson we can say that our algorthm wll outperform L BA when solvng for problems wth large number of cameras or frames lke n a vdeo data set. It s possble to make the present algorthm faster usng a parallel mplementaton and by a more effcent mplementaton of resecton and trangulaton. Acknowledgements The frst author wshes to thank Sameer Shrdhonkar and Ashok Veeraraghavan for dscussons and encouragement. Fgure 6. Behavor of BA and L BA wth mage feature nose for the sphere data set. References [] Q.Chen and G.Medon, Effcent teratve soluton to M- vew projectve reconstructon problem, EE Conf. on CVPR, 999. [] R.Hartley and A.Zsserman, Multple vew geometry n computer vson, Cambrdge Unversty press, nd edton, 4. [3] F.Kahl, Multple vew geometry and the -norm, EE ICCV, 5. [4] F.Kahl and R.Hartley, Multple vew geometry under the -norm, EE TPAMI, 7. [5] Q.Ke and T.Kanade, Quasconvex optmzaton for robust geometrc reconstructon, EE ICCV, 5. [6] Q.Ke and T.Kanade, Uncertanty Models n Quasconvex Optmzaton for Geometrc Reconstructon, EE CVPR, 6. [7] M.S.Lobo, L. Vandenberghe, S. Boyd and H. Lebret, Applcatons of second-order cone programmng, Lnear Algebra and ts Applcatons 84:93-8, November 998. [8] M.I.A. Louraks and A.A. Argyros, The Desgn and Implementaton of a Generc Sparse Bundle Adjustment Software Package Based on the Levenberg-Marquardt Algorthm, ICS/FORTH Techncal Report No. 34, 4. [9] S.Mahamud, M.Herbert, Y.Omor and J.Ponce, Provably-Convergent Iteratve Methods for Projectve Structure from Moton, EE Conf. on CVPR,. [] C.Olsson, A.P.Erksson and F.Kahl, Effcent Optmzaton for -problems usng Pseudoconvexty, Proc. EE ICCV, 7. [] J.Sturm, Usng SeDuM., A MATLAB toolbox for optmzaton over symmetrc cones., Optmzaton methods and software, -:65-653, 999. [] P.H.S.Torr, A Structure and Moton Toolkt n Matlab Interactve advantures n S and M, Techncal report, MSR-TR--56, June. [3] B.Trggs, Factorzaton methods for projectve structure and moton, Proc. EE Conf. on CVPR, 996. [4] B.Trggs, P.McLauchlan, R.Hartley and A.Ftzgbbon, Bundle adjustment - a modern synthess, In Vson Algorthms: Theory and Practce, Sprnger-Verlag,. 86