WITH the basic principles and algorithms of structurefrom-motion

Transcription

1 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 Coreses for Triangulaion Qianggong Zhang and Ta-Jun Chin arxiv:77.566v [cs.cg] 8 Jul 7 Absrac Muliple-view riangulaion by l minimisaion has become esablished in compuer vision. Sae-of-he-ar l riangulaion algorihms exploi he quasiconvexiy of he cos funcion o derive ieraive updae rules ha deliver he global minimum. Such algorihms, however, can be compuaionally cosly for large problem insances ha conain many image measuremens, e.g., from web-based phoo sharing sies or long-erm video recordings. In his paper, we prove ha l riangulaion admis a corese approximaion scheme, which seeks small represenaive subses of he inpu daa called coreses. A corese possesses he special propery ha he error of he l soluion on he corese is wihin known bounds from he global minimum. We esablish he necessary mahemaical underpinnings of he corese algorihm, specifically, by enacing he sopping crierion of he algorihm and proving ha he resuling corese gives he desired approximaion accuracy. On large-scale riangulaion problems, our mehod provides heoreically sound approximae soluions. Ieraed unil convergence, our corese algorihm is also guaraneed o reach he rue opimum. On pracical daases, we show ha our echnique can in fac aain he global minimiser much faser han curren mehods. Index Terms Coreses, approximaion, generalised linear programming, muliple view geomery, riangulaion. INTRODUCTION WITH he basic principles and algorihms of srucurefrom-moion well esablished, researchers have begun o consider large-scale reconsrucion problems involving millions of inpu images. Arguably such large-scale problems, which arise from, e.g., phoo sharing websies or long-erm video observaions in roboic exploraion, are more common and pracical. The significan problem sizes involved in such seings, however, compel praciioners o eiher use disribued compuaional archiecures (e.g., GPU) o perform he required opimisaion, or accep approximae soluions for he reconsrucion. This paper conains a heoreical conribuion under he second paradigm. We inroduce a corese approximaion scheme (more below) and prove is validiy for muliple view 3D reconsrucion, specifically for riangulaion. Triangulaion is he ask of esimaing he 3D coordinaes of a scene poin from muliple D image observaions of he poin, given ha he pose of he cameras are known []. The ask is of fundamenal imporance o 3D vision, since i enables he recovery of he 3D srucure of a scene. Whils in heory srucure and moion mus be obained simulaneously, here are many seings, such as large-scale reconsrucion [], [3] and SLAM [], where he camera poses are firs esimaed wih a sparse se of 3D poins, before a denser scene srucure is produced by riangulaing oher poins using he esimaed camera poses. An esablished approach for riangulaion is by l minimisaion [5]. Specifically, we seek he 3D coordinaes ha minimise he maximum reprojecion error across all views. Unlike he sum of squared error funcion which conains muliple local minima, he maximum reprojecion error funcion is quasiconvex and hus conains a single global minimum. Algorihms ha ake advanage of his propery have been developed o solve such quasiconvex problems exacly [6], [7], [8], [9], [], [], [], [3]. In The auhors are wih he School of Compuer Science, The Universiy of Adelaide, Adelaide, SA, 5, Ausralia. {qianggong.zhang, a-jun.chin}@adelaide.edu.au paricular, Agarwal e al. [] showed ha some of he mos effecive algorihms belong o he class of generalised fracional programming (GFP) mehods [], [5]. Alhough algorihms for l riangulaion have seadily improved, here is sill room for improvemen. In paricular, on large-scale reconsrucion problems or SLAM where here are usually a significan number of views per poin (recall ha he size of a riangulaion problem is he number of D observaions of a scene poin), he compuaional cos of many of he algorihms can be considerable; we will demonsrae his in Secion 5. A major reason is ha he algorihms need o repeaedly solve convex programs o deermine he updae direcion, which is of cubic complexiy in wors case. I is hus of ineres o invesigae effecive approximae algorihms.. Conribuions As alluded above, our main conribuion in his paper is heoreical. Specifically, we prove ha he l riangulaion problem admis a corese approximaion scheme [6], [7]. A corese is a small represenaive subse of he daa ha approximaes he overall disribuion of he daa. In he conex of l riangulaion from N views, our algorihm ieraively accumulaes a corese, such ha he error from solving he problem on he corese is bounded wihin a facor of ( + ɛ) from he heoreically achievable minimum. Given a desired ɛ, we esablish a sopping crierion for he algorihm such ha he oupu corese gives he required approximaion accuracy. This provides a mahemaically jusified way o deal wih large-scale problems where considering all available daa may no be desirable or worhwhile. Ieraed unil convergence, he corese algorihm is guaraneed o aain he globally opimal soluion. We experimenally demonsrae ha he algorihm can in fac find he global minimiser much faser han many sae-of-he-ar l riangulaion mehods. This superior performance was esablished on publicly available large scale 3D reconsrucion daases. From a pracical sandpoin, our algorihm hus

2 Residuals a opimal JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 provides a useful anyime behaviour, i.e., he algorihm can simply be run unil convergence, or unil he ime budge is exhaused. In he laer case, we have a guaraneed bound of he approximaion error w.r.. he opimum. The exisence of coreses for quasiconvex vision problems was speculaed by Li [8]. However, lile progress has been made on his subjec since. We provide a posiive answer on one such problem. Our work is also one of he firs o exend he idea of coreses in compuaional geomery [6], [7] o compuer vision. BACKGROUND Le {P i, u i } N i= be a se of daa for riangulaion, consising of camera marices P i R 3 and observed image posiions u i R of he same scene poin x R 3. In his paper, by a daum we mean a specific camera and image poin {P i, u i }. Le X = {,..., N} index he se of daa. The l echnique esimaes x by minimising he maximum reprojecion error where min max r(x P x i, u i ), () i X subjec o P 3 i x > i X. r(x P i, u i ) = u i P: i x P 3 i x () is he reprojecion error. Here, P : i and P 3 i respecively denoe he firs-wo rows and hird row of P i, and x is x in homogeneous coordinaes. The reprojecion error is basically he Euclidean disance beween he observed poin u i and he projecion of x ono he i-h image plane. The cheiraliy consrains P 3 i x > i X ensure ha he esimaed poin lies in fron of all he cameras. Problem () belongs o a broader class of problems called generalised linear programs (GLP) [9]. Two properies of GLPs ha will be useful laer in his paper, are saed in he conex of () as follows. Propery (Monooniciy). For any C X, min max x i C r(x P i, u i ) min max r(x P x i, u i ) (3) i X given he appropriae cheiraliy conrains on boh sides. Propery (Suppor se). Le x and δ respecively be he minimiser and minimised objecive value of (). There exiss a subse B X wih B, such ha for any C ha saisfies B C X, he following holds δ = min max r(x P x i, u i ) i B = min x max i C r(x P i, u i ) = min max r(x P x i, u i ) i X given he appropriae cheiraliy conrains. In fac, he hree problems in () have he same minimiser x. Furher, () r(x P i, u i ) = δ for any i B. (5) The subse B is called he suppor se of he problem. See [8], [9], [] for deails and proofs relaed o he above properies. Inuiively, (5) saes ha, a he soluion of (), he minimised maximum error occurs a he suppor (a) Camera index Fig.. Triangulaing a poin x observed in views. The red + is he l soluion x. Observe ha here are four views/measuremens wih he same residual a x. The index of he suppor se is hus B = {7, 8, 9, }. se B. Fig. illusraes his propery. Furher, () saes ha solving () amouns o solving he same problem on B. Many classical algorihms in compuaional geomery [], [], [3] exploi his propery o solve GLPs. 3 CORESET ALGORITHM We firs describe he corese algorihm and focus on is operaional behaviour, before embarking on a discussion of is convergence properies in Sec. 3. and he derivaion of he corese approximaion bound in Sec Main Operaion The corese algorihm for l riangulaion is lised in Algorihm. The primary objecive is o seek a represenaive subse C s X of he daa. This is accomplished by ieraively accumulaing he daa ha should appear in he subse, where he daum ha is seleced for inclusion a each ieraion is he mos violaing daum; see Sep 6. The size of he subse, and equivalenly he runime of he algorihm, is conrolled by he desired approximaion error ɛ. To achieve, for e.g., a % approximaion error, se ɛ =.. Observe ha Algorihm is a mea-algorihm, since i requires execuing a solver for () on he daa subse indexed by he curren subse C (see Seps 3 and ). Any of he previous l riangulaion algorihms [6], [7], [8], [9], [], [] can be applied as he solver. There are wo erminaing condiions for Algorihm : ) Ieraion couner reaches /ɛ. In his case, he oupu C s indexes a corese wih he desired approximaion accuracy ɛ. Secion 3.3 will esablish he error bound for approximaing () using he daa indexed by C s. ) The global minimiser has been found (Sep 8). The saisfacion of he condiion in Sep 7 implies ha C already conains he suppor se B, since he larges error across all X is no larger han he value of () on he daa indexed by C ; see Propery. To aid inuiion, a sample parial run of Algorihm is shown in Fig.. (b)

3 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 3 (a) = (iniialisaion) (b) = (c) = 3 (d) = Fig.. A sample run of Algorihm on he daa displayed in Fig.. (a) Four image measuremens/camera viewpoins (in red) were seleced o form he iniial corese C. The curren soluion x is shown as a red cross. (b) (d) Algorihm progressively insers new daa ino he corese. Daa in he curren corese is shown in black, and he newly insered daum (chosen according o Sep 6) is shown in red. Similary, he previous soluions x s are shown in black, and he curren esimae is shown in red. If erminaed a = /ɛ, he esimae is a ɛ-approximaion of he rue opimum. Ieraed unil convergence, he global opimum is achieved. For anyime behaviour, he error bound can be backracked (see Sec. 3.5) o obain he approximaion error of he las esimae a erminaion. Algorihm Corese algorihm for l riangulaion (). Require: Inpu daa {P i, u i } N i=, approximaion error ɛ. : Randomly permue he order of {P i, u i } N i=, and define X = {,..., N}. : s, γ, g, C {,, 3, }. 3: (x, δ ) Minimiser and minimised value of () on daa indexed by C : 5: while /ɛ do 6: q argmax i X r(x P i, u i ). 7: if r(x P q, u q ) δ hen 8: /* Found global minimum */ s, g, exi while loop. 9: end if : if r(x P q, u q ) < γ hen : /* Found a beer corese */ s, γ r(x P q, u q ). : end if 3: C C {q}. : (x, δ ) Minimiser and minimised value of () on daa indexed by C. 5: +. 6: end while 7: if g = hen 8: q argmax i X r(x /ɛ P i, u i ). 9: if r(x /ɛ P q, u q ) < γ hen : s /ɛ. : end if : end if 3: reurn C s, x s and δ s. 3. Convergence o Global Minimum If we are only ineresed in he global minimiser x, hen ɛ should be se o (or a value small enough such ha /ɛ N 3). We prove ha wih his seing Algorihm will always find x in a finie number of seps. Theorem. If /ɛ N 3, hen Algorihm finds x in finie ime. Proof. Le q be obained according o Sep 6. If q C, hen, by how x and δ were calculaed in Sep, he condiion in Sep 7 mus be saisfied and x is he global minimiser. If q / C and he condiion in Sep 7 is saisfied, hen equaion () is implied and x is he global minimiser. If q / C and he condiion in Sep 7 is no saisfied, hen Algorihm will inser q ino C. There are a mos N of such inserions (including he iniial four inserions ino C ). If /ɛ N 3, in he wors case all of X will finally be insered, and C /ɛ = X and x /ɛ = x. Noe ha, whils Algorihm needs o repeaedly call an l solver, i only invokes he solver on a small subse C of he daa. Second, he way a new daum is seleced (Sep 6) o be insered ino C basically by choosing he mos violaing daum w.r.. he curren soluion enables B o be found quickly. Secion 5 demonsraes ha Algorihm can in fac find he global minimiser much more efficienly han invoking an l solver [6], [7], [8], [9], [], [] in bach mode on he whole inpu daa X. Uilised as a global opimiser (i.e., se ɛ = ), Algorihm can be viewed as a Las Vegas syle randomised algorihm, since i always finds he correc resul bu a non-deerminisic runime. In addiion, as indicaed in he proof of Theorem, in he wors case Algorihm akes N ieraions since i considers each measuremen a mos once. 3.3 Corese Approximaion Our primary conribuion is o show ha he subse C s oupu by Algorihm is a corese of he l riangulaion problem (). This is conveyed by he following heorem, which bounds he error of approximaing () using C s.

4 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 Theorem. Le C s, x s and δ s be he oupu of Algorihm. Then max i X r(x s P i, u i ) ( + ɛ)δ, (6) where δ is he minimised objecive value for problem () on he full daa X. Inuiively, he above heorem saes ha he error of approximaing x wih x s (he laer was compued using he C s oupu by Algorihm ) is a mos ( + ɛ)-imes of he smalles possible error. This provides a mahemaically jusified way of dealing wih large scale problems. The res of his subsecion is devoed o proving Theorem. Firs, we define he se of geomerical quaniies in Fig. 3. For an arbirary camera marix P wih measuremen u, he reprojecion error of a given x is r(x P, u) = u P: x P 3 x = u f P (x), (7) where f P (x) is he projecion of x ono he image. Given a se of daa {P i, u i } N i=, le x be he global minimiser of (). Define a disc on he image plane wih cenre u and radius r(x P, u); backprojecing his disc creaes a solid ellipic cone c P (x ). Define h P (x ) as he angen plane on he surface of c P (x ) ha conains x. Lemma. The angle (u : f P (x ) : f P (x)) is acue iff here is a line segmen S = {x x = x + α(x x ), α < } (8) (i.e., S has a sar poin a x and lies along vecor x x ) such ha any poin x on S will give a sricly smaller reprojecion error han x, i.e., r(x P, u) < r(x P, u) x S. (9) Proof. If (u : f P (x ) : f P (x)) is acue, hen from Lemma, x mus be on he same side of h P (x ) as u. The line segmen joining x and x mus hus inersec he inside of c P (x ); his inersecion gives S. Since S is inside c P (x ), from Lemma any x S mus give a sricly smaller reprojecion error han x. The reverse direcion can be proven by realising ha any x which gives a sricly smaller reprojecion error han x mus lie in c p (x ). Any line segmen ha joins x and x wih x in he middle mus lie on he same side of h P (x ) as u. From Lemma, (u : f P (x ) : f P (x)) mus be acue. Of cenral imporance is he following resul. Lemma 5. Le {P i, u i } N i= be a se of daa, x be he global minimiser of () on he daa, and δ be he minimised value of (). For an arbirary x R 3 in fron of he camera, here exiss a daum {P j, u j } such ha (u j : f Pj (x ) : f Pj (x)) > 9. () Via he cosine rule, he above inequaliy can be re-expressed as r(x P j, u j ) f Pj (x) f Pj (x ) + r(x P j, u j ). () Proof. From (5), a he soluion x here mus exis a suppor se B such ha he daa indexed by B aain he minimised maximum residual δ. I is sufficien o consider B. We aim o conradic he following assumpion: x s.. (u i : f Pi (x ) : f Pi (x)) < 9 i B. () Fig. 3. Definiion of several geomerical quaniies for l riangulaion. We now esablish several inermediae resuls. In he following, we consider only x R 3 ha lies in fron of he camera, i.e., x is never on he same side of he image plane as he camera cenre. Lemma. x is inside c P (x ) iff r(x P, u) < r(x P, u). Lemma. x is on he same side of h P (x ) as u iff he angle θ formed by he hree poins u : f P (x ) : f P (x) is acue, i.e., θ < 9. Lemma 3. x is on he opposie side of h P (x ) as u iff he angle θ formed by he hree poins u : f P (x ) : f P (x) is obuse, i.e., θ > 9. The above hree lemmaa can be proven easily by inspecing Fig. 3. As an exension of Lemma, he following saemen can be made. Given an x ha saisfies (), Lemma saes ha for each i B, here is a line segmen S i ha lies compleely inside c Pi (x ). Amongs all he segmens S i, i B, pick he shores one and call i S. The segmen S mus lie simulaneously in all of he cones c Pi (x ), i B (recall from (8) ha all S i s begin a x and lie along vecor x x ). Any x S mus hus yield a sricly smaller reprojecion error han x for all {P i, u i }, i B. This conradics ha B is a suppor se, hus falsifying (). The falsiy of () implies ha for an arbirary x, here mus be an i B such ha (u i : f Pi (x ) : f Pi (x)) > 9 se j as ha i. Given he above resuls, we adap Bǎdoiu and Clarkson s derivaion [7] o yield he inequaliy (6) for riangulaion. Define δ := ( + ɛ)δ, λ := δ / δ, k P := f P (x ) f P (x ). (3)

5 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 5 Noe ha λ. Furher, since C X, from Propery, δ δ, hus λ δ / δ = /( + ɛ). () We aim o disprove he following assumpion: such ha max i X r(x P i, u i ) ( + ɛ)δ. (5) In words, (5) effecively saes ha none of he C accumulaed hroughou he ieraions in Algorihm gives a (+ɛ) approximaion o (). For any, Lemma 5 saes ha here exiss a j C such ha r(x P j, u j ) f Pj (x ) f Pj (x ) + r(x P j, u j ) = (k Pj ) + δ (6) (recall ha j indexes a daum in he suppor se of he daa indexed by C, hus r(x P j, u j ) = δ ). Then r(x P j, u j ) λ δ + (k Pj ). (7) For he q chosen in ieraion (Sep 6 in Algorihm ), via riangle inequaliy r(x P q, u q ) implying ha r(x P q, u q ) r(x P q, u q ) + f Pq (x ) f Pq (x ), r(x P q, u q ) f Pq (x ) f Pq (x ) = r(x P q, u q ) k Pq > δ k Pq. (8) (9) The las inequaliy follows from he assumpion in (5) which saes ha none of he C for gives a ( + ɛ) approximaion of (). Since boh j and q are in C, by combining (7) and (9) we obain λ δ = δ max (r(x P j, u j ), r(x P q, u q )) ( ) max λ δ + (k Pj ) P, δ k q. Recall he definiion of k Pj and k Pq : () k Pj = f Pj (x ) f Pj (x ), () k Pq = f Pq (x ) f Pq (x ). () Geomerically, hese quaniies represen he D projecion, respecively on cameras j and q, of he 3D shif x x beween he curren and previous esimaes. A his juncure, he res of he proof diverges based on he following condiions: k Pj Condiion. k Pj k Pq or k Pj < k Pq. (3) k Pq. Under Condiion, and following from (), ( λ δ max λ δ + (k Pj ) P, δ k q ( max λ δ + (k Pj ), δ k P j ) ), () where he second inequaliy follows since k Pj and k Pq are boh non-negaive quaniies. Inerpreing he argumens in he second max λ δ + (k Pj ) and δ k Pj (5) as wo funcions of k Pj, observe ha he firs funcion increases wih k Pj whils he second decreases wih k Pj. Therefore, he RHS of () achieves is minimum when Solving (6) for k Pj λ δ + λ λ δ + (k Pj ) = δ k Pj. (6) and replacing i in (), we arrive a δ = λ λ. (7) The second inequaliy in (7) can invered as λ = ( λ )( + λ ) + λ + λ + > (8) λ +, (9) where he las sep is due o λ <. By recursively expanding he above from,,...,, and recalling ha λ, we obain which implies > + > + λ λ, (3) λ > +. (3) For = /ɛ + he las inequaliy reduces o λ /ɛ + > + (/ɛ + ) = + ɛ (3) which conradics (). Thus, (5) canno be rue. Whils i may be disconcering ha we have chosen an ieraion coun = /ɛ + ha does no exis in Algorihm, for he purpose of a heoreical argumen we can always arbirarily exend he algorihm by one ieraion. Since (5) is false, here mus be a /ɛ (say ) ha yields a ( + ɛ) approximaion. The se index by C s, which saisfies max r(x s P i, u i ) max r(x i X i X P i, u i ) ( + ɛ)δ, (33) is hus a corese. Assuming Condiion is always saisfied, herefore, he proof for Theorem is complee. Condiion. k Pj < k Pq. The above derivaions for Condiion unforunaely do no cover Condiion. In oher words, if Condiion occurs during he ieraions in Algorihm, we canno guaranee ha he oupu corese C s saisfies Theorem. Forunaely his deficiency can be recified by a small weak o Algorihm. Specifically, we replace Seps 3 o 5 in he main algorihm wih he slighly more elaborae seps in Algorihm. Noe ha index j in Algorihm is appropriaely chosen from C according o Lemma 5. Inuiively, he modificaion causes Algorihm o skip a sep whenever Condiion presens iself; namely, he mos

6 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 6 Algorihm Pseudo-code o replace Seps 3 o 5 in Algorihm o accommodae Condiion. : if k Pj k Pq hen : C C {q}. 3: (x, δ ) Minimiser and minimised value of () on daa indexed by C. : +. 5: else 6: C C {q}. 7: (x, δ ) Minimiser and minimised value of () on daa indexed by C. 8: end if violaing daum q is sill insered ino he corese, bu he ieraion couner is no incremened. Wih he modificaion above and by Propery, we have ha wo successive coreses C and C give δ δ, (3) since C C. Thus, all he derivaions saring from () will hold, and he oupu corese C s from Algorihm wih he above modificaion will always saisfy Theorem. 3.5 Error Backracking for Anyime Operaion By anyime mode, we mean he allowance o sop Algorihm premaurely (i.e., before any of he erminaing condiions are me) wih he abiliy o bound he approximaion error of he curren bes esimae x s w.r.. x. If Algorihm is run up o = /ɛ (which implies ha global convergence has no occurred before ha), hen he bound (6) holds. To faciliae analysis wih non-inegral, we can equivalenly sae ha if Algorihm is run unil a leas = /ɛ, hen (6) holds. Invering he relaionship beween and ɛ, we can resae he bound as max i X r(x s P i, u i ) ( + /)δ, (38) i.e., if Algorihm is run up unil an arbirary, we can expec a (/)-facor approximaion o () (noe ha x s in his case is he bes esimae up o ieraion ). Care mus be aken o spell ou running up unil an arbirary. By his, we mean running he main loop (Seps 5 o 6) in is enirely (including he modificaion summarised in Algorihm ) under ha paricular value, and hen conducing he pos-hoc refinemen (Seps 7 o ) such ha he esimae x from he las corese C has a chance o be used o updae he incumben x s. If his las updae is no aemped, hen (38) is no guaraneed o hold. 3. Size of Oupu Corese and Runime Analysis Wih he modificaion o accoun for Condiion, we mus anicipae ha in general C C, (35) i.e., here could be occurrences of Condiion beween any wo successive incremens o he ieraion couner. This also implies ha he size of he oupu corese C s is /ɛ V, (36) where V is he oal number of occurrences of Condiion hroughou Algorihm (recall ha C is iniialised already wih four of he available measuremens). To faciliae he analysis of he runime, define α as he probabiliy of Condiion occurring a a paricular ieraion of he main loop (hus, V = α /ɛ /( α)). Therefore, he number of imes he main loop is raversed is /ɛ /( α). (37) We argue ha he value of α is dependen mainly on he disribuion of he cameras and he srucure of he scene, raher han on he number of measuremens N iself (see evidence in Sec. ). Under his assumpion, he number of effecive ieraions of Algorihm, and hence he size of he oupu corese, depends only on ɛ. Of course, he acual runime of Algorihm depends closely on he rouine used o solve () a each ieraion. Many previous sudies have shown ha () can be solved efficienly [6], [9], [], more so since he solver need only be invoked on a small subse C a each ieraion in Algorihm. Sec. 5 will invesigae he acual runimes of Algorihm on real image daases for 3D reconsrucion. VALIDATION ON SYNTHETIC DATA Here we validae our heoreical resuls above on synheically generaed daa for riangulaion.. Daa Generaion We synhesised four ypes of camera pose disribuions: Type A: Camera posiions were on a sraigh line. Type B: Camera posiions were randomly disribued. Type C: Camera posiions were on a circle. Type D: Sereo cameras wih fixed baselines and posiions randomly disribued. For Types B and D, he camera orienaions were randomly generaed. For Types A and C, angular noise was added o he orienaion/roaion marices. In all cases, he cameras could observe he 3D poin o respec cheiraliy. See Fig. for sample daa insances generaed. Type A simulaes a roboic exploraion scenario where he robo views a scene from a direced rajecory [], Type B simulaes large-scale 3D reconsrucion from crowd-sourced images [], Type C simulaes he usage of a roaing plaform for 3D modelling [], and Type D simulaes large-scale 3D reconsrucion using sereo cameras [5]. For each camera disribuion, N image measuremen/camera marix pairs {P i, u i } N i= are generaed by projecing he 3D poin ono each camera and corruping he projeced poin wih Gaussian noise σ = pixels.. Validaion of Approximaion Accuracy To experimenally validae Theorem, one insance of each of he camera disribuion ype in Fig. wih N = views were generaed. For each camera disribuion, 3D scene poins were creaed (in a way ha he 3D poins are observable in all N cameras) and projeced ono he N cameras. This creaed a oal of riangulaion insances ().

7 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 7 (a) Type A (b) Type B (c) Type C (d) Type D Fig.. The four ypes of synheically generaed riangulaion insances. Type A: Camera posiions are on a sraigh line; Type B: Camera posiions are randomly disribued; Type C: Camera posiions are on a circle; Type D: Sereo cameras wih fixed baselines and posiions randomly disribued. In all cases, he cameras are roughly oriened owards he 3D poin o respec cheiraliy. On each riangulaion insance, we execued Algorihm. The approximaion error raio max i X r(x s P i, u i ) δ (39) achieved by he curren esimae x s a each is ploed; hese are shown as red curves in Fig. 5 (one curve for each riangulaion insance). Noe ha he horizonal axis begins a =, since he bound (38) is no guaraneed o hold for he iniial corese C. Noe also ha due o he allowance of skipping by insering Algorihm ino Algorihm, each can involve several updaes o x s ; in Fig. 5 we ploed he error raio peraining o final x s in each. raio bound () o accoun for Condiion is also shown; specifically, as he corese is increased from size 8 o 9, Condiion occured and was no incremened. Hence he bound () does no decrease beween hese wo seps. Fig. 6(b) plos he approximaion error raios for all he synheic riangulaion insances from Fig. 5 agains corese size. In his figure, since Condiion occurred a differen ieraions for he respecive problem insances, he bounding curve is no ploed. Observe ha all he problem insances converged o a corese size ha is no very much larger han he value of a he ime of convergence, cf. Fig. 5. Approximaion error raio Couner Approximaion error raio Corese size (a) Skipping Fig. 5. Approximaion error raios (39) across couner (red curves) for synheically generaed riangulaion insances. The blue curve is he upper bound on he error raio ( + /) as prediced by Theorem. By he backracking formula (38), he approximaion error raios should lie below he curve + /; () his curve is ploed in blue in Fig. 5. As prediced, he bound is respeced across all. In Sec. 5, we will furher validae Theorem using real image daa. Approximaion error raio Corese size (b).. Illusraing Effec of Condiion The effec of Condiion on Algorihm is demonsraed in Fig. 6. On one of he synheic riangulaion insances, Fig. 6(a) plos he approximaion error raio (39) agains he corese size, as he corese is being accumulaed in Algorihm (he horizonal axis hus begins a 5 since he iniial corese C has size ). The effecs of skipping on he Fig. 6. (a) Approximaion error raio (39) ploed agains corese size for one of he synheic daa insances. In his insance, Condiion occurred as he corese was increased from size 8 o 9, hus he bound () does no decrease beween hese wo ieraions. (b) Approximaion error raio ploed agains corese size for all he synheic daa insances. Since Condiion occurred a differen ieraions for he respecive problem insances, he bounding curve is no ploed.

8 , JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST Probabiliy of Condiion We demonsrae ha he probabiliy α of he occurrence of Condiion in any problem insance is mainly affeced by he way he camera poses are disribued, and is no a facor of problem size N. For a complee execuion of Algorihm on he following paricular problem insances, we obain α empirically as he raio of he number of occurrence of Condiion over he effecive number of ieraions in he algorihm. The experimenal seings were as follows: for each ype of camera disribuion, 3D poins were randomly generaed and projeced ono N views, where N was varied from o,. On each problem insance, Algorihm was execued imes (wih random iniialisaions) and he α values were recorded. Fig. 7 shows he average α over all insances as a funcion of N. Evidenly α is almos consan across N, and he bigges facor in he difference in α is he ype of camera pose disribuion (he curves of Types B and D are similar since hey are essenially randomly disribued camera poses). This suppors he analysis in Sec. 3. ha he oal runime of Algorihm, and hence he size of he oupu corese, is mainly dependen on he desired approximaion facor ɛ. The nex secion will furher invesigae he size of he corese oupu by Algorihm. 5 EXPERIMENTS ON REAL DATA We conduced experimens on real daa o validae Theorem and invesigae he performance of Algorihm for l riangulaion. We used a sandard machine wih 3. GHz processor and 6 GB main memory. 5. Daases and Iniialisaion We esed on publicly available daases for large scale 3D reconsrucion, namely, Vercingeorix Saue, Sockholm Ciy Hall, Arc of Triumph, Alcaraz, Örebro Casle [6], [7], and Nore Dame []. The a priori esimaed camera poses and inrinsics supplied wih hese daases were used o derive camera marices. For riangulaion, he size of an insance is he number of observaions of he arge 3D poin. To avoid excessive runimes, we randomly sampled % of he scene poins in each daase - his reduces he number of problem insances, bu no he size of each of he seleced insances. A hisogram of he problem sizes for each of he above daases are shown in he op lef panel of Figs. o 5. Our corese mehod was iniialised as shown in he firs few seps in Algorihm, which amouns o randomly choosing four daa o insaniae x by solving (). For any oher algorihm ha requires iniialisaion, he same x or is curren maximum reprojecion error were provided as he iniial esimaes Type A Type B Type C Type D. 6 8 Number of views - N Fig. 7. Average α (probabiliy of occurrence of Condiion ) as he problem size N increases, separaed according o he ype of camera pose disribuion (see Fig. )..3 Size of Oupu Corese To invesigae he size of he oupu corese produced by Algorihm as a funcion of he approximaion error ɛ, we generaed riangulaion insances for he four ypes of camera disribuion, wih 3D poins in each insance bu wih varying problem size (number of views) N {, 5,, 5}. On each insance, he seing of ɛ for Algorihm was varied decreasingly from and he size of he oupu corese was recorded. Fig. 8 plos he maximum corese size as a funcion of ɛ, averaged across all insances, bu separaed according o ype of camera pose disribuions and problem size N. The resuls show ha he corese size depends mainly on ɛ and is independen of he problem size N. Furher, for all he daa seings/parameers, he corese size did no exceed. For some of he ypes of disribuion, he corese size also converged earlier due o earlier global convergence. 5. Validaion of Approximaion Accuracy The op righ panel in Figs. o 5 show he acual raio of errors versus he prediced raio (using he backracking formula in Sec. 3.5) across he ieraions of Algorihm for all problem insances in he daases. Again, he resuls confirm he validiy of Theorem. 5.3 Relaive Speed-up of Corese over Bach Here we invesigae he pracicaliy of Algorihm as a global opimiser for l riangulaion. As described in Secion 3, Algorihm is a mea-algorihm which requires a sub-rouine o solve () on he subse indexed by C. We hus compared running Algorihm wih a specific solver as a sub-rouine, and he direc execuion of he same solver in bach mode on he whole daa. Since he runime of Algorihm depends on he efficiency of he embedded solver, he key performance indicaor here is he relaive speed-up achieved by corese over bach. Based on he invesigaions in [], we have chosen o use bisecion [6] and Dinkelbach s mehod [] (equivalen o [9]) o embed ino Algorihm. Alhough he bes performing echnique in [] was Guga s algorihm [5], our experimens suggesed ha i did no ouperform Dinkelbach s mehod on he riangulaion problem. SeDuMi [8] was used o solve he SOCP sub-problems in [6], []. The boom diagrams of Figs. o 5 show he average runime of corese and bach as a funcion of problem size (number of views), for each respecive l solver. Observe ha he runime of bach increased linearly and hen exponenially, whils corese exhibied almos consan runime he laer observaion is no surprising, since Algorihm usually erminaed a ieraions regardless of he problem size a shown in he op righ panel.

9 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 9 Corese size Type A Type B Type C Type D Corese size Type A Type B Type C Type D (a) N= (b) N=5 Corese size Type A Type B Type C Type D Corese size Type A Type B Type C Type D (c) N= (d) N=5 Fig. 8. Average size of corese ploed agains decreasing ɛ, separaed according o he four ypes of camera disribuion as in Fig.. Panel (a),(b),(c), and (d) are he resuls respecively for N =, 5,, and 5. Of course, on all of he daases, mos of he riangulaion insances are small, as shown in he hisogram a he upper lef panel of Figs. o 5. However, in hese daases, here are sufficien numbers of moderae o large problem insances, such ha he oal runime of corese is sill much smaller han hen oal runime of bach. Table shows he oal and average runime of he varians considered. Evidenly, corese ouperformed bach in all he daases. Alhough in principle any l solver for l norm reprojecion error [6], [7], [9], [], [] can be used in Algorihm, we emphasise again ha he primary performance indicaor here is he relaive speed-up of corese over bach. If a faser solver is used, i would likely improve boh corese and bach by he same facor. 5. Exensions o l and l Reprojecion Error In he lieraure, apar from he more radiional l norm used in he reprojecion error (), differen p-norms have been considered [], i.e., r(x P i, u i ) = u i P: i x P 3 i x, p {,, }. () p Our corese heory was developed based on p =, hus he approximaion bound (Theorem ) will no hold for oher p. I is noneheless feasible o apply Algorihm as a meaalgorihm for solving l riangulaion () under differen reprojecion errors. Since problem () remains a GLP (Sec. ) for p = and p = in (), global convergence is guaraneed. I is hus of ineres o compare he relaive speed-up given by he corese mehod over a bach mehod in finding he globally opimal soluion. Table shows he resuls of repeaing he experimen in Sec. 5.3 wih p = and p =. For p =, he Dinkelbach mehod was used as he embedded solver for Algorihm. For p =, he sae-of-he-ar polyhedron collapsed mehod [3] was used as he embedded solver. Evidenly he resuls show ha he corese mehod is able o significanly speed up global convergence. 6 DEALING WITH OUTLIERS While he exisence of ouliers in he measuremens (e.g., from incorrec feaure associaions) is ransparen o Algorihm (i.e., he error bound in Theorem will sill hold), he resul will of course be biased by he ouliers afer all, he l framework is no inherenly robus [9]. Noneheless, Sim and Harley [] showed how an effecive oulier removal scheme can be consruced based on l esimaion. Basically, heir scheme recursively conducs l esimaion and removes he suppor se (see Propery ) from he inpu daa unil he maximum residual is below a pre-deermined inlier hreshold. The remaining daa hen forms an inlier se. Here, in he conex of riangulaion wih ouliers, we showed how he efficiency of Sim and Harley s scheme can be improved by using Algorihm (wih a high ɛ) as a fas l solver. Due o he approximaion by Algorihm, insead of removing he suppor se, we remove he measuremens wih he larges residuals. Fig. 9 compares he runime and number of remaining inliers produced by Sim and Harley s original scheme (wih Dinkelbach s mehod as he l solver) and our coreseenabled scheme (wih ɛ =.). The resuls are based on 3D scene poins projeced ono N views ( N 5), and where 9% of he D measuremens in each problem insance were corruped wih Gaussian noise of σ = 5

10 Runime ( s) JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 Daase Scene poins (number of riang. insances) Number of views (max. riang. size) Toal runime (seconds) Bisecion solver Dinkelbach solver Bach Corese Bach Corese Vercingeorix (3%) 3 7 (6%) Sockholm (3%) 9 7 (3%) Arc of Triumph (59%) 89 (56%) Alcaraz (57%) 5 39 (7%) Örebro Casle (8%) 35 (75%) Nore Dame (83%) (75%) TABLE Comparisons beween corese and bach in erms of oal runime in seconds. For corese, he number in parenheses indicaes he percenage of reducion in runime by using he corese mehod over he bach counerpar. Daase Scene poins (number of riang. insances) Number of views (max. riang. size) Toal runime (seconds) Dinkelbach (p = ) Polyhedron (p = ) Bach Corese Bach Corese Vercingeorix (3%).8.83 (-%) Sockholm (9%)..7 (% ) Arc of Triumph (9%) 3.5. (%) Alcaraz (73%) (8%) Örebro Casle (9%) (9%) Nore Dame (87%) (96%) TABLE Comparisons beween corese and bach in erms of oal runime in seconds, under he l and l reprojecion error (). For l reprojecion error, he Dinkelbach mehod was used as he embedded solver in Algorihm. For l reprojecion error, he sae-of-ar polyhedron collapse mehod [3] ( Polyhedron above) was used as he embedded solver in Algorihm. For corese, he number in parenheses indicaes he percenage of reducion in runime by using he corese mehod over he bach counerpar. pixels (he inliers), while he remaining % were corruped wih larger noise (σ = 3 pixels) o creae ouliers. The inlier hreshold was se o pixels. Evidenly our corese modificaion significanly improved he efficiency of he original scheme, wihou significanly affecing he qualiy of he resul (number of remaining inliers). 5 3 Approx. ` riang. solver Exac ` riang. solver 7 CONCLUSIONS AND OPEN QUESTIONS In his paper, we show ha l riangulaion admis corese approximaion. We also provided comprehensive experimenal resuls ha esablish he pracical value of he corese algorihm on large scale 3D reconsrucion daases. There are several open quesions: A deeper analysis of Condiion o hopefully remove i from consideraion in he corese bound, or a leas o beer characerise and predic is occurrence. The proof in Sec. 3.3 was inspired by he work of [7] on minimum enclosing ball (MEB) problems. There, he corese size bound /ɛ was proven o be igh if he dimensionaliy d is comparable o /ɛ. For lower dimensional MEBs, igher bounds have been proposed. I would be of ineres o consruc such igher bounds for l riangulaion (d = 3). Las bu no leas, we hope ha our work encourages more effor o seek heoreically jusifiable approximae algorihms for large scale geomeric compuer vision problems, especially he class of problems surveyed in [6], which can be seen as GLPs [9]. Number of remaining inliers 3 5 Number of views - N 5 3 (a) Approx. ` riang. solver Exac ` riang. solver Inliers in original inpu 3 5 Number of views - N (b) Fig. 9. Comparing Sim and Harley s oulier removal scheme [] wih an exac solver (Dinkelbach s mehod) and wih Algorihm. (a) Average runime; (b) Number of inliers remaining in he final inlier se (he dashed green line o indicae he number of inliers in original inpu).

11 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 5 Hisogram of problem sizes Theoreical bound vs acual error Number of scene poins 3 Approximaion error raio Couner.3.5 Runime of bach vs corese (bisecion) Corese-Bisecion Bach-Bisecion.. Runime of bach vs corese (dinkelbach) Corese-Dinkelbach Bach-Dinkelbach Fig.. Resuls for Vercingeorix Saue. (op lef) Hisogram of problem sizes. (op righ) Approximaion error raio versus error raio bound. (boom lef) Runime of corese vs bach, for bisecion solver. (boom righ) Runime of corese vs bach, for Dinkelbach solver. 5 Hisogram of problem sizes Number of scene poins Runime of bach vs corese (bisecion) Corese-Bisecion Bach-Bisecion Runime of bach vs corese (dinkelbach) Corese-Dinkelbach Bach-Dinkelbach Fig.. Resuls for Sockholm Ciy Hall. (op lef) Hisogram of problem sizes. (op righ) Approximaion error raio versus error raio bound. (boom lef) Runime of corese vs bach, for bisecion solver. (boom righ) Runime of corese vs bach, for Dinkelbach solver.

12 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 7 Hisogram of problem sizes Number of scene poins Runime of bach vs corese (bisecion) Corese-Bisecion Bach-Bisecion. Runime of bach vs corese (dinkelbach) Corese-Dinkelbach Bach-Dinkelbach Fig.. Resuls for Alcaraz. (op lef) Hisogram of problem sizes. (op righ) Approximaion error raio versus error raio bound. (boom lef) Runime of corese vs bach, for bisecion solver. (boom righ) Runime of corese vs bach, for Dinkelbach solver. 5 Hisogram of problem sizes Number of scene poins Runime of bach vs corese (bisecion) Corese-Bisecion Bach-Bisecion Runime of bach vs corese (dinkelbach) Corese-Dinkelbach Bach-Dinkelbach Fig. 3. Resuls for Arc of Triumph. (op lef) Hisogram of problem sizes. (op righ) Approximaion error raio versus error raio bound. (boom lef) Runime of corese vs bach, for bisecion solver. (boom righ) Runime of corese vs bach, for Dinkelbach solver.

13 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST Hisogram of problem sizes Number of scene poins Runime of bach vs corese (bisecion) Corese-Bisecion Bach-Bisecion.5.5 Runime of bach vs corese (dinkelbach) Corese-Dinkelbach Bach-Dinkelbach Fig.. Resuls for Örebro Casle. (op lef) Hisogram of problem sizes. (op righ) Approximaion error raio versus error raio bound. (boom lef) Runime of corese vs bach, for bisecion solver. (boom righ) Runime of corese vs bach, for Dinkelbach solver. Hisogram of problem sizes Number of scene poins Runime of bach vs corese (bisecion) Corese-Bisecion Bach-Bisecion 8 6 Runime of bach vs corese (dinkelbach) Corese-Dinkelbach Bach-Dinkelbach Fig. 5. Resuls for Nore Dame. (op lef) Hisogram of problem sizes. (op righ) Approximaion error raio versus error raio bound. (boom lef) Runime of corese vs bach, for bisecion solver. (boom righ) Runime of corese vs bach, for Dinkelbach solver.

14 JOURNAL OF L A T E X CLASS FILES, VOL., NO. 8, AUGUST 5 REFERENCES [] R. I. Harley and P. Surm, Triangulaion, Compuer vision and image undersanding, vol. 68, no., pp. 6 57, 997. [] N. Snavely, S. M. Seiz, and R. Szeliski, Modeling he world from inerne phoo collecions, Inernaional Journal of Compuer Vision, vol. 8, no., pp. 89, 8. [3] Y. Furukawa and J. Ponce, Accurae, dense, and robus muliview sereopsis, IEEE ransacions on paern analysis and machine inelligence, vol. 3, no. 8, pp ,. [] R. Mur-Aral and J. D. Tardós, Probabilisic semi-dense mapping from highly accurae feaure-based monocular slam, Proceedings of Roboics: Science and Sysems, Rome, Ialy, vol., 5. [5] R. Harley and F. Schaffalizky, L minimizaion in geomeric reconsrucion problems, in Compuer Vision and Paern Recogniion,. CVPR. Proceedings of he IEEE Compuer Sociey Conference on, vol.. IEEE,, pp. I 5. [6] F. Kahl, Muliple view geomery and he L -norm, in Tenh IEEE Inernaional Conference on Compuer Vision (ICCV 5) Volume, vol.. IEEE, 5, pp. 9. [7] Q. Ke and T. Kanade, Quasiconvex opimizaion for robus geomeric reconsrucion, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 9, no., pp , 7. [8] Y. Seo and R. Harley, A fas mehod o minimize L error norm for geomeric vision problems, in Compuer Vision, 7. ICCV 7. IEEE h Inernaional Conference on. IEEE, 7, pp. 8. [9] C. Olsson, A. P. Eriksson, and F. Kahl, Efficien opimizaion for L -problems using pseudoconvexiy, in 7 IEEE h Inernaional Conference on Compuer Vision. IEEE, 7, pp. 8. [] S. Agarwal, N. Snavely, and S. M. Seiz, Fas algorihms for L problems in muliview geomery, in Compuer Vision and Paern Recogniion, 8. CVPR 8. IEEE Conference on. IEEE, 8, pp. 8. [] Z. Dai, Y. Wu, F. Zhang, and H. Wang, A novel fas mehod for L problems in muliview geomery, in European Conference on Compuer Vision. Springer,, pp [] A. Eriksson and M. Isaksson, Pseudoconvex proximal spliing for l-infiniy problems in muliview geomery, in IEEE Conference on Compuer Vision and Paern Recogniion. IEEE,, pp [3] S. Donné, B. Goossens, and W. Philips, Poin riangulaion hrough polyhedron collapse using he L norm, in Proceedings of he IEEE Inernaional Conference on Compuer Vision, 5, pp [] W. Dinkelbach, On nonlinear fracional programming, Managemen Science, vol. 3, no. 7, pp. 9 98, 967. [5] M. Guga, A fas algorihm for a class of generalized fracional programs, Managemen Science, vol., no., pp , 996. [6] P. K. Agarwal, S. Har-Peled, and K. R. Varadarajan, Geomeric approximaion via coreses, Combinaorial and compuaional geomery, vol. 5, pp. 3, 5. [7] M. Bădoiu and K. L. Clarkson, Opimal core-ses for balls, Compuaional Geomery, vol., no., pp., 8. [8] H. Li, Efficien reducion of l-infiniy geomery problems, in Compuer Vision and Paern Recogniion, 9. CVPR 9. IEEE Conference on. IEEE, 9, pp [9] N. Amena, Helly-ype heorems and generalized linear programming, Discree & Compuaional Geomery, vol., no. 3, pp. 6, 99. [] K. Sim and R. Harley, Removing ouliers using he l\ infy norm, in Compuer Vision and Paern Recogniion, 6 IEEE Compuer Sociey Conference on, vol.. IEEE, 6, pp [] R. Seidel, Small-dimensional linear programming and convex hulls made easy, Discree & Compuaional Geomery, vol. 6, no. 3, pp. 3 3, 99. [] K. L. Clarkson, Las vegas algorihms for linear and ineger programming when he dimension is small, Journal of he ACM (JACM), vol., no., pp , 995. [3] J. Maoušek, M. Sharir, and E. Welzl, A subexponenial bound for linear programming, Algorihmica, vol. 6, no. -5, pp , 996. [] D. P. Sysems, 3d scanners, hp://3dpriningsysems.com/ producs/3d-scanners/. [5] P. F. Alcanarilla, C. Beall, and F. Dellaer, Large-scale dense 3d reconsrucion from sereo imagery. Georgia Insiue of Technology, 3. [6] O. Enqvis, C. Olsson, and F. Kahl, Sable srucure from moion using roaional consisency, Cieseer, Tech. Rep.,. [7] C. Olsson and O. Enqvis, Sable srucure from moion for unordered image collecions, in Scandinavian Conference on Image Analysis. Springer,, pp [8] J. F. Surm, Using sedumi., a malab oolbox for opimizaion over symmeric cones, Opimizaion mehods and sofware, vol., no. -, pp , 999. [9] F. Kahl and R. Harley, Muliple-view geomery under he L - norm, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 3, no. 9, pp , 8. Qianggong Zhang received he BEng degree in compuer science and echonology in and he MEng degree in compuer science and echonology in 7. Since 5, he has been a PhD candidae a The Universiy of Adelaide, Souh Ausralia. His primary research areas include approximaion algorihms for geomeric compuer vision problems. Ta-Jun Chin received he BEng degree in mecharonics engineering from Universii Teknologi Malaysia (UTM) in 3 and he PhD degree in compuer sysems engineering from Monash Universiy, Vicoria, Ausralia, in 7. He was a research fellow a he Insiue for Infocomm Research (IR) in Singapore from 7 o 8. Since 8, he has been a The Universiy of Adelaide, Souh Ausralia, and is now an Associae Professor. He is an Associae Edior of IPSJ Transacions on Compuer Vision and Applicaions (CVA). His research ineress include robus esimaion and geomeric opimisaion.