4 x 10 3 Residual during bilinear iteration error residual iteration

Transcription

1 Optmal Structure From Moton: Local Ambgutes and Global Estmates Stefano Soatto y and Roger Brockett z y Washngton Unversty, One Brookngs dr., St. Lous - MO 6, soatto@ee.wustl.edu and Dpartmento d Matematca ed Informatca, Unversta d Udne, Italy z Harvard Unversty, 9 Oxford st., Cambrdge MA 8, brockett@hrl.harvard.edu Abstract We present an analyss of SFM from the pont of vew of nose. Ths analyss results n an algorthm that s provably convergent and provably optmal wth respect to a chosen norm. In partcular, we cast SFM as a nonlnear optmzaton problem and dene a blnear projecton teraton that converges to xed ponts of a certan cost-functon. We then show that such xed ponts are \fundamental",.e. ntrnsc to the problem of SFM and not an artfact ntroduced by our algorthm. We classfy and characterze geometrcally local extrema, and we argue that they correspond to phenomena observed n vsual psychophyscs. Fnally, we show under what condtons t s possble - gven convergence toalocal extremum - to \jump" to the valley contanng the optmum; ths leads us to suggest a representaton of the scene whch s nvarant wth respect to such local extrema. Introducton The problem of \Structure From Moton" (SFM) deals wth extractng three-dmensonal nformaton about the envronment from the moton of ts projecton onto a twodmensonal surface. We restrct our attenton to a pontwse representaton of the world. Despte beng a rudmentary model, t allows us to touch upon some mportant ssues n SFM that have been addressed only margnally n the past. After years of work n SFM, we can safely say that the geometry of the problem s farly well understood (for the case of feature-ponts), and ncely summarzed n the forthcomng book of Faugeras and Luong [5]. At the same tme, the performance of many of the algorthms has been demonstrated on (more or less controlled) sequences of real mages, and ths has led many n the Computer Vson communty to conclude that SFM has been solved. On the other hand, we are wtnessng the frustraton of many others, especally engneers, who are mplementng exstng algorthms for SFM n real-world stuatons, and observe that they behave naway that s much derent from ther declared performance. Ths has led some to conclude that SFM s too dcult a problem to be solved n full generalty even n the case of pont features [9]. We beleve that such a dversty of feelngs comes from the fact that, whle the geometry has been studed extensvely, the ssue of nose has been touched upon only n a supercal way (wth some notable exceptons upon whch we wll comment later). Relaton to prevous work Ths paper relates to many prevous works n SFM, and some of the relatonshps are ponted out throughout the paper. In partcular, Weng et al. [8] test varous generalpurpose nonlnear optmzaton technques on cost functons that nclude the eppolar constrant as well as the average -norm of the mage-measurements. Cramer-Rao bounds are evaluated, and an extensve set of smulaton experments compares exstng lnear algorthms aganst the optmal. In [8], teratve optmzaton schemes are ntalzed usng a lnear algorthm (such asavaraton of the 8-pont algorthm), and therefore the process can be vewed as the optmal renement of exstng SFM algorthms. Under condtons n whch the lnear algorthms do not gve a satsfactory answer, Weng et al. only guarantee convergence to a local mnmum, wth no ndcaton as to how ths s related to the optmum. Recently, Szelsk and Kang have also addressed the ssue of optmal estmaton of SFM and characterzaton of ambgutes [4, 5]. They use the Levemberg-Marquardt algorthm to nd the extrema of the sum of reprojecton errors. They perform an analyss of the bas-relef ambguty based on the sngulartes of the Hessan of the nformaton matrx. However, ther results are essentally local, snce they rely on a lnearzed model and assume small measurement errors. The nherent ambgutes n SFM have been studed before also by Adv [], Young and Chellappa [] (who also address the aperture problem), Olenss [9] (who gves a provably convergent algorthm ), and Spetsaks and Alomonos [] (who also propose an optmal algorthm). Our paper extends the above results: When other optmal algorthms converge to ther global mnmum, the soluton s dentcal to the one we obtan. However, we acheve useful results for noses that are one order of magntude hgher than what commonly handled n the lterature, wthout mposng restrctons on the ntal condtons. We also classfy and nterpret geometrcally local extrema, and we observe that they correspond to phenomena observed n vsual psychophyscs. We show how t s possble, under certan condtons, to obtan the global soluton to SFM gven convergence to a local extremum. Sphercal Least-Squares Suppose we are gven p unt-norm vectors x ;:::;x p S and an unknown transformaton a IR that acts on each x ; = :::p va the cross-product a x T x S

2 (the tangent plane to the unt-sphere S at x ). Suppose further that we can measure each transformed vector up to an unknown scalng factor IR + : y = ax +n = :::p where each n represents the uncertanty (or error) n the measurement y, and hence n T x S. Now, gven anumber p of vectors x ;:::p and the correspondng measurements y,wewant to nd the transformaton a IR and the scales =[ ;:::; p ] T IR p + that mnmze the norm of the uncertanty n: mn kn k air w subject to y = a x +n Tx ;IR p S + () where w ndcate the weghts chosen for the components of the cost functon. In the next three subsectons we wll analyze three dstnct cases. Obvously, n the absence of nose (n = ) the three problems of mnmzng the cost functon above under the derent choces of weghts have the same exact soluton. In ths and the followng sectons we use the \hat" operator to ndcate a skew-symmetrc matrx bx so() that performs the cross product between two vectors x and v: bxv = x v. Snce n all equatons the parameters a and appear as a product, t s clear that they can only be recovered up to a common scale. Therefore, we choose to normalze a, so that kak =, although any other choce for the normalzaton would do. Usng ths notaton, we can formulate the \Sphercal Least-Squares Problem" (SLS) as a opt; opt = arg mn as ;IR p + ky + bx a k w : () We refer to the sum above as the cost functon of SLS.. Weghted Sphercal Least Squares In ths secton we consder w (a) : = ka x k[; ]. Clam. Gven p> ponts x S ; =:::p and the correspondng measurements y, under general poston condtons the SLS problem () wth w = w (a) admts a unque soluton a opt and opt up to a P sgn. If we dene M to be the symmetrc matrx M = p yyt, then the optmal unt-norm soluton a opt s the egenvector of M correspondng to ts smallest egenvalue: a opt = v mn(m) and the optmal scales opt are obtaned from a opt va opt =, yt bx a opt a T optbx aopt =:::p: () Proof: Consder the cost functon of SLS, dened nequaton (). For any gven a, the that mnmzes t s gven, as a functon of a, by (a) =,(bx a) y y =, at bx y =:::p: (4) a T bx a Once we substtute (a) back nto (), the components of the cost functon become k(bx a)? y k. We now use the fact that x? =,bx to replace (bx a)? y wth (bx a) kbx ak y, endng up P wth mnmzng for a the cost functon k(bx a)y k w : Snce kbx ak T (bx a) = ax, xa T, and y s orthogonal to x,wecan further smplfy the SLS problem, that becomes arg mn as kx a T y k = a T y y T! a: (5) It s mmedate to see that the least-squares unt-norm soluton for a s gven by the egenvector of M correspondng to the smallest egenvalue. Note that, snce M s symmetrc, the egenvalues are real and postve, and the egenvectors are unt-norm orthogonal vectors. Once the optmal a has been computed, the correspondng optmal can be obtaned as (a) from equaton (4). Note that there s a sgn ambguty n a, that reects onto the sgn of the vector.. Balanced Sphercal Least Squares Let us assume for a moment that the weghts w are all dentcal to. We can stll follow the procedure outlned n the proof of clam., but rather than endng up wth mnmzng the cost functon P p have P p kxat y k kx ak kx a T y k n (5), we, whch can be re-wrtten as < y ; a > kx : (6) ak If we assume that x span a small sold angle (compared to the full sphere), then t makes sense to talk about an average drecton x. If we weght each pont wth w = kx ak, the cost functon to be mnmzed becomes at Ma kxak a T Na, where M : = P p y y T and N : = b x. The soluton for ths problem has to do wth Sngular Raylegh quotents and s derved n []. We summarze the result n the followng Clam. The soluton a opt for the problem of mnmzng the cost functon (6) s obtaned by mnmzng the correspondng Sngular Raylegh Quotent at Ma a T Na. The soluton : s gven by the egenvector of the matrx M s = M M xx T M, x T M x relatve to the matrx N correspondng to the smallest nonzero egenvalue.. Unweghted Sphercal Least Squares When we choose all weghts w to be dentcally equal to one, the problem of Sphercal Least Squares can be reduced, followng the proof of clam., to mnmzng (6) subject to the constrants < y ; x >= and kx k =. In the presence of nose, we have not found a closed-form optmal soluton for the Unweghted Sphercal Least-Squares problem. However, the followng smple teraton can be easly proven to be contractve and therefore to converge to a xed pont:, ak+ =pr S I, H, D T ak (7) where D = P yy T,aa T bx +a T yy T abx,aa T yy T (a T bxa) P yy T (a T bx a),bx (a T yy T a) (a T bx a), H s, 4,yyT aa T bx,bx aa T yy T aa T bx (a T bxa) and pr S denotes renormalzaton (projecton onto the sphere). Of course ths teraton s only guaranteed to converge to a local soluton of the Unweghted SLS. However, we have notced that usng a Weghted SLS or a Balanced SLS as an ntalzaton step usually places the teraton n the basn of attracton of the global mnmum. In partcular, for small levels of nose (up to 5% of the average sgnal), we have observed that both the Weghted SLS and,

3 the Balanced SLS approxmate well the soluton of the unweghted SLS, and therefore runnng an teraton of the type above mproves only margnally the soluton. For hgher nose levels we have observed that when the devaton of the x s large, the soluton to the Weghted SLS provdes an accurate ntalzaton, whle when devaton of the x s small (on the order of 4 o or less), the soluton of a Balanced SLS s more accurate. In both cases, however, the teraton converges n a few steps (less than ). Structure From Moton Consder a varaton of the Sphercal Least-Squares problem of equaton (), where we add to the cost functon the ane term bx b, wth b IR unknown: mn as ;bir IR p r (a; b;) where we dene + r (a; b;)= ky + bx a, bx bk (8) as the cost functon of SFM, and we neglect the subscrpts w that ndcate the choce of weghts. Ths problem can be nterpreted as that of estmatng the drecton of translaton a S, the rotatonal velocty b IR and the depth = = :::p of a number p of movng ponts n -D, from the nosy projecton of ther velocty onto the retna, modeled as a unt-sphere [7]. The estmaton crteron s to mnmze the (weghted) norm of the \reprojecton error". In the specal case b =(no rotaton), (8) reduces to a SLS problem. Dependng upon the weghts chosen, we have shown n secton how to obtan a (local) soluton for the drecton of translaton a and the scalng parameters (nverse depths). In the presence of rotaton b 6=, the problem dened by equaton (8) s no longer a standard SLS problem. Many n the Computer Vson lterature have proposed methods to \undo" the rotaton, by ether warpng the mage [,, ], by applyng a lnear transformaton to the measured data [6, 7], or by estmatng a rotatonal velocty assumng small translaton [9]. In all cases, however, the transformaton acts on the nose as well as on the data, therefore spolng the goal of achevng an optmal estmate. Here we take a derent approach, that s somewhat more aware of nose and results n an optmal (although non-lnear) estmator. Followng the dervaton of the soluton of SLS n clam., we can transform the problem of SFM nto arg mn as ;bir kx a T (y, bx b)k (9) Now, for any gven b, the a that mnmzes the norm of the cost functon of SFM n (8) can be obtaned by solvng the SLS problem () wth y beng substtuted by ~y = y,bx b: In the same fashon { gven a { the b that mnmzes the norm of the n s obtaned mmedately from (9) as b(a) = bx aa T bx!, bx aa T y : () Therefore, the \condtonal" problems of estmatng a gven b { or b gven a { are partcularly smple. Based on the smplcty of the condtonal problems, one may be tempted to try the followng Blnear Projecton Algorthm (BPA): let k = and choose any ntal value for b IR terate the followng: { a k = arg mn as P p kx a T (y, bx b k)k { b k+ =,P p bx a ka T k bx, P p bx a ka T k y : { k = k + We are mplctly excludng the case a =,for that case can be easly detected and treated separately (see []). It s straghtforward to prove that ths teraton converges \somewhere". It s less obvous to make sure that the teraton converges to some meanngful local extrema. Luckly we have the followng Clam. Gven p>5 ponts n general poston, the Blnear Projecton Algorthm converges to a local extremum of the blnear cost functon (BCF) r(a; b) = : P p kxat (y,bx b)k : Extrema of the blnear cost functon r are n one-to-one correspondence wth those of the cost functon of SFM n (8). In order to prove the clam we need to establsh that the blnear teraton does not ntroduce \phantom" statonary ponts to the cost functon. Ths s guaranteed by the followng: Lemma. Let (a) be the p-dmensonal vector wth -th component x a T y, and the p matrx wth -th row equal to x a T bx. Then r(a; b) = k (a), (a)bk; dene r : (a) = k(a)? (a)k. Furthermore, assume that has constant rank =n some open subset of IR p. If a s a crtcal pont (or a global mnmzer) of r, and b : = y (a ) (a ), then (a ; b ) s a crtcal pont (or a global mnmzer) of r and r (a )=r(a ; b ). If (a ; b ) s a global mnmzer of r, then a s a global mnmzer of r and r (a )=r(a ; b ). y denotes the (least-squares) pseudo-nverse. Proof: Ths s aaspecal case of theorem. on page 46 of [6], wth the smple extenson of allowng the ane term to depend on a. Proof of clam.: The Blnear Projecton Algorthm s a Gauss-Newton teraton for the cost functon r(a; b). The lemma guarantees that the teraton performed by alternatng the varables a and b has the same xed ponts of the teraton performed smultaneously on a and b. The rst part of the clam follows from standard propertes of Gauss-Newton teratons. That such extrema correspond to those of the orgnal cost functon n (8) follows by applyng the lemma agan to the SLS problem. The above clam guarantees that, followng the BPA outlned n ths paragraph, we do not ntroduce spurous solutons to the problem of SFM. Therefore, t remans to be establshed whether the orgnal problem of SFM, as formulated n (8), admts spurous local solutons n the presence of nose n. Ths s the subject of the next secton. Here the notaton? stands for the projector operator onto the orthogonal complement of the range space of, dened as? = I, y, where y denotes the pseudo-nverse.

4 4 \Bas-relef ambguty", \rubbery moton percept" and a robust representaton of shape In order to detect and classfy the local extrema assocated wth the cost functon of SFM, one can start by settng up a random samplng smulaton. Note that, by vrtue of lemma., t s equvalent to checkany of the cost functons r (a; b;), r(a; b) orr (a), dened n (8) and n lemma. respectvely. Ths makes the random search partcularly favorable for r (a), snce t only depends upon parameters, and can therefore be vsualzed, as we wll see n secton 5. We rst observe that local extrema tend to cluster n a small number of groups (eght), and then gve analytcal explanaton for the geometrc/nose conguratons that gve rase to such local mnma. The \bas-relef" ambguty One of the common complants to SFM algorthms from perspectve s that they become unrelable n the presence of small elds of vew, and when the rotatonal component of moton s \confused" wth the translatonal component (see for nstance [8]). It s our goal n ths secton to formalze ths concept and establsh that ths eect s \ntrnsc", and not an artfact of the algorthm. We wll also dscuss what nformaton can be robustly recovered under these condtons. When an object occupes a small porton of the vsual eld, x tend to be smlar. We call ther average drecton x. Rotaton about an axs orthogonal to the lne of sght passng through the object corresponds to a and b beng orthogonal to each other and both orthogonal to x. We wll now show that these condtons, n presence of nose, gve rse to a mnmum of the cost functon (8). Clam 4. Let y =,bx a + bx b + n, and x? a? b. Furthermore, let b = b + b, wth kbk = knk, the average norm of the measurement error. Then the cost functon r (a; b;) = : P p ky + bxa, bx bk P has a local extremum at b ~ = b, ~ =,, p and ~a = v mn( yyt ), where the bar denotes the average, and v mn denotes the egenvector correspondng to the smallest egenvalue. Proof Wthout loss of generalty, assume b =, so that b = b s of sze comparable to the nose: kbk = knk. Snce x? b and x S, we have kbx bk = kbk 8 = :::p and therefore the terms bx b are comparable wth the nose n, and they can be lumped as a bas nto ~n = bx b + n : The value of b that mnmzes the norm of ~n s b ~ =, and the correspondng ~a s obtaned P as the soluton to the SLS problem (), e.g. ~a = p v mn( yyt ). In order to evaluate the correspondng, ~ we observe that =,(bx a) y y +(bx a) y bx b = +(bxa) y bx b = + where are obtaned assumng b = and s the average of the scales. Therefore, the scales correspondng to the local soluton b =are the zero-mean verson of the orgnal ones ~ =, : Remark 4. From the clam we can conclude that, n presence of hgh nose levels, a porton of the rotatonal velocty b can be confused wth nose and compensated by a \bas" n the translatonal velocty a, and the correspondng nverse depths are oset towards the orgn. Ths statement conrms the observatons of Weng et al. [8], although they attrbute the eect to the geometry of the eppolar constrant. The consequence of clam 4. seems n contrast wth the observatons of [7], that the axs of rotaton has no mpact on the bas of the estmate of translaton. However, n order for the bas-relef eect to show, the condtons of clam 4. must be met, n partcular the aperture angle must be small, the nose level must be hgh and the algorthm must be ntalzed far away from the true soluton. Some of these condtons were not explored n the expermental setup of [7], and therefore the eect was not observed. Other extrema In the proof of clam 4. we have computed the local mnmum ~a assocated wth ~ b = as the soluton of the correspondng weghted SLS problem. Such a soluton s the egenvector of the matrx M (dened n clam.) correspondng to ts smallest egenvalue. In the absence of nose, such egenvalue s. For small nose levels, t stll s dstnctvely smaller than the remanng two. However, n the presence of large nose, the egenvalues become comparable and therefore the actual soluton of the SLS problem can be any of the three egenvectors of M. Ths ndeed happens { for large nose levels { and t accounts for three of the extrema of the cost functon found expermentally. Detectng local mnma and swtchng between extrema We have establshed that there are (at least) two extrema of the blnear cost functon (BCF) for b: one correspondng to the true soluton, and one correspondng to the bas-relef ambguty ~ b =. Correspondngly, there are three extrema for a, the egenvectors of the matrx M dened n clam.. Out of these three extrema, there s a mnmum, a saddle, and a maxmum, as a consequence of the analyss n secton. It s possble to detect whether a statonary pont ~a s a local mnmum or not. In fact, gven ~a, we can compute ts orthogonal complement (the remanng two egenvectors of M), and compute the correspondng resdual. We then just choose the egenvector that carres the smallest resdual. By dong so, we rule out 4 local extrema (+), and we are left wth possbltes: ~ b = b, or ~ b =, the bas-relef ambguty. As t turns out, ths stuaton can also be detected easly, even wthout knowng the nose level knk (whch gves a lower bound on the resdual). In fact, the correct b leads to reconstructed scales that are postve, whle n the bas-relef ambguty they are oset towards the orgn and, as long as not all are equal (.e. when the structure s a perfect fronto-parallel plane), some ~ wll be negatve, as a consequence of clam 4.. Note that even n the latter case, whch corresponds to the bas-relef ambguty, t s stll possble to retreve a useful representaton of shape, for ~ can be re-scaled by choosng so that ~ + >, whch leads to the bas-relef. These scaled parameters can be chosen to eectvely re-ntalze the algorthm, as we wll see n the expermental secton.

5 Z \Rubbery moton" percept An nterestng phenomenon that has been observed n psychophyscal experments occurs under the same condtons of the bas-relef ambguty. However, nstead of rotatonal velocty beng underestmated, t s perceved as beng the opposte of the true one. In order to analyze ths phenomenon from the pont of vew of nose, let us consder the same condtons of the basrelef ambguty as expressed n clam 4., and assume that ~b =,b, and ~ =,,. In order for ths to be a legtmate local extremum, as a consequence of lemma., there must exst some ~a that makes the nose ~n small, where y =,bx ~a ~ +bx ~ b+ ~n : If we substtute the expressons for ~ and ~ b we get y = bx ~a,( b ~ax)(bax) y b x b, b x b+~n : From that expresson t s possble to see that, under the assumptons of the bas-relef ambguty, ~a =,a, where a s the soluton to the SLS problem obtaned by assumng b =. In fact, n that case we have y = bx ~a, b (bax)? x b + ~n but b x b = b and bax = b, so that (bax) kbk ( b x b)=and the second term n the prevous expresson s neglgble. We have therefore y = bx ~a + ~n : () The value of ~a that mnmzes the sum of the norms of ~n s obtaned as the soluton of the SLS problem assocated to (). Remark 4. Note that ths soluton, whch we call the \rubbery moton" eect, s not just the correct soluton wth the pped sgn, for that would correspond to ~ =,, whle here we have ~ =,. A robust representaton of shape The averaged nverse depth s nvarant under the bas-relef ambguty. The rubbery moton percept conssts n a sgn change of the averaged nverse depth. Therefore, the averaged nverse depth represents shape up to a global scalng factor and a sgn, and t s nvarant under the bas-relef ambguty and the rubbery moton percept. 5 Experments How much nose s too much nose? Typcal feature-trackng/optcal-ow algorthms declare accuracy n locatng correspondng feature-ponts n the order of : pxels std []. It s our experence that ths s ndeed the case for about % of the feature-ponts extracted automatcally accordng to a SSD (Sum of Squared Derences) crteron. However, for 7% of the features, a more realstc gure for the localzaton error s pxel std. Now consder a camera wth a o eld of vew and an magng sensor of 5 5 pxels, translatng forward at :5m=s. Dependng upon the scene beng vewed, the average norm of the ow vectors on the magng sensor s n the order of - pxels (for a 5 frames/second capture rate). Therefore, an error n the order of pxel corresponds to 5%-% of the measurements. Consder agan the camera just descrbed, but now lookng at an object error resdual that s m ahead of the camera and rotatng about an axs passng through ts centrod at o =s. In ths case the average norm of ow vectors s : pxels/frame, and pxel error corresponds to an ntolerable %. Therefore, even the use of the most accurate featuretracker does not dspense us from dealng wth nose n scenaros that are very often encountered n real-world stuatons. Sample tests on real mages Snce the algorthm descrbed n secton s optmal by constructon, we should expect t to work at least as well as any other algorthm. Most real mage sequences avalable do not have a relable ground-truth, and therefore a far comparson s mpossble. Later n ths secton, however, we report the results of a smulaton to compare the optmal algorthm versus ones based upon eppolar geometry and lnear subspace constrants. Here we just report the use of the algorthm on a real mage sequence for the sake of example. We have chosen as a sample experment a \box-sequence" that was avalable on the Web n Matlab format wth calbraton data (gure ); the moton pattern s the one leadng to the bas-relef ambguty. The box rotates about a vertcal axs at a rate of o =frame, whch s a farly large moton. Under these condtons even the 8-pont algorthm of Longuet-Hggns works top vew of the reconstructed box scene X Fgure : Box scene A box rotates about a vertcal axs at o =frame. The top-vew of the reconstructed scene s shown on the rght n normalzed unts (unts of translatonal velocty). No relable ground-truth s avalable. Sample of convergence behavor durng smulatons In gure we show a typcal case where the teraton converges to the global mnmum. The resdual decreases up to the level of the nose (left), and both the parameters a; b (center) and the scales (rght) are wthn bounds from the true values. 4 x Resdual durng blnear teraton teraton estmaton error Velocty error velocty component scaled nverse depth Estmated scales pont ndex Fgure : Convergence to the global mnmum (left) resdual cost functon, (center) parameter estmaton error, (rght) estmated scales. Ground truth s n dotted lnes (f you can see them, that s). However, sometmes the resdual stablzes at a level derent from the nose level, as n gure (top-row left), but both the parameters (center) and the scales (rght) are qute far from ther true values, ndcated n dotted lnes. Ths s a clear sgn

6 that the algorthm has converged to a local extremum. However, f we check all three egenvectors of the matrx M and the scales that they generate ( mddle-row), we see that one of them (center) produces an estmate that corresponds to the averaged verson of the correct scales. Therefore, there has been a swtch of the egenvalues of M. We can now swtch to the soluton for a and correspondng to the egenvector that generates the smallest resdual, and use that as ntal condton for a second run of the algorthm, that converges n 5 teratons ( bottomrow). Fgure 4 shows convergence to the bas-relef ambguty. error resdual..8.6 Resdual durng blnear teraton estmaton error Velocty error nverse depth Estmated scales elevaton), we expect a maxmum and a saddle n the orthogonal drecton. These are showed n gure 5 (top left). The saddle concdes wth the sngularty of the sphercal coordnates: n fact, the two lnes (,=;), and (=;) correspond to a pont on the sphere. The rubbery nterpretaton corresponds to the local mnmum dametrcally opposed to the true moton (gure 5 top rght). the local extrema correspondng to the bas-relef ambguty are reported n gure 5 (bottom left), whle n (bottom rght) we show the locaton of the extrema correspondng to the \rubbery" bas-relef ambguty. We expect that our smulatons wll show convergence to some or all of these local extrema. ntensty of reduced cost functon, a = [ ], b = [.698 ] ntensty of reduced cost functon, a = [ ], b = [.698 ] teraton velocty component pont ndex elevaton elevaton. nverse depth correspondng to the saddle. nverse depth correspondng to the mnmum. nverse depth correspondng to the maxmum azmuth azmuth scaled nverse depth.5.5 scaled nverse depth.5.5 scaled nverse depth.5.5 ntensty of reduced cost functon, a = [ ], b = [.698 ] ntensty of reduced cost functon, a = [ ], b = [.698 ] elevaton elevaton pont ndex pont ndex pont ndex Resdual durng blnear teraton Velocty error.5 Estmated scales azmuth azmuth error resdual estmaton error scaled nverse depth Fgure 5: Predcted extrema for xatng moton Local extrema are plotted as black astersks supermposed to the resdual of the cost functon r (top left). We also show the local extrema correspondng to the rubbery nterpretaton (top rght), the bas-relef ambguty (bottom left), and the rubbery bas-relef ambguty (bottom rght). 5 5 teraton velocty component pont ndex error resdual Fgure : Convergence to a local extremum (top) the resdual stablzes (left), but the parameter error (center) and scales (rght) are far away from ther true values (n dotted lnes). The normalzed scales correspondng to the egenvectors of M, plotted n the mddle row, show that one of them corresponds to the correct estmate. We may then swtch to the correct soluton and re-ntalze the algorthm, that converges to the correct soluton wthn 5 steps (bottom row). Resdual durng blnear teraton teraton estmaton error Velocty error velocty component scaled nverse depth Estmated scales pont ndex Fgure 4: Convergence to the bas-relef ambguty The resdual stablzes (left), the parameters (center) are far away from the true values, but the estmated scales (rght) are an averaged verson of the true ones. Predcted behavor based on the analyss Based upon the analyss carred out n secton, we know that local extrema of the orgnal functon of SFM r n (8) are n correspondence wth the local extrema of the reduced functon r (a). Snce kak =,we can represent a n sphercal coordnates and plot the cost functon r (g. 5). The moton s a xatng one smlar to that of the box experment (gure ). In addton to the global mnmum, correspondng to the coordnates (;=) (azmuth, Expermental trals We have consdered p = ponts on a volume of sde m centered m from the center of projecton. The ponts have mages on the unt sphere. Frst we have consdered forward translaton at :m=f rame, and nose levels of :, and pxels std, correspondng to 4%, 4% and 4% of the measurements respectvely. For each nose level we have performed trals. In gure 7 we show the plot of the resdual of the cost functon supermposed to the pont where the Blnear Projecton teraton converged (a black astersk). On the second column, the same ponts have been checked aganst local mnma, and the global mnmum has been chosen correctly n all cases. The stuaton s very derent for a xatng moton. We have consdered the same stuaton just descrbed, but where the cloud of dots rotates of o =frame about an axs passng through the centrod. In ths case we have consdered nose of :5, :5 and 5 pxels std, correspondng to %, % and % of the measurements. Already at % nose we notce that the algorthm converges to local mnma correspondng to both the saddle, the bas-relef ambguty, and the rubbery moton percepton. If we check for local mnma, however, we can get to the correct estmate n all trals (rght). For % nose, however, the rubbery moton percepton becomes stable, so that about 4% of the trals return the rubbery moton soluton even after correcton.

7 azmuth local extrema of blnear teraton, nose level = 4% local extrema of blnear teraton, nose level = 9% azmuth local extrema of blnear teraton, nose level = 89% azmuth swtched extrema of blnear teraton, nose level = 4% azmuth azmuth swtched extrema of blnear teraton, nose level = 9% swtched extrema of blnear teraton, nose level = 89% azmuth local extrema of blnear teraton, nose level = % azmuth local extrema of blnear teraton, nose level = 9% azmuth local extrema of blnear teraton, nose level = 86% azmuth swtched extrema of blnear teraton, nose level = % azmuth swtched extrema of blnear teraton, nose level = 9% azmuth swtched extrema of blnear teraton, nose level = 86% azmuth Estmates of the components of a.5 Estmates of the scales lambda 5 x Estmaton resdual: mean and standard devaton errorbars = std 4 A (m/s) errorbars = std lambda (m).5 nose level (m) elevaton elevaton elevaton elevaton.5 4 ( th component of a) ( th scale lambda() ( th component of the resdual) Estmates of the components of a.5 Estmates of the scales lambda 5 x Estmaton resdual: mean and standard devaton elevaton elevaton elevaton elevaton errorbars = std 4 A (m/s) errorbars = std lambda (m).5 nose level (m).5 4 elevaton elevaton elevaton elevaton ( th component of a) ( th scale lambda() ( th component of the resdual) A (m/s) Estmates of the components of A errorbars = std ( th component of A) lambda (m) Estmates of the scales lambda errorbars = std ( th scale lambda() nose level (m) Estmaton resdual: mean and standard devaton ( th component of the resdual) Fgure 6: Comparson wth eppolar geometry and lnear subspace methods (Top row) the estmates of translaton (left) and the scales (center) obtaned wth the BPA are plotted wth errorbars for trals of the experment. Nose s 5% of the measurements. The resdual (rght) s small, but t can be no smaller than the nose level, plotted n dashed lnes. (Center row) algorthms based on the eppolar constrant perform consderably worse (left and center), despte the fact that the resdual s smaller (rght). The fact that the resdual s smaller than the nose level s ndeed a consequence of the fact that the eppolar constrant does not mnmze the reprojecton error. Lnear subspace methods (bottom row) exhbt a bas both n the estmates (left and center) and n the resdual (rght). Comparson wth other algorthms In ths secton we compare the optmal algorthm proposed n secton wth other approaches based upon eppolar geometry [4], and upon lnear subspace methods [7]. We use as a representatve of the rst class the algorthm descrbed n [9], and as a representatve of lnear subspace methods the one n [6]. We consder p = ponts dstrbuted unformly n a cube of sde m centered at m, rotatng at o =frame, wth measurements on the unt sphere corrupted by 5% nose. In gure 6 we show the result of trals. The upshot s, not surprsngly, that the optmal algorthm works better. The outcome of the experments for lnear subspace methods show that the estmates are based, as reported n [6]. 6 Conclusons The assumpton of \small nose" s often llegtmate n condtons normally encountered n real-world experments wth SFM. Therefore, SFM needs to be addressed from the pont of vew of nose. We have proposed a provably convergent algorthm (the blnear projecton teraton), characterzed the set of local extrema, gven a geometrc nterpretaton and proven that they are ntrnsc to SFM, and proposed a representaton of shape that s nvaranttolocal mnma. Fgure 7: Forward translaton (left) The resdual of the cost functon r s supermposed to the xed ponts of the Blnear Projecton teraton before (left) and after (rght) correcton for local extrema. Nose s 4% (top), 4% (center) and 4% (bottom) of the measurements. Convergence to the valley of the global optmum s acheved n all trals. Bas-relef ambguty (rght) The Blnear Projecton algorthm converges to local mnma correspondng to both the bas-relef ambguty and the rubbery nterpretaton (left). For noses of % (top row) and % (center row), checkng for local extrema s sucent toacheve the correct soluton n all trals. For % nose (bottom), the rubbery nterpretaton s stable n % of the trals. Acknowledgements We thank J. Olenss and C. Tomas for comments. References [] G. Adv. Inherent ambgutes n recoverng -D moton and structure from a nosy flow feld. IEEE Trans. on Patt. Anal. and Mach. Intell., (5), 989. [] J. Barron, D. Fleet, and S. Beauchemn. Performance of optcal flow technques. Int. J. of Computer Vson, ():4{78, 994. [] J. Bergen, R. Kumar, P. Anandan, and M. Iran. Representaton of scenes from collectons of mages. In Proc. of the IEEE Workshop on Vsual Scene Representaton, Boston, June 995. [4] O. D. Faugeras. Three Dmensonal Vson, a geometrc vewpont. MIT Press, 99. [5] O. D. Faugeras and Q. T. Luong. n preparaton, 997. [6] G. Golub and V. Pereyra. The dfferentaton of pseudo-nverses and nonlnear least-squares problems whose varables separate. SIAM J. Numer. Anal., ():4{ 4, 97. [7] A. Jepson and D. Heeger. Lnear subspace methods for recoverng rgd moton. Spatal Vson n Humans and Robots, Cambrdge Unversty Press, 99. [8] K. Kanatan. Statstcal Optmzaton for Geometrc Computaton. Unpublshed manuscrpt 997. [9] J. Olenss. Provably correct algorthms for mult-frame structure from moton. Proc. of the IEEE Conf. on Computer Vson and Pattern Recognton, 996. [] P. Anandan R. Kumar and K. Hanna. Shape recovery from multple vews: a parallax based approach. Proc. of the Image Understandng Workshop, 994. [] S. Soatto and R. Brockett. Optmal Structure From Moton. Techncal Report, Aprl 997, revsed Oct [] H. S. Sawhney. Smplfyng moton and structure analyss usng planar parallax and mage warpng. Proc. of the Int. Conf. on Pattern Recognton, Seattle, June 994. [] M. Spetsaks and J. Alomonos. Optmal moton estmaton. Proc. of the IEEE workshop on vsual moton, Irvne, March 989. [4] R. Szelsk and S. Kang. Recoverng D shape and moton from mages usng nonlnear least squares. J. Vsual Communcaton and Image Representaton, vol.5 n., 994. [5] R. Szelsk and S. Kang. Shape ambgutes n structure from moton. IEEE Trans. on Patt. An. and Mach. Intell., vol.9 n. 5, 997. [6] I. Thomas and E. Smoncell. Lnear Structure from Moton. Techncal Report IRCS 94-6, Unversty ofpennsylvana, 994. [7] T. Tan, C. Tomas, and D. Heeger. Comparson of approaches to egomoton computaton. In proc. of the IEEE Conf. on Computer Vson and Pattern Recognton, 996. [8] J. Weng, N. Ahuja, and T. Huang. Optmal moton and structure estmaton. IEEE Trans. Pattern Anal. Mach. Intell., 5:864{884, 99. [9] J. Weng, T. S. Huang, and N. Ahuja. Moton and structure from two perspectve vews: algorthms, error analyss and error estmaton. IEEE Trans. Pattern Anal. Mach. Intell., (5):45{476, 989. [] G. Young and R. Chellappa. Statstcal analyss of nherent ambgutes n recoverng -D moton from a nosy flow feld. IEEE Trans. Pattern Anal. Mach. Intell., 4(), 99.