Robust Recovery of Camera Rotation from Three Frames. B. Rousso S. Avidan A. Shashua y S. Peleg z. The Hebrew University of Jerusalem

Transcription

1 Robust Recovery of Camera Rotaton from Three Frames B. Rousso S. Avdan A. Shashua y S. Peleg z Insttute of Computer Scence The Hebrew Unversty of Jerusalem 994 Jerusalem, Israel e-mal : roussocs.huj.ac.l Abstract Computng camera rotaton from mage sequences can serve many computer vson applcatons. One drect applcaton s mage stablzaton, and when the camera rotaton s known the computaton of camera translaton and 3D scene structure are much smpled. A new approach for recoverng camera rotaton s presented n ths paper, whch proves to be much more robust than exstng methods by avodng the computaton of the eppole. Another benet of the new approachsthattdoes not assume any specc scene structure. The rotaton matrx of the camera s computed explctly from three homography matrces, recovered usng the trlnear tensor whch descrbes the relatons between the projectons of a 3D pont nto three mages. The entre computaton s lnear for small angles, and s therefore fast and stable. Iteratng the lnear computaton can then be used to recover larger rotatons as well. Introducton Recoverng camera rotaton s one of the basc steps n many mage sequence applcatons, such as electronc mage stablzaton. Most exstng methods take one of the followng two approaches. One approach s to compute the camera rotaton only after computng the camera translaton (the eppole) [8, 4, 8, 2]. The second approach assumes a specc 3D scene structure, e.g. assumng the exstence and the detecton of a 3D plane n the scene [8, 7,3,]. We propose a new method to recover rotatons usng three homography matrces, wthout usng the eppoles and wthout assumng any specc 3D model. A homography s a transformaton that maps the mage of a 3D plane n one frame nto ts mage n the second frame, Ths research was sponsored by ARPA through the U.S. Oce of Naval Research under grant N , R&T Project Code y A. Shashua s wth the Dept. of Computer Scence, Technon, Hafa, Israel. z S. Peleg s vstng Davd Sarno Research Center, Prnceton, NJ 854. and t can be representedasa33 matrx (see Secton 2). The homographes used do not correspond to any planes that have to be present n the mage, and therefore there s no restrcton on the scene structure. The benets of our method are ncreased accuracy, as eppoles are beleved to be a source for error [22], and more general applcablty as no specc 3D model s assumed. The theoretcal background of ths paper ncludes a recently dscovered property of homography matrces between mage pars, namely that homography matrces between two mages form a lnear space of rank 4 (See [6]). Wetake ths general result further and show that under small-moton assumptons (whch stypcally the case n vdeo sequence processng) the rotatonal component of camera moton (whch s the homography dueto the plane at nnty) s spanned by only three homography matrces. There are several possbltes for ndng the three homography matrces whch are needed to compute the camera rotaton [7,, 2,, 3, 7]. However, most methods to compute homographes between two mages assume that each homography corresponds to an actual planar surface n the scene. Even when several planar surfaces do exst n the scene, the accuracy of these methods reduces as more homographes are extracted. The best method we found for computng the needed homographes s by usng the trlnear tensor between three mages [5, 4, 8, 5]. The trlnear tensor s relatvely accurate snce t s computed from three frames, rather than only two, and no 3D scene structure s assumed. We then use the recently dscovered contracton property nto three homography matrces [8] to obtan a closed-form soluton for the rotatonal component of camera moton from the trlnear tensor. We thereby obtan lnear expressons for the camera rotatons that are clear and smple, and do not suer from naccuraces n eppole's recovery. 2 The Homography Matrx In ths secton we wll brey dene the homographymatrces, and prove that all homography matrces between two mages form a lnear space of rank 4. For more detaled nformaton on homography matrces n 3D-from- 2D geometry see [3, 2,4,6,7,3,9,,8], and for more detals on the rank-4 result see [6].

2 Let P beapont n 3D space projectng onto mages ;. Let p 2 and p 2 be the matchng ponts n the two mage planes descrbed by: p = M[I;]P p = M [R; T ]P where = denotes equalty up to scale, and M denotes the transformaton from the observed coordnates of the mage plane to the camera coordnate system, of the rst camera, and M of the second camera. In the smplest conguraton, M s of the form: f xo M = f y o where f denotes the focal length of the camera and x o ;y o s the orgn of the mage plane (known as the \prncple pont). The rgd camera moton s represented by the rotaton matrx R and translaton vector T. Taken together, P =(x; y; ; =z) >, where x; y are the mage coordnates of the rst vew, z s the depth of the pont,.e., zm, p are the Eucldean coordnates of the pont P n the rst camera frames, and: p = M RM, p + z M T: When the ponts P lveonaplane,thenn > (zm, p)= d where n (normal vector) and d (scalar) are the parameters of n the rst camera coordnate system. We obtan, p = M RM, p + M T (zn > M, p) z d = M (R + Tn > )M, p d = H p: In other words, the homography matrx assocated wth s H = M (R + Tn> )M, : () d Therefore, A homography H s a transformaton assocated wth the two mages and, and wth the 3D plane. For any pont P n, the homography H maps p to p (see Fg. ). For a xed par of cameras (M;M ;T and R are constant), gven a homography matrx H of some 3D plane, all other homography matrces can be descrbed by H + Tn > (2) for some scale factor and a normal to some plane n, snce the homographes der only n scale and n the plane parameters. Consder homography matrces H ;H 2 ;:::;H k each as a column vector n a 9k matrx. Let H = H +Tn >. The followng can be vered by nspecton: = 9k T T T T H T T =[ H k H ] 9k [n n k] 3k = k n n k 4k (3) P π p p 2 ψ ψ 2 O O 2 Fgure : The homography nduced by the plane maps p to p. p and p are the perspectve projectons of any pont P n the 3D plane on the mage planes and. We have thus proven that the space of all homography matrces between two xed vews s embedded n a 4 dmensonal lnear subspace of R 9. 3 The Rotaton Matrx Gven a sequence of mages taken by a camera movng n a statc scene, we would lke to recover the rotaton parameters of the camera. The rotaton matrx R s an orthonormal matrx (up to scale). The orthonormalty s the only constrant onr, whch can generate ve non-lnear constrants on the elements of R. Note also that M RM, s the homography of the plane at nnty (lettng d go to nnty n eqn. ). Snce solvng non-lnear equatons s n general less stable and harder to compute than lnear equatons, we wll rst examne the case of small rotatons. In addton we wll assume that M = M = I,.e., that the nternal parameters of the cameras are known. Wewllshow that n addton to workng wth lnear constrants, the case of small rotatons places a strong constrant on the famly of admssble homographes: the only skew-symmetrc homography matrx corresponds to the plane at nnty. When small rotatons are nvolved, the rotaton matrx R can be approxmated by a matrx havng the followng form (up to scale): R ^R = Z, Y, Z X Y, X : (4) In ths representaton, the vector = ( X ; Y ; Z ) > s the rotaton axs, and the magntude of the vector s the magntude of the rotaton around ths axs. Lkewse, the famly of all approxmate homography matrces ^H s dened by: H ^H ^R + d Tn > ; (5) where d s the dstance from the orgn to the plane, n > s the unt vector perpendcular to the plane toward the orgn, and T =(T X ;T Y ;T Z ) > s the camera's translaton vector. Note that when d!,wehave ^H = ^R. Therefore, the approxmate rotaton matrx ^R s also an approxmate homography matrx of the plane at nnty.

3 There are two man advantages of ths framework. Frst, as we shall see next, the non-lnear constrants assocated wth orthonormal matrces s replaced wth a skewsymmetrc condton on the famly of approxmate homographes whch, moreover, leads to a unque choce for the approprate rotaton (unlke the general dscrete case n whch the space of homography matrces s represented by a 4 parameter famly). Second, shown later, s that one can obtan a drect lnk between the trlnear tensor and the rotaton matrx, thereby avodng the computaton of the translatonal component ofcamera moton (the eppolar geometry). We show next that the asymmetrc form of ^R s unque. Lemma The only approxmate homography matrx that has an asymmetrc form r u,s,u r t : (6) s,t r s the one assocated wth the plane at nnty. Proof : f ^H has the asymmetrc form (6), then Tn > = d ( ^H, ^R) (usng Equaton 5) also has the asymmetrc form (6), because ^R has ths asymmetrc form (Equaton 4). However, the asymmetrc form (6) has rank 3 (or rank 2 f r = ), whle the matrx Tn > has rank, and we got a contradcton. Ths Lemma together wth the rank 4 result mples that the matrx ^R we are lookng for s the asymmetrc matrx spanned by four approxmate homography matrces. In the followng secton we wll take ths result a step further and show that, rst, only three homography matrces are requred, and second, that ^R has a closed-form representaton n terms of the coecents of the trlnear tensor. 4 From Homographes to Rotatons We have shown (Secton 2) that the space of all homography matrces between two magessalnearspaceof rank 4 (See also [6]). Ths holds also for the space of all approxmate homographes (Equaton 5), usng the same proof. Gven the rank-4 property we should nd 4 lnearly ndependent (approxmate) homography matrces ^E ; =::4; between the two gven mages, and solve the equaton: Z, Y, Z X = ^R = c ^E + c 2 ^E2 + c 3 ^E3 + c 4 ^E4 Y, X (7) where c ; = ::4; are scalars. We should actually compute the c ; =::4; that satsfy the asymmetrc form constrants (sx constrants). We can reduce the need for 4 matrces to 3 by choosng the fourth matrx to be Tx ^E 4 = T y ; (8) T z where T =(T x ;T y ;T z ) > s the translaton vector from to, whch s unknown yet. Furthermore, we can set c 4 = because the translaton vector s dened up to scale, and so does ^E 4. Lemma 2 The matrx ^E4 s a vald homography matrx from to. Proof : Gven the rotaton matrx ^R and a translaton vector T, the homography matrx from to can be dened by ^H = d ( ^R + d Tn> ) for any value of the plane parameters d and n (Equaton 5). When usng ths equaton for the plane havng n > =(; ; ) at the lmt of d! we get lm d! ^H = Tn > = Tx T y T z ; (9) whch s the matrx ^E 4. To solve Equaton (7), wth ^E4 dened n Equaton (8), we need to solve for the sx unknowns: c ;c 2 ;c 3 ;T x ;T y ;T z, usng the sx constrants on the asymmetrc form of ^R : Z, Y, Z X = ^R = Y, X Tx c ^E + c 2 ^E2 + c 3 ^E3 + T y : () T z Any three homography matrces recovered n any known method can be used to recover the camera rotaton (assumng small angle rotatons). Our method of choce, however, s the trlnear tensor computed for three mages from pont correspondences [5, 8, 5]. There are two advantages for ths choce. Frst, the tensor provdes drectly three homography matrces [8], hence as a technque for producng homography matrces one s explotng three nstead of two vews (thereby ganng numercal redundancy) and also no assumpton on scene structure s beng made (.e., that the scene contans domnant and dstnct physcal planes). Second, we wll show that there s a closed-form soluton for ^R as a functon of the tensor coecents, thereby obtanng a lnear method that uses all the matchng ponts across three vews (for computng the tensor), does not requre solvng for the translatonal component of camera moton as an ntermedate stage, and s very smple. The trlnear tensor s descrbed n Appendx A. The tensor jk contans 27 entres (coecents) as ; j; k = ; 2; 3. The coecents can be recovered lnearly from at least 7 matchng ponts across three vews. The connecton between the tensor and homography matrces comes from contracton propertes as follows: for any vector s j =(s ;s 2 ;s 3 ), the matrx s j jk s a homography matrx from vew to vew 2, where s descrbes the orentaton of the assocated plane (smlarly, s k jk s a homography matrx from vew to vew 3). In partcular, when s =(; ; ); (; ; ) and (; ; ) we get our three ndependent homography matrces [8].

4 In the case of small rotatons we can enforce the constrants on the rotaton ^R on the homography matrces recovered from the tensor. Usng Maple for symbolcally solvng for the system of equatons we found that the translatonal component T x ;T y ;T z have been elmnated from the result, and we are left wth a very smple, closed-form, expresson relatng the tensor jk and X ; Y ; Z : X = det Y = det Z = det K = det 2, j2 j2 2 A =K A =K A =K A () where j2 2 stands for (2 2 ; 22 2 ; 32 2 ), etc. Ths expresson recovers drectly and smply small rotatons from the trlnear tensor. 5 Vdeo Stablzaton In ths secton we present the algorthm for vdeo stablzaton whch contans two steps. The rst step s to compute the trlnear tensors, and the second step s to compute the camera rotatons and perform derotaton on the frames. 5. Trlnear Tensor Computaton Followng are the steps we performed to compute the trlnear tensor from a set of three mages. For computaton stablty, all coordnates are normalzed to the range of (-, ). Selecton of correspondng ponts Optcal ow s computed between all three possble pars of the three mages. As correspondng ponts we selects only those ponts havng a hgh gradent, and for whch all the parwse optcal ows are consstent. Ths process results n the selecton of several hundred ponts as correspondng n all three frames (\correspondng trplets). Robust Estmaton From all correspondng trplets computed n the prevous step, several hundred subsets of ten trplets are randomly selected [2]. For each subset the trlnear tensor s computed usng Eq. 4. Each computed tensor s then appled to all matchng trplets, and the sngle tensor for whch the maxmal number of trplets satsfy Eq. 4 s selected. Least Square Step As a nal step we use all the ponts whch sats- ed the selected tensor n the prevous step to solve a) b) Fgure 2: Frst sequence - outdoor scene. a) Frst orgnal frame. b) Last (fourth) orgnal frame. The camera was rotatng whle movng forward. c) average of the four orgnal mages. d) average of the four mages after rotaton cancellaton. The remanng moton s only due to the orgnal translaton. The sequence looks as f t was taken usng a stablzed camera. the tensor agan from Eq. 4 usng a least squares method. From ths tensor the homography matrces wll be computed. 5.2 Derotaton of Vdeo Frames The mage sequence s stablzed wth regard to the rst mage. Ths s done by selectng an arbtrary mage to serve as the thrd mage and sequentally gong through the mages. For each such trplet of mages the trlnear tensor s computed as descrbed above. From the compute tensor we compute X ; Y ; Z usng Eq.. The mage s then warped back to cancel the rotaton, thus gettng a stable sequence. 6 Expermental Results: Stablzaton Gven a sequence of mages, a trlnear tensor s recovered from the rst frame to all other frames, usng an arbtrary frame as the thrd frame. The rotaton from the rst frame to all other mages s then recovered by usng the the tensor values n Eq.. By warpng back every mage usng the calculated rotaton, we obtan a new sequence of mages havng no rotaton compared to the rst mage. The remanng moton n the new sequence s only due to the orgnal translaton, thus the new sequence s smooth and clear, as can be observed n the average mages. The method was tested on outdoor scene (Fg. 2), ndoor scene (Fg. 4), and on a scene wth objects that are very close to the camera (Fg. 3). The method proved to be robust and ecent.

5 a) b) a) b) Fgure 3: Second sequence - close objects. a) Frst orgnal frame. b) Second orgnal frame. The camera was movng and rotatng around the objects. c) average of the two orgnal mages. d) average of the two mages after rotaton cancelaton. The remanng moton s only due to the orgnal translaton. 7 Concludng Remarks A new robust method to recover the rotaton of the camera was descrbed. The man contrbuton to the robustness s the fact that we do not have to recover the eppoles, and the rotaton s computed drectly from three homography matrces assumng small rotatons. The homography matrces are obtaned from three mages usng the trlnear tensor parameters, and the recovery process does not assume any 3D model. The method can be extended to handle also the case of general rotatons by usng teratons. A Appendx: The Trlnear tensor In ths secton we brey present the trlnear tensor and gve an example to measure ts qualty. The trlnear tensor s an extenson of the fundamental matrx to the case of three mages, and t descrbes the spatal relaton of three cameras. The qualty of the tensor can be evaluated by reprojectng the thrd mage from the rst two mages usng the trlnear tensor. More detals about the trlnear tensor can be found n [5, 8]. Let P be a pont n 3D projectve space projectng onto p; p ;p three vews ; ; represented by the two dmensonal projectve space. The relatonshp between the 3D and the 2D spaces s represented by the 3 4 matrces, [I;], [A; v ] and [B;v ],.e., p =[I;]P p = [A; v ]P p = [B;v ]P We may adopt the conventon that p = (x; y; ) >, p = (x ;y ; ) > and p = (x ;y ; ) >, and therefore Fgure 4: Thrd sequence - ndoor scene. a) Frst orgnal frame. b) Second orgnal frame. The unstable camera was movng towards the man. c) average of the two orgnal mages. The rotatons make the average mage unclear. d) average of the two mages after rotaton cancellaton. The remanng moton s only due to the orgnal translaton. The sequence looks as f t was taken usng a stablzed camera. P = [x; y; ;]. The coordnates (x; y); (x y ); (x ;y ) are matchng ponts (wth respect to some arbtrary mage orgn say the geometrc center of each mage plane). The matrces A and B homography matrces from to and, respectvely, nduced by some plane n space (the plane = ). The vectors v and v are known as eppolar ponts (the projecton of O, the center of projecton of the rst camera, onto vews and, respectvely). The trlnear tensor s an array of27entres: jk = v k b j, v j a k : ; j; k =; 2; 3 (2) where superscrpts denote contravarant ndces (representng ponts n the 2D plane, lke v ) and subscrpts denote covarant ndces (representng lnes n the 2D plane, lke therows of A). Thus, a k s the element ofthe k'th row and 'th column of A, andv k s the k'th elementofv. The tensor jk forms the set of coecents of certan trlnear forms that vansh on any correspondng trplet p; p ;p (.e., functons of vews that are nvarant to object structure). These functons have the followng form: let s l k be the matrx, s = and, smlarly, letr m j r =,x,y be the matrx,,x,y Then, the tensoral equatons are: s l kr m j p jk =; (3)

6 a) b) Fgure 5: Reprojecton n derent methods. a) Reprojecton usng eppolar lne ntersecton. Fundamental Matrces computed wth code dstrbuted by INRIA. b) Reprojecton usng eppolar lne ntersecton. Fundamental Matrces computed from tensor. c) Reprojecton usng the tensor equatons. d) Orgnal thrd mage. Presented for comparson. wth the standard summaton conventon that an ndex that appears as a subscrpt and superscrpt s summed over (known as a contracton). For detals on the dervaton of ths equaton see Appendx A. Hence, we have four trlnear equatons (note that l; m = ; 2). In more explct form, these functons (referred to as \trlneartes) are: x 3 p, x x 33 p + x 3 p, p =; y 3 p, y x 33 p + x 32 p, 2 p =; x 23 p, x y 33 p + y 3 p, 2 p =; y 23 p, y y 33 p + y 32 p, 22 p =: Snce every correspondng trplet p; p ;p contrbutes four lnearly ndependent equatons, then seven correspondng ponts across the three vews unquely determne (up to scale) the tensor jk. More detals and applcatons can be found n [5]. Also worth notng s that these trlnear equatons are an extenson of the three equatons derved by [9] under the context of unfyng lne and pont geometry. The connecton between the tensor and homography matrces comes from contracton propertes descrbed n Secton 4, and from the homography matrces one can obtan the \fundamental matrx F (the tensor produces 8 lnear equatons of rank 8 for F, for detals see [8]). Fg. 5 shows an example of mage reprojecton (transfer) usng the trlneartes, compared to usng the eppolar geometry (recovered usng INRIA code or usng F recovered from the tensor). One can see that the best results are obtaned from the trlneartes drectly. References [] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hngoran. Herarchcal model-based moton estmaton. In European Conference on Computer Vson, pages 237{252, Santa Margarta Lgure, May 992. [2] O.D. Faugeras. Three-Dmensonal Computer Vson, A Geometrc Vewpont. MIT Press, 993. [3] O.D. Faugeras and F. Lustman. Let us suppose that the world s pecewse planar. In O. D. Faugeras and Georges Gralt, edtors, Internatonal Symposum on Robotcs Research, pages 33{4. MIT Press, Cambrdge, MA, 986. [4] O.D. Faugeras and B. Mourran. On the geometry and algebra of the pont and lne correspondences between n mages. In Internatonal Conference on Computer Vson, pages 95{956, Cambrdge, MA, June 995. [5] R. Hartley. A lnear method for reconstructon from lnes and ponts. In Internatonal Conference on Computer Vson, pages 882{887, Cambrdge, MA, June 995. [6] R. Hartley and R. Gupta. Computng matched-eppolar projectons. In IEEE Conference on Computer Vson and Pattern Recognton, pages 549{555, New York, NY, 993. [7] M. Iran, B. Rousso, and S. Peleg. Detectng and trackng multple movng objects usng temporal ntegraton. In European Conference on Computer Vson, pages 282{287, Santa Margarta Lgure, May 992. [8] M. Iran, B. Rousso, and S. Peleg. Recovery of ego-moton usng mage stablzaton. In IEEE Conference on Computer Vson and Pattern Recognton, pages 454{46, Seattle, June 994. [9] R. Kumar and P. Anandan. Drect recovery of shape from multple vews: A parallax based approach. In Internatonal Conference on Pattern Recognton, Jerusalem, Israel, October 994. [] J.M. Lawn and R. Cpolla. Robust egomoton estmaton from ane moton parallax. In European Conference on Computer Vson, pages 25{2, May 994. [] Q.T. Luong and T. Vevlle. Canonc representatons for the geometres of multple projectve vews. In European Conference on Computer Vson, pages 589{599, Stockholm, Sweden, May 994. Sprnger Verlag, LNCS 8. [2] S. Negahdarpour and S. Lee. Moton recovery from mage sequences usng rst-order optcal ow nformaton. In IEEE Workshop on Vsual Moton, pages 32{39, Prnceton, NJ, October 99. [3] Harpreet Sawhney. 3d geometry from planar parallax. In IEEE Conference on Computer Vson and Pattern Recognton, June 994. [4] A. Shashua. Projectve structure from uncalbrated mages: structure from moton and recognton. IEEE Trans. on Pattern Analyss and Machne Intellgence, 6(8):778{79, 994. [5] A. Shashua. Algebrac functons for recognton. IEEE Trans. on Pattern Analyss and Machne Intellgence, 7:779{789, 995. [6] A. Shashua and S. Avdan. The rank4 constrant n multple vew geometry. In European Conference on Computer Vson, Cambrdge, UK, Aprl 996. Also n CIS report 952, November 995, Technon. [7] A. Shashua and N. Navab. Relatve ane structure: Theory and applcaton to 3D reconstructon from perspectve vews. In IEEE Conference on Computer Vson and Pattern Recognton, pages 483{489, Seattle, Washngton, 994. [8] A. Shashua and M. Werman. Trlnearty of three perspectve vews and ts assocated tensor. In Internatonal Conference on Computer Vson, pages 92{925, Cambrdge, MA, 995. [9] M.E. Spetsaks and J. Alomonos. A uned theory of structure from moton. In ARPA IU Workshop, 99. [2] P.H.S Torr, A. Zsserman, and D. Murray. Moton clusterng usng the trlnear constrant over three vews. In Workshop on Geometrcal Modelng and Invarants for Computer Vson. Xdan Unversty Press., 995. [2] R. Tsa and T.S. Huang. Estmatng three-dmensonal moton parameters of a rgd planar patch, II: sngular value decomposton. IEEE Trans. on Acoustc, Speech and Sgnal Processng, 3, 982. [22] A. Zsserman and S. Maybank. A case agant eppolar geometry. In J.L. Mundy, A. Zsserman, and D. Forsyth, edtors, Applcatons of Invarance n Computer Vson, pages 35{5, Ponta Delgada, Azores, 994. Sprnger.