Amnon Shashua Shai Avidan Michael Werman. The Hebrew University, objects.

Transcription

1 Trajectory Trangulaton over Conc Sectons Amnon Shashua Sha Avdan Mchael Werman Insttute of Computer Scence, The Hebrew Unversty, Jerusalem 91904, Israel e-mal: Abstract We consder the problem of reconstructng the 3D coordnates of a movng pont seen from a monocular movng camera,.e., to reconstruct movng objects from lne-of-sght measurements only. The task s feasble only when some constrants are placed on the shape of the trajectory of the movng pont. We con the famly of such tasks as \trajectory trangulaton". In ths paper we focus on trajectores whose shape s a conc-secton and show that generally 9 vews are suf- cent for a unque reconstructon of the movng pont and fewer vews when the conc s a known type (lke a crcle n 3D Eucldean space for whch 7 vews are suf- cent). Experments demonstrate that our solutons are practcal. The paradgm of Trajectory Trangulaton n general pushes the envelope of processng dynamc scenes forward. Thus statc scenes become a partcular case of a more general task of reconstructng scenes rch wth movng objects (where an object could be a sngle pont). 1 Introducton We wsh to remove the statc scene assumpton n 3D-from-2D reconstructon. Ths paper ntroduces another stage n a new paradgm we call \trajectory trangulaton" that pushes the envelope of processng \dynamc scenes" from \segmentaton" to 3D reconstructon. Consder the stuaton n whch a 3D scene contanng a mx of statc and movng objects s vewed from amovng monocular camera. The typcal queston addressed n ths context s that of \segmentaton": can one separate the statc from dynamc n order to calculate the camera ego-moton (and 3D structure of the statc porton)? ths queston s bascally a robust estmaton ssue and has been extensvely (and successfully) treated as such n the lterature (cf. [6, 5]). Abyproduct of the robust estmaton s the segmentaton of the scene to the statc and dynamc portons, or to the portons correspondng to multply movng objects. However, consder the next (natural) queston n ths context: can one reconstruct the 3D coordnates of a (sngle) pont onamovng object? Unlke the segmentaton problem, the reconstructon problem s not feasble, unless further constrants are mposed. In order to reconstruct the coordnates of a 3D pont, the pont must be statc n at least two vews (to enable trangulaton) f the pont smovng generally then the task of trangulaton s not feasble. Note that the feasblty ssue arses regardless of whether we assume the ego-moton of the camera to be known or not. Knowledge of camera ego-moton does not change the feasblty of the problem. The feasblty status changes when we constran the trajectory of the movng pont to belong to some (parametrc) famly of trajectores. We call the topc of reconstructng movng ponts, whose moton (n 3D) s constraned parametrcally, from general multple 2D projectons as \trajectory trangulaton". In the sequel we assume that the camera ego-moton (projecton matrces) s known. We acknowledge the dculty of recoverng the camera ego-moton n general, and under dynamc scene condtons n partcular, but beleve t to be reasonable n vew of the large body of theoretcal and appled lterature on the subject. Thus, we treat the problem of ego-moton as a "blackbox" and a rst layer n a herarchy of tasks that are possble n a "3D-from-2D" famly of problems. In ths paper we extend the noton of \lnear trajectory trangulaton" (see secton below) to second-order trajectores (See Fgure 1). In other words, we nvestgate the problem of a pont movng along some 3D conc trajectory and show that the reconstructon can be done n a practcal manner. Extensons and future work are dscussed n Secton 5. 1

2 Image 4 Image 4 Image 3 Image 1 Image 2 Image 3 Image 1 Image 2 Fgure 1: (a) "Trajectory Trangulaton" along a lne [1]. A pont s movng along a lne whle the camera s movng. (b) "Trajectory trangulaton" along a planar conc. A pont s movng along a planar conc whle the camera s movng. 2 Related Work The problem of trajectory trangulaton of a pont movng along some straght lne path was dscussed n [1]. Also related to the problem addressed n ths paper, conc trajectores, s the problem of \orbt determnaton" n astro-dynamcs (cf. [3]). We brey dscuss below these two related sources. The case of a lnear trajectory (straght-lne path) has a smple geometrc ntuton: the projecton of a movng pont gves rse to a collecton of 3D rays whch are the lnes of sght to the movng pont. Because the trajectory of the movng pont s a straght lne the collecton of rays form a \lnear lne complex",.e., they have a common ntersectng lne (the trajectory of the movng pont) as ther kernel. Thus, the problem formulaton s to gure out the condtons (number of vews) for a unque kernel and to follow t wth an algebrac soluton. The algebrac method for recoverng the kernel s based on representng the kernel (the trajectory of the movng pont) wth ts Plucker coordnates. One can then show that each projecton provdes a lnear equaton for the kernel, and thus 5 equatons are necessary for a unque soluton (wth 4 vews one can obtan two solutons usng the quadratc constrant of plucker representaton). So, 5 vews are sucent to lnearly recover for the trajectory of a pont movng on a lne. Further detals and mplementaton can be found n [1]. The problem of orbt determnaton n astrodynamcs s about determnng the orbt (conc secton, typcally ellptc) of body A around body B under a gravtatonal eld. A branch of ths problem ncludes the Fgure 2: In kepleran moton a body sweeps equal areas n equal tme. determnaton of an orbt from drectonal measurements only (lnes of sght). However, the assumpton of moton under a gravtatonal eld constrans not only the shape of the trajectory (conc secton) but also the law of moton along the trajectory n ths case the moton s Kepleran, whch s to say that equal areas are swept durng equal tmes (See Fgure 2). Our work on determnng a conc trajectory from lne of sght measurements (trajectory trangulaton over concs) ders from the classc work on orbt determnaton by that the moton of the pont along the trajectory s arbtrary. In other words, the only assumpton we make s about the shape of the trajectory (conc secton) whle the moton of the pont along the trajectory s unconstraned. Therefore, what we wsh to recover are the followng parameters: the poston

3 of the plane on whch the conc resdes (3 parameters) and the poston, shape and type of the conc (5 parameters). Once these parameters are recovered t becomes a smple matter to determne the 3D coordnates of the movng pont ateach frame of the mage sequence. 3 Trajectory Trangulaton over Concs As mentoned above we wsh to recover from measurements of lne-of-sght only (2D projectons of the movng pont) 8 parameters n general: 3 for the poston of the plane on whch the conc resdes on, and 5 for the conc tself. Once these parameters are recovered the 3D coordnates of the movng pont can be recovered by ntersectng the lne-of-sght wth the conc secton. We propose two methods for recoverng the parameters. The rst method performs a 2D optmzaton (based on conc ttng) on some arbtrary vrtual common plane. The method s very smple, but can only deal wth general concs a-pror constrants on the shape of the 3D conc cannot be enforced due to the projectve dstorton from the conc plane to the vrtual common plane. The second method s slghtly more complex as the optmzaton s performed n 3D (projectve or Eucldean) but enables the enforcement of a-pror constrants on the shape of the conc when the cameras are calbrated. Numercal stablty s greatly enhanced when a-pror nformaton s ntegrated nto the estmaton process. We wll derve the second method for the case of calbrated cameras and when the conc n 3D s a crcle. The extenson to general concs follows n a straghtforward manner but wll not be derved here. 3.1 Method I: 2D Optmzaton on a Common Plane We denote the 3D poston of the movng pont and the camera matrx (projecton matrx) at tme ; = 1::k by P =[X ;Y ;Z ; 1] T and M =[H ; t ], respectvely. The mage measurements are thus p = M P. Our goal s to recover the 3D ponts P, gven the uncalbrated camera matrces M and the mage measurements p. Ths can be formulated as a non-lnear optmzaton problem n whch 8 parameters are to be estmated. The 3 parameters of the normal to the plane n and the 5 parameters of the conc as dened (up to scale) by a symmetrc 3 3 matrx C. Let the sought-after plane on whch the conc resdes on be denoted by. Let A be the 2D homography from mage to some common arbtrary plane (mage plane =1fM 1 =[I; 0]) through the plane, Image 1 Image 2 Image 3 Image 4 Fgure 3: Sketch of method I. The true plane s shown n bold, the guessed plane s shown wth dashed lnes. Choosng a derent plane aects the projecton of the ponts on the rst mage (or common plane n general). If the plane s not the correct one, then the ponts on the rst mage wll not form a conc..e., A p, =1; :::; k must be a conc on the common plane (see Fg. 3). The followng relaton must hold: A =(knkh + t n > ),1 : Therefore, each vew provdes one (non-lnear) constrant: p > A> CA p =0; =1; :::; k: Snce the total number of parameters are 8, and each vew contrbutes one (non-lnear) equaton, then 8 vews are necessary for a soluton (up to a nte-fold ambguty) and 9 vews for a unque soluton. It s possble to solve for n and C by means of numercal optmzaton, or to use an nterleavng approach descrbed below: 1. Start wth an ntal estmate of n. 2. Compute ^p = A p, where A =(knkh +t n),1. 3. Ft a conc C to the ponts ^p. 4. Search over the space of all possble n to mnmze the error term: mnn; err(c; ^p )) There are a number of ponts worth mentonng. The mnmzaton s over 3 parameters only due to step 3 of conc ttng. A large body of lterature s

4 devoted to conc ttng and the numercal bases assocated wth ths problem (cf. [5, 2, 4]). The error term n step 4 s also an mportant choce: the algebrac error p > Cp between a pont p and a conc C s least recommended because of numercal bases. In our mplementaton, for example, we have chosen to mnmze the dstance to the polar lne Cp,.e., err(c; p) = dst(cp;p). Fnally, the search n step 4sacheved (n our mplementaton) by Levenberg- Marquardt optmzaton usng numercal derentaton. Usng Matlab, the optmzaton step conssts of smply callng the leastsq functon. To summarze, ths approach has the two advantages. It s smple and s carred over the 2D plane only. The dsadvantages are, rst, that the method does not facltate a-pror constrants on the shape of the conc, and second, the method nvolves a conc ttng (and evaluaton) stage whch could be challengng on the numercal front. 3.2 Method II: Conc ttng n 3D In ths method the objectve functon s mnmzed n 3D space and s desgned such that t can express a-pror shape constrants, when avalable and when cameras are calbrated. The general dea s that a conc n 3D s represented by the ntersecton of the plane and a quadrc surface. By denng a sutable coordnate system of the quadrc surface one can obtan an 8 parameter objectve functon. In case of calbrated projecton matrces and f a-pror nformaton about the type of conc s gven, say a crcle, then the quadrc surface representaton can be smpled further. We wll derve here a specal case n whch the sought-after conc s a crcle n 3D. In the case of a crcle, we wsh to represent the arrangement of a sphere and a cuttng plane. We expect the total number of parameters to be 6 (three for n and 3 for representng a crcle n the plane), yet a sphere s dened by 4 parameters. Therefore an addtonal constrant s necessary and ths s obtaned by constranng the plane to concde wth the center of the sphere. The detals are below. Let p and M be the projecton and camera matrces of frame =1; :::; k as dened prevously. In case the cameras are calbrated, then the projecton matrces represent the mappng from an Eucldean coordnate system to the mage plane,.e., M = K [R ; u ] where R ;u are the rotatonal and translatonal components of the mappng, and K s an upper-dagonal matrx contanng the nternal parameters of the camera (focal length, aspect rato, prncple pont). For our needs, snce we assume M to be known, we can stll denote M by the composton M =[H ; t ]as was done prevously (thus, at ths juncture t doesn't really matter whether the camera are calbrated or not). Let the 3D coordnates of the movng pont P be denoted (as before) by P =[X ;Y ;Z ] T at tme =1; :::; k. We rst represent P as a functon of n as follows: p = M P : (1) Whch after substtuton becomes: P p = [H t ] 1 (2) H,1 p = P + H,1 t (3) thus, P as a functon of M and p becomes: P = H,1 p, H,1 t : (4) Next, we know that the movng pont resdes on the plane, thus After substtuton we obtan P n +1=0: (5) = (H,1 t ) T n, 1 (H,1 p ) T n : (6) Taken together, eqn. 4 and above, gve rse to: 2 P = 4 X Y n 1X +n 2Y +1,n 3 n whch X ;Y are functons of n (and Z s elmnated by beng expressed as a functon of X ;Y ; n). Let the center of sphere be at the coordnates P c = [X c ;Y c ;Z c ] and ts radus R, thus the ponts P satsfy the constrant: (X, X c ) 2 +(Y, Y c ) 2 +(Z, Z c ) 2, R 2 =0 (7) whch can be wrtten as [P T 1]Q P 1 The 4 4 symmetrc matrx Q s gven by: q 1 Q = B q q 3 A q 1 q 2 q 3 q 4 where q 1 = X c ; q 2 = Y c ; q 3 = Z c ; 3 5 (8) q 4 = X 2 c + Y 2 c + Z 2 c, R2 (9)

5 Snce the center of the crcle P c s on the plane we have: Z c = n 1X c + n 2 Yc+1,n 3 (10) so we need to solve for the three parameters q 1 ;q 2 ;q 4. Taken together, each vew provdes one (non-lnear) constrant (Eq. 7) over 6 parameters n and q 1 ;q 2 ;q 4. Thus, 7 vews are necessary for a unque soluton. As wth Method I, t s possble to solve for the system over 6 parameters or to adopt an nterleavng approach: 1. Start wth an ntal estmate of n. 2. Compute the pont ^P from eqn. 4 and Solve for q 1 ;q 2 ;q 4 (lnear least-squares). P c ;Rfollow by substtuton. 4. Search over the space of all possble n to mnmze the error term: (a) mnn; (dst( ^P ;P c ), R) 2 ; =1; :::; k where the search s done usng numercal optmzaton (leastsq functon of Matlab). 4 Experments We have conducted a number of experments on both synthetc and real mage sequences. We report here a typcal example of a real mage sequence experment. A sequence of 16 mages was taken wth a handheld movng camera vewng a small Lego pece on a turntable. The Lego pece s therefore movng along a crcular path. The rst, mddle and last mages of the sequence are shown n Fg. 4. The projecton matrces were recovered from matchng ponts on the statc calbraton object (the folded chess-board n the background). The corners of the chess-board were the control ponts for a lnear system for solvng for M for each mage. The lnear soluton s not optmal but was good enough for achevng reasonbale results for the trajectory trangulaton experments. A pont on the Lego cube was then (manually) tracked over the sequence and ts mage postons p was recorded. We tested both methods I,II. In general, the 3Dbased optmzaton (method II) always converged from any ntal guess of n (the poston of the plane ). Fg. 6a shows the conc due to the ntal guess that was used for ths experment, for example. The 2Dbased optmzaton (method I) was more senstve to the ntal guess of n, and Fg. 5a showsatypcal ntal guess. The remanng dsplays n Fgs. 5 and 6 (b) (c) Fgure 4: The orgnal mage sequence. (a),(b) and (c) are the rst, mddle and last mages, respectvely n a sequence of 16 mages. The camera s movng manly to the left whle the Lego cube traces a crcle on the turntable.

6 show the projecton of the nal conc (followng convergence of the numercal optmzaton) on the rst, mddle and last mages of the sequence. In method II, the reconstructed ponts n 3D dene a crcle (as t was constraned to begn wth) of a radus 5% o from the ground truth, and around 4 o n orentaton. In method I, the resultng conc had an aspect rato of 0.9 (recall that we solved for a general conc), radus roughly 8% o, and orentaton of the plane was 6 o o. To summarze, both methods generally behave well n terms of convergence from reasonable ntal guesses. Method II was much less senstve to the ntal guess (converged n all our experments) and generally produced more accurate results. 5 Summary and Future Research We have ntroduced a new approach for handlng scenes wth dynamcally movng objects vewed by a monocular movng camera. In a general stuaton, when both the camera and the target are movng wthout any constrans, the problem s not solvable,.e., one cannot recover the 3D poston of the target even when the camera ego-moton s known. In prevous work we have shown that by assumng that the target s movng along a straght 3D lne the problem of recoverng the target's trajectory s unquely solved gven at least ve vews of the movng target. In ths paper we have extended the famly of trajectores to nclude conc sectons as well. In ths context we have ntroduced two methods. The rst method performs the optmzaton on some arbtrary vrtual plane and s very smple. However, t can only deal wth general concs only a-pror constrants on the shape of the 3D conc cannot be enforced due to the projectve dstorton from the conc plane to the vrtual common plane. The second method performs the optmzaton n 3D. The advantage of the second method s that under calbrated cameras t s possble to enforce a-pror constrants on the shape of the conc. For example, we have derved the equatons necessary for recoverng a 3D crcular path. We beleve that future work on the famly of trajectory trangulaton tasks may nclude the followng drectons: Sldng-wndow lnear or conc trajectory ttng. A reconstructon of a generally movng pont can be decomposed onto smaller sub-problems n case many (dense) samples of the movng pont are avlable (lke n contnous moton). A uncaton of statc and dynamc reconstructon. It s possble to estmate whether a pont s statc or movng smply by the sze of the kernel n the case of lnear trajectory trangulaton. A onedmensonal kernel corresponds to a straght-lne path, whereas hgher dmensonal kernels correspond to a sngle pont (statc stuaton). The possblty of recoverng both the camera egomoton and the trajectory (lnear or conc) of the pont. The task s of a mult-lnear nature (for the lnear trajectory trangulaton) and thus there may be an elegant way of decouplng the system as s done n the statc case. Handlng more complex trajectores by trackng multple ponts. If a sucent number of ponts are tracked on a rgd body than the full moton of the object (relatve to the camera ego-moton) can be recovered. It may be nterestng to nvestgate the possble trajectory shapes when fewer ponts are avalable such as two ponts. References [1] S. Avdan and A. Shashua. Trajectory trangulaton of lnes: Reconstructon of a 3D pont movng along a lne from a monocular mage sequence. In Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton, June [2] F.L. Booksten. Fttng conc sectons to scattered data. In Computer Graphcs and Image Processng, pages (9):56{71, [3] P.R. Escobal. Methods of Orbt Determnaton. Kreger Publshng Co., [4] K. Kanatan. Statstcal bas of conc ttng and renormalzaton. In IEEE Transactons on Pattern Analyss and Machne Intellgence, pages 16(3):320{326, [5] P. Meer and Y. Leedan. Estmaton wth blnear constrants n computer vson. In Proceedngs of the Internatonal Conference on Computer Vson, pages 733{738, Bombay, Inda, January [6] Torr P.H.S., Zsserman A., and Murray D. Moton clusterng usng the trlnear constrant over three vews. In Workshop on Geometrcal Modelng and Invarants for Computer Vson. Xdan Unversty Press., 1995.

7 (a) (b) (c) (d) Fgure 5: Usng 2D conc ttng (method I) to recover the planar conc secton. The results are shown by projectng the recovered planar conc (and the 3D ponts traced along the conc) on several reference mages from the sequence. (a) shows the ntal guess wth the rst mage as the reference mage. (b),(c), (d) shows the the results of the 2D conc ttng when the reference mage s the rst, mddle and the last mages of the sequence, respectvely. The resultng conc had an aspect rato of 0.9, radus roughly 8% o, and orentaton of the plane was 6 o o.

8 (a) (b) (c) (d) Fgure 6: Usng 3D sphere ttng (method II) to recover a planar conc secton. The results are shown by projectng the recovered planar conc (and the 3D ponts traced along the conc) on several reference mages from the sequence. (a) shows an extereme ntal guess wth the rst mage as the reference mage. (b),(c), (d) shows the the results of the 3D sphere ttng when the reference mage s the rst, the mddle and the last mages of the sequence, respectvely. The resultng radus of the crcular path was 5% o from the ground truth, and around 4 o o n orentaton.