Deformable Surface Tracking Ambiguities

Transcription

1 Deformable Surface Trackng mbgutes Matheu Salzmann, Vncent Lepett and Pascal Fua Computer Vson Laboratory École Polytechnque Fédérale de Lausanne (EPFL) 5 Lausanne, Swtzerland {matheu.salzmann,vncent.lepett,pascal.fua}epfl.ch bstract We study from a theoretcal standpont the ambgutes that occur when trackng a generc deformable surface under monocular perspectve projecton gven 3 D to 2 D correspondences. We show that, addtonally to the known scale ambguty, a set of potental ambgutes can be clearly dentfed. From ths, we deduce a mnmal set of constrants requred to dsambguate the problem and ncorporate them nto a workng algorthm that runs on real nosy data.. Introducton Wthout a strong model, 3 D shape recovery of nonrgd surfaces from monocular vdeo sequences s a severely under-constraned problem. Pror models are requred to resolve the nherent ambgutes. Many approaches to creatng such models have been proposed, such as physcs-based models [,, 3, 5, 9], feature pont-based structure from moton algorthms [3, 7, 6] and machne learnng technques [2, 2, 8]. However, as wll be dscussed n Secton 2, these methods typcally make restrctve assumptons that prevent them from beng completely general. Furthermore, we are not aware of any formal study of the ambgutes when explctly reconstructng deformable surfaces n the total absence of pror knowledge, or of the number of constrants that must be suppled to resolve them. In ths paper, we address ths ssue from a theoretcal standpont and show how such a theoretcal understandng can be translated nto workng algorthms that make mnmal assumptons on the range of possble surface deformatons. s shown n Fg., we focus here on surfaces that are textured enough to let us establsh 3 D to 2 D correspondences between nterest ponts on the surface and ther mage locatons but whose physcal propertes may be very dfferent. Requrng texture s a lmtng assumpton but our approach nevertheless represents a key step towards desgnng vdeo-based trackng algorthms able to reconstruct the deformatons of classes of deformable surfaces whose be- Fgure. Reconstructon of deformable surfaces from vdeo sequences wth mnmal a pror knowledge. We constran the reconstructon of the deformng sheet of paper and of the much flexble plastc sheet n the same manner, even though they have very dfferent physcal propertes. havor s not known a pror: Gven a robust algorthm able to recover the deformatons of such a surface when t s suffcently textured, t wll become feasble to construct large tranng sets of such deformatons; to use them to learn lowdmensonal deformaton models; and fnally to use these models to recover the shape of surfaces of the same class even though they may be less textured. More specfcally, we model our surfaces as trangulated meshes seen under perspectve projecton. Computng the 3 D coordnates of ts vertces can be acheved by solvng a large lnear system, whose rank and sngular values we can easly compute. Ths wll allow us to show that: Gven suffcently many nose-free correspondences, the coordnates can be retreved up to a sngle scale ambguty. In practce, the mage locatons of the correspondences are never perfect and the resultng ambgutes can be attrbuted to the presence of very small sngular values n the lnear system. These ambgutes actually correspond to those of a pecewse affne model, whch ntroduces an extra depth ambguty for each vertex. The ambgutes can be resolved by consderng a sequence of mages nstead of a sngle one and mposng a very smple dynamcs model that lnks the reconstructons n consecutve mages. Ths results n a much larger lnear system but of full rank thanks to the addtonal moton constrants.

2 We wll show that for surfaces wth physcal propertes as dfferent as those of the sheet of paper and the pece of plastc of Fg., the same set of generc constrants allows us to resolve the ambgutes. s a result, we can retreve ther overall shape as they deform, even though the correspondences we use are automatcally establshed and therefore contan many errors. 2. Related Work Recoverng the shape of a deformng surface n a monocular sequence requres pror knowledge to make the problem tractable. Many dfferent approaches have been studed over the years, most of whch make very strong and restrctve assumptons about the object of nterest. Physcs-based deformaton models have been used extensvely to add a qualtatve knowledge about the object s behavour. The orgnal 2 D models were frst appled to shape recovery [6], but have also been used for 2 D surface regstraton []. They have rapdly been adapted to 3 D under the form of superquadrcs [], trangulated surfaces [3], or thn-plate splnes [9]. To reduce the dmensonalty of the problem, lnearty assumptons have also been made on those models through modal analyss [, 3, 5]. Even though these models have been extremely successful, the mply some knowledge of the pseudo-physcal propertes of the surface, whch may not be avalable. Furthermore, the complexty of modelng a true nonlnear behavor tends to restrct them to cases where nonlneartes are small. Structure from moton methods have also been shown to be effectve. They rely on feature ponts tracked through a sequence to retreve the deformed shape of a surface. common assumpton n such methods s to consder the deformatons as beng a lnear combnaton of bases vectors [7], whch can be learned durng the process [3]. Ths of course does not correspond to the true behavour of a surface whch, by nature, deforms nonlnearly. slghtly dfferent approach s to consder pecewse rgd deformatons [6]. In ths case, rgd objects are movng ndependently, and the moton of the whole scene s consdered as a deformaton. Ths, agan, ntroduces a strong pror, whch n general s not vald for a deformable surface. Machne learnng technques have known an ncreasng popularty n the past few years. They make use of tranng data to buld a model that can then be appled to track objects from monocular mages. Even though nonlnear dmensonalty reducton methods have proved successful n the case of human moton [4], most applcatons to deformable surfaces have been lnear. ctve appearance models [4] poneered ths approach n the 2 D case and have snce been extended to 3 D [8]. Morphable models [2] rely on the same phlosophy to buld 3 D deformable face models. Recently [2] appled ths dea to deformable surface trackng, and created a tranng set of deformed shapes by varyng angles between the facets of a mesh. However, ther tranng data are far from correspondng to realty. Machne learnng methods have proved effcent, but suffer from the need of good tranng sets whch mght be hard to obtan, especally n the case of deformable surfaces. Fnally, t was recently shown that texture and shadng nformaton could be combned to retreve the shape of a deformable surface [5]. However, very strong assumptons on the lghtng envronment must be made, and therefore the method lacks generalty. The ultmate goal of our research s to solve the problem of buldng tranng sets of deformable surfaces from textured objects and wth mnmal pror knowledge of the feasble deformatons. Ths would consttute a good startng pont to learn accurate deformaton models that could then be appled to less textured surfaces of the same knd. We therefore see ths theoretcal study as a necessary step towards fully understandng the problem we are facng and showng that a few reasonable assumptons can make t tractable. 3. Sngle-Image mbgutes We represent surfaces as 3 D trangulated meshes and assume that we are gven a set of 3 D to 2 D correspondences between surface ponts and mage locatons. In ths secton, we show that recoverng the shape amounts to solvng an ll-condtoned lnear system. We then show that the degeneraces, or near-degeneraces, of ths system correspond to depth ambgutes that can be explaned n terms of a pecewse affne projecton model. Snce we use a sngle camera and assume ts nternal parameters to be known, we express all world coordnates n the camera referental for smplcty and wthout loss of generalty. 3.. mbgutes under Perspectve Projecton In ths secton, we formulate the computaton of the 3 D mesh vertex coordnates gven the correspondences n terms of solvng a lnear system and dscuss ts degeneraces. We start wth a mesh contanng a sngle trangle and extend our result to a complete one. Projecton of a 3 D Surface Pont Let x be a 3 D pont whose coordnates are expressed n the camera referental. We wrte ts perspectve projecton as u v = [ I 3 3 ] ( ) x, () k where s the nternal parameters matrx, and k a scalar. If x les on the facet of a trangulated mesh, t can be expressed as a weghted sum of the facet vertces. Eq.

3 becomes u v = (a v + b v 2 + c v 3 ), (2) k where v, 3 are the 3-D coordnate vectors of the vertces and (a, b, c ) the barycentrc coordnates of x. Reconstructng a Sngle Facet Let us assume that we are gven a lst of n such 3 D to 2 D correspondences for ponts lyng nsde one sngle facet. The v, 3 coordnates of ts vertces can be computed by solvng M f `v T v T 2 v T 3 k... k... k n T = wth (3) a b c u v M f = a b c... u v B a n b n c n un C v n where the k are treated as auxlary varables to be recovered as well. We could have hoped that, for n > 4, the columns of M f would become lnearly ndependent and that the system would then have a unque soluton. However, ths s not what happens. To prove that M f s rank-defcent, we show that ts last column can always be wrtten as a lnear combnaton of the others as follows. From Eq. 2 we can wrte u n v n = a n λ + b n λ 2 + c n λ 3 (4) where λ j = v j /k n for j 3. For all < n, we have a λ + b λ 2 + c λ 3 = a v b v 2 c v 3 k n k n k n = k u v. k n Ths mples that the last column of the M f matrx of Eq. 3 s ndeed a lnear combnaton of the prevous ones wth coeffcents (λ T, λ T 2, λ T 3, k /k n,..., k n /k n ). Thus, n general, M f has full rank mnus. Reconstructng the Whole Mesh If we now consder a mesh made of n v > 3 vertces wth a total of m correspondences, Eq. 3 becomes M m B v... v nv k... k m =, M m = ˆ M C L M R, (5) wth a b c b M L = j c j d j... and B C a l c l e l... `u v T M R = `u j v j T B C... `u l v l T Coeffcents smlar to those of Eq. 4 can be derved to compute (u m, v m, ) T as a lnear combnaton of the non-zero columns of the last row. Snce these coeffcents only depend on k m, on the mesh vertces and on the projecton matrx, t can easly be checked that, as n the sngle trangle case, the last column of the matrx can be expressed as a lnear combnaton of the others. Thus matrx M m of Eq. 5 has stll full rank mnus. Ths was to be expected and reflects the well-known scale ambguty n monocular vson. Representng the problem as n Eq. 5 was convenent to dscuss the rank of the matrx. However, n practce, we want to recover the vertex coordnates but are not nterested n havng the k as unknowns. We therefore elmnate them by rewrtng Eq. 5 as a T b T c T b j T j c j T j d j T j... v... = (6) B C a l T l c l T l e l T l... v nv wth «u T = 2 3 3, v 3 where 3 represents the last row of matrx and 2 3 ts frst two rows. By constructon, the matrx n Eq. 6 has the same rank as matrx M m, therefore the followng results are vald for both representatons of the problem. Effectve Rank of the Matrces In the prevous paragraph, we showed that M m has at most full rank mnus one. However, ths does not tell the whole story: In general, t s ll-condtoned and many of ts sngular values are small enough so that, n practce, t should be treated as a matrx of even lower rank. To llustrate ths pont, we projected randomly sampled ponts on the facets of the synthetc 88 vertces mesh of the top row of Fg. 2 usng a known camera model. We then computed the sngular values of matrx of Eq. 6, whch we plot n Fg. 3. In Fg. 4, we show the effect of addng two of the correspondng sngular vectors one assocated to the zero sngular value and the other to a small one to the mesh n ts reference poston.

4 Fgure 2. Reconstructng an 88-vertex mesh usng perfect correspondences that were corrupted usng zero-mean Gaussan nose wth varance fve, whch s much larger than what can be expected of automated matchng technque. Top. The orgnal mesh and reconstructed one projected n the synthetc vew used to create the correspondences. s expected, the projectons match very closely. Bottom. The two meshes seen from a dfferent vewpont. Fgure 3. Sngular values of the matrx of Eq. 6 for the 88 vertex mesh of Fg.. Note how the values drop down after the 2n v = 76 th one, as predcted by the affne model of Secton 3.2. The small graph on the rght s a magnfed verson of the part of the graph contanng the small sngular values. The last one s zero up to the precson of the matlab routne used to compute t and the others are not very much larger. (a) (b) (c) Fgure 4. Vsualzng vectors assocated to small sngular values. (a) Reference mesh and mesh to whch one the vectors has been added seen from the orgnal vewpont, n whch they are almost ndstngushable. (b) The same two meshes seen from a dfferent vewpont. (c) The reference mesh modfed by addng the vector assocated to the zero sngular value. Note that the resultng deformaton s close to beng a scalng. Even though only one of these values s exactly zero, we can see that they drop down drastcally after the frst 2n v = 76. Ths shows that, even though the matrx may have full rank mnus, the soluton of the lnear system would be very senstve to nose. Therefore, n a real stuaton, we would actually be closer to havng n v ambgutes, whch can be understood n terms of the pecewse affne model we ntroduce below mbgutes under Pecewse ffne Projecton pecewse affne camera model s one that nvolves an affne transform for each facet of the mesh. Ths approxmaton s warranted f the facets are small enough to neglect depth varatons across them. Projecton of a 3 D Surface Pont Let x be a 3 D pont whose coordnates are agan expressed n the camera referental. We wrte ts projecton to a 2 D mage plane as ( ) u k = Px v, P = [ I 2 2 ], (7) where k s a depth factor assocated to the affne camera and s a 2 2 matrx representng the nternal parameters. s n Secton 3., we study the ambgutes for a mesh contanng frst a sngle trangle and then many. Reconstructng a Sngle Facet We can agan wrte a lnear system for a sngle trangle contanng n 3 D to 2 D correspondences, wth 3 D ponts gven by ther barycentrc coordnates «u a P b P c P v «v u a P b P c P Bv 2 C v v 3 =. (8) B «C k un a np b np c np v n Snce we only have one facet, we also only have one projecton matrx, therefore a sngle k correspondng to the average depth of the facet s necessary, and all (u, v ) T can be put n the same column. Snce P s of sze 2 3, t has at most rank 2. Moreover, we can show that the last column of the global matrx also s a lnear combnaton of the two frst columns of P «u v = P k (a v + b v 2 + c v 3 ) = ˆ k (a v + b v 2 + c v 3 ) = a «v k + b v 2 k «v2 + c v 22 k «v3. (9) v 32 The coeffcents of Eq. 9 are ndependent of the correspondence consdered and are therefore vald for any row of the matrx. Ths fnally means that, when n 3, the rank of the matrx of Eq. 8 s always 6.

5 Reconstructng the Whole Mesh s dscussed above, when there are several trangles, usng the pecewse affne model amounts to ntroducng a projecton matrx per facet. However, snce n realty we only have one camera, ts nternal parameters, rotaton matrx, and center are bound to be the same for each trangle. Ths only lets us wth a varable depth factor k for each facet among the n f facets of the mesh. We can then wrte the system M m wth v T... v nv T k... k nf T =, M m = ˆ M L M R «u a P b P c P v « uj M m = b jp c jp d jp v j «B ul C a l P c l P e l P v l M L, whch s of sze 2m 3n v, m beng the number of correspondences, has at most rank 2n v because P has rank 2. Smlarly, M R, whch s of sze 2m n f, has at most rank n f, because we can agan show that ts last column s a lnear combnaton of the prevous one n a smlar manner as was done for the perspectve case, wth the coeffcents of Eq. 9. Ths means that for a full mesh, M m has at most rank 2n v + n f. Ths leaves us wth n v + ambgutes. Ths agan seems natural due frst to the same scale ambguty as n the perspectve case, and second to the fact that now each vertex s free to move along the lne of sght. Ths number corresponds to the number observed n the perspectve case of Secton 3., except that, n the affne case, a global scale s dfferent from all vertces sldng along the lne of sght, whch produces an extra sngular value. 4. Weak but Broadly pplcable Constrants (). Snce the lnear systems of Secton 3 are rank-defcent, we need to ntroduce addtonal constrants to obtan acceptable solutons. In essence, ths s what all the model-based methods dscussed n Secton 2 do. However, they typcally nvolve very specfc assumptons about ether the physcal propertes or the range of possble deformatons of the surfaces at hand, whch s very restrctve. In ths secton, we show that a much weaker and more broadly applcable set of constrants suffces: Snce we deal wth vdeo sequences, we can assume that the surface does not move randomly between two frames, whatever the physcal propertes of the target surface. We therefore perform the reconstructon over several frames smultaneously and smply lmt the range of moton from frame to frame. We show here that ths can be expressed as a set of addtonal lnear constrants that make our lnear systems wellcondtoned, frst n the affne case and then n the projectve one. 4.. Constranng the ffne Reconstructon Gven a temporal sequence of n I mages and the correspondng matrces M mt, t n I of Eq., we can create a block dagonal matrx whose blocks are the M mt and use t to wrte a bg lnear system that the vertex coordnates n all frames must satsfy smultaneously. However, wthout temporal consstency constrants, the ambgutes reman: s dscussed n Secton 3.2, when the camera coordnates are algned wth the world coordnates, reconstructon s only possble up to an unknown moton along the z-axs for each vertex at each tme step. To mtgate ths problem, t s therefore natural to lnk the z value of vertces across tme. The smplest way to do ths s to wrte v t+ z v t z = () for all vertces and all tmes. These constrants and those mposed by the 3 D to 2 D correspondences can then be mposed smultaneously by solvng wth respect to Θ where Θ = M s = M U = M B = z frst v T... v T nv k... kn f» MU, b = M B M s Θ = b, (2) z frst... zn frst v... v n I T n... v I T n n v k I... k n T I n f, M m M m t B M m t+..., C M m n I C C C C C......, B C C C C T z last... zn last v, and z last are the z-coordnate of vertex n the frst and last frames, n whch we assume that the shape s known, and C s an n v 3n v matrx contanng a sngle n each row, whch corresponds to the z-coordnate of one vertex. The number of constrants we add n ths manner s equal to the number n v of ambgutes that we derved n Secton 3.2. Therefore t affects the rank of M s, and reduces the number of ambgutes to zero as shown n Fg. 5. Moreover, these constrants do not overlap wth the ones mposed by the correspondences and can then be consdered as mnmal.

6 Fgure 5. Sngular values for a 5 frames sequence under affne projecton. Left Wthout temporal consstency constrants between frames, the lnear system has many zero sngular values, whch mples severe reconstructon ambgutes. Rght Constranng the z coordnates as dscussed n Secton 4. leaves the non zero sngular values unchanged but ncrease the value of the others, thus removng the ambgutes. In practce the correspondences are never perfect and nclude nose and outlers. We therefore solve Eq. 2 n the least-squares sense and take Θ to be Θ = argmn Θ (M s Θ b) T W(M s Θ b), (3) where W s a dagonal matrx of ones for the lnes correspondng to projecton constrants and a user-defned weght for those that correspond to the depth constrants. The weght s desgned to gve comparable nfluence to both classes of constrants and drectly affects how much the small sngular values ncrease Constranng the Perspectve Reconstructon In Secton 3.2, we showed that ambgutes under perspectve projecton are smlar to those under pecewse affne projecton. It s therefore natural to constran the reconstructon n a smlar way, that s by lmtng the moton along the lne-or-sght. However, snce t s not parallel to the z-axs anymore, the constrants become more dffcult to express. Let us consder one vertex v of the mesh at tmes t and t +. We can try mnmzng d = v t v t+ e, the length of the projecton on the lne-of-sght of the v t v t+ vector, where e s the vector cv t after normalzaton, and c represents the optcal center of the camera. The dffculty comes from the fact that ths constrant s nonlnear and can therefore not be ntroduced nto our lnear formulaton. We overcome ths problem by replacng the exact formulaton of d by an upper bound that can be expressed lnearly as follows: d 2 = (v t v t+ e) 2, = (e x(xc t+ x t c) + e y(yc t+ yc) t + e z(zc t+ zc)) t 2, (e x(xc t+ x t c ))2 + (e y(yc t+ yc t ))2 + (e z(zc t+ zc t ))2 + ( 2e x(x t+ c x t c)) 2 + ( 2e y(y t+ c y t c)) 2 + ( 2e x(x t+ c x t c ))2 + ( 2e z(z t+ c z t c ))2 + ( 2e y(y t+ c y t c ))2 + ( 2e z(z t+ c z t c ))2, (sn(θ max x )( + 2 2)(x t+ c x t c ))2 + (sn(θ max y )( + 2 2)(y t+ c y t c ))2 + (( + 2 2)(z t+ c z t c ))2, (4) Fgure 6. Sngular values for a 5 frames sequence under perspectve projecton. Left Wthout temporal consstency constrants between frames, the lnear system s rank-defcent. Rght Boundng the frame-to-frame dsplacements along the lne of sght usng the lnear expresson of Eq. 4 transforms the ll-condtoned lnear system nto a well-condtoned one. The smaller sngular values have ncreased and are now clearly non-zero. Snce our moton model ntroduces more equatons than strctly necessary, the other values are also affected, but only very slghtly. where x c, y c and z c are the coordnates of a vertex n the camera reference system, and θx max and θy max are the maxmum angles between the camera center and the ponts projectng on the left/rght, and upper/lower border of the mage, respectvely. s n Secton 4., these constrants and those mposed by the 3 D to 2 D correspondences can be mposed smultaneously. We rewrte Eq. 2 by replacng the M mt matrces of Eq. by the matrx of Eq. 6 and the C matrces by 3n v 3n v matrces, contanng a sngle value n each row that wll constran the x-, y-, or z-coordnate of one vertex. Ths value s set to one of the three coeffcents of Eq. 4, dependng on whch coordnate the row corresponds to. Fg. 6 shows how the sngular values of the system are affected by ntroducng our depth constrants. s n the affne case, we can see that the smaller sngular values have ncreased and now clearly dfferent from zero. Snce ths was our only goal n addng constrants, ths justfes our approach to lberalzaton by mnmzng the upper bound of d of Eq. 4 nstead of d tself. Note that because we added more equatons than was strctly necessary, the other sngular values also ncreased, but only very slghtly. 5. Experments In the prevous sectons, we developed theoretcal bass for reconstructng the shape of a deformable surface from 3 D to 2 D correspondences n a vdeo sequence. We showed that constranng the varatons n depth from frame to frame s suffcent, n theory, to formulate the reconstructon problem n terms of solvng a well-condtoned lnear system. In ths secton, we show that ths ndeed produces vald reconstructons n practce. We present results obtaned usng both synthetc data and real mages. In both cases, the deformatons of the meshes were retreved by solvng the lnear system of Secton 4 for whole sequences wth known deformatons n the frst and last frames. Ths was done usng Matlab s mplementaton of sparse matrces and resoluton of lnear systems

7 Fgure 7. Dstance between the orgnal mesh and ts reconstructon for each one of the 9 deformed versons of the mesh of Fg. 2. We plot fve curves correspondng to vertex-to-surface dstances obtaned wth varance one to fve gaussan nose on the correspondences. The dstances are expressed as percentages of the length of the mesh largest sde. wth known covarance matrx n the least square sense. In our experments, the covarance matrx smply s the weght matrx of Eq. 3, whch weghs dfferently the correspondences equatons and the constrants. 5.. Synthetc Data We deformed the 88-vertex mesh of Fg. 4(a) to produce 9 dfferent shapes and 9 correspondng sets of 3 D to 2 D correspondences usng a perspectve projecton matrx. We then added Gaussan nose wth mean zero and varance rangng from one to fve to the mage locatons of these correspondences. Fg. 2 depcts the reconstructon results overlad on the orgnal mesh wth nose varance fve. The dfferences are hard to see, even though ths represents far lower precson than what can be expected of good featurepont matchng algorthms. To quantfy the dfferences between the meshes, we plot the dstances between the two meshes n Fg. 7 for each one of 9 dfferent shapes, gven ncreasng nose varance. The dstances are expressed as percentages of the mesh largest sde. Wth a nose varance one, they are of the order of.25% for vertex-to-surface dstance, whch works out to.25cm for a cm 7cm mesh. Ths s very small gven that we ncorporate very lttle a pror knowledge nto our reconstructon algorthm Real Data We now present results on two real monocular vdeo sequences acqured wth an ordnary dgtal camera. The longest one s 25 frames long, whch shows that, even though our approach nvolves solvng a very large system, t s sparse enough to use a standard Matlab routne. In both cases, we automatcally establsh 3 D to 2 D correspondences between the frst frame, where the 3 D pose s assumed to be known, and the others by frst trackng the surface n 2 D usng normalzed cross-correlaton. We then compute correspondences by pckng random samples n each facet and lookng n each frame n an area lmted by the 2 D trackng result for 2 D ponts matchng ther projectons n the frst frame. To ths end we use standard crosscorrelaton, whch results n nosy correspondences wth a number of msmatches at places where there s not enough texture to guarantee relable matches. Fg. 8 depcts our reconstructon results for a relatvely nelastc pece of paper n a 25-frame sequence and Fg. 9 those for a much more flexble sheet of plastc n a 47- frame sequence. In both cases, the global shape s correct, whch confrms that the ambgutes have been correctly handled. However, because we mpose no smoothness constrant of any knd, there are also local errors that are caused by the msmatches present n our nput data. If the goal were to derve a perfect shape from a set of nosy correspondences, we could mtgate the effect of erroneous matches by ntroducng a robust estmator nto the leastsquares mnmzaton of Eq. 3. However, we wll argue n the followng secton that ths may not actually be necessary for the applcaton we have n mnd. Snce our technque does not ntroduce any pror on the physcal propertes of the target surface, we were able to reconstruct both the paper and plastc wthout changng anythng to our system. It s not clear that ths would have been the case had we used a physcs-based approach or any other that mplctly lmts the range of deformaton of the surface. 6. Concluson In ths paper we have presented a theoretcal study of the ambgutes that arse when reconstructng deformable 3 D surfaces from monocular vdeo sequences. We showed that they can be nterpreted n terms of those nherent to a pecewse affne model and can be removed by smply constranng the frame-to-frame varaton n depth. These are very weak constrants that are broadly applcable because they do restrct the range of possble surface deformatons. When used n conjuncton wth real correspondences, ncludng nose and outrght msmatches, these constrants are suffcent to recover the surface, not perfectly, but wth good accuracy nevertheless. More specfcally, we do not smooth our results at all because t would defeat our basc purpose, whch s to ntroduce as lttle a pror knowledge of the surface s physcal propertes as possble. s a result, our reconstructons may contan local devatons from the true surface. However, we do not beleve ths to be a major ssue gven our ultmate purpose: If the goal s to track many surfaces to create a moton database from whch a moton model can be learned, the devatons can be treated as random perturbatons that wll be elmnated when observng a large number of sequences. Provng ths to be the case wll be the focus of our future work.

8 Fgure 8. Reconstructng a deformng sheet of a paper from a 25-frames sequence. Top The reconstructed mesh s reprojected nto the orgnal mages and closely matches the outlne of the paper. Bottom The same mesh seen from the sde. In spte of local naccuraces n depth, the overall shape s correct, whch ndcates that the ambgutes have been successfully resolved. complete vdeo s submtted as supplementary materal. Fgure 9. Reconstructon results for a plastc sheet, whch s much more flexble than the sheet of paper of Fg. 8. In spte of ths, the overall shape s agan correctly recovered up to small errors due to erroneous correspondences. complete vdeo s submtted as supplementary materal. References []. Bartol and. Zsserman. Drect Estmaton of Non- Rgd Regstraton. In Brtsh Machne Vson Conference, Kngston, UK, September 24. [2] V. Blanz and T. Vetter. Morphable Model for The Synthess of 3 D Faces. In CM SIGGRPH, pages 87 94, Los ngeles, C, ugust 999. [3] L. Cohen and I. Cohen. Deformable models for 3-d medcal mages usng fnte elements and balloons. In Conference on Computer Vson and Pattern Recognton, pages , 992. [4] T. Cootes, G. Edwards, and C. Taylor. ctve ppearance Models. In European Conference on Computer Vson, pages , Freburg, Germany, June 998. [5] H. Delngette, M. Hebert, and K. Ikeuch. Deformable surfaces: free-form shape representaton. In SPIE Geometrc Methods n Computer Vson, volume 57, pages 2 3, 99. [6] M. Kass,. Wtkn, and D. Terzopoulos. Snakes: ctve Contour Models. Internatonal Journal of Computer Vson, (4):32 33, 988. [7] X. Llado,. D. Bue, and L. gapto. Non-rgd 3D Factorzaton for Projectve Reconstructon. In Brtsh Machne Vson Conference, Oxford, UK, September 25. [8] I. Matthews and S. Baker. ctve ppearance Models Revsted. Internatonal Journal of Computer Vson, 6:35 64, November 24. [9] T. McInerney and D. Terzopoulos. Fnte Element Model for 3D Shape Reconstructon and Nonrgd Moton Trackng. In Internatonal Conference on Computer Vson, pages , Berln, Germany, 993. [] D. Metaxas and D. Terzopoulos. Constraned deformable superquadrcs and nonrgd moton trackng. IEEE Transactons on Pattern nalyss and Machne Intellgence, 5(6):58 59, 993. []. Pentland. utomatc extracton of deformable part models. Internatonal Journal of Computer Vson, 4(2):7 26, 99. [2] M. Salzmann, S. Ilć, and P. Fua. Physcally Vald Shape Parameterzaton for Monocular 3 D Deformable Surface Trackng. In Brtsh Machne Vson Conference, Oxford, UK, September 25. [3] L. Torresan,. Hertzmann, and C. Bregler. Learnng nonrgd 3d shape from 2d moton. In dvances n Neural Informaton Processng Systems. MIT Press, Cambrdge, M, 23. [4] R. Urtasun, D. Fleet,. Hertzman, and P. Fua. Prors for people trackng from small tranng sets. In Internatonal Conference on Computer Vson, Bejng, Chna, October 25. [5] R. Whte and D. Forsyth. Combnng cues: Shape from shadng and texture. In Conference on Computer Vson and Pattern Recognton, 26. [6] J. Xao and T. Kanade. Uncalbrated perspectve reconstructon of deformable structures. In Internatonal Conference on Computer Vson, 25.