A Factorization Approach to Structure from Motion with Shape Priors

Transcription

1 A Factorzaton Approach to Structure from Moton wth Shape Prors Alesso Del Bue Insttute for Systems and Robotcs Insttuto Superor Técnco Av. Rovsco Pas Lsboa Portugal adb/ Abstract Ths paper presents an approach for ncludng 3D pror models nto a factorzaton framework for structure from moton. The proposed method computes a closed-form affne ft whch mxes the nformaton from the data and the 3D pror on the shape structure. Moreover, t s general n regards to dfferent classes of objects treated: rgd, artculated and deformable. The ncluson of the shape pror may ad the nference of camera moton and 3D structure components whenever the data s degenerate (.e. nearly planar moton of the projected shape). Afnal non-lnear optmzaton stage, whch ncludes the shape prors as a quadratc cost, upgrades the affne ft to metrc. Results on real and synthetc mage sequences, whch present predomnant degenerate moton, make clear the mprovements over the 3D reconstructon. 1. Introducton Extractng 3D data from monocular mage sequences s a problem extensvely studed n Computer Vson. The dea embracng varous scenaros s relatvely smlar: to nfer both the 3D structure parametrzaton and the camera parameters from a set of 2D ponts extracted from a sequence whch depcts a movng object. In ths context, methods based on the blnear factorzaton of the mage data have been very successful n both proposng smple and closedform approaches wth the least assumptons as possble. Back n the early 90s, Tomas and Kanade [14] ntroduced the frst factorzaton algorthm dealng wth a rgd object vewed by a smple orthographc camera. Later on, studes were mostly focused on upgradng the approach to more complex vewng condtons such as paraperspectve [11] and projectve [13]. Only recently, the framework was extended to deal wth objects whch may also vary ther shape. Deformable [5] and artculated [17, 20] factorzatons are among these examples. However, one of the man problems of Structure from Moton (SfM) conssts n the hgher complexty n whch shapes may vary ther 3D structure. Ths may add nonlneartes and strong dependences between both shape and moton components resultng n non-trval solutons of the problem. Lkewse, the hgher number of degrees of freedom and the possble moton degeneraces n the measured data may lead to a local soluton whch correctly mnmzes the 2D reprojecton error but, however, results n a poor or even meanngless 3D reconstructon. In order to counter ths effect, pror nformaton may be ncluded to obtan relable 3D reconstructons. Dfferent prors have been shown to mprove performances n rgd and non-rgd SfM. Forsyth et al. [7] frst proposed prors over the specfc camera constrants n a consstent Bayesan framework. Xao et al. [19] use the pror nformaton over a set of ndependent shapes to compute a closed-form soluton. In a face modellng context, Solem and Kahl [12] used a learned shape model to ad the 3D nference over regons wth no 2D nformaton avalable. Del Bue et al. [6] enforce prors over the rgdty of some ponts to obtan relable estmatons of the object rgd components. Torresan et al. [15] propose the use of gaussan prors over the deformaton parameters n order to avod arbtrarly varatons. Fnally, Olsen and Bartol [10] mpose a pror over temporal varatons of the camera parameters combned wth constrants over the proxmty of projected 2D ponts and reconstructed 3D ponts. Dfferently from the mentoned solutons, prors are here ntroduced n the form of prevously computed 3D shapes. In such way, t s possble to obtan a descrpton of the object shape jontly gven by the measured data and the pror nformaton avalable. Ths approach especally supports the computaton of relable 3D shapes whenever the mage sequence contans strong degeneraces. For nstance, a common case s a talkng head n front of a camera performng tny pose changes. In ths case depth would be lost f not rremedably mxed wth the ongong deformatons. A known 3D metrc descrpton of the subject face may however recover the lost depth and dsambguate better moton and shape components. At ths end, the proposed computatonal methods drectly nclude the 3D metrc prors n a /08/$ IEEE

2 factorzaton framework for SfM. Frst, the approach uses the data and the pror to compute an ntal affne soluton whch s then upgraded to metrc wth an teratve non-lnear optmzaton procedure. The next secton provdes an ntroducton to SfM methods showng how dfferent problems can be treated wth a unque approach. Secton 3 shows how to compute a metrc soluton for the moton and structure components The proposed algorthm s then explaned n Secton 4 showng computatonal tools to compute 3D models gven pror nformaton. The experments n Secton 5 show real and synthetc examples of 3D reconstructon n such cases. 2. Factorzaton for SfM The key dea n SfM s to gather all the 2D mage coordnates lyng on a generc shape at each frame n a sngle measurement matrx W. The locaton of a pont j n a certan frame can be defned by a vector w j =(u j v j ) T where u j and v j are the horzontal and vertcal mage coordnates respectvely. A compact matrx representaton can be expressed as: w w 1p W =..... w f1... w fp = W 1. W f (1) where f s the total number of mage frames and p the number of mage ponts. The mage trajectores stored n W can be expressed as a blnear product as W = M 2f r S r p.the matrces M and S refer to the moton and shape subspaces respectvely wth dmenson r where r mn{2f,p}. As a result, the rank of W s constraned to be rank{w} r. The matrces M and S can be further decomposed n: M = M 1. M f S = S 1 S p where M wth =1...f s a 2 r matrx projectng the shape onto the mage frame. The sze of M drectly depends on the type of camera and moton that appears n the scene. The component S j wth j =1...ps a r-vector that defnes the 3D parametrzaton for each pont j and ts sze depends on the shape propertes (.e. rgd or non-rgd) Rgd factorzaton The projecton of a 3D rgd shape by means of an orthographc camera s hstorcally the frst factorzaton problem studed [14]. In ths case the camera moton M and the 3D (2) pont S j canbeexpressedas: r1 r M = 2 r 3 r 4 r 5 r 6 t u t v = R t S j = X j Y j Z j 1 T = Xj 1 (3) where R contans the frst two rows of a rotaton matrx (.e. R R T = I 2 2 ), S j s a 4-vector contanng the homogeneous metrc coordnates of the 3D pont X j,andt s a 2-vector representng a translaton nto the mage plane. Every pont belongng to the rgd structure shares the same rotaton and translaton, thus we can compact 3D ponts n a sngle 4 p matrx S gvng: W = R t S1 S p = M S. (4) Stackng the rows of W for every frame, we obtan the full measurement matrx as: W = MS = R t S (5) where R s the 2f 3 collecton of f rotaton matrces, t s a 2f-vector whch contans the translaton for every frame. The dmenson of M and S s fxed to r =4. If the 2D ponts n W are regstered to the shape centrod (.e. W1 T = 0), the maxmum dmensonalty of each subspace s r =3and equaton (5) can be wrtten as: W = MS = R X 1 X p. (6) 2.2. Artculated factorzaton If the measurements n W belong to two ndependent movng objects, the overall rank sums to eght snce t s possble to wrte moton and shape components as: h such that: M = S = R (1) t (1) R (2) t (2) S (1) 0 0 S (2) h W = W (1) W (2) (7) = M S (8) where W (1) and W (2) are the measured data at frame for the frst and second shape respectvely. The components of M and S for each shape are n the form as shown by equaton (4) for a sngle rgd object. However, the rank r may decrease f the movng objects show a dependency such as a common rotatonal axs. In artculated SfM [17, 20], ths dependency s gven by the jonts whch constran the degrees of freedom of the movng objects. In the case of a unversal jont [17] the dstance between the center of the shapes s constant (for nstance, the head and the torso of a human body) but they

3 show ndependent rotaton components. At each frame the shapes connected by a jont satsfy: t (1) + R (1) d (1) = t (2) + R (2) d (2) (9) where t (1) and t (2) are the 2D mage centrod of the two objects, R (1) and R (2) the 2 3 orthographc camera matrces and d (1) and d (2) the 3D dsplacement vectors of each shape from the jont. The relaton n equaton (9) gves the reduced dmensonalty n the moton and shape subspaces. Thus, the shape matrx S can be wrtten as: S (1) d (1) S = 0 S (2) d (2) (10) 1 1 where S s a full rank-7 matrx. The moton for a frame has to be accordngly arranged to satsfy equaton (9) as: h M = R (1) R (2) t (1). (11) Further detals can be found n [17], alongsde a descrpton of addtonal jont models. Notce that s necessary to solve an assgnment problem n order to form W as n equaton (8). At ths end, a recent approach able to detect artculated parts from mages s presented n [20] Deformable factorzaton In the case of deformable objects, a sngle shape vares ts 3D structure wth respect to a set of deformaton modes. The number of modes used to defne the shape deformatons results n a specfc rank-constrant over the mage trajectores n W. The representaton for the deformatons s a smple model where any specfc3dconfguraton X s approxmated by a lnear combnaton of a set of k bass shapes B d whch represent the prncpal modes of deformaton: kx X = c d B d X, B d < 3 p c d < (12) d=1 Bregleretal. [5]werethefrst to propose an extenson of factorzaton algorthms able to deal wth the case of deformable shapes assumng an orthographc camera model. In ths case, the coordnates of the 2D mage ponts observed at each frame are related to the coordnates of the 3D ponts accordng to the followng equaton: Ã kx! W = R c d B d + T (13) d=1 where c d s the confguraton weght for bass d at frame. When the mage coordnates are regstered to the object s centrod, equaton (13) can be rewrtten as: W = B 1 c 1 R... c k R. = M S (14) B k Agan, by stackng the measurements at each frame we obtan the followng compact matrx form: c 11 R 1... c 1k R 1 B 1 W = = MS (15) c f1 R f... c fk R f B k Snce M s a 2f 3k matrx and S s a 3k p matrx, the rank of W when no nose s present must be r 3k. 3. Affne and metrc reconstructon In order to extract the moton and shape components, the classcal procedure solves two separate problems: 1. Fnd an affne ft M and S to the measured data W gven the rank r. 2. Enforce the metrc structure by computng a correctve transform Q r r such that W = MQ Q 1 S = MS. The frst step can be trvally solved usng any rank revealng technque such as SVD. However ths soluton s merely one means of numercal computatons, other approaches may be used as ponted out n [9]. Gven the chosen rgd/non-rgd model, the numercal rank t s fxed to r and thus t s possble to approxmate the decomposton as: W SV D nx u σ v T = =1 rx u σ v T = U r Σ r V T r (16) =1 where U r s a 2f r orthogonal matrx, Σ r a r r dagonal matrx and V T r a p r orthogonal matrx. After smple operatons the product can be arranged n the affne form W = M S. Notce that ths ntal soluton s ndependent from the problem consdered, ether the consdered shape s rgd or non-rgd. The second step, the computaton of Q, s where the metrc propertes are mposed. For nstance, n the rgd orthographc case, ths leads to the computaton of the transformaton Q 3 3 whch renders each row of M orthonormal (.e. M R ). In the non-rgd case the computaton s more complex snce the relatonshps n M are strongly non-lnear. For nstance, n the deformable case, a closed form soluton can be found only f a correct set of ndependent bases s chosen [19]. A wrong set of bases may lead to naccurate solutons as shown n [4]. On the other hand, non-lnear optmzaton or hybrd approaches have been proposed to solve the problem [4, 15]. In the case of artculated shapes, solutons are avalable [17] but based on the correct knowledge of the type of jont connectng the bodes. 4. Factorzaton wth shape prors It s clear from the prevous secton that the metrc upgrade s dependent on the ntal affne decomposton. An

4 naccurate affne ft of M and S may rremedably compromse the followng metrc upgrade. In ths sense, the ntroducton of a pror on the values of the shape subspace S can brng the estmaton close to the desred soluton. Moreover, shape prors can support the computaton n case of defcent components n the moton subspace M a lkely case when the mage moton s weak. In such cases, t s always possble to compute a factorzaton whch well ft W numercally but nonetheless the factors M and S may contan meanngless components. Ths problem s hard to solve unless pror nformaton s ncluded n the computaton Measurements and prors Shape prors are ntroduced as a matrx L < l p whch holds a parametrzaton of each mage trajectory stored n W. A pont trajectory w j can be wrtten as a 2f-vector such that W =[w 1 w p ]. Thus accordngly, we can wrte the shape pror 1 as L =[l 1 l p ]. The sze of each l-vector l j depends on the type of pror chosen. Often L may represent a rgd 3D shape leadng to l =3. Such pror may be used n deformable SfM to obtan a more relable estmaton of an object wth complex shape varatons. Alternatvely, t may support the estmaton of the object s depth when the shape s movng planarly. In general, L may as well store more complex shape descrptons (.e. l > 3) suchasa set of deformable bass shapes computed prevously from a smlar shape. The man dea s to jon the nformaton stored n L wth the avalable measurements n W n order to extract an affne ft whch s dependent on both components. Ths can be formulated as two blnear models for the data: SJ W = M 2f t S t p =[M J M I ] (17) S I and for the shape pror: L = N l l S J (18) where the J subscrpt refers to the components obtaned by the jont space between pror and mage measurements whle the I refers to the remanng ones. Notce that we always consder L beng full rank thus the followng propertes hold: µ W rank (W)=r, rank (L)=l and rank =t (19) L where t =max{r, l} s the overall rank for both pror and measurements. 1 Notce that both the pror and measurements are regstered to the respectve centrods.e. W1 p 1 = 0 2f 1 and L1 p 1 = 0 l Generalzed sngular value decomposton Once the shape pror and the data are defned, we seek a computatonal soluton able to fnd the jont factorzaton for W and L. Ths can be obtaned usng a Generalzed Sngular Value Decomposton (GSVD) whch can decompose the matrx par {W, L} as: W = UD U X T L = VD V X T (20) where X T s a p p matrx whch span the common row space of {W, L }, U s a 2f 2f matrx wth orthonormal columns (U T U = I)andVs a l l matrx such that V T V = I. Notce that the matrx X s rank defcent wth t non-zero sngular values (for more detals and proofs on GSVD see [2], Sec ). The dagonal value matrces D U and D V of sze 2f p and l p are arranged as: D U = ΣU 0 0 I and D V = ΣV (21) The dagonal matrces Σ U = dag(σ 1,...,σ l ) and Σ V = dag(μ 1,...,μ l ) of sze l l are constraned such that Σ 2 U + Σ 2 V = I and they have the dagonal elements ordered as: 0 σ 1... σ l 1 and 1 μ 1... μ l > 0 The rato between the dagonal values are called Generalzed Sngular Values (GSV) and they are defned as γ = σ /μ. As a further note, the data and pror matrces are usually pre-scaled such that kwk 2 =klk 2. Ths condton [8] guarantees a well-condtonng of the matrx X and t s the only data scalng performed n the decomposton Generalzed factorzaton for SfM The GSVD decomposes the mage measurements n W wth a common row space whch s dependent on both the measured data stored n W and the 3D pror nformaton stored n L. However, to obtan a soluton n the form of equaton (17), we must reduce the decomposton of W gven by GSVD nto the two standard affne components M 2f t and S t p. Thus, t s convenent to splt X n the components whch are dependent on the pror (the frst l) and the one dependent on the data (the remanng p l)suchthat: X = X J X I. (22) In such way, we am to preserve the common row space X T J component whch was computed by GSVD. Thus, equaton (20) can be accordngly separated n two components W J and W I gvng: W = W J + W I = U J Σ U X T J + U IX T I. (23) The matrx X T J of sze l p alone contans the row space components whch mxes measurements and prors. The

5 row space n X I of sze (p l) l stll entals the rank defcency and t requres a further decomposton to extract the remanng t l components. In order to obtan ths further affne ft, the proposed method performs two projectons of W I : frst over the subspace defned by X T J and then along ts orthogonal complement. Ths s obtaned by defnng the orthogonal projector P such that: P = X J X T 1 J X J X T J (24) gvng: W I = W I P + W I P = W 0 + W 00 (25) where P = I P whch has the result to further splt the remanng components n W 0 whch stll belongs to the subspace gven the pror and W 00 whch s ts orthogonal complement. The components n W 00 can be reduced va SVD to obtan the remanng t l components gvng: whch can be re-arranged as: W 00 SV D U c D c V T c (26) M I = U c D c and S I = V T c (27) where M I s a 2f (t l) matrx and S I a (t l) p matrx. The remanng data W 0 projected along X T J s merged to the jont space obtanng the measurements W g such that: W g = W J + W 0 = U J Σ U X T J + U I X T I X J X T 1 J X J X T J = ³U J Σ U + U I X T I X J X T 1 J X J X T J = M J S J where S J = X T J and M J s gven by the remanng factors. Gven the factors M J, M I, S J and S I, t s possble now to form the blnear decomposton as n equaton (17) Fndng a metrc soluton For the case of rgd shapes, the affne ft gvenbythe prors can be fnally upgraded to metrc by forcng metrc constrants n closed form for dfferent type of cameras [9]. Dfferently for non-rgd shapes, we opt for a non-lnear optmzaton stage based on bundle adjustment [18] where a pror on the rgd bass shape s ncluded as an addtonal quadratccost (.e. L s a 3 p matrx). In prncple, the pror may correspond to a full parametrzaton of a deformable shape,.e. l>3, however here we focus on prors whch are descrbng the rgd component of a deformable object. Thus, the cost functon mnmzed reflects the deformable model presented n Secton 2.3 gvng: X X mn kw j (R c d B dj )k 2 + X kb 1j Cl j k 2 R B dj c d,j j d where B dj s the 3 1 bass component for the pont j such that B d =[B d1 B dp ]. The matrx C performs a metrc Neutral Anger Surprse Fgure 1. Three mages sampled from a 160 frames sequence showng dfferent facal expressons. algnment of L to the frst bass shape B 1 and t can be computed usng standard Procrustes analyss. The mnmzaton of the frst sum of quadratc costs s equvalent to a Maxmum Lkelhood (ML) estmate of the model parameters f..d gaussan nose s affectng the measurements. However, by ncludng the second sum, we obtan a Maxmum A Posteror (MAP) estmate gven the shape pror. In order to ntalze the non-lnear optmzaton, the affne ft gven by equaton (17) s used to compute an ntal metrc soluton for the pror constraned components M J and S J. Ths procedure has analoges wth the approach frst proposed n [16] where the rgd component was used to ntalze a non-lnear optmzaton procedure. The man dfference here s that the rgd shape s gven by a mxture of pror and measured data. To compute each rotaton matrx R and the frst confguraton weght c 1 each frame-wse component of M J can be decomposed usng a orthonormal decomposton [3]. The remanng values c d wth d =2...k are ntalzed close to zero. Fnally notce that the non-lnear optmzaton wth k =1can be used to nfer the 3D structure of a rgd object. For the artculated case, the algorthm presented n [17] can be extended to compute an affne ft usng shape prors whch can be then corrected by forcng specfc metrc constrants for the gven jont. Non-lnear optmzaton can be then appled by addng addtonal prors for each artculated part. 5. Experments The experments are manly focused on deformable and artculated shapes. Frst, synthetc tests are performed on a deformable face n order to verfy the valdty of the reconstructon usng shape prors. The results from non-lnear optmzaton are compared aganst ground truth obtaned from a VICON moton capture system. Then, two further real tests show the method performances wth real magng condton for the deformable and artculated case Deformable face wth ground truth In ths experment 37 pont tracks from a 160 frames long sequence were obtaned wth a VICON system whch cap-

6 (a) (b) (c) Fgure 2. Shape pror and ts effect over the estmated bass shapes. (a) The 1 st deformable bass computed wthout prors. (b) The 3D prorusednthetest.(c)the1 st bass computed usng prors. No Prors Wth Prors Fgure 3. Comparson between the soluton wth and wthout prors. Even f the frontal vew appears correct n both cases, only the soluton wth prors can properly estmate the shape depth. subject was performng a neutral pose as shown n Fgure 2(b). After performng non-lnear optmzaton wth prors, the bass shape B 1 much resemble the pror wth some varatons located over the temple area (Fgure 2(c)). Fgure 3 shows a comparson between 3D reconstructons for the surprse expresson. Notce that both the frontal vews of the 3D reconstructons apparently estmated correctly the face shape. However, the ncluson of the pror s crtcal as shown n the sde vews. Gven a tny rgd moton, the ML soluton alone s very ambguous snce strong varatons n depth results n small dsplacements onto the mage plane. The ncluson of a pror over the rgd shape component constrans the object depth and deformaton estmates. Fgure 4 presents front and sde vews of the fnal 3D reconstructon after 21 teratons of non-lnear optmzaton. Facal symmetry s well preserved and generally the depth of the shape s correctly estmated. In order to compute quanttatvely the algorthm performances, we performed 100 trals for each test wth dfferent condtons. Gaussan nose of dfferent levels was added to the mage ponts whle the pror accurateness was as well altered by addng gaussan nose to L. These tests were performed n order to show how much the algorthm s reslent to naccurate prors and ncreasng mage nose. Results are presented n Fgure 5 showng that the algorthm can delver satsfactory performances even wth some degrees of naccuracy on the shape pror. 2D error (pxels) Pror σ = 0 Pror σ = 10 Pror σ = 20 Pror σ = Nose σ Relatve 3D error (%) Pror σ = 0 Pror σ = 10 Pror σ = 20 Pror σ = Nose σ Neutral Anger Surprse Fgure 4. Front and sde vews of the 3D Reconstructons after non-lnear optmzaton wth shape prors of the neutral, anger and surprse facal expressons. Fgure 5. Synthetc experments results for both 2D reprojecton error n pxel (left) and 3D error n unts (rght). The mage shape s approxmately of sze pxels and zero-mean gaussan nose of varance σ = {0, 0.5, 1, 1.5, 2} pxels s added to W. The 3D pror L s contaned n a box of sze and zero-mean gaussan nose of varance σ = {0, 10, 20, 30} unts s added to smulate naccurate shape prors. tured 2D and 3D locatons of a set of markers overlad on a deformng face (see Fgure 1). The subject was performng tny head pose changes whch n turn affected the extracton of satsfactory 3D reconstructons from 2D trajectores. Fgure 2(a) shows the frst bass shape B 1 obtaned usng deformable non-lnear optmzaton wthout shape prors. The bass shape was generally qute flat and the mouth bend nward nto the head. The shape pror L 3 37 s taken from a 3D reconstructon of the VICON system tself when the 5.2. Image data wth moton degeneracy The am of ths experment s to enforce a shape pror belongng to a subject wth measurements obtaned from a dfferent subject. A rgd 3D shape of a face s frst extracted from the mage sequence shown n Fgure 6 usng a rgd factorzaton approach. Then, the pror s used to nfer the 3Dstructure of a 45 frames sequence wth deformatons manly localzed n the mouth regon. In both sequences,

7 the 65 mage ponts were extracted usng an AAM tracker. The second row of Fgure 7 shows the prors and the dfference between the computed rgd bass shape B 1.From Fgure 7(b) t s evdent that the soluton wthout prors presents depth estmates whch were rather compromsed. Dfferently, after the shape pror ncluson, depth s computed correctly and the face characterstcs are adapted to the new subject. It s possble to notce ths n Fgure 7(c) where the nose s more elongated and eyebrows are less bent. Fnally, Fgure 8 shows the 3D reconstructon for three frames after non-lnear mnmzaton. Fgure 6. The frst two mages show snapshots from a bref sequence of 75 frames showng domnant rgd moton. The rght mage shows the 3D rgd shape pror computed from the sequence. Frame 1 Frame 25 Frame 35 Fgure 9. The left mage shows a frame from a sequence wth two artculated bodes wth a unversal jont. The rght mage shows the overall 3D reconstructon whch has wrong depth estmates. (a) (b) (c) Fgure 7. The frst row shows three frames of a sequence wth nearly planar moton and deformatons manly located n the mouth. The second row presents the 3D shape comparsons between the pror (a), the 1 st bass extracted wthout pror (b) and the same bass computed usng shape prors (c). Box Pror Head Pror 3D reconstructon Fgure 10. The two mages on the left show the prors used to nfer the 3D artculated structure. On the rght t s shown the full reconstructon along wth the estmaton of the jont poston (green). Frame 1 Frame 25 Frame 35 Fgure 8. Front and sde vews after non-lnear optmzaton. Deformatons are manly localzed n the mouth regon Artculated shape wth prors Ths test was amed to show the relevance of the nference wth prors also n the case of artculated SfM. The experment dealt wth a unversal jont between two shapes asshownbyfgure9.the61 frames mage sequence however contaned moton degeneraces only n the box shape whle the head was rotatng enough to assure relable 3D reconstructons. The shape prors were obtaned from separated rgd factorzatons of both objects from dfferent sequences. Generalzed factorzaton s then appled to obtan a better ntal ft for the box shape. The overall affne

8 structure s fnally upgraded to metrc wth the closed-form soluton proposed n [17]. Fgure 10 shows the box sdes now preservng orthogonalty and the depth was accurately estmated along wth the poston of the unversal jont. 6. Concluson Ths paper presented a method capable of ncludng shape prors n a factorzaton framework for SfM. These prors were n the form of prevously computed 3D shapes whch represented a close representaton of the measured data. In such way t was possble to obtan relable 3D reconstructons especally when the moton appearng n the mage sequence was degenerate. At ths end, a closed-form soluton s used to compute an affne ft of moton and shape components whch are then upgraded to metrc usng nonlnear optmzaton wth shape prors. Notce the strong relaton of ths approach to other methods based on factorzaton. For nstance, n the case of photometrc stereo [1], prors on the normals of the object may be drectly appled usng slght varatons of our method. As ongong work, n order to wden the approach applcablty, t s necessary to developaneffcent algorthm for automatcally matchng 2D pont trajectores and 3D shape prors. Acknowledgments Ths work was supported by Fundação para a Cênca e a Tecnologa (ISR/IST plurannual fundng) through the POS Conhecmento Program (nclude FEDER funds) and grant PTDC/EEA-ACR/72201/2006, MODI - 3D Models from 2D Images. E. Muñoz, J. Xao and P. Tresadern provded the sequences used n the expermental sectons for the synthetc, deformable and artculated test respectvely. Thanks to L. Agapto and X. Lladó for suggestons and for carefully readng ths paper. References [1] R. Basr, D. Jacobs, and I. Kemelmacher. Photometrc stereo wth general, unknown lghtng. Internatonal Journal of Computer Vson, 72(3): , [2] Å. Björck. Numercal methods for least squares problems. SIAM Phladelpha, [3] M. Brand. Morphable models from vdeo. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, Kaua, Hawa, volume 2, pages , December [4] M. Brand. A drect method for 3d factorzaton of nonrgd moton observed n 2d. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, San Dego, Calforna, pages , [5] C. Bregler, A. Hertzmann, and H. Bermann. Recoverng non-rgd 3d shape from mage streams. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, Hlton Head, South Carolna, pages , June , 3 [6] A. Del Bue, X. Lladó, and L. Agapto. Non-rgd metrc shape and moton recovery from uncalbrated mages usng prors. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, New York, NY, pages , New York, June [7] D. Forsyth, S. Ioffe, and J. Haddon. Bayesan structure from moton. Proc. 7th Internatonal Conference on Computer Vson, Kerkyra, Greece, 01:660, [8] P. Hansen. Rank-Defcent and Dscrete Ill-Posed Problems: Numercal Aspects of Lnear Inverson. Socety for Industral Mathematcs, [9] K. Kanatan and Y. Sugaya. Factorzaton wthout factorzaton: complete recpe. Memores of the Faculty of Engneerng, Okayama Unversty, 38(1 2):61 72, , 5 [10] S. Olsen and A. Bartol. Usng prors for mprovng generalzaton n non-rgd structure-from-moton. Proc. Brtsh Machne Vson Conference, [11] C. J. Poelman and T. Kanade. A paraperspectve factorzaton method for shape and moton recovery. In Proc. 3rd European Conference on Computer Vson, Stockholm, volume 2, pages , [12] J. Solem and F. Kahl. Surface reconstructon usng learned shape models. Advances n Neural Informaton Processng Systems, 17, [13] P. Sturm and B. Trggs. A factorzaton based algorthm for mult-mage projectve structure and moton. In Proc. 4th European Conference on Computer Vson, Cambrdge, pages , Aprl [14] C. Tomas and T. Kanade. Shape and moton from mage streams under orthography: A factorzaton approach. Internatonal Journal of Computer Vson, 9(2): , , 2 [15] L. Torresan, A. Hertzmann, and C. Bregler. Non-rgd structure-from-moton: Estmatng shape and moton wth herarchcal prors. IEEE Transactons on Pattern Analyss and Machne Intellgence, , 3 [16] L. Torresan, D. Yang, E. Alexander, and C. Bregler. Trackng and modelng non-rgd objects wth rank constrants. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, Kaua, Hawa, [17] P. Tresadern and I. Red. Artculated structure from moton by factorzaton. In Proc. IEEE Conference on Computer Vson and Pattern Recognton, San Dego, Calforna, volume 2, pages , June , 2, 3, 5, 8 [18] B. Trggs, P. McLauchlan, R. I. Hartley, and A. Ftzgbbon. Bundle adjustment A modern synthess. In W. Trggs, A. Zsserman, and R. Szelsk, edtors, Vson Algorthms: Theory and Practce, LNCS, pages Sprnger Verlag, [19] J. Xao, J. Cha, and T. Kanade. A closed-form soluton to non-rgd shape and moton recovery. Internatonal Journal of Computer Vson, 67(2): , Aprl , 3 [20] J. Yan and M. Pollefeys. A factorzaton-based approach for artculated non-rgd shape, moton and knematc chan recovery from vdeo. IEEE Transactons on Pattern Analyss and Machne Intellgence, 30(5), May , 2, 3