Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors

Transcription

1 878 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 Nonrigid Srucure-from-Moion: Esimaing Shape and Moion wih Hierarchical Priors Lorenzo Torresani, Aaron Herzmann, Member, IEEE, and Chrisoph Bregler Absrac This paper describes mehods for recovering ime-varying shape and moion of nonrigid 3D obecs from uncalibraed 2D poin racks. For example, given a video recording of a alking person, we would like o esimae he 3D shape of he face a each insan and learn a model of facial deformaion. Time-varying shape is modeled as a rigid ransformaion combined wih a nonrigid deformaion. Reconsrucion is ill-posed if arbirary deformaions are allowed, and hus addiional assumpions abou deformaions are required. We firs sugges resricing shapes o lie wihin a low-dimensional subspace and describe esimaion algorihms. However, his resricion alone is insufficien o consrain reconsrucion. To address hese problems, we propose a reconsrucion mehod using a Probabilisic Principal Componens Analysis (PPCA) shape model and an esimaion algorihm ha simulaneously esimaes 3D shape and moion for each insan, learns he PPCA model parameers, and robusly fills-in missing daa poins. We hen exend he model o represen emporal dynamics in obec shape, allowing he algorihm o robusly handle severe cases of missing daa. Index Terms Nonrigid srucure-from-moion, probabilisic principal componens analysis, facor analysis, linear dynamical sysems, expecaion-maximizaion. Ç 1 INTRODUCTION AND RELATED WORK A cenral goal of compuer vision is o reconsruc he shape and moion of obecs from images. Reconsrucion of shape and moion from poin racks known as srucure-from-moion is very well undersood for rigid obecs [17], [26] and muliple rigid obecs [10], [16]. However, many obecs in he real world deform over ime, including people, animals, and elasic obecs. Reconsrucing he shape of such obecs from imagery remains an open problem. In his paper, we describe mehods for Nonrigid Srucure-From-Moion (NRSFM): exracing 3D shape and moion of nonrigid obecs from 2D poin racks. Esimaing ime-varying 3D shape from monocular 2D poin racks is inherenly underconsrained wihou prior assumpions. However, he apparen ease wih which humans inerpre 3D moion from ambiguous poin racks (for example, [18], [30]) suggess ha we migh ake advanage of prior assumpions abou moion. A key quesion is wha should hese prior assumpions be? One possible approach is o explicily describe which shapes are mos likely (for example, by hard-coding a model [32]), bu his can be exremely difficul for all bu he simples cases. Anoher approach is o learn a model from raining daa. Various auhors have described mehods for learning linear subspace models wih Principal Componens Analysis (PCA). L. Torresani is wih Microsof Research, 7 J.J. Thompson Ave., Cambridge, CB3 0FB, UK. lorre@microsof.com.. A. Herzmann is wih he Deparmen of Compuer Science, Universiy of Torono, 40 S. George Sree, Rm. 4283, Torono, Onario M5S 2E4 Canada. herzman@dgp.orono.edu.. C. Bregler is wih he Couran Insiue, New York Universiy, 719 Broadway, 12h Floor, New York, NY bregler@couran.nyu.edu. Manuscrip received 31 Oc. 2006; revised 15 June 2007; acceped 18 June 2007; published online 2 Aug Recommended for accepance by P. Torr. For informaion on obaining reprins of his aricle, please send o: pami@compuer.org, and reference IEEECS Log Number TPAMI Digial Obec Idenifier no /TPAMI for recogniion, racking, and reconsrucion [4], [9], [24], [31]. This approach works well if appropriae raining daa is available; however, his is ofen no he case. In his paper, we do no assume ha any raining daa is available. In his work, we model 3D shapes as lying near a lowdimensional subspace, wih a Gaussian prior on each shape in he subspace. Addiionally, we assume ha he nonrigid obec undergoes a rigid ransformaion a each ime insan (equivalenly, a rigid camera moion), followed by a weakperspecive camera proecion. This model is a form of Probabilisic Principal Componens Analysis (PPCA). A key feaure of his approach is ha we do no require any prior 3D raining daa. Insead, he PPCA model is used as a hierarchical Bayesian prior [13] for he measuremens. The hierarchical prior makes i possible o simulaneously esimae he 3D shape and moion for all ime insans, learn he deformaion model, and robusly fill-in missing daa poins. During esimaion, we marginalize ou deformaion coefficiens o avoid overfiing and solve for MAP esimaes of he remaining parameers using Expecaion- Maximizaion (EM). We addiionally exend he model o learn emporal dynamics in obec shape, by replacing he PPCA model wih a Linear Dynamical Sysem (LDS). The LDS model adds emporal smoohing, which improves reconsrucion in severe cases of noise and missing daa. Our original presenaion of his work employed a simple linear subspace model insead of PPCA [7]. Subsequen research has employed variaions of his model for reconsrucion from video, including he work of Brand [5] and our own [27], [29]. A significan advanage of he linear subspace model is ha, as Xiao e al. [34] have shown, a closed-form soluion for all unknowns is possible (wih some addiional assumpions). Brand [6] describes a modified version of his algorihm employing low-dimensional opimizaion. However, in his paper, we argue ha he PPCA model will obain beer reconsrucions han simple subspace models, because PPCA can represen and learn more accurae models, hus avoiding degeneracies ha can occur wih simple subspace models. Moreover, he PPCA formulaion can auomaically /08/$25.00 ß 2008 IEEE Published by he IEEE Compuer Sociey

2 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 879 esimae all model parameers, hereby avoiding he difficuly of manually uning weigh parameers. Our mehods use he PPCA model as a hierarchical prior for moion and sugges he use of more sophisicaed prior models in he fuure. Toward his end, we generalize he model o represen linear dynamics in deformaions. A disadvanage of his approach is ha numerical opimizaion procedures are required in order o perform esimaion. In his paper, we describe he firs comprehensive performance evaluaion of several NRSFM algorihms on synheic daa ses and real-world daa ses obained from moion capure. We show ha, as expeced, simple subspace and facorizaion mehods are exremely sensiive o noise and missing daa and ha our probabilisic mehod gives superior resuls in all real-world examples. Our algorihm akes 2D poin racks as inpu; however, due o he difficulies in racking nonrigid obecs, we anicipae ha NRSFM will ulimaely be used in concer wih racking and feaure deecion in image sequences such as in [5], [11], [27], [29]. Our use of linear models is inspired by heir success in face recogniion [24], [31], racking [9] and compuer graphics [20]. In hese cases, he linear model is obained from complee raining daa, raher han from incomplee measuremens. Bascle and Blake [2] learn a linear basis of 2D shapes for nonrigid 2D racking, and Blanz and Veer [4] learn a PPCA model of human heads for reconsrucing 3D heads from images. These mehods require he availabiliy of a raining daabase of he same ype as he arge moion. In conras, our sysem performs learning simulaneously wih reconsrucion. The use of linear subspaces can also be moivaed by noing ha many physical sysems (such as linear maerials) can be accuraely represened wih linear subspaces (for example, [1]). 2 SHAPE AND MOTION MODELS We assume ha a scene consiss of J ime-varying 3D poins s ; ¼½X ; ;Y ; ;Z ; Š T, where is an index over scene poins, and is an index over image frames. This ime-varying shape represens obec deformaion in a local coordinae frame. A each ime, hese poins undergo a rigid moion and weakperspecive proecion o 2D p ; 21 ¼ c 11 R 23 ð s ; 31 þ d 31 Þþ n 21 ; ð1þ where p ; ¼½x ; ;y ; Š T is he 2D proecion of scene poin a ime, d is a 3 1 ranslaion vecor, R is a 2 3 orhographic proecion marix, c is he weakperspecive scaling facor, and n is a vecor of zero-mean Gaussian noise wih variance 2 in each dimension. We can also sack he poins a each ime-sep ino vecors p 2J1 ¼ G ð s 2J3J 3J1 þ D Þþ N ; ð2þ 3J1 2J1 where G replicaes he marix c R across he diagonal, D sacks J copies of d, and N is a zero-mean Gaussian noise vecor. Noe ha he rigid moion of he obec and he rigid moion of he camera are inerchangeable. For example, his model can represen an obec deforming wihin a local coordinae frame, undergoing a rigid moion, and viewed by a moving orhographic camera. In he special case of rigid shape (wih s ¼ s 1 for all ), his reduces o he classic rigid SFM formulaion sudied by Tomasi and Kanade [26]. Our goal is o esimae he ime-varying shape s and moion ðc R ; D Þ from observed proecions p. Wihou any consrains on he 3D shape s, his problem is exremely ambiguous. For example, given a shape s and moion ðr ; D Þ and an arbirary orhonormal marix A,we can produce a new shape A s and moion ðc R A 1 ; A D Þ ha ogeher give idenical 2D proecions as he original model, even if a differen marix A is applied in every frame [35]. Hence, we need o make use of addiional prior knowledge abou he naure of hese shapes. One approach is o learn a prior model from raining daa [2], [4]. However, his requires ha we have appropriae raining daa, which we do no assume is available. Alernaively, we can explicily design consrains on he esimaion. For example, one may inroduce a simple Gaussian prior on shapes s Nðs; IÞ or, equivalenly, a penaly erm of he form P ks sk 2 [35]. However, many surfaces do no deform in such a simple way, ha is, wih all poins uncorrelaed and varying equally. For example, when racking a face, we should penalize deformaions of he nose much more han deformaions of he lips. In his paper, we employ a probabilisic deformaion model wih unknown parameers. In Bayesian saisics, his is known as a hierarchical prior [13]: shapes are assumed o come from a common probabiliy disribuion funcion (PDF), bu he parameers of his disribuion are no known in advance. The prior over he shapes is defined by marginalizing over hese unknown parameers. 1 Inuiively, we are consraining he problem by simulaneously fiing he 3D shape reconsrucions o he daa, fiing he shapes o a model, and fiing he model o he shapes. This ype of hierarchical prior is an exremely powerful ool for cases where he daa come from a common disribuion ha is no known in advance. Suprisingly, hierarchical priors have seen very lile use in compuer vision. In he nex secion, we inroduce a simple prior model based on a linear subspace model of shape and discuss why his model is unsaisfacory for NRSFM. We hen describe a mehod based on PPCA ha addresses hese problems, followed by an exension ha models emporal dynamics in shapes. We hen describe experimenal evaluaions on synheic and real-world daa. 2.1 Linear Subspace Model A common way o model nonrigid shapes is o represen hem in a K-dimensional linear subspace. In his model, each shape is described by a K-dimensional vecor z ; he corresponding 3D shape is s 3J1 ¼ s 3J1 þ V 3JK z K1 þ m ; ð3þ 3J1 where m represens a Gaussian noise vecor. Each column of he marix V is a basis vecor, and each enry of z is a corresponding weigh ha deermines he conribuions of he basis vecor o he shape a each ime. We refer o he weighs z as laen coordinaes. (Equivalenly, he space of possible shapes may be described by convex combinaions of basis shapes by selecing K þ 1 linearly independen poins in he space.) The use of a linear model is inspired by he 1. For convenience, we esimae values of some of hese parameers insead of marginalizing.

3 880 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 observaion ha many high-dimensional daa ses can be efficienly represened by low-dimensional spaces; his approach has been very successful in many applicaions (for example, [4], [9], [31]). Maximum likelihood esimaion enails minimizing he following leas squares obecive wih respec o he unknowns L MLE ¼ ln pðp 1:T c 1:T ; R 1:T ; V 1:K ; d 1:T ; z 1:T Þ ¼ 1 X 2 2 kp ; c R ðs þ V z þ d Þk 2 ; þ JT lnð2 2 Þ; where V denoes he row of V corresponding o he h poin. Ambiguiies and degeneracies. Alhough he linear subspace model helps consrain he reconsrucion problem, many difficulies remain. Suppose he linear subspace and moion ðs; V; G ; D Þ were known in advance and ha G V is no full rank, a some ime. For any shape represened as z, here is a linear subspace of disinc 3D shapes z þ w ha proec o he same 2D shape, where w lies in he nullspace of G V, and is an arbirary consan. (Here, we assume ha V is full rank; if no, redundan columns should be removed). Since we do no know he shape basis in advance, he opimal soluion may selec G V o be low rank and use he above ambiguiy o obain a beer fi o he daa a he expense of very unreliable deph esimaes. In he exreme case of K ¼ 2J, reconsrucion becomes oally unconsrained, since V represens he full shape space raher han a subspace. We can avoid he problem by reducing K, bu we may need o make K arificially small. In general, we canno assume ha small values of K are sufficien o represen he variaion of realworld shapes. These problems will become more significan for larger K. Ambiguiies will become increasingly significan when poin racks are missing, an unavoidable occurrence wih real racking. In general, we expec he linear subspace model o be sensiive o he choice of K.IfK is oo large for he obec being racked, hen he exra degrees of freedom will be unconsrained by he daa and end up fiing noise. However, if K is oo small, hen imporan degrees of variaion will be los. In pracice, here may no be a clear bes value of K ha will capure all variaion while discarding all noise. Empirically, he eigenvalue specrum obained from PCA on real-world 3D shapes ends o fall off smoohly raher han being bounded a a small value of K. An example from facial moion capure daa is shown in Fig. 1. An addiional ambiguiy occurs in he represenaion of he subspace; specifically, we can apply an arbirary affine ransformaion A o he subspace (replacing V wih VA 1 and z wih Az). However, his does no change reconsrucion error or he underlying subspace, so we do no consider i o be a problem. Alhough he subspace model can be made o work in simple siuaions, paricularly wih limied noise and small values of K, he above ambiguiies indicae ha i will scale poorly o larger problems and become increasingly sensiive o manual parameer uning. As he number of basis shapes grows, he problem is more likely o become unconsrained, evenually approaching he oally unconsrained case described in he previous secion, where each frame may have an enirely disinc 3D shape. ð4þ ð5þ Fig. 1. (a) Two-dimensional coordinaes obained by applying convenional PCA o aligned 3D face shapes. The bes-fi Gaussian disribuion is illusraed by a gray ellipse. (b) Eigenvalue specrum of he face daa. (Deails of he original daa are given in Secion 4.2.) Mos NRSFM mehods make an addiional assumpion ha he recovered shape and moion can be obained by ransforming a low-rank facorizaion of he original poin racks [5], [6], [7], [34]. The main appeal of hese approaches is ha hey decompose he problem ino much simpler ones. However, his approach is only usified when measuremen noise is negligible; wih non-negligible noise, hese mehods give no guaranee of saisical opimaliy, 2 and may in pracice be highly biased. We do no expec noise in real NRSFM problems o be negligible, and he imporance of noise modeling is borne ou by our experimens. 2.2 Probabilisic PCA Model We propose using PPCA [22], [25] o describe he disribuion over shapes. In PPCA, we place a Gaussian prior disribuion on he weighs z and define he res of he model as before z Nð0; IÞ; s ¼ s þ Vz þ m ; p ¼ G ðs þ D Þþn ; ð8þ where m and n are zero-mean Gaussian vecors, wih variance 2 m and 2. Moreover, when esimaing unknowns in PPCA, he laen coordinaes z are marginalized ou: we never explicily solve for z. Because any linear ransformaion of a Gaussian variable is Gaussian, he disribuion over p is Gaussian. 3 Combining (6)-(8) gives p NðG ðs þ D Þ; G VV T þ 2 m I G T þ 2 IÞ: ð10þ 2. NRSFM can be posed as a consrained leas squares problem: facor he daa ino he produc of wo marices ha minimize reproecion error while saisfying cerain consrains. Singular Value Decomposiion (SVD) provides an opimal leas squares facorizaion bu does no guaranee ha any consrains are saisfied. One approach has been o find a subspace ransformaion o he SVD soluion o aemp o saisfy he consrains, bu here is no guaranee ha such a ransformaion exiss. Hence, such mehods canno guaranee boh ha he consrains are saisfied and ha he soluion is opimal. For example, Tomasi and Kanade s algorihm [26] guaranees opimal affine reconsrucions bu no opimal rigid reconsrucions. In pracice, i ofen finds accepable soluions. However, in he NRSFM case, he consrains are much more complex. 3. This may also be derived by direcly marginalizing ou z : pðp Þ¼ R pðp ; z Þdz ¼ R pðp z Þpðz Þdz ; where pðp z Þ is Gaussian (as given by (7) and (8)), and pðz Þ¼Nðz 0; IÞ, assuming ha we condiion on fixed values of s, V, G, D, 2, and 2 m. Simplifying he above expression gives (10). ð6þ ð7þ ð9þ

4 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 881 In his model, solving NRSFM esimaing moion while learning he deformaion basis is a special form of esimaing a Gaussian disribuion. In paricular, we simply maximize he oin likelihood of he measuremens p 1:T or, equivalenly, he negaive logarihm of he oin likelihood L ¼ 1 X ðp 2 ðg ðs þ D ÞÞ T ðg ðvv T þ 2 m IÞGT þ 2 IÞ ðp ðg ðs þ D ÞÞ þ 1 X ln G ðvv T þ 2 m 2 IÞGT þ 2 I þ JT lnð2þ: ð11þ We will describe an esimaion algorihm in Secion 3.2. Inuiively, he NRSFM problem can be saed as solving for shape and moion such ha he reconsruced 3D shapes are as similar o each oher as possible. In his model, shapes arise from a Gaussian disribuion wih mean s and covariance VV T þ 2 mi. Maximizing he likelihood of he daa simulaneously opimizes he 3D shapes according o boh he measuremens and he Gaussian prior over shapes while adusing he Gaussian prior o fi he individual shapes. An alernaive approach would be o explicily learn a 3J 3J covariance marix. However, his involves many more parameers han necessary, whereas PPCA provides a reduced-dimensionaliy represenaion of a Gaussian. This model provides several advanages over he linear subspace model. Firs, he Gaussian prior on z represens an explici assumpion ha he laen coordinaes z for each pose will be similar o each oher, ha is, he z coordinaes are no unconsrained. Empirically, we find his assumpion o be usified. For example, Fig. 1 shows 2D coordinaes for 3D shapes aken from a facial moion capure sequence, compued by convenional PCA. These coordinaes do no vary arbirarily bu remain confined o a small region of space. In general, we find his observaion consisen when applying PCA o many differen ypes of daa ses. This Gaussian prior resolves he imporan ambiguiies described in he previous secion. Deph and scaling ambiguiies are resolved by preferring shapes wih smaller magniudes of z. The model is robus o large or misspecified values of K, since very small variances will be learned for exraneous dimensions. A roaional ambiguiy remains: replacing V and z wih VA T and Az (for any orhonormal marix A) does no change he likelihood. However, his ambiguiy has no impac on he resuling disribuion over 3D shapes and can be ignored. Second, his model accouns for uncerainy in he laen coordinaes z. These coordinaes are ofen underconsrained in some axes and canno necessarily be reliably esimaed, especially during he early sages of opimizaion. Moreover, a concern wih large K is he large number of unknowns in he problem, including K elemens of z for each ime. Marginalizing over hese coordinaes removes hese variables from esimaion. Removing hese unknowns also makes i possible o learn all model parameers including he prior and noise erms simulaneously wihou overfiing. This means ha regularizaion erms need no be se manually for each problem and can hus be much more sophisicaed and have many more parameers han oherwise. In pracice, we find ha his leads o significanly improved reconsrucions over user-specified shape PDFs. I migh seem ha, since he parameers of he PDF are no known a priori, he algorihm could esimae wildly varying shapes and hen learn a correspondingly spread-ou PDF. However, such a spreadou PDF would assign very low likelihood o he soluion and hus be subopimal; his is a ypical case of Bayesian inference auomaically employing Occam s Razor [19]: daa fiing is auomaically balanced agains he model simpliciy. One way o see his is o consider he erms of he log probabiliy in (11): he firs erm is a daa-fiing erm, and he second erm is a regularizaion erm ha penalizes spread-ou Gaussians. Hence, he opimal soluion rades-off beween 1) fiing he daa, 2) regularizing by penalizing disance beween shapes and he shape PDF, and 3) minimizing he variance of he shape PDF as much as possible. The algorihm simulaneously regularizes and learns he regularizaion. Regularized linear subspace model. An alernaive approach o resolving ambiguiies is o inroduce regularizaion erms ha penalize large deformaions. For example, if we solve for laen coordinaes z in he above model raher han marginalizing hem ou, hen he corresponding obecive funcion becomes L MAP ¼ ln pðp 1:T R 1:T ; V 1:K ; d 1:T ; z 1:T Þ ¼ 1 X 2 2 kp ; c R ðs þ V z þ d Þk 2 ; þ z X kz k 2 þ JT 2 lnð22 Þþ TK 2 lnð22 z Þ; ð12þ ð13þ which is he same obecive funcion, as in (5) wih he addiion of a quadraic regularizer on z. However, his obecive funcion is degenerae. To see his, consider an esimae of he basis ^V and laen coordinaes ^z 1:T.Ifwe scale all of hese erms as ^V 2 ^V; ^z 1 2 ^z ; ð14þ hen he obecive funcion mus decrease. Consequenly, his obecive funcion is opimized by infiniesimal laen coordinaes bu wihou any improvemen o he reconsruced 3D shapes. Previous work in his area has used various combinaions of regularizaion erms [5], [29]. Designing appropriae regularizaion erms and choosing heir weighs is generally no easy; we could place a prior on he basis (for example, penalize he Frobenius norm of V), bu i is no clear how o balance he weighs of he differen regularizaion erms; for example, he scale of he V weigh will surely depend on he scale of he specific problem being addressed. One could require he basis o be orhonormal, bu his leads o an isoropic Gaussian disribuion, unless separae variances were specified for every laen dimension. One could also aemp o learn he weighs ogeher wih he model, bu his would almos cerainly be underconsrained wih so many more unknown parameers han measuremens. In conras, our PPCA-based approach avoids hese difficulies wihou requiring any addiional assumpions or regularizaion. 2.3 Linear Dynamics Model In many cases, poin racking daa comes from sequenial frames of a video sequence. In his case, here is an addiional emporal srucure in he daa ha can be modeled in he disribuion over shapes. For example, 3D human facial moion shown in 2D PCA coordinaes in Fig. 1 shows disinc

5 882 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 emporal srucure: he coordinaes move smoohly hrough he space, raher han appearing as random independen and idenically disribued (IID) samples from a Gaussian. Here, we model emporal srucure wih a linear dynamical model of shape z 1 Nð0; IÞ; ð15þ z ¼ z 1 þ v ; v Nð0; QÞ: ð16þ In his model, he laen coordinaes z a each ime sep are produced by a linear funcion of he previous ime sep, based on he K K ransiion marix, plus addiive Gaussian noise wih covariance Q. Shapes and observaions are generaed as before s ¼ s þ Vz þ m ; p ¼ G ðs þ D Þþn : ð17þ ð18þ As before, we solve for all unknowns excep for he laen coordinaes z 1:T, which are marginalized ou. The algorihm is described in Secion 3.3. This algorihm learns 3D shape wih emporal smoohing while simulaneously learning he smoohness erms. 3 ALGORITHMS 3.1 Leas Squares NRSFM wih a Linear Subspace Model As a baseline algorihm, we inroduce a echnique ha opimizes he leas squares obecive funcion (5) wih block coordinae descen. This mehod, which we refer o as BCD- LS, was originally presened in [29]. No prior assumpion is made abou he disribuion of he laen coordinaes, and so, he weak-perspecive scaling facor c can be folded ino he laen coordinaes by represening he shape basis as ~V ½s; VŠ; ~z c c ½1; z T ŠT : ð19þ We hen opimize direcly for hese unknowns. Addiionally, since he deph componen of rigid ranslaion is unconsrained, we esimae 2D ranslaions T G D ¼ ½c R d ;...;c R d Š½ ;...; Š. The variance erms are irrelevan in his formulaion and can be dropped from (5), yielding he following wo equivalen forms: L MLE ¼ X kp ; R ~V ~z c k 2 ; ð20þ ¼ X kp H ~V~z c T k 2 ; ð21þ where H is a 2J 3J marix conaining J copies of R across he diagonal. This obecive is opimized by coordinae descen ieraions applied o subses of he unknowns. Each of hese seps finds he global opimum of he obecive funcion wih respec o a specific block of he parameers while holding he ohers fixed. Excep for he roaion parameers, each updae can be solved in closed form. For example, he updae o is derived by MLE =@ ¼ 2 P ðp ; R ~V ~z c Þ¼0. The updaes are as follows: vecð ~V Þ M þ ðp ;1:T T Þ; ð22þ ~z c ðh VÞ ~ þ ðp T Þ; ð23þ 1 X ðp J ; R ~V ~z c Þ; ð24þ where p ;1:T ¼½p T ;1 ;...; pt ;T ŠT, M ¼½~z c 1 RT 1 ;...; ~zc T RT T ŠT, denoes Kronecker produc, and he vec operaor sacks he enries of a marix ino a vecor. 4 The shape basis updae is derived by rewriing he obecive as L MLE / X kp ;1:T Mvecð ~V Þ T k 2 ð25þ and by MLE =@vecð ~V Þ¼0. The camera marix R is subec o a nonlinear orhonormaliy consrain and canno be updaed in closed form. Insead, we perform a single Gauss-Newon sep. Firs, we parameerize he curren esimae of he moion wih a 3 3 roaion marix Q, so ha R ¼ Q, where ¼ : We define he updaed roaion relaive o he previous esimae as Q new ¼ Q Q. The incremenal roaion Q is parameerized in exponenial map coordinaes by a 3D vecor ¼½! x ;!y ;!z ŠT Q ¼ e^ ¼ I þ ^ þ ^ 2 =2! þ...; ð26þ where ^ denoes he skew-symmeric marix 2 0! z! y 3 ^ ¼ 4! z 0! x 5: ð27þ! y! x 0 Dropping nonlinear erms gives he updaed value as Q new ¼ðIþ ^ ÞQ. Subsiuing Q new ino (20) gives L MLE / X kp ; ði þ ^ ÞQ ~V ~z c k 2 ð28þ ; / X k^ a ; b ; k 2 ; ð29þ ; where a ; ¼ Q ~V ~z c and b ; ¼ðp ; R ~V ~z c Þ. Le a ; ¼½a x ; ;ay ; ;az ; ŠT. Noe ha we can wrie he marix produc ^ a ; direcly in erms of he unknown wis vecor ¼½! x ;!y ;!z ŠT : 2 ^ a ; ¼ 0!z! y a x 3 ; 6 a y 7! z 0! x 4 ; 5 ð30þ a z ; " # 0 a z ; a y ; ¼ a z ; 0 a x : ð31þ ; We use his ideniy o solve for he wis vecor minimizing (29):! 1! X ¼ C T ; C X ; C T ; b ; ; ð32þ a 4. For example, vec 0 a 2 ¼½a a 1 a 0 ;a 1 ;a 2 ;a 3 Š T. 3

6 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 883 where 0 a z ; a y ; C ; ¼ a z ; 0 a x : ð33þ ; We finally compue he updaed roaion as Q new e^ Q, which is guaraneed o saisfy he orhonormaliy consrain. Noe ha, since each of he parameer updaes involves he soluion of an overconsrained linear sysem, BCD-LS can be used even when some of he poin racks are missing. In such even, he opimizaion is carried ou over he available daa. The rigid moion is iniialized by he Tomasi-Kanade [26] algorihm; he laen coordinaes are iniialized randomly. 3.2 NRSFM wih PPCA We now describe an EM algorihm o esimae he PPCA model from poin racks. The EM algorihm is a sandard opimizaion algorihm for laen variable problems [12]; our derivaion follows closely hose for PPCA [22], [25] and facor analysis [14]. Given racking daa p 1:T, we seek o esimae he unknowns G 1:T, T 1:T, s, V, and 2 (as before, we esimae 2D ranslaions T, due o he deph ambiguiy). To simplify he model, we remove one source of noise by assuming 2 m ¼ 0. The daa likelihood is given by pðp 1:T G 1:T ; T 1:T ; s; V; 2 Þ¼ Y pðp G ; T ; s; V; 2 Þ; ð34þ where he per-frame disribuion is Gaussian (8). Addiionally, if here are any missing poin racks, hese will also be esimaed. The EM algorihm alernaes beween wo seps: in he E sep, a disribuion over he laen coordinaes z is compued; in he M sep, he oher variables are updaed. 5 E-sep. In he E sep, we compue he poserior disribuion over he laen coordinaes z given he curren parameer esimaes, for each ime. Defining qðz Þ o be his disribuion, we have qðz Þ¼pðz p ; G ; T ; s; V; 2 Þ ¼Nðz ðp G s T ÞÞ; I G VÞ; ¼ V T G T ðg VV T G T þ 2 IÞ 1 : ð35þ ð36þ ð37þ The compuaion of may be acceleraed by he Marix Inversion lemma ¼ 2 I G VðI þ 2 V T G T G VÞ 1 V T G T 4 : ð38þ Given his disribuion, we also define he following expecaions: E½z Š¼ðp G s T Þ; E½z z T Š¼I G V þ T ; ð39þ ð40þ where he expecaion is aken wih respec o qðz Þ. M-sep. In he M sep, we updae he moion parameers by minimizing he expeced negaive log likelihood: Q E½ log pðp 1:T G 1:T ; T 1:T ; s; V; 2 ÞŠ ð41þ ¼ 1 X E½kp 2 2 ðg ðsþvz Þ T Þk 2 ŠþJT logð2 2 Þ:ð42Þ 5. Technically, our algorihm is an insance of he Generalized EM algorihm, since our M sep does no compue a global opimum of he expeced log likelihood. This funcion canno be minimized in closed form, bu closed form updaes can be compued for each of he individual parameers (excep for he camera parameers, discussed below). To make he updaes more compac, we define he following addiional variables: ~V ½s; VŠ; ~z ½1; z T ŠT ; ð43þ ~ ½1; T ŠT ; ~ 1 T : ð44þ The unknowns are hen updaed as follows; derivaions are given in he Appendix. vecð ~VÞ c! 1 X ð ~ T ðg T G ÞÞ vec X! G T ðp T Þ~ T ; 2 1 2JT X kp T k 2 2ðp T Þ T G ~V~ þ ð45þ ð46þ r ~V T G T G V ~ ~ ; ð47þ X ~ T V ~ T RT ðp ; Þ= X 1 J r ~V T RT RT ~ V ~ ; ð48þ X p ; c R ~V ~ : ð49þ The sysem of equaions for he shape basis updae is large and sparse, so we compue he shape updae using conugae gradien. The camera marix R is subec o a nonlinear orhonormaliy consrain and canno be updaed in closed form. Insead, we perform a single Gauss-Newon sep. Firs, we parameerize he curren esimae of he moion wih a 3 3 roaion marix Q, so ha R ¼ Q, where ¼ : The updae is hen vecðþ A þ B; ð50þ R e Q ; ð51þ where A and B are given in (70) and (71). If he inpu daa is incomplee, he missing racks are filled in during he M sep of he algorihm. Le poin p 0 ; 0 be one of he missing enries in he 2D racks. Opimizing he expeced log likelihood wih respec o he unobserved poin yields he updae rule p 0 ; 0 c 0R 0 ~V 0 ~ 0 þ 0 : ð52þ Once he model is learned, he maximum likelihood 3D shape for frame is given by s þ V ; in camera coordinaes, i is c Q ðs þ V þ D Þ. (The deph componen of D canno be deermined and, hus, is se o zero). Iniializaion. The rigid moion is iniialized by he Tomasi- Kanade [26] algorihm. The firs componen of he shape basis V is iniialized by fiing he residual using separae

7 884 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 shapes S a each ime sep (holding he rigid moion fixed) and hen applying PCA o hese shapes. This process is ieraed (ha is, he second componen is fi based on he remaining residual, and so forh) o produce an iniial esimae of he enire basis. We found he algorihm o be likely o converge o a good minimum when 2 is forced o remain large in he iniial seps of he opimizaion. For his purpose, we scale 2 wih an annealing parameer ha decreases linearly wih he ieraion coun and finishes a NRSFM wih Linear Dynamics The linear dynamical model inroduced in Secion 2.3 for NRSFM is a special form of a general LDS. Shumway and Soffer [15], [23] describe an EM algorihm for his case, which can be direcly adaped o our problem. The sufficien saisics,, and E½z z T 1Š can be compued wih Shumway and Soffer s E sep algorihm, which performs a linear-ime Forward-Backward algorihm; he forward sep is equivalen o Kalman filering. In he M sep, we perform he same shape updae seps as above; moreover, we updae he and Q marices using Shumway and Soffer s updae equaions. 4 EXPERIMENTS We now describe quaniaive experimens comparing NRSFM algorihms on boh synheic and real daa ses. Here, we compare he following models and algorihms: 6. BCD-LS. The leas squares algorihm described in Secion EM-PPCA. The PPCA model using he EM algorihm described in Secion EM-LDS. The LDS model using he EM algorihm described in Secion XCK. The closed-form mehod by Xiao e al. [34].. B05. Brand s direc mehod [6]. We do no consider here he original algorihm by Bregler e al. [7], since we and ohers have found i o give inferior resuls o all subsequen mehods; we also omi Brand s facorizaion mehod [5] from consideraion. To evaluae resuls, we compare he sum of squared differences beween esimaed 3D shapes o ground-ruh deph: k^s C 1:T sc 1:T k F, measured in he camera coordinae sysem (ha is, applying he camera roaion, ranslaion, and scale). In order o avoid an absolue deph ambiguiy, we subrac ou he cenroid of each shape before comparing. In order o accoun for a reflecion ambiguiy, we repea he es wih he sign of deph invered ( Z insead of Z) for each insan and ake he smaller error. In he experimens involving noise added o he inpu daa, we perurbed he 2D racks wih addiive Gaussian noise. The noise level is ploed as he raio of he noise variance o he norm of he 2D racks, ha is, JT 2 =kp 1:T k F. Errors are averaged over 20 runs. 4.1 Synheic Daa We performed experimens using wo synheic daa ses. The firs is a daa se creaed by Xiao e al. [34], conaining six rigid poins (arranged in he shape of a cube) and hree linearly deforming poins, wihou noise. As repored previously, he 6. The auhors are graeful o Brand and o Xiao e al. for providing he source code for heir algorihms. Fig. 2. Reconsrucion error as a funcion of measuremen noise for he cube-and-poins daa in [34]. XCK and B05 algorihms yield he exac shape wih zero error in he absence of measuremen noise. In conras, he oher mehods (EM-PPCA, EM-LDS) have some error; his is o be expeced, since he use of a prior model or regularizer can add bias ino esimaion. Addiionally, we found ha EM-PPCA and EM-LDS did no obain good resuls in his case unless iniialized by XCK. For his paricular daa se, he mehods of XCK and B05 are clearly superior; his is he only daa se on which Xiao e al. [34] perform quaniaive comparisons beween mehods. However, his daa se is raher arificial, due o he absence of noise and he simpliciy of he daa. If we inroduce measuremen noise (Fig. 2), EM-PPCA and EM- LDS give he bes resuls for small amouns of noise, when iniialized wih XCK (his is he only example in his paper in which we used XCK for iniializaion). Our second synheic daa se is a 3D animaion of a shark, consising of 3D poins. The obec undergoes rigid moion and deformaion corresponding o K ¼ 2 basis shapes; no noise is added. Reconsrucion resuls are shown in Fig. 3, and errors ploed in Fig. 4, he ieraive mehods (BCD-LS, EM- PPCA, and EM-LDS) perform significanly beer han B05 and XCK. The ground-ruh shape basis is degenerae (ha is, individual elemens of he deformaion are no full rank when viewed as J 3 marices), a case ha Xiao e al. [34] poin o as being problemaic (we have no esed heir soluion o his problem). Performance for BCD-LS ges significanly worse as superfluous degrees of freedom are added ðk >2Þ, whereas EM-PPCA and EM-LDS are relaively robus o choice of K; his suggess ha BCD-LS is more sensiive o overfiing wih large K. EM-LDS performs slighly beer han EM-PPCA, mos likely because he very simple deformaions of he shark are well modeled by linear dynamics. In order o es he abiliy of EM-PPCA and EM-LDS o esimae noise variance ð 2 Þ, we compare he acual wih esimaed variances in Fig. 5. The esimaion is generally very accurae, and error variance across he muliple runs is very small (generally less han 0.04). This illusraes an advanage of hese mehods: hey can auomaically learn many of he parameers ha would oherwise need o be se by hand. 4.2 Moion Capure Daa We performed experimens wih wo moion capure sequences. The firs sequence was obained wih a Vicon

8 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 885 Fig. 3. Reconsrucions of he shark sequence using he five algorihms. Each algorihm was given 2D racks as inpus; reconsrucions are shown here from a differen viewpoin han he inpus o he algorihm. Ground-ruh feaures are shown as green circles; reconsrucions are blue dos. Fig. 4. Reconsrucion error as a funcion of he number of basis shapes ðkþ for he synheic shark daa. The ground-ruh shape has K ¼ 2. The plo on he lef compares all mehods discussed here, and he plo on he righ compares only our mehods.

9 886 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 Fig. 6. (a) The facial moion capure session, which provided es daa for his paper. (b) The full-body moion capure session (from he Carnegie Mellon Universiy (CMU) mocap daabase) used in his paper. Fig. 5. Noise esimaion for he synheic shark daa se. For each rue noise variance (x-axis), he variance esimaed by our algorihm is shown on he y-axis. The diagonal line corresponds o ground ruh. Resuls are averaged over 20 runs. Error bars are no shown because he sample variance is very small. opical moion capure sysem, wih 40 markers aached o he subec s face (Fig. 6). The moion capure sysems rack he markers and riangulaes o esimae he 3D posiion of all markers. We subsampled he daa o 15 Hz, yielding a sequence 316 frames long. The subec performed a range of facial expressions and dialogue. Tes daa is generaed by orhographic proecion. Reconsrucion resuls are shown in Fig. 7 and reconsrucion error ploed in Fig. 8. As is visible in he figure, he XCK and B05 algorihms boh yield unsaisfacory reconsrucions, 7 regardless of he choice of K, whereas he ieraive mehods (BCD-LS, EM-PPCA, EM-LDS) perform significanly beer. EM-PPCA yields he bes resuls on he original daa. The performance of BCD-LS degrades as K increases, suggesing an overfiing effec, whereas EM-PPCA only improves wih larger K. We also performed his es wih EM- PPCA using a pure orhographic proecion model ðc 1Þ, and he error curve was very similar o ha of scaled orhographic proecion. We esed a MAP version of he algorihm ha opimizes L MAP (13) plus a penaly on he Frobenius norm of V by block coordinae descen. We found his mehod o give worse resuls han he leas squares opimizaion (ha is, opimizing L MLE by BCD-LS), for all regularizaion weighs ha we esed. This suggess ha selecing appropriae regularizaion is no rivial. We also performed experimens wih noise added o he daa and wih random racks removed. The missing daa case is imporan o es, because 3D poins will necessarily become occluded in real-image sequences and may also disappear for oher reasons such as dropped racks or speculariies. We simulae missing daa by omiing each measuremen uniformly a random wih a fixed probabiliy. (In real siuaions, occlusions ypically occur in a much more srucured manner [8]). Fig. 9 demonsraes he sensiiviy of 7. One possible explanaion would be ha his daa suffers from degenerae bases; however, his did no appear o be he case, as we deermined by esing he PCA bases of he aligned ground-ruh daa. he differen ieraive esimaion mehods o missing daa; hese figures sugges ha EM-PPCA and EM-LDS are more robus o missing daa, whereas BCD-LS degrades much faser. These resuls are compued by averaging over 30 random runs. Again, EM-LDS performs bes as he amoun of missing daa increases. We did no es XCK and B05 on hese daa ses, as hese mehods assume ha no daa is missing, and will herefore depend on how his missing daa is impued in he iniial facorizaion sep. In order o visualize he model learned by EM-PPCA, Fig. 10 shows he mean shape and he modes of deformaion learned wih K ¼ 2. We addiionally esed he algorihms sensiiviy o he size of he daa se (Fig. 11). Tess were conduced by sampling he face sequence a differen emporal raes. (Due o local minima issues wih BCD-LS, we performed 30 muliple resars for each BCD-LS es.) We found ha he daa se size did no have a significan effec on he performance of he algorihm; surprisingly, reconsrucion error increased somewha wih larger daa ses. We suspec ha his reflecs nonsaionariy in he daa, for example, some frames having significanly greaer variaions han ohers, or non-gaussian behavior. We also performed he same experimen on synheic daa randomly generaed from a linear-subspace model and found he behavior o be much more as prediced, wih error monoonically decreasing as he daa se grew, and hen leveling off. In some applicaions of NRSFM, here may be significan srucure in he deformaions ha are no represened in he model or no known in advance. In order o explore his case, we performed experimens on a full-body human moion capure daa of a person walking. This human body can be approximaely modeled as an ariculaed rigid-body sysem. The ariculaion is no modeled by he NRSFM mehods considered here, and we canno expec perfec resuls from his daa. However, if he simple NRSFM models work well in his case, hey may provide useful iniializaion for an algorihm ha aemps o deermine he ariculaion srucure or he kinemaics. We chose walking daa ha includes urning (Fig. 6), 8 in order o ensure adequae roaion of he body; as in rigid SFM, wihou roaion, here is inadequae informaion o esimae shape. The inpu o he algorihm is an orhographic proecion of 3D marker measuremens. 8. We used sequence from he CMU moion capure daabase (hp://mocap.cs.cmu.edu). The sequence was subsampled by discarding every oher frame and mos of he markers. The resuling daa has 260 frames and 55 poins per frame.

10 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 887 Fig. 7. Reconsrucion of he facial moion capure daa. The op row shows seleced frames from he inpu daa. The remaining rows show reconsrucion resuls (blue dos), ogeher wih ground ruh (green circles), viewed from below. Fig. 8. Reconsrucion error for he face moion capure daa, varying he number of basis shapes ðkþ used in he reconsrucion. Reconsrucions are shown in Fig. 12. As ploed in Fig. 13, all of he algorihms exhibi nonrivial reconsrucion error. However, EM-PPCA gives he bes resuls, wih BCD-LS somewha worse; XCK and B05 boh yield very large errors. Addiionally, B05 exhibis significan sensiiviy o he choice of he number of basis shapes ðkþ, and as before, he reconsrucion from BCD-LS degrades slowly as K grows, whereas EM-PPCA is very robus o he choice of K.

11 888 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 Fig. 9. Reconsrucion error for he face moion capure daa. (a) shows he dependence on added measuremen noise and (b) shows increasing amouns of missing daa. Noe ha 0 percen noise corresponds o zero noise added o he daa, in addiion o any noise already presen in he measuremens. Fig. 10. Three-dimensional mean shape and modes recovered by EM-PPCA wih K ¼ 2. Shape modes are generaed by adding each deformaion vecor (scaled) o he mean shape. The lines are no par of he model; hey are shown for visualizaion purposes only. 5 DISCUSSION AND FUTURE WORK In his work, we have inroduced NRSFM. Due o he ineviable presence of measuremen noise, missing daa and high-dimensional spaces, we argue ha NRSFM is bes posed as a saisical esimaion problem. This allows us o build explici generaive models of shape o marginalize ou hidden parameers and o use prior knowledge effecively. As shown by our experimens, closed-form mehods while obaining perfec resuls on noiseless synheic daa yield much higher errors on noisy daa and real measuremens. The superioriy of EM-PPCA o BCD-LS in all of our ess illusraes he imporance of marginalizing ou laen coordinaes. The superioriy of EM-LDS over EM-PPCA for highly noisy real daa illusraes he value of he use of a moion model, alhough a firs-order linear dynamical model was oo weak for our daa ses. We did find ha, on synheic, noiseless daa, our mehods had issues wih local minima, whereas he closed-form mehods performed very well on hese cases. This indicaes ha esing on pure synheic daa, while informaive, canno replace quaniaive esing on real daa, and may in fac give opposie resuls from real daa. The cube-and-poins daa se is one for which our prior disribuion may no be appropriae. Linear models provide only a limied represenaion of shape and moion, and here is significan work o be done in deermining more effecive models. For example, nonlinear ime-series models (for example, [21], [33]) can represen emporal dependencies more effecively;

12 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 889 Fig. 11. Dependence on daa se size for he face daa. We suspec ha he odd behavior of he plos is due o nonsaionariy of he facial moion daa; some frames are fi much beer by he model han ohers. Fig. 12. Reconsrucion of he walking moion capure daa. The op row shows seleced frames from he inpu daa. The remaining rows show reconsrucion resuls (blue dos), ogeher wih ground ruh (green circles), viewed from below. perspecive proecion is a more realisic camera model for many image sequences. However, we believe ha, whaever he model, he basic principles of saisical esimaion should be applied. For example, NRSFM for ariculaed rigid-body models will likely benefi from marginalizing over oin angles. We do no address he selecion of K in his paper, alhough our resuls sugges ha he mehods are no exremely sensiive o his choice. Alernaively,

13 890 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 Fig. 13. Reconsrucion error as a funcion of basis shapes ðkþ for full-body moion capure daa. This noise is added in addiion o any noise already presen in he measuremens. mehods such as Variaional PCA [3] could be adaped in order o esimae K or inegrae i ou. Anoher imporan direcion is he inegraion of NRSFM wih image daa. An advanage of he saisical esimaion framework is ha i can be direcly ied o an appearance model [27], whereas oher mehods mus somehow exrac reliable racks wihou he benefi of 3D reconsrucion. Alhough we have chosen o use he EM algorihm for esimaion, i is possible ha oher numerical opimizaion mehods will give beer resuls. For example, conugae gradien could be applied direcly o he log poserior. APPENDIX We now derive he M-sep sep updaes used in Secion 3.2. The expeced negaive log likelihood is Q ¼ 1 X 2 2 E½kp ðg ~V~z þ T Þk 2 ŠþJT logð2 2 Þ: ð53þ To derive updaes, we solve for he minimizing value of Q wih respec o each of he unknowns, holding he ohers fixed. Closed-form updaes exis for each of he individual unknowns, aside from he roaion marices. To derive he shape basis ~V updae, we solve for he ~V ¼ 1 X 2 2 E½G T ðp ðg ~V~z þ T ÞÞ~z T Š ð54þ ¼ 1 X 2 2 G T ðp T Þ~ T þ 1 X 2 2 G T G V ~ ~ : ð55þ Applying he vec operaor o boh sides and using he ideniies vecðabcþ ¼ðC T AÞvecðBÞ ~VÞ ¼ vec X! G T ðp T Þ~ T ð56þ þ 1 X ð 2 ~ T 2 ðg T G ÞÞvecð ~VÞ: ð57þ ¼ 0 yields he shape basis updae ~VÞ To solve for he variance updae, we can 2 ¼ 0 and hen simplify 2 ¼ 1 X E½kp 2JT ðg ~V~z þ T Þk 2 Š ð59þ ¼ 1 X kp 2JT T k 2 2ðp T Þ T G ~V~ ð59þ þ E½~z T ~ V T G T G V~z ~ Š : ð60þ The final erm in his expression is a scalar, and so, we can apply a race, and using he ideniy rðabþ ¼rðBAÞ, we ge: E½~z T ~ V T G T G V~z ~ Š¼rð ~V T G T G VE½~z ~ ~z T ŠÞ ¼ rð ~V T G T G V ~ ~ Þ: To solve for he camera updaes, we firs rewrie he obecive funcion using (1) and, for breviy, drop he dependence on 2 Q ¼ X E½kp ; ðc R ~V ~z þ Þk 2 Š; ð61þ ; where ~V are he rows of ~V corresponding o he h poin (ha is, rows 3 2 hrough 3), and are he x and y componens of he ranslaion in image space. The parial for ranslaion ¼ X ; ðc R ~V ~z þ ÞÞŠ ; ¼ 2 X ðp ; c R ~V ~ Þþ2J : The updae o c is derived ¼ X ¼ 2 X E½ 2~z T ~ V T RT ðp ; ðc R ~V ~z þ ÞÞŠ ð62þ ð63þ ð64þ ~ T ~ V T RT ðp X ; Þþ2c r ~V T RT RT ~ V ~ : ð65þ

14 TORRESANI ET AL.: NONRIGID STRUCTURE-FROM-MOTION: ESTIMATING SHAPE AND MOTION WITH HIERARCHICAL PRIORS 891 The camera roaion is subec o a orhonormaliy consrain, for which we canno derive a closed-form updae. Insead, we derive he following approximae updae. Firs, we differeniae X ¼ R c 2 ~V ~ ~V c X ðp ; Þ~ T V ~ T : ð66þ Since we canno obain a closed-form soluion ¼ 0, we linearize he roaion. We parameerize he curren roaion as a 3 3 roaion marix, such ha R ¼ Q, parameerize he updaed roaion relaive o he previous esimae: Q new ¼ Q Q. The incremenal roaion Q is parameerized by an exponenial map wih wis marix Q ¼ e ¼ I þ þ 2 =2! þ...: ð67þ Dropping nonlinear erms gives he updaed value as Q new ¼ðIþÞQ. Subsiuing Q new ino (66) X ði þ c 2 ~V ~ ~V T c X ðp ; Þ~ T V ~ T : ð68þ Applying he vec operaor gives X A ¼ c 2 ~V ~ ~V T QT ; B ¼ c 2 Q X ~V ~ ~V T c X ðp ; Þ~ T ~ V T : We minimize kavecðþþbk F updae, giving vecðþ A þ B. ACKNOWLEDGMENTS ð69þ ð70þ ð71þ wih respec o for he Earlier versions of his work appeared in [7], [28], [29]. Porions of his work were conduced while L. Torresani was a Sanford Universiy, New York Universiy, and Riya Inc., while A. Herzmann was a New York Universiy and Universiy of Washingon, and while C. Bregler was a Sanford Universiy. The auhors would like o hank Mahew Brand and Jing Xiao for providing heir source code, Jacky Bibliowicz for providing he face mocap daa, and he Carnegie Mellon Universiy (CMU) Moion Capure Daabase for he full-body daa. They also hank Gene Alexander, Hrishikesh Deshpande, and Danny Yang for paricipaing in earlier versions of hese proecs. Thanks o Sefano Soao for discussing shape deformaion. This work was suppored in par by he Office of Naval Research (ONR) Gran N , US Naional Science Foundaion (NSF) Grans IIS , , , and , he Universiy of Washingon Animaion Research Labs, he Alfred P. Sloan Foundaion, a Microsof Research New Faculy Fellowship, he Naional Sciences and Engineering Research Council of Canada, he Canada Foundaion for Innovaion, and he Onario Minisry of Research and Innovaion. In memory of Henning Biermann. REFERENCES [1] J. Barbic and D. James, Real-Time Subspace Inegraion for S. Venan-Kirchhoff Deformable Models, ACM Trans. Graphics, vol. 24, no. 3, pp , Aug [2] B. Bascle and A. Blake, Separabiliy of Pose and Expression in Facial Tracking Animaion, Proc. In l Conf. Compuer Vision, pp , Jan [3] C.M. Bishop, Variaional Principal Componens, Proc. In l Conf. Arificial Neural Neworks, vol. 1, pp , [4] V. Blanz and T. Veer, A Morphable Model for he Synhesis of 3D Faces, Proc. ACM In l Conf. Compuer Graphics and Ineracive Techniques (SIGGRAPH 99), pp , Aug [5] M. Brand, Morphable 3D Models from Video, Proc. Compuer Vision and Paern Recogniion, vol. 2, pp , [6] M. Brand, A Direc Mehod for 3D Facorizaion of Nonrigid Moion Observed in 2D, Proc. Compuer Vision and Paern Recogniion, vol. 2, pp , [7] C. Bregler, A. Herzmann, and H. Biermann, Recovering Non- Rigid 3D Shape from Image Sreams, Proc. Compuer Vision and Paern Recogniion, pp , [8] A.M. Buchanan and A.W. Fizgibbon, Damped Newon Algorihms for Marix Facorizaion wih Missing Daa, Proc. Compuer Vision and Paern Recogniion, vol. 2, pp , [9] T.F. Cooes and C.J. Taylor, Saisical Models of Appearance for Medical Image Analysis and Compuer Vision, Proc. SPIE Medical Imaging, [10] J.P. Coseira and T. Kanade, A Mulibody Facorizaion Mehod for Independenly Moving Obecs, In l J. Compuer Vision, vol. 29, no. 3, pp , [11] F. Dellaer, S.M. Seiz, C.E. Thorpe, and S. Thrun, EM, MCMC, and Chain Flipping for Srucure from Moion wih Unknown Correspondence, Machine Learning, vol. 50, nos. 1-2, pp , [12] A.P. Dempser, N.M. Laird, and D.B. Rubin, Maximum Likelihood from Incomplee Daa via he EM Algorihm, J. Royal Saisical Soc. Series B, vol. 39, pp. 1-38, [13] A. Gelman, J.B. Carlin, H.S. Sern, and D.B. Rubin, Bayesian Daa Analysis, second ed. CRC Press, [14] Z. Ghahramani and G.E. Hinon, The EM Algorihm for Mixures of Facor Analyzers, Technical Repor CRG-TR-96-1, Univ. of Torono, [15] Z. Ghahramani and G.E. Hinon, Parameer Esimaion for Linear Dynamical Sysems, Technical Repor CRG-TR-96-2, Univ. of Torono, [16] M. Han and T. Kanade, Muliple Moion Scene Reconsrucion from Uncalibraed Views, Proc. In l Conf. Compuer Vision, vol. 1, pp , July [17] R. Harley and A. Zisserman, Muliple View Geomery in Compuer Vision, second ed. Cambridge Univ. Press, [18] G. Johansson, Visual Percepion of Biological Moion and a Model for Is Analysis, Percepion and Psychophysics, vol. 14, pp , [19] D.J.C. MacKay, Probable Neworks and Plausible Predicions A Review of Pracical Bayesian Mehods for Supervised Neural Neworks, Nework: CNS, vol. 6, pp , [20] F.I. Parke, Compuer Generaed Animaion of Faces, Proc. ACM Ann. Conf., pp , [21] V. Pavlovic, J.M. Rehg, and J. MacCormick, Learning Swiching Linear Models of Human Moion, Advances in Neural Informaion Processing Sysems 13, pp , [22] S.T. Roweis, EM Algorihms for PCA and SPCA, Proc. Ann. Conf. Advances in Neural Informaion Processing Sysems (NIPS 97), pp , [23] R.H. Shumway and D.S. Soffer, An Approach o Time Series Smoohing and Forecasing Using he EM Algorihm, J. Time Series Analysis, vol. 3, no. 4, pp , [24] L. Sirovich and M. Kirby, Low-Dimensional Procedure for he Characerizaion of Human Faces, J. Opical Soc. Am. A, vol. 4, no. 3, pp , Mar [25] M.E. Tipping and C.M. Bishop, Probabilisic Principal Componens Analysis, J. Royal Saisical Soc. Series B, vol. 61, no. 3, pp , [26] C. Tomasi and T. Kanade, Shape and Moion from Image Sreams Under Orhography: A Facorizaion Mehod, In l J. Compuer Vision, vol. 9, no. 2, pp , 1992.

15 892 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 2008 [27] L. Torresani and A. Herzmann, Auomaic Non-Rigid 3D Modeling from Video, Proc. European Conf. Compuer Vision, pp , [28] L. Torresani, A. Herzmann, and C. Bregler, Learning Non-Rigid 3D Shape from 2D Moion, Proc. Ann. Conf. Advances in Neural Informaion Processing Sysems (NIPS 04), pp , [29] L. Torresani, D. Yang, G. Alexander, and C. Bregler, Tracking and Modeling Non-Rigid Obecs wih Rank Consrains, Proc. Compuer Vision and Paern Recogniion, pp , [30] N.F. Troe, Decomposing Biological Moion: A Framework for Analysis and Synhesis of Human Gai Paerns, J. Vision, vol. 2, no. 5, pp , [31] M. Turk and A. Penland, Eigenfaces for Recogniion, J. Cogniive Neuroscience, vol. 3, no. 1, pp , [32] S. Ullman, Maximizing Rigidiy: The Incremenal Recovery of 3- D Srucure from Rigid and Nonrigid Moion, Percepion, vol. 13, no. 3, pp , [33] J.M. Wang, D.J. Flee, and A. Herzmann, Gaussian Process Dynamical Models, Proc. Ann. Conf. Advances in Neural Informaion Processing Sysems (NIPS 06), pp , [34] J. Xiao, J. Chai, and T. Kanade, A Closed-Form Soluion o Non- Rigid Shape and Moion Recovery, In l J. Compuer Vision, vol. 67, no. 2, pp , [35] A.J. Yezzi and S. Soao, Deformoion: Deforming Moion, Shape Averages, and he Join Regisraion and Approximaion of Srucures in Images, In l J. Compuer Vision, vol. 53, pp , Lorenzo Torresani received a degree in compuer science from he Universiy of Milan, Ialy, in 1996 and he MS and PhD degrees in compuer science from Sanford Universiy in 2001 and 2005, respecively. He worked a Riya Inc., a he Couran Insiue of New York Universiy, and a Digial Persona Inc. He is currenly an associae researcher a Microsof Research Cambridge and a research assisan professor of compuer science a Darmouh College. His main research ineress include compuer vision, machine learning, and compuer animaion. He was he recipien of he Bes Suden Paper Prize a he IEEE Conference on Compuer Vision and Paern Recogniion Aaron Herzmann received he BA degree in compuer science and in ar and ar hisory from Rice Universiy in 1996, and he MS and PhD degrees in compuer science from New York Universiy in 1998 and 2001, respecively. He worked a he Universiy of Washingon, Microsof Research, Misubishi Elecric Research Laboraories, Inerval Research Corporaion, and NEC Research Insiue. He is an associae professor of compuer science a he Universiy of Torono. His research ineress include compuer vision, compuer graphics, and machine learning. He serves as an associae edior of he IEEE Transacions on Visualizaion and Compuer Graphics, an area coordinaor for he Inernaional Conference and Exhibiion on Compuer Graphics and Ineracive Techniques (SIGGRAPH) 2007, and cochaired he Inernaional Symposium on Non-Phoorealisic Animaion and Rendering (NPAR) His awards include an MIT TR100 in 2004, an Onario Early Researcher Award in 2005, a Sloan Foundaion Fellowship in 2006, and a Microsof New Faculy Fellowship in He is a member of he IEEE. Chrisoph Bregler received he Diplom degree from Karlsruhe Universiy in 1993 and he MS and PhD degrees in compuer science from he Universiy of California, Berkeley, in 1995 and 1998, respecively. He is currenly an associae professor of compuer science a New York Universiy s (NYU) Couran Insiue. Prior o NYU, he was on he faculy of Sanford Universiy and worked for several companies including Hewle Packard, Inerval, and Disney Feaure Animaion. He received he Olympus Prize for achievemens in compuer vision and arificial inelligence in He was named as a Sanford Joyce Faculy Fellow and Terman Fellow in 1999 and Sloan Research Fellow in He was he chair of he Inernaional Conference on Compuer Graphics and Ineracive Techniques (SIG- GRAPH) 2004 Elecronic Theaer and Compuer Animaion Fesival.. For more informaion on his or any oher compuing opic, please visi our Digial Library a