THE goal of this work is to develop statistical models for

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL Nonsaionary Shape Aciviies: Dynamic Models for Landmark Shape Change and Applicaions Samarji Das, Suden Member, IEEE, and Namraa Vaswani, Member, IEEE Absrac Our goal is o develop saisical models for he shape change of a configuraion of landmark poins (key poins of ineres) over ime and o use hese models for filering and racking o auomaically exrac landmarks, synhesis, and change deecion. The erm shape aciviy was inroduced in recen work o denoe a paricular sochasic model for he dynamics of landmark shapes (dynamics afer global ranslaion, scale, and roaion effecs are normalized for). In ha work, only models for saionary shape sequences were proposed. Bu mos aciviies of a se of landmarks, e.g., running, jumping, or crawling, have large shape changes wih respec o iniial shape and hence are nonsaionary. The key conribuion of his work is a novel approach o define a generaive model for boh 2D and 3D nonsaionary landmark shape sequences. Grealy improved performance using he proposed models is demonsraed for sequenially filering noise-corruped landmark configuraions o compue Minimum Mean Procruses Square Error (MMPSE) esimaes of he rue shape and for racking human aciviy videos, i.e., for using he filering o predic he locaions of he landmarks (body pars) and using his predicion for faser and more accurae landmarks exracion from he curren image. Index Terms Landmark shape sequence analysis, nonsaionary shape sequences, Kendall s shape space, angen space, racking, paricle filering. Ç 1 INTRODUCTION THE goal of his work is o develop saisical models for he shape change of a configuraion of landmark poins (key poins of ineres) over ime and o use hese models for filering, racking (o auomaically exrac landmarks), synhesis, and change deecion applicaions. The shape of an ordered se of landmarks was defined by Kendall e al. [3] as all of he geomeric informaion ha remains when locaion, scale, and roaional effecs are filered ou. The erm shape aciviy was inroduced in [4] o denoe a paricular sochasic model for he dynamics of landmark shapes (dynamics afer global ranslaion, scale, and roaion effecs are normalized for). A model for shape change is invarian o camera moion under he weak perspecive model (also referred o as he scaled orhographic camera) [5], which is a valid assumpion when he scene deph is small compared o disance from he camera. The models sudied in [4] were primarily for modeling saionary shape aciviies (SSA) of 2D landmarks (assume consan mean shape ). In his work, we propose models for he dynamics of nonsaionary shape sequences (referred o as nonsaionary shape aciviies (NSSA)) of 2D and 3D landmarks. Mos aciviies of a se of landmarks, for example, see Fig. 6, are no saionary, and hence, his more. The auhors are wih he Deparmen of Elecrical and Compuer Engineering, Iowa Sae Universiy, Ames, IA {samarji, namraa}@iasae.edu. Manuscrip received 7 Nov. 2008; revised 20 Mar. 2009; acceped 9 Apr. 2009; published online 24 Apr Recommended for accepance by A. Srivasava, J.N. Damon, I.L. Dryden, and I.H. Jermyn. For informaion on obaining reprins of his aricle, please send o: pami@compuer.org, and reference IEEECS Log Number TPAMISI Digial Objec Idenifier no /TPAMI general model is needed. Even if he aciviy is acually saionary, i sill ges racked using our model. Two-dimensional landmarks are usually he 2D coordinaes of feaure poins of ineres in an image sequence, e.g., hese could be he joins of he differen body pars of he human body and he goal could be o model and rack ariculaed human body moion (see Figs. 6 and 7). Alernaively, hese could be he locaions of a se of ineracing poin objecs and he goal could be o rack heir collecive behavior over ime and deec abnormaliies [4]. Threedimensional landmark shape sequences are ofen obained from a ime sequence of volume images, e.g., by manually or auomaically exracing landmarks from a 3D hear MR image sequence or from a ime sequence of brain MRI volumes. Two-dimensional or 3D landmarks may also be obained from moion capure (MOCAP) [6] daa where sensors are aached o various joins of he human body and heir 3D coordinaes measured over ime. The Carnegie Mellon Moion Capure daabase is a common example. Modeling Mocap daa have applicaions in biomechanics and graphics o undersand he moion of human joins in various acions. 1.1 Our Conribuions and Relaed Work The key conribuion of his work is a novel approach o define a generaive model for 2D and 3D nonsaionary landmark shape sequences. 1 The main idea is o compue he angen space represenaion of he curren shape in he 1. We could have jus defined he model for m-d landmark shape sequences and 2D or 3D would follow as special cases. Bu we model he 2D case separaely since boh shape compuaion from preshapes (compare (1) versus (16)) and Procruses mean compuaion are more efficien in 2D han in general m-d (where he mean is compued using an ieraive algorihm) [7] /10/$26.00 ß 2010 IEEE Published by he IEEE Compuer Sociey

2 580 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 angen space a he previous shape. This can be referred o as he shape velociy vecor since i quanifies he difference beween wo consecuive shapes projeced ino he angen space a he firs one. The coefficiens of shape velociy along he orhogonal basis direcions spanning he curren angen space ( shape speed ) can be modeled using sandard vecor ime-series models. An imporan requiremen in doing his is o ensure ha he basis direcions of he curren angen space are aligned wih hose of he previous one. For boh 2D and 3D shape sequences, we use he angen space projecions defined in [7, pages 71-77]. A second conribuion of our work is demonsraing he use of our nonsaionary model for 1. sequenially filering noise-corruped landmark configuraions o compue Minimum Mean Procruses Square Error (MMPSE) esimaes of he rue shape; 2. racking, i.e., for using he filering o predic he locaions of he landmarks a he curren ime and using his predicion for faser and more accurae landmarks exracion from he curren image; 3. synhesis; 4. change deecion. Mos of our experimens focus on 1 and 2. Grealy improved performance of our racking and filering algorihm over exising work [8], [4], [9] is demonsraed. Due o he nonlineariies in he shape dynamics model and he non-gaussian observaion model (similar o ha of Condensaion [10]), we use a paricle filer (PF) [11] for filering and racking. Our racking problem is a ypical example of a large-dimensional problem wih frequenly mulimodal observaion likelihoods (due o background cluer and missing landmarks), and hence, we replace he basic PF used in previous work by he recenly proposed PF wih Efficien Imporance Sampling (PF-EIS). We demonsrae ha PF-EIS has a much beer performance for landmark shape racking han he basic PF, when he number of paricles used is small. In recen years, here has been a large amoun of work on modeling sequences of landmark shapes boh in saisics [7], [12] and in compuer vision and medical image analysis [8], [4], [13], [14], [10], [15], [16], [17], [18]. Acive shape models (ASMs) [8] and SSAs [4] boh assume saionariy of he shape sequence (single mean shape plus saionary deviaions abou i). Bu, in mos real applicaions, here is large shape variaion over a long sequence, and herefore, a single mean shape plus an ASM or SSA model does no suffice. This is explained in more deail in Secion 2.2. For example, consider a running sequence (see Fig. 6). Anoher example is he changes in shape wihin a single hear cycle. In exising work, he ASM is usually replaced by piecewise ASMs [13], for example, differen ASMs are used for sysolic and diasolic moions in [13] or SSA is replaced by piecewise SSA [14]. Piecewise ASMs are good for recogniion problems, bu no for auomaic racking or for compression since hey do no model he ransiions beween pieces well. When piecewise SSA was used for racking in [14], i needed o use separae change deecion and shape recogniion procedures o deec when and which piece o swich o. In his work, we demonsrae hrough exensive experimens ha boh filering and racking using our model significanly ouperform eiher ASMs or SSAs. Smoohing splines [12] is, o he bes of our knowledge, he only oher exising work ha ruly models nonsaionary landmark shape sequences (oher han he piecewise models discussed above). Bu i does no provide a generaive model for he shape sequences, which is he key requiremen in racking, compression, or synhesis applicaions. A key difference of our work from Condensaion [10] is ha he laer only models and racks global affine deformaion beween wo landmark configuraions. This is a valid model for rigid or approximaely rigid objec moion, bu no for modeling shape change of differen pars of he human body performing acions such as running or jumping, where here is significan local shape deformaion which is no affine. Our modeling approach is similar in spiri o [19], which also uses piecewise geodesic priors o define a generaive model bu in a very differen conex. Oher relaed work includes Acive Appearance Models [15] and Acive Appearance Moion Models [16], which also model appearance, and hence, are no invarian o inensiy changes beween raining and es daa, and work on ariculaed human body racking [17], [18], [20]. The paper is organized as follows: In Secion 2, we explain he nonsaionary model for 2D landmark shape sequences and is parameer esimaion. We discuss he same for 3D shape sequences in Secion 3. Algorihms for filering and racking are developed in Secion 4. Experimenal resuls are given in Secion 5. We conclude he paper in Secion 6. 2 MODELING 2D SHAPE SEQUENCES For modeling human moion aciviy or any aciviy involving muliple ineracing objecs, we represen body joins/objecs as he landmark poins and he corresponding aciviy is represened as a sequence of deforming landmark shapes over ime. I is done in wo seps. Firs, we ransform he shape sequence o a vecor ime series using he nonsaionary shape deformaion model. Then, we fi sandard saisical models o he ime series D Landmark Shape Analysis Preliminaries The configuraion S is an ordered se of K landmarks. In he 2D case, i can be represened as a K-dimensional complex vecor [7, page 39], wih he x (and y) coordinaes forming he real (and imaginary) pars. The preshape w is obained by ranslaion and scale normalizaion, and he shape z is obained by roaion normalizaion w.r.. a given shape. For deails, see [7, Chaper 3] or [4, Secion 2]. The Procruses disance and Procruses mean are also defined here. The complex eigenvecor soluion for compuing he Procruses mean shape can be found in [7, Chap. 3] and [21]. As explained in [7, Chap. 4], he shape space M is a manifold in C K 1, and hence, is acual dimension is C K 2. Thus, he angen plane a any poin of he shape space is a C K 2 -dimensional hyperplane in C K [7]. The projecion of a configuraion S ino he angen space a a pole can be compued [7] as follows:

3 DAS AND VASWANI: NONSTATIONARY SHAPE ACTIVITIES: DYNAMIC MODELS FOR LANDMARK SHAPE CHANGE AND APPLICATIONS 581 y ¼ C K S; C K ¼ 4 I K 1 K 1 T K =K; w ¼ y=kyk; ðw; Þ ¼angleðw Þ; zðw; Þ ¼we jðw;þ ; vðz; Þ ¼½I K Šz: ð2þ Here, I K is a K K ideniy marix and 1 K is a column vecor wih K rows wih all enries as 1. The noaion x T denoes ranspose and x denoes conjugae ranspose. The firs hree equaions involve ranslaion, scale, and roaion normalizaion, respecively, and he las one involves projecing he shape z ino he angen space a. The projecion from angen space o shape space is given by [7]: z ¼ð1 v vþ 1 2 þ v: 2.2 Problem wih SSA and ASM Models The SSA model proposed in [4] compued a single mean shape for a raining sequence and aligned each preshape w in he sequence o o obain he shape sequence z. Tangen projecions vðz ;Þ of each z were compued in he angen space a and heir ime series was modeled using an auoregressive (AR) model. The work of [9] replaced AR by ARMA models and used he models for recogniion problems. The ASM of [8] assumed ha z belongs o a vecor space and replaced he angen space projecion given in (2) by is linear version vðz ;Þ¼ z and modeled he ime series of vðz ;Þ. Since boh SSA and ASM assumed a single mean shape, hey could model only small deviaions from mean, which is only possible for saionary sequences. Bu, in many applicaions, his assumpion may no hold, for example, a crawling or a dancing sequence or see Fig. 6. In hese cases, he mean shapes for differen ime inervals are differen. Or, in oher words, considering he enire sequence, he shape aciviy is essenially nonsaionary. Now, if we force a fixed mean shape o such a deforming shape sequence, he resuling shapes z would drif oo far away from. Iis imporan o noe ha a single angen space approximaion works as long as each elemen of vðz ;Þ for all shapes is less han 1 (oherwise, he square roo in (3) will be of a negaive number). Also, a ime-invarian AR or ARMA model on vðz ;Þs is a valid one only if he magniudes of each elemen of vðz ;Þ are significanly smaller han 1 (his is because, when vðz ;Þ is large, i.e., when z is far from, small changes in vðz ;Þ would correspond o very large changes in z ). Bu, for large shape variaion, vðz ;Þ will be large. In such a scenario, boh SSA and ASM would fail o correcly model he shape dynamics. 2.3 Modeling Nonsaionary Shape Sequences To model a nonsaionary shape sequence, we use ¼ z 1 a ime. Thus, w z :¼ w z 1 jw z 1j ; v :¼ vðz ;z 1 Þ¼½I z 1 z 1 Šz : The inverse map is given by z ¼ð1 v v Þ 1 2 z 1 þ v : ð1þ ð3þ ð4þ ð5þ Since he projecion of z 1 in he angen space a z 1, T z 1, is zero, v can be inerpreed as he difference ðz z 1 Þ projeced ino T z 1, i.e., i is he shape velociy a ime. The ranslaion, scale, and roaion normalizaion in 2D removes wo complex dimensions (four real dimensions), and hus, he shape space is a K 2-dimensional manifold in C K and so he angen space is a K 2-dimensional hyperplane in C K [7]. Thus, he shape velociy v has only K 2-independen complex dimensions, i.e., i can be rewrien as v ¼ U ~c, where he columns of ðu Þ KK 2 conain he K 2 orhonormal basis direcions spanning T z 1 and ~c 2C K 2 are he basis coefficiens. ~c may be inerpreed as a shape speed vecor. Noe ha, by definiion, T z 1 is perpendicular o z 1 and 1 K. Also, z 1 is perpendicular o 1 K (due o ranslaion normalizaion). Thus, he projecion marix for T z 1 =KŠ. In is ½I K z 1 z 1 ŠC K ¼½I K z 1 z 1 1 K1 T K oher words, U saisfies U U ¼ I K z 1 z 1 1 K1 T K =K : One way o obain U is by compuing he Singular Value Decomposiion (SVD) of he righ-hand side (RHS) of (6) and seing he columns of U equal o he lef singular vecors wih nonzero singular values. Denoe his operaion by U ¼ lef:singular:vecorsðmðz 1 ÞÞ, where MðzÞ ¼ 4 IK zz 1 K 1 T K =K : This was used in [1], [4]. Bu, if his is done a each, he columns of U and U 1 may no be aligned. As an exreme example, consider he following. Le K ¼ 4. I may happen ha U 1 ¼ and 2 3 0:1 0:995 0:995 0:1 U ¼ : 0 0 In his case, i is obvious ha he firs column of U corresponds o he second column of U 1 and vice versa for he second column of U. Or, in oher words, ~c ;1 corresponds o ~c 1;2 and ~c ;2 o ~c 1;1. Thus, if SVD is used o obain U a each, he ~c s canno be assumed o be idenically disribued, i is herefore incorrec o model hem by an AR model, which assumes saionariy of ~c. Noice he large modeling error of his mehod (NSSAunaligned) in Fig. 2a. We fix his problem as follows (also, see Fig. 1): To obain an aligned sequence of basis direcions over ime, we obain he mh column of U by saring wih he mh column of U 1, making i perpendicular o z 1 (by subracing z 1 z 1 ), and hen using Gram-Schmid orhogonalizaion o also make he resuling vecor perpendicular o he firs m P 1 columns of U (i.e., by furher subracing ou m 1 j¼1 ðu Þ j ðu Þ j ). This procedure can be summarized as ð6þ ð7þ

4 582 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 Fig. 1. This figure shows he alignmen of successive angen spaces for NSSA. When using [4], [1], he axes (here, x and y) of he consecuive angen planes may no be aligned (op). Our mehod gives aligned axes (boom). U ¼ gðu 1 ;z 1 Þ; where " # gð:þ m ¼ 4 I z 1 z 1 8m ¼ 1;...ðK 2Þ: Xm 1 j¼1 gð:þ j gð:þ j ðu 1 Þ m ; ð8þ Here, gð:þ m denoes he mh columns of gðu 1 ;z 1 Þ. U 0 is iniialized as U 0 ¼ lef:singular:vecorsðmðz 0 ÞÞ. Now, since he columns of U are aligned, i is fair o assume ha ~c ;j s are idenically disribued for each j over ime. Since hey are also emporally correlaed, we model hem by an auoregressive model wih lag 1 (AR(1) model). For simpliciy of noaion, we firs conver ~c ino a 2K 4- dimensional real vecor. We denoe his operaion by c ¼ vecð~c Þ; and he inverse operaion (obaining he complex vecor) is denoed by ~c ¼ vec 1 ðc Þ. Thus, in summary, he dynamical model of he sae X ¼½U ;z ;c Š is given by c ¼ A c c 1 þ c; ; c; Nð0; c Þ; U ¼ gðu 1 ;z 1 Þ; z ¼ 1 c T c 1=2z 1 þ U vec 1 ðc Þ; ð9þ ð10þ where Nð; Þ denoes Gaussian pdf wih mean and covariance marix. The las equaion follows from (3) and he fac ha v ¼ U ~c, U U ¼ I, and ~c ~c ¼ c T c. The above model is iniialized wih z 0 ¼ w 0 ;U 0 ¼ lef:singular:vecorsðmðz 0 ÞÞ; c 0 ¼ 0: ð11þ 2.4 Model Parameer Esimaion The above model is compleely specified by z ini ¼ w 0, A c, c. A c is he AR ransiion marix and c is he modeling error covariance marix in he AR model. Given a raining sequence of landmark configuraions, fs g N 1 ¼0, a maximumlikelihood (ML) esimae of he parameers can be obained as follows: 1. Obain he shape sequence fz g by ranslaion, scale, and roaion normalizaion of fs g ðn 1Þ ¼0, i.e., compue y ¼ C K S and w ¼ y ky k for each. Se z 0 ¼ w 0. Compue Fig. 2. (a) The ME for NSSA, ASM, and ASM for a few aciviies using 2D MOCAP daa. I is imporan o noe ha NSSA wihou he basis alignmen has a very large modeling error. While afer he basis alignmen is aken ino accoun, NSSA has much lower ME han SSA and ASM. (b) The ME for NSSA, ASM, and ASM for a few aciviies using 3D MOCAP daa. Again, NSSA had much lower ME compared o ha SSA and ASM. Tha is why he corresponding bar plo for NSSA modeling error has almos disappeared. w z ¼ w z 1 jw z ; 8 >0: 1j ð12þ 2. For all, obain he shape velociy coefficiens fc g from fz g. This involves compuing U ¼ gðu 1 ;z 1 Þ; ~c ¼ U v ¼ U z ; c ¼ vecð~c Þ; ð13þ saring wih U 0 ¼ lef:singular:vecorsðmðz 0 ÞÞ. The second equaion above follows because U v ¼ U ½I K z 1 z 1 Šz ¼ U IK z 1 z 1 1 K1 T K =K z ¼ U U U z ¼ U z (he second equaliy follows because z is ranslaion normalized so ha 1 T K z ¼ 0 and hird one follows because, by definiion, U U ¼½I K z 1 z 1 1 K 1 T K =KŠ). 3. Obain an ML esimae of he AR model parameers A c ; c from fc g by using he Yule-Walker equaions, i.e.,

5 DAS AND VASWANI: NONSTATIONARY SHAPE ACTIVITIES: DYNAMIC MODELS FOR LANDMARK SHAPE CHANGE AND APPLICATIONS 583 A c ¼ R c ð1þr c ð0þ 1 ; where R c ð0þ ¼ 1 N XN 1 ¼0 c c T ;R cð1þ ¼ 1 XN 1 c c T 1 N 1 ; ¼1 c ¼ 1 XN 1 ðc A c c 1 Þðc A c c 1 Þ T : N 1 ¼ Using Muliple Training Sequences ð14þ If more han one raining sequence is available, one can compue a mean z ini (denoed by ~z ini ) by aligning he iniial preshapes w 0 of all he sequences. We se z 0 for each sequence as he corresponding w 0 aligned o ~z ini. These operaions make sure ha he iniial shapes of all he raining sequences are aligned. Now, saring wih z 0,we can obain he shape speed c s for each sequence. Say, we have a oal of q raining sequences for a given moion aciviy, each wih lengh N. We denoe c s corresponding o he ih sequence as fc i g, where i ¼ 1;...;q. Now, we can esimae R c ð0þ;r c ð1þ as R c ð0þ ¼ 1 P q q i¼1 1 P N 1 N ¼0 ci ci T and R c ð1þ ¼ 1 P q P 1 N 1 q i¼1 N 1 ¼1 ci cit 1. Finally, we compue A c ¼ R c ð1þr c ð0þ 1 and c ¼ 1 P q q i¼1 i c, where i c ¼ 1 N 1 XN 1 ¼1 c i A c c i 1 c i A c c i T 1 : The enire procedure is summarized in Algorihm 1. Algorihm 1. 2D NSSA: Training wih Muliple Training Sequences For a Given Moion Aciviy Inpu: Preshapes corresponding o q raining sequences (fw i gn 1 ¼0, i ¼ 1;...;q) Oupu: Compued parameers ~z ini ;A c ; c. 1) Compue ~z ini ¼ ðw 1 0 ;w2 0 ;...;wq 0 ; Þ where ð:þ is he Procruses mean shape [7], [21]. 2) Compue U ~ ini ¼ lef:singular:vecorsðmð~z ini ÞÞ where M(.) is given in (7). 3) For each i, i ¼ 1; 2;...q a) Compue z i 0 ¼ zðwi 0 ; ~z iniþ using (1), compue U0 i ¼ gð U ~ ini ;z i 0 Þ using (8) and se ci 0 ¼ 0. b) For each, ¼ 1;...N 1 do i) Compue z i ;Ui ;ci using (12), (13). 4) Compue A c ¼ R c ð1þr c ð0þ 1 where, R c ð0þ ¼ 1 q P q i¼1 1 N R c ð1þ ¼ 1 q P q i¼1 5) Compue c ¼ 1 q ðc i A cc i 1 ÞT P N 1 ¼0 ci ci T and P 1 N 1 N 1 ¼1 ci cit 1 P 1 q N 1 i¼1 P N 1 ¼1 ðci A cc i 1 Þ 3 MODELING 3D SHAPE SEQUENCES A 3D configuraion is represened by a se of K ordered landmarks as a ðk 3Þ marix whose each row corresponds o he ðx; y; zþ coordinaes of he corresponding landmark. In his secion, we discuss he basics of 3D landmark shape analysis [7] and hen develop 3D nonsaionary shape aciviy model (3D-NSSA) D Landmark Shape Analysis Preliminaries For 3D shapes, he compuaion of preshape ðwþ K3 from raw shape ðsþ K3 is similar o he 2D case, i.e., firs ge he cenered shape ðyþ K3 and hen perform size normalizaion: y ¼ C K S; w ¼ y : kyk F where C K is given in ð1þ ð15þ Here, k:k F denoes Frobenius norm of a marix. The roaion aligned shape z is obained from preshape w in he following way: Say, we wan o align ðwþ K3 w.r.. ðþ K3. We do his as z ¼ wuv T ; where VU T ¼ SV Dð T wþ; ð16þ where V; U are he lef and righ singular vecors of he 3 3 marix ð T wþ. As poined ou by an anonymous reviewer, while performing 3D shape alignmen, we may have reflecions unlike he 2D case. This happens if deðuþ ¼ 1 or deðvþ ¼ 1, where deð:þ denoes deerminan. Thus, 3D alignmen is a bi differen from 2D since reflecions are allowed in 3D bu no in 2D. Anoher imporan hing abou 3D shape analysis is he vecorizaion operaion [7]. Say, z is he shape a a given insan which is a ðk 3Þ marix wih columns z 1 ; z 2 ; z 3. We vecorize z o a 3K lengh vecor as follows: vec 3D ðzþ ¼ z T 1 ; zt 2 ; zt 3 T : ð17þ The inverse operaion is given by vec 1 3Dð:Þ, which forms a K 3 marix from a 3K lengh vecor. The angen space coordinae vðz; Þ of a shape z w.r.. he shape is given as follows: vðz; Þ ¼½I 3K vec 3D ðþvec 3D ðþ T ŠvecðzÞ: ð18þ The inverse map (i.e., from angen space o shape space) is given as z ¼ vec 1 3D ðð1 vt vþ 1 2 vec3d ðþþvþ: 3.2 3D Nonsaionary Shape Aciviy (3D-NSSA) ð19þ To define an NSSA model on 3D shape daa, we firs obain he ranslaion- and scale-normalized preshape sequence fw g from he 3D configuraion sequence fs g using (15). As in he 2D case, we use ¼ z 1 o compue he shape sequence followed by compuing he shape velociy and shape speed vecors in an exacly analogous fashion. The final procedure can be summarized as follows: Firs, we have z ini ¼ z 0 ¼ w 0 and hen we compue he iniial angen space basis marix as U ini ¼ U 0 ¼ lef:singular:vecorsðm 3D ðz 0 ÞÞ, where where M 3D ðzþ ¼ 4 ½I 3K vecðzþvecðzþ T ŠC K;3D ; 2 C K 0 KK 3 0 KK C K;3D ¼ 4 0 KK C K 0 KK 5: 0 KK 0 KK C K ð20þ

6 584 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 Here, 0 KK is a ðk KÞ marix wih all zero enries and C K is defined in (1). Now, saring wih z 0 and U 0, he compuaion of he corresponding ime sequence of shape speed vecors is done as follows: z ¼ w UV T ; where VU T ¼ SV D z T 1 w ; ð21þ U ¼ gðu 1 ; vec 3D ðz 1 ÞÞ; c ¼ U T v ¼ U T vec 3Dðz Þ; ð22þ ð23þ where gð:þ is defined in (8). The reason why U T v ¼ U T vec 3Dðz Þ is similar o he 2D case (see discussion below (13)). We model c using a firs order AR model (in general, his may be replaced by any appropriae model for c ). Thus, he forward model for generaing a 3D shape sequence is c ¼ A c c 1 þ n ;n Nð0; c Þ; U ¼ gðu 1 ; vec 3D ðz 1 ÞÞ; z ¼ vec 1 1 3D 1 c T c 2 vec 3D ðz 1 ÞþU c : ð24þ The las equaion follows from (19) and he fac ha v T v ¼ ðu c Þ T U c ¼ c T ðut U Þc ¼ c T c and U T U ¼ I. 3.3 Model Parameer Esimaion The parameer esimaion algorihm for he 3D case can be summarized as follows: 1. For all, obain fw g from a given 3D landmark configuraion sequence fs g. 2. For all, compue fz g from fw g using (21). 3. For all, compue fc g from fz g using (22) and (23). 4. Esimae he AR model parameers for c using he Yule-Walker equaions given in (14). 4 FILTERING AND TRACKING The goal of filering is o filer ou he noise and ge a good esimae of he rue landmark shape from noisy observed landmarks. In our algorihm, he paricle filer akes noisy observed landmark daa as inpu and oupus he MMPSE esimae of he rue landmark shape. The MMPSE esimae of shape can be derived by following he Procruses mean derivaion [7] as: ^z ¼ arg min E½d 2 ðz ;ÞjY 1: Š ¼ arg min E½kz z k2 jy 1: Š ¼ arg max E½z z jy 1:Š; ð25þ where E½jY 1: Š denoes he condiional expecaion given Y 1:, d denoes he Procruses disance [7], and Y 1: are he observaions unil. The las equaliy follows because z z ¼ 1 and ¼ 1. Under a paricle filering seup, he MMPSE esimae is compued as ^z ¼ principal eigenvecor of P N pf i¼1 zi zi w i, where N pf denoes number of paricles, i denoes he ih paricle, and w i represens he imporance weigh corresponding o he ih paricle, i.e., pðz jy 1: Þ XNpf w i z z i : i¼1 The configuraion parameers, i.e., scale, ranslaion, and roaion are also esimaed in he process of filering. Apar from removing random addiive noise, paricle filering can also be used o clean up he effecs of occlusion and cluer. Tracking is used o exrac and filer ou landmark configuraions from a sequence of images. Filering plays a very crucial role in he process. In fac, racking can be considered as observaion exracion coupled wih filering. I works as follows: A shape deformaion model (as described in Secion 2.3) predics he shape a he curren insan using he previous shape esimaes. Similarly, scale, ranslaion, and roaion models are used o predic heir values as well. These, coupled wih he prediced shape, give he prediced landmark locaions (i.e., prediced configuraion) a he curren insan. Using hese prediced landmark locaions in he curren image, he landmarks can be exraced for he curren image using any echnique, for example, edge deecion or opical flow. Our mehod for doing his is described in Algorihm 4 and Secion 4.5. Once he observed landmarks are obained, hey are filered o ge an MMPSE esimae of he rue landmark shape and MMSE esimaes of scale, roaion, and ranslaion. These esimaes are again uilized o exrac he observed landmark locaions a he nex ime insan as described above. We describe our sae ransiion model and observaion model in Secions 4.1 and 4.2. We develop he PF algorihms for filering in Secions 4.3 and 4.4. The PF-based racking algorihm o exrac landmarks from video sequences is described in Secion Sysem Model (Sae Transiion Model) Since he observaions are landmark configuraions, o exrac hem, we need o esimae boh he shape and he moion (scale, ranslaion, and roaion). Thus, our sae vecor is X ¼½s ; ; ;c ;z ;U Š, where s is he logarihm of global scale, is he global roaion, is he xy ranslaion, c is he shape speed vecor, z is he shape, and U is he basis se spanning he curren angen space. The shape dynamics model is given in (10). I is a second order model on z which is equivalen o a firs order model on he shape speed c. We use a firs order model on logarihm of global scale s, global 2D roaion (his ypically models he random moion of camera), and ranslaion : s ¼ s s 1 þ s; ; s; N 0; 2 s ; ¼ 1 þ ; ; ; N 0; 2 ; ¼ 1 þ ; ; ; N 0; 2 : ð26þ Noe ha, in case of filering (when landmark observaions are already available), ranslaion can be normalized for. Since i is a linear process, he form of he observaion noise pdf does no change. Bu, in case of racking o predic and exrac landmarks from image sequences, ranslaion does need o be racked o predic where he configuraion of landmarks ranslaed o.

7 DAS AND VASWANI: NONSTATIONARY SHAPE ACTIVITIES: DYNAMIC MODELS FOR LANDMARK SHAPE CHANGE AND APPLICATIONS 585 The resuling sae ransiion prior becomes: pðx jx 1 Þ¼N s s 1 ; 2 s N 1 ; 2 N 1 ; 2 NðAc c 1 ; c Þ ðu gðu 1 ;z 1 ÞÞðz fðz 1 ;U ;c ÞÞ; where denoes he Dirac dela funcion and fðz 1 ;U ;c Þ¼ 4 ð1 c T c Þ 1=2 z 1 þ U vec 1 ðc Þ and gð:þ is defined in (8). 4.2 Observaion Model There are various ways o exrac landmarks from image sequences e.g., edge deecion followed by exracing he K sronges edges closes o prediced landmark locaions or using he Kanade-Lucas-Tomasi (KLT) Feaure Tracker [22] (block opical flow esimaion) algorihm a or around prediced landmark locaions. As explained in Secion 4.5 and Algorihm 4, we use a modificaion of KLT for his purpose. Algorihm 2. PF-Gordon for Landmark Shape Filering Iniializaion: A ime ¼ 0, sample s ðiþ 0 Nðs 0 ; 2 s Þ, ðiþ 0 Nð 0 ; 2 Þ;cðiÞ 0 ¼ 0; z ðiþ 0 ¼ z 0 and U ðiþ 0 ¼ U 0. Here, i ¼ 1;...;N pf where, N pf is he number of paricles. For >0, 1) Imporance sample X ðiþ pðx jx ðiþ 1 Þ as s i Nð ss i 1 ;2 s Þ, i Nði 1 ;2 Þ, c i NðA cc i 1 ; cþ, U i ¼ gðui 1 ;zi 1 Þ, z i ¼ fðzi 1 ;Ui ;ci Þ. i ¼ 1; 2;...;N pf 2) Weigh and Resample. Compue w i ¼ ~wi P Npf j¼1 ~wj where, ~w Q i ¼ wi 1 pðy jx iþ wih, pðy jx iþ¼pðy jhðs i ;i ;zi ÞÞ ¼ K k¼1 ½ð1 pþn ð½hðsi ;i ;zi ÞŠ k ;2 o ÞþpNð0; 1002 oþš; 8i 3) Compue he MMPSE esimae ^z as he principal eigenvecor of P N pf i¼1 zi zi w i. Esimae configuraion parameers as ^s ¼ P N pf i¼1 wi si and ^ ¼ P N pf i¼1 wi i 4) Se þ 1 and go o sep 1. The configuraion of landmarks is obained from he shape, scale, and roaion by he ransformaion hðs ; ;z Þ¼z e s e j. The simples observaion model is of he form Y ¼ hðs ; ;z Þþw ;w N 0; 2 o I ; ð27þ where w is a complex Gaussian noise vecor. This assumes ha here is no background cluer: each of he K sronges edges or he K KLT-feaure poins are always generaed by he rue landmark locaion plus some error modeled as Gaussian noise. Bu his is ofen a simplisic model since here is always background cluer ha generaes false edges or false KLT-feaure maches or here migh be missing landmarks due o blur or occlusion. Thus, i may happen ha, ou of he K observed landmark locaions, some landmark a some ime is acually generaed by cluer (e.g., if a rue landmark is blurred or occluded, while a nearby cluer poin has a sronger edge). We model his as follows: wih a small probabiliy p, he kh landmark Y ;k is generaed by a cluer poin (model a cluer poin locaion as a large variance Gaussian or by a uniform), independen of oher landmarks. Wih probabiliy ð1 pþ, i is generaed by a Gaussian-noise-corruped acual landmark (independen of oher landmarks), i.e., Y ;k ð1 pþn ½hðs ; ;z ÞŠ k ; 2 o þ pn 0; o : ð28þ The above model has been adaped from he observaion model used in Condensaion [10]. The resuling observaion likelihood erm is pðy jx Þ¼ YK ð1 pþn ½hðs ; ;z ÞŠ k ;o 2 þ pn 0; o : k¼1 4.3 Paricle Filer wih Efficien Imporance Sampling The firs PF algorihm, PF-Gordon [11], used he sae ransiion prior (i.e., pðx jx 1 Þ) as he imporance densiy. This assumes nohing and has very small compuaion burden per paricle. Bu since i does no use knowledge of he curren observaion, he weighs variance can be large, paricularly when he observaions are more reliable han he prior model. Thus, i requires more paricles for a given accuracy level compared o he case when he knowledge of observaions is used. The opimal imporance densiy [23] is given by he poserior condiioned on he previous sae, denoed by p, where p ðx Þ¼ 4 pðx jx 1 ;Y Þ: ð29þ Bu, in mos problems, including ours, p canno be compued analyically. When i is unimodal, PF-Douce [23] approximaes i by a Gaussian abou is mode (Laplace s approximaion [24]) and samples from he Gaussian. Oher work ha also implicily assumes ha p is unimodal includes [25]. Bu, in our case, he observaion likelihood is a raised Gaussian as a funcion of ½hð:ÞŠ k and is hus heavy ailed. If he equivalen sae ransiion prior of ½hð:ÞŠ k is broad (e.g., his will happen if STP of s or is broad), whenever Y ;k is generaed from he oulier disribuion (i.e., is far from he prediced landmark locaion), he resuling poserior given he previous sae p ðx Þ¼ 4 pðx jx 1 ;Y Þ will be mulimodal. For such problems where p is ofen mulimodal, a paricle filer wih efficien imporance sampling (PF-EIS) was proposed in [26] which combines he ideas of boh PF- Gordon and PF-Douce o handle mulimodal observaion likelihoods. This algorihm relies on he fac ha even hough p is mulimodal, for mos real-life problems, i is possible o spli he sae vecor X ino an effecive basis X ;s and residual space X ;r in such a way ha p, condiioned on X ;s, is unimodal, i.e., p ;i ðx ;r Þ¼ 4 p X jx;s i ¼ p X;r jx 1 i ;Xi ;s ;Y ð30þ is unimodal. Here, he index i represens he sample from he ih paricle. We sample he X ;s paricle, X;s i, from is sae ransiion prior (STP) bu use Laplace s approximaion [24], [23] o approximae p ;i by a Gaussian and sample X ;r from i. Thus, we sample X;r i from Nðmi ; i ISÞ, where where m i ¼ arg min½ log p ;i ðx ;r ÞŠ ¼ arg min L i ðx ;r Þ; X ;r X ;r i IS ¼ r 2 L i ðm i Þ 1 ; L i ðx ;r Þ¼ 4 log p Y jx i ;s ;X ;r þ log p X;r jx i 1 ;Xi ;s ;

8 586 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 and Nð; Þ denoes a Gaussian pdf wih mean and covariance marix. As shown in [26], unimodaliy of p ;i is ensured if he variance of STP of X ;r is small enough compared o disance beween he modes of OL given X ;s in any direcion. Even if X ;s is chosen so ha his holds for mos paricles, a mos imes, he proposed algorihm will work. 4.4 PF-Gordon and PF-EIS for Our Problem We summarize he basic PF (i.e., PF-Gordon [11]) for landmark shape filering in Algorihm 2. In order o develop he PF-EIS, we follow he seps as explained in Secion 4.3. The choices of X ;s and X ;r under his problem seup are jusified as follows: Since he STP of s ; is usually broad (o allow for occasional large camera moion or zoom), we use X ;s ¼½s ; Š and X ;r ¼½c ;z ;U Š. Noe ha, for he purpose of imporance sampling, only s ; ;c are he imporance sampling saes since z ;U are deerminisically compued from c and X 1. The paricles of X ;s are sampled from is sae ransiion prior, i.e., using he firs wo equaions of (26). Condiioned on he sampled scale and roaion, X;s i, i is much more likely ha p is unimodal, i.e., p ;i ðc ;U ;z Þ defined below is unimodal: p ;i ðc ;z ;U Þ¼p Y j h s i ;i ;z N c ; A c c i 1 ; c U g U 1 1 i ;zi z f z i 1 ;U ;c ; where Nðx; ; Þ denoes he value of a Gaussian pdf wih mean and variance compued a he poin x and is a proporionaliy consan. Since he pdfs of U, z, condiioned on c, X 1, are Dirac dela funcions, he above simplifies o p ;i ¼ p Y j h s i ;i ;f z i 1 ;gi ;c N c ; A c c i 1 ; c ðu g i Þ z f z i 1 ;gi ;c ¼ 4 p ;i ðc Þ ðu g i Þ z f z i 1 ;gi ;c ; ð31þ where g i ¼ 4 gðu 1 i ;zi 1Þ. The imporance sampling par of X ;r is only c. We compue he imporance densiy for c by approximaing p ;i ðc Þ by a Gaussian a is unique mode. The mode is compued by minimizing L i ðc Þ¼ log p ;i ðc Þ defined below: L i ðc Þ¼ log p Y j h s i ;i ;f z i 1 ;gi ;c þ log N c ; A c c i 1 ; ð32þ c : The PF-EIS algorihm for landmark shape racking is summarized in Algorihm 3. Algorihm 3. PF-EIS for Landmark Shape Filering Iniializaion: A ime ¼ 0, sample s ðiþ 0 Nðs 0 ; 2 s Þ, ðiþ 0 Nð 0 ; 2 Þ;cðiÞ 0 ¼ 0; z ðiþ 0 ¼ z 0 and U ðiþ 0 ¼ U 0. Here, i ¼ 1;...;N pf where, N pf is he number of paricles. For >0, 1) Imporance sample s i Nð ss i 1 ;2 s Þ, i Nði 1 ;2 Þ. i ¼ 1; 2;...;N pf 2) Compue m i ¼ arg min c L i ðc Þ and i IS ¼½r2 L i ðm i ÞŠ 1 where L i is defined in (32). 3) Imporance sample c i Nðmi ; i ISÞ. Compue U i ¼ gðui 1 ;zi 1 Þ and zi ¼ fðzi 1 ;Ui ;ci Þ. 4) Compue Imporance weighs as, w i ¼ ~wi P Npf where j¼1 ~wj ~w i ¼ wi pðy jhðs i ;i ;zi ÞÞN ðci ;A cc i 1 ; cþ 1 Nðc i ;mi ;i IS Þ. 5) Compue he MMPSE esimae ^z as he principal eigenvecor of P N pf i¼1 zi zi w i. Resample. 6) Se þ 1 and go o sep Tracking o Auomaically Exrac Landmarks In his secion, we describe ou echnique o rack and auomaically exrac landmark configuraions over a sequence of images or a video. The sysem comprises of a opical flow (OF) racker coupled wih filering. We compue opical flow a a cluser of poins around each currenly esimaed landmark locaion ½ ^S Š k (k denoes he kh landmark) and use his o move he cluser of poins ino he nex frame (frame þ 1). The cenroid of he moved cluser serves as he new observaion for he kh landmark a þ 1. The same hing is done for all landmark poins o ge Y þ1 (he vecor of observed landmark locaions a þ 1). This observaion is fed ino he NSSA-based PF which oupus he esimaed landmark locaions (and esimaed shape) a þ 1. The enire procedure is summarized in Algorihm 4. For compuing he opical flow, we used he code/mehod developed by [27]. Algorihm 4. Auomaic Landmark Exracion over a Sequence of Images Inpu: image(-1), image(), ^S 1 (esimaed landmark configuraion a 1) Oupu: fx i;wi g;i¼ 1; 2;..., ^S ¼ P N pf i¼1 ðzi esi þji þ i Þw i (esimaed landmark configuraion a ime ) where, z is he shape and e s,, are he global scale, roaion and ranslaion respecively. 1) For each esimaed landmark ½ ^S 1 Š k ;k¼ 1; 2;...K, compue opical flow a a cluser of poins around ½ ^S 1 Š k and use his o move he poins ino image(). Use he cenroid of he moved cluser as he kh observed landmark a ime, ½Y Š k. Do his for all landmark poins o ge Y 2) Run PF-Gordon using Y, i.e., implemen seps 1 and 2 of Algorihm 2. Bu his ime, we include he global ranslaion in he sae-space as well. 3) Display he esimaed landmarks locaion, ^S ¼ P N pf i¼1 ðzi esi þji þ i Þw i (esimaed landmark configuraion a ime ), se þ 1, and go o sep 1. 5 EXPERIMENTAL RESULTS We began by comparing he abiliy of NSSA o model reallife landmark shape sequences wih ha of SSA, ASM, and he wrong NSSA model (NSSA-unaligned) [1]. This is discussed in Secion 5.1. I was observed ha NSSA had much smaller modeling error han all hree. This comparison gave us he firs indicaion ha NSSA would provide a much more accurae prior dynamic model for Bayesian filering or racking applicaions, as well as also for synhesis/exrapolaion applicaions. Nex, we simulaed muliple realizaions of a nonsaionary shape aciviy along wih scale and roaion variaions and aemped o rack i using hree differen PF

9 DAS AND VASWANI: NONSTATIONARY SHAPE ACTIVITIES: DYNAMIC MODELS FOR LANDMARK SHAPE CHANGE AND APPLICATIONS 587 Fig. 3. Comparison of he MSE of esimaed configuraions compued using PF-EIS, PF-Douce, and PF-Gordon for simulaed shape sequence. EIS has he smalles MSE (discussed in Secion 5.2). algorihms: he original PF (PF-Gordon) was compared wih PF-Douce [23] and PF-EIS [26] (described in Secion 4.3). These comparisons are discussed in Secion 5.2. I was observed ha, when he number of available paricles is small, PF-EIS has he bes performance. Since mos exising work ha used SSA or ASM for racking used PF-Gordon, we reained his PF for mos of our comparisons beween NSSA, SSA, and ASM. The following four ses of experimens were done. In Secion 5.3, we demonsrae he superior abiliy of NSSA-based PF (PF- Gordon wih N pf ¼ 1;000 and PF-EIS wih N pf ¼ 50) o filer ou he landmark shapes from heavily noise-corruped and cluered observed landmark configuraions. In Secion 5.4, we compare he racking abiliy of NSSA-based PF wih SSAbased PF and ASM-based PF, i.e., heir abiliy o accuraely exrac ou landmarks from image sequences. Once again NSSA is found o be significanly superior o SSA and ASM and also o direc landmark exracion (wihou any modelbased filering). The use of 3D-NSSA o accuraely synhesize new human aciviy sequences is discussed in Secion 5.5. In Secion 5.6, we give a preliminary experimen ha demonsraes ha NSSA is able o remain in rack even when a model change occurs. 5.1 Modeling Error Comparison We used he Carnegie Mellon Moion Capure (CMU Mocap) daabase [6] for our experimens. Each file in he daabase had he coordinaes of he markers placed a he body landmark locaions (especially he body joins) for successive frames of a specific human moion aciviy, e.g., running, jumping, ec. An example is shown in Fig. 6. The corresponding 2D and 3D landmark shape sequences were used as he raining daa for learning he NSSA (or SSA or ASM) model parameers. We define modeling error (ME) as he race of he noise covariance marix of he AR modeling error, i.e., ME ¼ raceð c Þ, where c ¼ E½ðc A c c 1 Þðc A c c 1 Þ T Š and c are he aligned coefficiens of shape velociy. We also compue he modeling error when he c s are no aligned, i.e., when U is compued using SVD a each (as in [1]). When compuing error for SSA, c s are he angen space coefficiens of shape (no of shape velociy), i.e., all shapes z are projeced ino a single angen space a he common mean shape. For Fig. 4. Filering noise and cluer corruped MOCAP daa: comparing NSSA-, SSA-, and ASM-based PF-Gordon and also NSSA-based PF-EIS. NSSA significanly ouperforms SSA and ASM. NSSA-based PF-EIS wih jus 50 paricles has comparable performance o ha of 1,000 paricle NSSA-based PF-Gordon (discussed in Secion 5.3). ASM, modeling error is sill he same bu now c ¼ z, i.e., he shape space is assumed o be euclidean. We compued he modeling error of SSA, ASM, and NSSA (unaligned) for various human acions and compared wih ha of NSSA. I was found ha NSSA has much lower modeling error han all hree. The modeling error bar plos of 2D and 3D shape sequences have been shown in Figs. 2a and 2b for a few moion aciviies. Modeling error ells us how well we are able o capure he shape deformaion dynamics using he given model. I hus quanifies how well we can predic he shape a a ime insan given he informaion from he previous ime insan. Thus, lower modeling error will resul in beer racking abiliy and also a beer abiliy o synhesize a new sequence or o exrapolae an exising sequence (graphics problems). 5.2 Comparing PF Algorihms We simulaed landmark shape change of a se of K ¼ 5 landmarks (a deforming penagon) and racked i using PF-EIS (Algorihm 3), PF-Gordon (Algorihm 2) [11], and PF-Douce [23] wih N pf ¼ 50 paricles. The iniial shape z 0 was a regular penagon. The shape and global moion change of he configuraion followed (10), (26) wih c ¼ 0:0025I 6, 2 s ¼ 0:0001, 2 ¼ 0:25, A c ¼ 0:6I 6, and s ¼ 0:9. The observaions followed (28) wih 2 o ¼ 0:04 and p ¼ 0:2. I 6 denoes a 6 6 ideniy marix. I is o be noed ha he STP of scale (e s ) is a log-normal, and hence, even 2 s ¼ 0:0001 resuls in a fairly broad disribuion. Whenever one or more landmarks are generaed by cluer, he observaion likelihood (OL) of log-scale (s )is eiher heavy-ailed wih he wrong (oulier) mode or is mulimodal. When many landmarks are generaed by cluer, he same happens for. This combined wih a broad prior of s ; resuls in mulimodal p ðs ; Þ. Whenever his happens, mos paricles of PF-Douce end up sampling from a Gaussian abou he wrong mode of p ðs ; Þ or p ðs Þ, resuling in loss of rack. Bu, PF-EIS does no suffer from his problem since i samples from he prior of s ;. Also, since he prior of c is narrow compared o he disance beween likelihood modes, p ;i ðc Þ is usually unimodal, and hus, sampling from is Laplace s

10 588 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 Fig. 5. Filering ou rue shape daa from noisy/cluered observaions using PF-Gordon (moion aciviy: jump). The landmark poins (denoed by for ground ruh and u for he filer oupu) are fied wih recangular paches o visualize he body posure a a given ime insan. The racking performances of NSSA, SSA, and ASM are shown over four frames. (a) Ground ruh wih observaions (þ), (b) racked wih NSSA, (c) racked wih SSA, and (d) racked wih ASM. I can be clearly seen ha NSSA way ouperforms SSA and ASM. approximaion is valid. On he oher hand, for small N pf, PF-Gordon ofen loses rack because i samples all saes from he prior, hus resuling in small effecive paricle size. In Fig. 3, he mean squared error (MSE) plo averaged over 80 Mone Carlo runs is shown. Also, see Fig. 4 for MOCAP daa, where PF-EIS wih N pf ¼ 50 paricles is able o achieve almos he same racking accuracy as PF-Gordon wih N pf ¼ 1; Filering from Noisy and Cluered Observaions In his experimen, 2D landmark shape sequences from CMU Mocap daabase (refer Secion 5.1) were used as he ground ruh (as shown in Fig. 5a). We incorporaed random scale variaions, addiive noise, and cluer o he ground ruh. The observaions were simulaed using (28). This experimen helped us o quaniaively compare performance since ground ruh was available. We used he PF-Gordon (i.e., he basic PF) wih N pf ¼ 1;000 paricles for filering. Our primary objecive was o compare he performance of NSSA-based sysem model wih ha of SSA and ASM. PF-Gordon solely depends on he STP for imporance sampling, and hus, is performance heavily relies on he accuracy of he sysem model. Also, all pas works on SSA- or ASM-based racking used PF-Gordon. We considered wo moion aciviies, namely, run and jump, wih K ¼ 16 landmarks. Differen daa sequences were used for raining and esing. Our resuls are shown in Appendix I (run), which can be found on he Compuer Sociey Digial Library a hp://doi.ieeecompuersociey. org/ /tpami , and Fig. 5 (jump), and he Procruses disance comparison plos are given in Fig. 4. The landmark locaions were fied wih recangular paches o visualize he body posure a a given insan. The firs row shows he ground ruh and also he observaion. I can be seen ha he observed landmark locaions (represened as þ on op of he ground ruh) were severely corruped wih cluer and random noise. Thus, visually, he observed configuraions hardly conform o he human body. Bu, as shown in Fig. 5b, NSSA has been able o filer ou he rue shape from hose observaions wih very good accuracy. SSA (Fig. 5c) and ASM

11 DAS AND VASWANI: NONSTATIONARY SHAPE ACTIVITIES: DYNAMIC MODELS FOR LANDMARK SHAPE CHANGE AND APPLICATIONS 589 Fig. 6. Tracking landmark configuraions over he video of a running sequence (discussed in Secion 4.5). (a) NSSA, (b) SSA, and (c) ASM. I can be clearly seen ha NSSA way ouperforms SSA and ASM. (d) Landmark observaions exraced using purely opical-flow-based approach. I can be seen ha he observed landmark loses posure informaion gradually over ime. Such observaion leads o poor filering/racking performance of he sysem. (Fig. 5d), however, perform poorly in his job. Similar resuls were found for moion aciviy run (see Appendix I, which can be found on he Compuer Sociey Digial Library a hp://doi.ieeecompuersociey.org/ / TPAMI ). In Fig. 4, we plo he Procruses disance beween he esimaed shape and he rue shape a each insan as he quaniaive measure of filering accuracy. Also, noe ha PF-EIS (wih NSSA model) is able o achieve almos he same accuracy as PF-Gordon (wih NSSA model) wih as few as N pf ¼ 50 paricles. In case of SSA and ASM, filering performed around he fixed mean shape ends up generaing shape samples far away from he desired shape space where he original se of deforming shapes lies. This, in urn, led o heir poor performances. Bu NSSA, on he oher hand, does an excellen job in filering because of is ime varying mean shape assumpion. Of course, we assume ha he shape deviaions over successive ime frames are small enough o make sure he curren shape is in he neighborhood of he curren mean shape (i.e., he previous shape) for our mahemaical reamens o be valid. This is a reasonable assumpion for mos of he real-life moion aciviies. 5.4 Tracking and Auomaic Landmark Exracion For auomaic landmark exracion and racking, we used 50 frames of real-life videos of human aciviies (as shown in Fig. 6 (run) and Fig. 7 (jump)). A oal of 13 body landmarks were considered as shown in Fig. 7. The body landmarks were: righ shoulder, righ elbow, righ hand, righ hip, righ knee, righ foo, head, lef shoulder, lef elbow, lef hand, lef hip, lef knee, and lef foo. For run sequences, he sysem model was learned from he hand-marked ground ruh daa corresponding o he raining daabase. In case of jump, however, we used he mocap daa iself. We appropriaely subsampled he mocap daa frames in order o roughly synchronize hem o he video frames. Implemenaions of Algorihm 4 using sysem models based on NSSA, SSA, and ASM were compared. Their performances over muliple ime frames are shown in

12 590 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 Fig. 7. Tracking landmark configuraions over he video of a jump sequence wih NSSA-based sysem model (discussed in Secion 4.5). I can be seen ha a frame 15, NSSA los rack of one of he landmarks (righ hand). Bu afer ha, i regains rack of he landmark. Fig. 8. Synhesis of a 3D shape sequence corresponding o moion aciviy run. Four frames are shown. The sysem model was learned using 3D-NSSA on 3D landmark shape sequences for running. The sequence shown above visually resembles a running sequence. Fig. 6 (run). I can be clearly seen ha NSSA (Fig. 6a) performs much beer han SSA (Fig. 6b) or ASM (Fig. 6c) in erms of keeping rack of he body landmarks. I is very imporan o noe ha filering and predicion play a very crucial role in he process of exracing landmark observaions. To verify his, we exraced he landmark observaions purely based on opical flow, i.e., saring wih he iniial locaions, we used he opical flow beween successive frames o drif he poins, and hus, geing observaions over ime. This procedure does no use he knowledge of he sae esimaes a each insan o decide where he expeced landmark migh be while compuing he opical flow. As can be seen from Fig. 6d, quie a few of he observed landmarks in his case were found o ge suck a wrong locaions (due o background inerference, occlusion, or cluer) and from ha poin on hey were never in sync wih he body movemens. In fac, he poin of using he sae esimaes is o correc he locaions of such landmarks and making sure ha we do no end up compuing he OF a he wrong regions for geing he landmark locaion for he nex frame. Nex, we es he sysem wih NSSA-based sysem model on he video of a person jumping. I can be seen in Fig. 7 ha NSSA did a very good job in erms of racking he landmarks over various ime frames. I is o be noiced ha, a frame 15, i loses rack of he landmark corresponding o he righ hand. Bu laer, i regains he rack (see frame 39, Fig. 7) D-NSSA Synhesis Moivaed by he drasically small modeling error of 3D-NSSA for human acions (see Fig. 2b), we aemped o use 3D NSSA for a moion synhesis applicaion. We learned he deformaion model for 3D body shape of he human body while running from MOCAP daa. The synhesized run sequence using his model is shown in Fig. 8. Since here is no ground ruh in his case, we visually verified he sysems abiliy o synhesize run and he sysem performance was found o be promising. 5.6 Change Deecion wih NSSA We did a simple experimen o es he abiliy of he NSSAbased racker o deec changes in aciviy while sill no compleely losing rack. We used a sequence where a person begins by running, and hen, leaps (hp:// mocap.cs.cmu.edu:8080/subjecs/49/49_04.avi). Noice ha his is a fairly sudden change. The ELL-based change deecion saisic [28] was able o deec he change o leap and, for some ime afer he change also, our racker did no lose rack (see Fig. 9 racking error does no increase unil laer). More resuls and comparison wih SSA are shown in [1]. Deailed resuls and all MATLAB codes can be found a hp:// Fig. 9. The ELL and racking error (TE) plo for he shape sequence wih run followed by leap. PF-Gordon was used wih NSSA-based sysem model. The acual aciviy ransiion occurred a ¼ 132. I can be clearly seen ha ELL deeced he change. There is disinc spike of ELL around ¼ 132. However, he racker sill keeps racking even afer he change has occurred.

13 DAS AND VASWANI: NONSTATIONARY SHAPE ACTIVITIES: DYNAMIC MODELS FOR LANDMARK SHAPE CHANGE AND APPLICATIONS Shorcomings of NSSA: Classificaion Despie very good performances while racking/filering, in is curren form, NSSA does no perform as well for classificaion. We ried o perform a model-based maximum-likelihood classificaion among various moion aciviies (e.g., run, jump, si, ec.). The inpu o he classifier was he ime sequence of shape velociy coefficiens for NSSA, angen space coefficiens for SSA, and shape deviaion vecors for ASM. The oupu was he mos likely aciviy o have generaed he sequence. In our preliminary experimens wih run, si, jump, and dance aciviy sequences, NSSA had a 4 percen misclassificaion rae, while SSA and ASM had 2.5 and 2 percen, respecively. The reason NSSA does no perform as well as he res is he same as he reason i significanly ouperforms SSA and ASM for racking i is a more generic model for shape change. I consiss of a zero mean random walk model on shape and a zero mean AR model on shape velociies. The effec of iniial shape is los in a long sequence. To use NSSA for classificaion, we should modify he curren model and define a nonzero-mean shape velociy change model. Alernaively, a good idea would be o use NSSA for racking, i.e., for exracing landmark shape sequences from video, and hen, feeding hese ino a piecewise SSA [14] or piecewise ASM [13] based classifier. 6 CONCLUSIONS AND FUTURE WORK The key conribuion of his work is a novel approach o define a generaive model for boh 2D and 3D nonsaionary landmark shape sequences, which we call NSSA. The main idea is o compue he angen space represenaion of he curren shape in he angen space a he previous shape. This can be referred o as he shape velociy vecor since i quanifies he difference beween wo consecuive shapes projeced ino he angen space a he firs one. The aligned shape velociy coefficiens (shape speed vecor) are modeled using vecor ime-series mehods. Applicaions in filering, racking, synhesis (using 3D-NSSA models), and change deecion are demonsraed. Filering and racking are sudied in deail and significanly improved performance over relaed work is demonsraed (see Figs. 2, 3, 4, 5, 6, 7, 8, and 9). Wih he excepion of smoohing splines [29], [12], mos oher exising work does no model nonsaionary shape sequences. In oher work [30], we have also successfully demonsraed he use of he NSSA for model-based compression of landmark shape sequence daa. Fuure work includes developing he 3D synhesis, change deecion, and classificaion applicaions (see Secions 5.5, 5.6, and 5.7 for deails). Anoher possible fuure research direcion is combining our models wih GPVLMbased observaion exracion echniques [17], [31]. Also, our opical-flow-based landmark exracor could be improved by using ideas from [32]. ACKNOWLEDGMENTS Par of his paper appeared in [1], [2]. This work was parially funded by US Naional Science Foundaion (NSF) gran ECCS REFERENCES [1] N. Vaswani and R. Chellappa, Nonsaionary Shape Aciviies, Proc. IEEE Conf. Decision and Conrol, Dec [2] N. Vaswani and S. Das, Paricle Filer wih Efficien Imporance Sampling and Mode Tracking (PF-EIS-MT) and Is Applicaion o Landmark Shape Tracking, Proc. Asilomar Conf. Signals, Sysems and Compuers, [3] D. Kendall, D. Barden, T. Carne, and H. Le, Shape and Shape Theory. John Wiley and Sons, [4] N. Vaswani, A. RoyChowdhury, and R. Chellappa, Shape Aciviy : A Coninuous Sae HMM for Moving/Deforming Shapes wih Applicaion o Abnormal Aciviy Deecion, IEEE Trans. Image Processing, vol. 14, no. 10, pp , Oc [5] R. Harley and A. Zisserman, Muliple View Geomery in Compuer Vision. Cambridge Univ. Press, [6] CMU Moion Capure Daa, hp://mocap.cs.cmu.edu/, [7] I. Dryden and K. Mardia, Saisical Shape Analysis. John Wiley and Sons, [8] T. Cooes, C. Taylor, D. Cooper, and J. Graham, Acive Shape Models: Their Training and Applicaion, Compuer Vision and Image Undersanding, vol. 61, pp , Jan [9] A. Veeraraghavan, A. RoyChowdhury, and R. Chellappa, Maching Shape Sequences in Video wih an Applicaion o Human Movemen Analysis, IEEE Trans. Paern Analysis and Machine Inelligence, vol. 27, no. 12, pp , Dec [10] M. Isard and A. Blake, Condensaion: Condiional Densiy Propagaion for Visual Tracking, In l J. Compuer Vision, vol. 29, pp. 5-28, [11] N.J. Gordon, D.J. Salmond, and A.F.M. Smih, Novel Approach o Nonlinear/Nongaussian Bayesian Sae Esimaion, IEE Proc.-F (Radar and Signal Processing), vol. 140, no. 2, pp , [12] A. Kume, I. Dryden, and H. Le, Shape Space Smoohing Splines for Planar Landmark Daa, Biomerika, vol. 94, pp , [13] N. Paragios, M. Jolly, M. Taron, and R. Ramaraj, Acive Shape Models and Segmenaion of he Lef Venricle in Echocardiography, Proc. Fifh In l Conf. Scale Space and PDE Mehods in Compuer Vision, Apr [14] B. Song, N. Vaswani, and A.K. Roy-Chowdhury, Closedloop Tracking and Change Deecion in Muli-Aciviy Sequences, Proc. IEEE Conf. Compuer Vision and Paern Recogniion, [15] T.F. Cooes, G. Edwards, and C.J. Taylor, Acive Appearance Models, Proc. European Conf. Compuer Vision, vol. 2, pp , [16] B.P. Lelieveld, R.J. van der Gees, J.H.C. Reiber, J.G. Bosch, S.C. Michel, and M. Sonka, Time-Coninuous Segmenaion of Cardiac Image Sequences Using Acive Appearance Moion Models, Proc. In l Conf. Informaion Processing in Medical Imaging, pp , Jan [17] T. Tai-Peng, L. Rui, and S. Sclaroff, Tracking Human Body on a Learned Smooh Space, Technical Repor No , Boson Univ. Compuer Science, [18] S. Hou, A. Gaala, F. Caillee, N. Thacker, and P. Bromiley, Real- Time Body Tracking Using a Gaussian Process Laen Variable Model, Proc. IEEE In l Conf. Compuer Vision, Oc [19] A. Srivasava and E. Klassen, Bayesian and Geomeric Subspace Tracking, Advances in Applied Probabiliy, vol. 36, pp , Mar [20] S.X. Ju, M.J. Black, and Y. Yacoob, Cardboard People: A Parameerized Model of Ariculaed Image Moion, Proc. Second In l Conf. Auomaic Face and Gesure Recogniion, [21] J. Ken, The Complex Bingham Disribuion and Shape Analysis, J. Royal Saisical Soc., Series B, vol. 56, pp , [22] J. Shi and C. Tomasi, Good Feaures o Track, Proc. IEEE Conf. Compuer Vision and Paern Recogniion, pp , [23] A. Douce, On Sequenial Mone Carlo Sampling Mehods for Bayesian Filering, Technical Repor CUED/F INFENG/TR. 310, Dep. of Eng., Cambridge Univ., [24] L. Tierney and J.B. Kadane, Accurae Approximaions for Poserior Momens and Marginal Densiies, J. Am. Saisical Assoc., vol. 81, no. 393, pp , Mar [25] S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, A Tuorial on Paricle Filers for On-Line Non-Linear/Non-Gaussian Bayesian Tracking, IEEE Trans. Signal Processing, vol. 50, no. 2, pp , Feb

14 592 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 4, APRIL 2010 [26] N. Vaswani, Paricle Filering for Large Dimensional Sae Spaces wih Mulimodal Observaion Likelihoods, IEEE Trans. Signal Processing, vol. 56, no. 10, pp , Oc [27] S. Khan, Malab Code for Opical Flow Using Lucas Kanade Mehod, [28] N. Vaswani, Addiive Change Deecion in Nonlinear Sysems wih Unknown Change Parameers, IEEE Trans. Signal Processing, vol. 55, no. 3, pp , Mar [29] A. Kume, I. Dryden, H.L. Le, and A. Wood, Fiing Cubic Splines o Daa in Shape Spaces of Planar Configuraions, Proc. Saisics of Large Daases, [30] S. Das and N. Vaswani, Model-Based Compression of Nonsaionary Landmark Shape Sequences, Proc. IEEE In l Conf. Image Processing, [31] N. Lawrence, Probabilisic Non-Linear Principal Componen Analysis wih Gaussian Process Laen Variable Models, J. Machine Learning Research, vol. 6, pp , Nov [32] A. Srivasava and I.H. Jermyn, Looking for Shapes in Two- Dimensional Cluered Poin Clouds, IEEE Trans. Paern Analysis and Machine Inelligence, vol. 31, no. 9, pp , Sep Samarji Das received he BTech degree in elecronics and communicaions engineering from he Indian Insiue of Technology (IIT), Guwahai, in May He is currenly working oward he docoral degree in he Deparmen of Elecrical and Compuer Engineering a Iowa Sae Universiy. His research ineress are in compuer vision and image processing, saisical signal processing, and machine learning. He is a suden member of he IEEE. Namraa Vaswani received he BTech degree in elecrical engineering from he Indian Insiue of Technology (IIT), Delhi, in Augus 1999, and he PhD degree in elecrical and compuer engineering from he Universiy of Maryland, College Park, in Augus Her PhD hesis was on change deecion in sochasic shape dynamical models and applicaions o aciviy modeling and abnormal aciviy deecion. She was a posdocoral fellow and research scienis a he Georgia Insiue of Technology during She is currenly an assisan professor in he Deparmen of Elecrical and Compuer Engineering a Iowa Sae Universiy. Her research ineress are in esimaion and deecion problems in sequenial signal processing, biomedical image sequence analysis, and compuer vision. Her curren research focus is on sequenial compressed sensing and largedimensional paricle filering. She is a member of he IEEE.. For more informaion on his or any oher compuing opic, please visi our Digial Library a