Multi-Layer Joint Gait-Pose Manifold for Human Motion Modeling

Transcription

1 Mult-Layer Jont Gat-Pose Manfold for Human Moton Modelng Meng Dng and Guolang Fan School of Electrcal and Computer Engneerng, Oklahoma State Unversty, USA Abstract We present a mult-layer jont gat-pose manfold (mult-layer JGPM) for human moton modelng to enhance the representatve capablty of the orgnal JGPM that represents gat knematcs by two varables. One s the pose to denote a seres of stages n a walkng cycle and the other s the gat to reflect the ndvdual walkng styles. Couplng pose and gat varables n the same latent space was shown effectve for human moton estmaton. However, the orgnal JGPM s lmted to one knd of human gats, and ts learnng cannot be scaled up to a large dataset due to a hgh computatonal load. Ths work overcomes the lmtatons of the prevous method by nvolvng a mult-layer topology pror that s able to accommodate a varety of walkng styles, leadng to better moton synthess results. Moreover, to learn mult-layer JGPM effectvely and effcently, we adopted two technques, tranng data dversfcaton and topology-aware local learnng. The expermental results confrm the advantages and superorty of our proposed method over several exstng Gaussan process-based moton models. I. INTRODUCTION Human moton modelng s an actve research topc n the feld of computer vson and machne learnng due to ts wde applcatons, such as survellance, human computer nteracton, bomechancs, etc. It s commonly beleved that the hgh dmensonal human moton data le ntrnscally on low dmensonal manfolds. Varous manfold learnng algorthms were proposed to explore a compact moton model as a pror to well constran the soluton space of posture estmaton. For example, Local Lnear Embeddng (LLE) [1] and Isometrc Feature Mappng (Isomap) [2] seek to preserve the local geometrcal neghborhood among hgh dmensonal (HD) data n the low dmensonal (LD) latent space. Whle they were appled n [3], [4] for human moton modelng, LLE and Isomap provde nether a probablty dstrbuton over the latent space nor a LD-HD mappng functon. Some probablstc dmensonalty reducton methods that can learn a latent space along wth a LD-HD mappng were also proposed, such as Gaussan Process Latent Varable Model (GPLVM) [5] and ts varants [6], [7], [8], [9], [10] that were developed for human moton modelng. GPLVM can capture the ntrnsc structure of redundant hgh dmensonal data. For example, Fg. 1 shows a latent space wth two crcular-shaped concentrc manfolds learned by GPLVM from two mage dataset that have the same rotated dgts at two dfferent scales. It s easy to see ths from the Prncpal Component Analyss (PCA) perspectve. The two mage subsets should have smlar egenvectors used to span a low dmensonal space. The radus of the crcular manfold s Ths work s supported by two OHRS awards (HR and HR12-30) from the Oklahoma Center for the Advancement of Scence and Technology. represented by the magntude of the data projecton on the frst two egenvectors. Fg. 1. Two approxmately crcular manfold of the rotated dgts dataset are learned by GPLVM n a 2D latent space. In ths work, we wll explore ths mult-layer manfold learnng dea n the context of complex gat modelng, where our objectve s to enhance the representatveness and dversty of the moton model. Specfcally, a term of pose manfold was often used to represent the sequental and cyclc pattern of human moton. The dea of gat manfold was proposed n [11] to represent the varablty of dfferent walkng styles from multple ndvduals, where both the pose and gat manfolds are assumed to have a crcular-shaped closed-loop structure. To capture the couplng effect between pose and gat manfolds, a jont gat-pose manfold (JGP- M) was further proposed n [12], where a torodal-shaped structure was employed to unfy pose and gat varables nto one manfold structure and a modfed topology-constraned GPDM (LL-GPDM) [10] was used for learnng JGPM. It was shown that JGPM outperforms exstng GPLVM-based models n the case of normal human gats. However, JGPM may not be applcable to more complex gats wth smaller or larger moton ranges. Also, just lke tradtonal GPLVMbased models, learnng JGPM s computatonally expensve and may not be scaled-up to a large tranng dataset. In ths work, we propose a mult-layer JGPM that s capable of dealng wth a varety of walkng styles wth both small or large moton ranges. Interestngly, ths model can stll be learned effcently from lmted tranng data. There are two new deas proposed. The frst s tranng data dversfcaton that creates a seres of smulated tranng gats wth dfferent moton ranges from a lmted tranng dataset. The second s topology-aware local learnng that extends the stochastc gradent descent algorthm proposed n [13] by selectng local neghbors accordng to the topology pror. The experments show that our mult-layer JGPM has great flexblty and capablty of representng a wde ranges of gats wth dfferent moton ranges. Moreover, t s shown that mult-layer JGPM s more effectve n humanod moton synthess than the orgnal JGPM and other GPLVM-based moton models.

2 II. PRELIMINARY A. GPLVM, GPDM and LL-GPDM Let Y = [y 1,...,y N ] T (y R D ) represent the HD data and X = [x 1,...,x N ] T (x R d ) are latent ponts. GPLVM nvolves a lkelhood functon of the HD data gven latent postons p(y X,β) = 1 (2π) ND K exp ( 1 2 tr ( K 1 YY T )), (1) where K s a N N covarance matrx whose entres are defned by the kernel functon, K(, j) = k(x,x j ). GPLVM s learned by maxmzng the lkelhood n (1). Consderng the sequental nature of human moton data, GPDM [7] augments GPLVM by a Gaussan Process (GP)- based latent dynamcal model p(x α) defned as p(x α) = p(x 1) exp Z 1 ( 1 2 tr ( K 1 X X 2:NX T 2:N) ), (2) where Z 1 s a normalzaton factor, p(x 1 ) s a Gaussan pror, X 2:N = [x 2,...,x N ] T, and K X s the (N 1) (N 1) kernel matrx constructed from X 1:N 1 = [x 1,...,x N 1 ] T and α s the kernel hyperparameters. Furthermore, to model dfferent moton types ( walkng and runng ) n the same latent space, LL-GPDM [10] ncorporates a LLE energy functon p(x W) n GPDM to encourage a cylnder-shaped latent structure. Then, learnng LL-GPDM was to maxmze the posteror, where p(x W) was nvolved as a topology pror: p(x,α,β Y,W) p(y X,β)p(X α)p(α)p(β)p(x W), (3) where p(α) and p(β) are pror models for hyperparameters. B. Jont Gat-Pose Manfold (JGPM) In [12], a torodal structure was used to learn JGPM that can unfy the pose and gat manfolds n one latent space, where a horzonal and vertcal crcle represents a pose-specfc gat manfold and a gat-specfc pose manfold respectvely. In the polar coordnate system, a torodal structure can be parameterzed by four varables p,g [0,2π) and R, r, whch represent two angular varables pose and gat, as well as two rad of the horzontal (gat manfold) and vertcal (pose manfold) crcles. The four varables are shown n Fg. 2 where JGPM s ntalzed as a torus. The learnng of JGPM s to optmze p,g,r,r by maxmzng the posteror probablty smlar to the one defned n (3) where the torodal topology s ncorporated. III. MULTI-LAYER JGPM We propose a mult-layer JGPM that focuses on the two major lmtatons of the orgnal JGPM,.e., modelng flexblty and computatonal complexty. A. Motvaton The orgnal JGPM only nvolves a small tranng dataset 20 gats wth a smlar moton range from the CMU Mocap Lbrary [14]. Intutvely, ncludng more dverse walkng styles as the tranng data s helpful to enhance the flexblty and robustness of general moton modelng. However, t may not be practcal to collect a large moton dataset from many subjects, and t would be practcally useful f we can generate more smulated moton data from a lmted tranng set. Gven a set of body jonts defned by a tree-structured skeleton model, human moton data are usually represented by two forms: the 3D postons and 3D Euler angles at each jont. Especally, the latter representaton can drectly reflect the moton range of each body segment durng a gat. It s ntrgung to use multple scalng factors to dversfy the tranng data by adjustng the standard devaton whle mantanng the mean of Euler angles at each jont, by whch a mult-layer manfold could be learned to represent a varety of walkng styles wth dverse moton ranges. We use the followng example to test ths dea tentatvely. Fg. 3 (top three rows) shows the moton scalng results, where two scalars, 1.25 and 0.5, are used to create two scaled gat sequences, and the correspondng latent spaces generated by GPLVM wth back constrant [6] and PCA are shown n Fg. 3(a) and (b). The two latent spaces reveal some nterestng relatonshp between the moton ranges and the rad of pose manfolds,.e., a wder moton range results n a larger radus of the learned pose manfold, vce versa. Fg. 2. The llustraton of pose,gat,r and r varables on JGPM. Fg. 3. Illustraton of the scaled moton and two latent spaces generated by GPLVM wth back constrant and PCA respectvely.

3 B. Tranng Data Dversfcaton One major assumpton behnd ths moton scalng dea s that a new gat can be approxmated by a tranng gat by scalng the dynamc range of Euler angles whle mantanng the mean at each jont. Although ths assumpton s worth further scrutny, we wll take ths dea to dversfy the ntal tranng data n order to learn a more robust and flexble moton model. Let y (k) u,v represents a 3D Euler angle vector ncludng three rotatons,.e., ptch, yaw and roll, where u, v denote u th pose n v th gat sequence and k s the bone jont ndex. The new smulated moton data y (k) u,v s generated by y (k) u,v = 1 n n u=1 y (k) (y(k) u,v + s u,v 1 n n u=1 y (k) u,v), (4) where n s the number of poses n a gat sequence and s s the scalng factor. In practce, t was found that a scalng factor between ( ) can lead to a realstc lookng gat. Wthout loss of generalty, we wll consder two scalng factors (0.4 and 1.25) to trple the sze of tranng data. As shown by latter experments, ths smple yet effectve way can multply a lmted tranng dataset wth more dversty and varablty. relatonshp of all tranng gats and all poses n a gat along ther respectve manfolds) s the same as the orgnal JGPM. D. LLE-based Topology Constrants After constructng the mult-layer torodal structure shown n Fg. 4, we need to ncorporate ths specfc topology pror nto a GPLVM-based learnng framework. Dfferent from the orgnal LLE method, where the local neghborhood relatonshp of HD data was preserved n LD manfold, our am s to mantan the neghborhood of a specfc LD structure so that the learned manfold could resemble our topology pror. Therefore, nstead of fndng the K nearest neghbors n the HD data, we frst defne a set of adjacent ponts {x j } j η for each pont x, where η s the collecton of all neghbors for the th pont. To preserve both the topologcal structure wthn a layer and across layers, η should nclude some wthn-layer and cross-layer neghbors. Specfcally, for a gven pont x, η = {ϕ () 1 m,ψ() 1 n } whch store the ndexes of m wthn-layer and n cross-layer neghbors. The basc prncple of neghbor selecton s that we expect to have a stronger wthn-layer constrant than the cross-layer one,.e., m > n. Consderng the computaton complexty, we select 16 (m = 10 and n = 6) neghbors for a reference pont as shown n Fg. 5. For a pont on the mddle layer, 3 cross-layer neghbors are selected from each of the outer and nner layers. For a pont on the outer layer, 6 cross-layer neghbors are from the mddle layer only, whle for a pont on the nner layer, 6 cross-layer neghbors are from the mddle layer only. Fg. 4. A three-layer torodal structure as a topology pror. C. Mult-layer Torodal Structure Correspondngly, we ntroduce a three-layer torodal structure as shown n Fg. 4 as a new topology pror to ntalze the mult-layer JGPM. The outer layer represents the moton data whch have a larger range (scalar 1.25); those wth a smaller range (scalar 0.4) are embedded nto the nner layer; the mddle layer represents the orgnal moton data. Hence, every ntal pont ndexed by (p,g,s) on the topology pror can be unquely defned by a 3D coordnate [t (p,g,s) x,t y (p,g,s),t z (p,g,s) ] T as t x (p,g,s) = (R + r (s) cos(α))cos(β), t y (p,g,s) = (R + r (s) cos(α))sn(β), (5) t z (p,g,s) = r (s) sn(α), where p,g,s are the ndexes of pose, gat and scale; α and β are two angular values correspondng pose p of gat g. R and r (s) (s = 1,2,3) are the rad of one horzontal (along the gat manfold) and three vertcal crcles (along three pose manfolds). In ths work, r (1) : r (2) : r (3) = 0.4 : 1 : Ths three-layer structure wll be used to ntalze the multlayer JGPM where the topology of each layer (the orderng Fg. 5. Neghborhood confguratons of a reference pont (red cross) on (a) nner layer (b) mddle layer (c) outer layer. Dfferent colors mean the neghbors are from dfferent layers. In LLE, the defnton of covarance C jk = (y y j ) T (y y k ) wth j,k η s used to compute the wght matrx W n hgh dmensonal space. To reflect the pror knowledge,.e., the mult-layer torodal topology, we specfy a unque covarance matrx for each latent dmenson usng the coordnates of latent ponts n (5): C x jk = ( t (p,g,s ) x t (p j,g j,s j )) T ( (p x t,g,s ) x t (p k,g k,s k )) x, C y jk = ( t (p,g,s ) y t (p j,g j,s j )) T ( (p y t,g,s ) y t (p k,g k,s k )) y, (6) C z jk = ( t (p,g,s ) z t (p j,g j,s j )) T ( (p z t,g,s ) z t (p k,g k,s k )) z, where j,k η. (p,g,s ), (p j,g j,s j ), and (p k,g k,s k ) are ndexes of x, x j and x k, respectvely, from whch we can fnd the 3D coordnates of three ponts accordng to (5).

4 To compute w n each dmenson,.e., {w (τ) τ (x,y,z)}, we solve the followng equatons: C x jk wx j = 1, k C y jk wy j = 1, (7) k k C z jk wz j = 1, where C x jk,cy jk,cz jk are defned n (6), and then normalze the weght vector. Gven the whole weght matrx W, whch s comprsed by w (τ), where = 1,...,N and τ (x,y,z), the LLE energy functon p(x W) s defned as p(x W) τ (x,y,z) exp{ 1 σ 2 N =1 x (τ) w (τ) j x (τ) j 2 }, (8) j η where x (τ) represents a coordnate of x along dmenson τ, w (τ) j s an element of w (τ) and σ represents a scalng term. Usng the energy functon above, we can ncorporate the topology constrant nto the LL-GPDM learnng framework defned n (3) to encourage the manfold to resemble the topologcal pror. IV. TOPOLOGY-AWARE LOCAL LEARNING Tradtonal GPLVM-based learnng algorthms struggle to learn a model from a large-scale dataset, because the computaton complexty grows cubcally wth the number of tranng samples. Exstng sparsfcaton technques approxmate the kernel matrx wth an actve set [15], [16] or a set of nducng varables [17], [18], however, they stll have been lmted to around one or only a few thousand tranng examples, because the number of nducng varables s proportonal wth the amount of tranng data. Here we seek to a fast and effectve sparsfcaton GPLVM-based learnng algorthm to the dversfed tranng data. A. Neghborhood-based Local Learnng GPLVM s learned by maxmzng the lkelhood n (1), whch s equvalent to mnmze the negatve log lkelhood L = lnp(y X,β) = DN 2 ln(2π) D 2 ln K 1 2 tr ( K 1 YY T ), (9) To mnmze L, the gradent of L wth respect to X s computed as L X = L K K X = ( K 1 YY T K 1 DK 1) K X, (10) K s the N N kernel matrx, where N s the number of tranng data. The computaton complexty of K 1 s O(N 3 ), whch consderably lmts the applcaton of GPLVM for large-scale tranng dataset. The man dea of exstng sparsfcaton technques s to reduce the dmensonalty of the kernel matrx K. Inspred by [13], where a stochastc gradent descent algorthm for the GPLVM was proposed, we develop a smlar strategy to teratvely approxmate the gradent by usng a small number of local samples, whch supports effcent mult-layer JGPM learnng. Compared wth the standard GPLVM algorthm, where all the tranng samples are taken nto account at the same tme to compute the gradent, our local learnng algorthm nvolves only a small number of tranng examples at one tme to approxmate the gradent locally. Frst, a reference pont x l s selected randomly and a neghborhood X L centered at x l s defned. Then, all the ponts n the neghborhood X L are used to compute the local gradent for updatng the latent varable X locally and the kernel parameters. The local gradent can be represented only by the ponts wthn the neghborhood L = ( K 1 L X Y LY T L K 1 L ) K L DK 1 L, (11) L X L where K L s the kernel matrx for X L, Y L s the correspondng HD moton data n the neghborhood and D s the dmensonalty of moton data. Because the dmensonalty of K L s small, the computaton cost s rather low. Dfferent wth [13], there are two specal treatments for the local learnng n ths work. The frst one s the ntegraton of our multlayer topology nto the GPLVM-based learnng framework, and the other s topology-based neghborhood selecton (to be dscussed n detals n Secton IV-B). To ncorporate the topology constrants, we use p(x L W L ) from the LLE energy functon n (8) to express the local topology constrant, where W L s the correspondng weght matrx of latent ponts wthn the neghborhood. Every tme we randomly choose a latent pont as the reference pont and repeat the above local gradent operaton to optmze one patch of the model wth respect to the maxmum a posteror probablty (MAP). The posteror probablty s defned as p(x L,α,β Y L,W L ) p(y L X L,β)p(X L α) p(α)p(β)p(x L W L ), (12) After suffcent teratons, all the latent ponts may have been updated many tmes and the mult-layer JGPM s optmzed. Next, we wll further dscuss our treatment for the neghborhood selecton. B. Topology-based Neghbor Selecton In [13], a neghborhood selecton strategy of subsamplng k neghbors from a larger neghborhood was suggested for allowng suffcent coverage of the latent space. As ponted by the authors, ths method may not mantan the neghborhood confguraton. In our case, ths subsamplng method s not sutable as t may nterrupt the contnuty of latent varables and the layered structure n the mult-layer JGPM. Thus we have two specal consderatons for neghborhood selecton. Frst, both wthn-layer and cross-layer neghbors are nvolved durng the learnng process rather than learnng each layer separately. Second, because the torodal structure s symmetrc, we can pre-compute a set of neghbors to have suffcent coverage of the latent space, at the same tme, to avod the stuaton that the gradent estmatons are too local to capture the global structure of the latent space. Note that ths neghborhood selecton for the local learnng s dfferent

5 Topology-based neghbor selecton at three locatons (a, b, c) n the mddle layer: a reference pont (n red) and ts neghbors (n green, magenta and cyan). Fg. 6. wth the neghborhood choosng for the LLE-based topology constrant n Sec. III-D. In ths work, we have 60 gats (scaled from 20) and each gat ncludes 30 poses n the tranng dataset, that s there are 1800 ntal ponts n three layers. To have a tradeoff between a suffcent coverage n the latent space and a reasonable computatonal load, we select the 120 nearest neghbors accordng to the Eucldean dstance gven the mult-layer torodal structure to determne the neghborhood for each reference pont, as shown n Fg. 6. Ths topologybased neghbor selecton wll lead to a topology-aware local learnng process that ensures the learned manfold structure comples wth the topologcal pror. Fg. 6 exhbts that for a pont (n red) n the mddle layer, neghbors (n green, magenta and cyan) wth dfferent pose/gat/scalng ndexes are ncluded n ts neghborhood. Ths reveals that both the wthn-layer and cross-layer constrants are nvolved durng the topology-aware local learnng. V. E XPERIMENTAL R ESULTS In ths secton, we evaluate the proposed mult-layer JGPM by comparng t wth the orgnal JGPM [12] and LL-GPDM [10] n terms of three aspects,.e., latent space llustraton, moton extrapolaton and flterng, and moton synthess. A. Latent Space Illustraton Frst, we compared the mult-layer JGPM wth JGPM and LL-GPDM by llustratng the volumetrc representaton of ther latent space n Fg. 7, where the color ndcates the predcton confdence (the warmer colors, the hgher confdence of moton reconstructon). LL-GPDM has a cylnderlke latent structure, but t only represents the pose manfold explctly and treats the gat varable mplctly. Both JGPM and mult-layer JGPM acheved a well organzed, smooth and compact latent space that s our expectaton for the human moton modelng. However, from the cross-secton vew, t s obvous that mult-layer JGPM has larger hghconfdence areas than JGPM, mplyng ts more powerful moton modelng capablty. It s expected that mult-layer JGPM s more flexble and robust for moton synthess and pose estmaton. Next, we wll evaluate the mult-layer JGPM n terms of moton extrapolaton and nosy flterng. Volumetrc vsualzaton of predcton confdence n latent spaces; warmer colors, (.e., red) depct hgher confdence of moton reconstructon. (a) LL-GPDM (b) JGPM (c) mult-layer JGPM Fg. 7. B. Moton Extrapolaton and Flterng To verfy the advantage of the proposed mult-layer JGPM, we compare t wth JGPM and LL-GPDM usng the unknown test data wth dfferent moton ranges for two specfc tasks,.e., moton extrapolaton and flterng. 1) Moton Extrapolaton: We chose twenty walkng sequences whch are dfferent wth the tranng data from the CMU Mocap Lbrary as our orgnal unknown test data for moton extrapolaton. Each test sequence has 30 poses downsampled from one walkng cycle. Then, as defned n (4), we generated four sets of smulated moton data by

6 Fg. 8. Moton extrapolaton results, where the red and blue ponts represent the ground-truth and estmated results respectvely. Fg. 9. Moton extrapolaton results of the real strde sequences, where the red and blue ponts represent the ground-truth and estmated results respectvely. usng four scalars 1.25, 0.667, 0.5 and 0.4, whch represent a seres of moton ranges. We notce scalars and 0.5 are dfferent wth those (0.4 and 1.25) used for tranng data dversfcaton. In addton, we also acqured two sets of real strde sequences from CMU Mocap dataset (Subject No.7, tral No.11 and Subject No.8, tral No.5). We developed a valdaton method, by whch new moton data were extrapolated to represent the unknown test data from a GPLVM-based moton model, and we appled ths method to all models. We computed the averaged 3D jont poston errors (mm) between the estmated motons and ground truth ones n all of the test sequences. The extrapolaton results are llustrated n Fg. 10. Error (mm) Fg. 10. Extrapolaton Comparson mult layer JGPM orgnal JGPM LL GPDM s=1.25 s=1 s=0.667 s=0.5 s=0.4 strde Scalng factor Comparson of extrapolaton results. It s shown that the mult-layer JGPM s more accurate than the orgnal JGPM and LL-GPDM to represent the unknown data, especally when the moton data wth larger or smaller scalng factors, whch mples the superor flexblty and robustness of mult-layer JGPM. Fg. 8 vsualzes the extrapolaton results of some smulated test data usng stck man, where the red ponts represent the ground-truth and the blue ponts are the extrapolaton results. Also, Fg. 9 shows the extrapolaton results of real strde moton sequence. Obvously, the mult-layer JGPM has better performance. 2) Nosy Moton Flterng: A better moton model should provde better flterng results. In ths experment, we utlzed the same unknown test data as we used n the prevous experment to compare the flterng performance of all moton models. For each scaled dataset and strde moton, three nosy moton datasets were generated by addng addtve whte Gaussan nose (AWGN) at three levels (5%, 10% and 15%) wth respect to the standard devaton of each jont angle. The flterng process was repeated by fve tmes usng fve sets of random nose and then we obtaned the mean errors for each nose level. Fg.11 shows that the mult-layer JGPM s more accurate (less errors) and robust (less standard devatons) than JGPM to flter the unknown moton data n all three nose level as well as all the moton ranges. It s nterestng to fnd that not only for the scaled and real strde moton sequences, but also for the orgnal unscaled moton data (s = 1), the proposed mult-layer JGPM demonstrate sgnfcant advantages.

7 Flterng Comparson ( Nose 5%) Flterng Comparson ( Nose 10%) Flterng Comparson ( Nose 15%) mult layer JGPM orgnal JGPM LL GPDM mult layer JGPM orgnal JGPM LL GPDM mult layer JGPM orgnal JGPM LL GPDM Error (mm) Error (mm) Error (mm) s=1.25 s=1 s=0.667 s=0.5 s=0.4 strde Scalng factor 0 s=1.25 s=1 s=0.667 s=0.5 s=0.4 strde Scalng factor 0 s=1.25 s=1 s=0.667 s=0.5 s=0.4 strde Scalng factor Fg. 11. Nosy subjects flterng results usng mult-layer JGPM, the orgnal JGPM and LL-GPDM. C. Moton Synthess va Latent Space Samplng To further evaluate the two JGPM models, we can sample ther latent spaces along certan trajectory and vsualze the reconstructed moton data accordngly. In ths experment, we used three samplng trajectores,.e., a horzonal straght lne, a large crcular spral outsde and a small crcular spral nsde, as shown n Fg. 12. For the frst trajectory, we expect there should be a gradual moton range ncrease under the same pose. For the latter two trajectores, we expect to see two walkng sequences wth two extreme moton ranges. As shown n Fg. 12, the orgnal JGPM offer lmted capablty to nterpolate humanod walkng moton wth dfferent styles, especally very large or small moton ranges. The dstorton becomes more severe when samples are away from the learned manfold structure. Compared wth JGPM, the multlayer JGPM has great flexblty to synthesze humanod walkng moton wth varous styles. VI. CONCLUSIONS In ths paper, we have proposed a mult-layer jont gatpose manfold (JGPM) for human moton modelng that drectly addresses two major lmtaton of the orgnal sngle layer one by two technques, tranng data dversfcaton and topology-aware local learnng. The frst technque allows us to learn a more powerful model wthout ncreasng the amount of the orgnal tranng data and s smple yet effectve to generate a rch set of smulated tranng moton wth dfferent walkng styles. Ths data multplcaton technque naturally leads a mult-layer torodal structure as a topologcal pror for manfold learnng. Also, the second technque was nspred by the stochastc gradent descent algorthm wth some specal consderatons on the topologcal pror to ensure that model learnng can be mplemented effcently and effectvely on a larger tranng dataset and the resultant manfold s complant wth the topologcal pror both locally and globally. The expermental results show that the new mult-layer JGPM outperforms other state-of-the-art GPLVM models n terms of moton extrapolaton, flterng and synthess. REFERENCES [1] S. T. Rowes and L. K. Saul, Nonlnear dmensonalty reducton by locally lnear embeddng, Scence, vol. 290, no. 5500, pp , [2] J. B. Tenenbaum, V. de Slva, and J. C. Langford, A global geometrc framework for nonlnear dmensonalty reducton, Scence, vol. 290, no. 5500, pp , [3] A. Elgammal and C.-S. Lee, Separatng style and content on a nonlnear manfold, n Proc. CVPR, [4] C.-S. Lee and A. Elgammal, Modelng vew and posture manfolds for trackng, n Proc. ICCV, [5] N. Lawrence, Probablstc non-lnear prncpal component analyss wth gaussan process latent varable models, Journal of Machne Learnng Research, vol. 6, pp , [6] N. Lawrence and J. Q. Candela, Local dstance preservaton n the gplvm through back constrants, n Proc. ICML, [7] J. Wang, D. Fleet, and A. Hertzmann, Gaussan process dynamcal models for human moton, IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 30, pp , [8] K. Grochow, S. L. Martn, A. Hertzmann, and Z. Popovć, Stylebased nverse knematcs, n ACM SIGGRAPH 2004 Papers, 2004, SIGGRAPH 04, pp [9] R. Urtasum, D. Fleet, and P. Fua, 3D people trackng wth gaussan process dynamcal models, n Proc. CVPR, [10] R. Urtasun, D. Fleet, A. Geger, J. Popovc, T. Darrel, and N. Lawrence, Topologcally-contrant latent varable models, n Proc. ICML, [11] X. Zhang and G. Fan, Dual gat generatve models for human moton estmaton from a sngle camera, IEEE Trans. on System, Man, and Cybernetcs, vol. 40, pp , [12] X. Zhang and G. Fan, Jont gat-pose manfold for vdeo-based human moton estmaton, n Proc. CVPR Workshop on Machne Learnng for Vson-based Moton Analyss, [13] A. Yao, J. Gall, L. V. Gool, and R. Urtasun, Learnng probablstc non-lnear latent varable models for trackng complex actvtes, NIPS, [14] CMU Human Moton Capture Database, Avalable at [15] N. Lawrence, Learnng for larger datasets wth the gaussan process latent varable model, AISTATS, [16] N. Lawrence, M. Seeger, and R. Herbrch, Fast sparse gaussan process methods: The nformatve vector machne, NIPS, vol. 15, pp , [17] J. Quñonero Candela and C. E. Rasmussen, A unfyng vew of sparse approxmate gaussan process regresson, J. Mach. Learn. Res., vol. 6, pp , Dec [18] E. Snelson and Z. Ghahraman, Sparse gaussan processes usng pseudo-nputs, n Advances n Neural Informaton Processng Systems , pp , MIT press.

8 Fg. 12. Moton synthess by samplng JGPM (left) and the mult-layer JGPM (rght).