A Neural Network Approach to Missing Marker Reconstruction

Transcription

1 A Neural Nework Approach o Missing Marker Reconsrucion Taras Kucherenko Hedvig Kjellsröm Deparmen of Roboics, Percepion, and Learning KTH Royal Insiue of Technology, Sockholm, Sweden {arask,hedvig}@kh.se Absrac Opical moion capure sysems have become a widely used echnology in various fields, such as augmened realiy, roboics, movie producion, ec. Such sysems use a large number of cameras o riangulae he posiion of opical markers. These are hen used o reconsruc he moion of rigid objecs or human ariculaed bodies, o which he markers are aached. The marker posiions are esimaed wih high accuracy. However, especially when racking ariculaed bodies, a fracion of he markers in each imesep are missing from he reconsrucion. In his paper, we propose o use a neural nework approach o learn how human moion is emporally and spaially correlaed, and reconsruc missing markers posiions hrough his model. We experimen wih wo differen models, one LSTM-based and one window-based. Experimens on he CMU Mocap daase show ha we ouperform he sae of he ar by 20% - 400%. I. INTRODUCTION Capuring errors (some marker posiions no observed) Ground ruh moion Observed moion Ofen a digial represenaion of human moion is needed. This represenaion is useful in a wide range of scenarios: mapping an acor performance o a virual avaar (in movie producions or in he game indusry) [1]; predicing or classifying a moion (in roboics or human-robo ineracion) [2]; rying clohes in a digial mirror [3]; ec. A common way o obain his digial represenaion is marker-based opical moion capure (mocap) sysems. Such sysems use a large number of cameras o riangulae he posiion of opical markers. These are hen used o reconsruc he moion of rigid objecs or human ariculaed bodies, o which he markers are aached. All moion capure sysems suffer o a higher or lower degree from missing marker deecions, due o occlusion problems ha less han wo of he cameras see he marker or marker deecion failures ha he marker is no correcly deeced in less han wo of he cameras. In his paper, we propose a mehod for reconsrucion of missing markers o creae a more complee pose esimae (see Fig. 1). The mehod explois knowledge abou spaial and emporal correlaion in human moion, learned from daa examples o fill in missing pars of he pose esimae (Sec. IV). A number of mehods have been proposed wihin he Graphics communiy o address he problem of missing marker reconsrucion. The radiional approach [4], [5] is inerpolaion wihin he curren sequence. However, recenly a learning approach has been proposed, which explois a body of moion examples o learn ypical correlaions [6]. The novely wih Training daa NN Reconsruced moion Fig. 1. Illusraion of our mehod for missing marker reconsrucion. Due o errors in he capuring process, some markers are no capured. The proposed mehod explois spaial and emporal correlaions o reconsruc he pose of he missing markers. respec o his mehod is ha while hey learn linear dependencies, we employ a Neural Nework (NN) mehodology which enables modeling of more complicaed spaial and emporal correlaions in sequences of a human pose. In Sec. V we demonsrae he effeciveness of our nework, showing ha our mehod significanly ouperforms he sae of he ar in missing marker reconsrucion. Finally, we discuss our resuls and give suggesions for fuure work (Sec. VI).

2 II. RELATED WORK The ask of modeling human moion from mocap daa has been sudied quie exensively in he pas. We here give a shor review of he works mos relaed o ours. A. Missing Marker Reconsrucion I is possible o do 3D pose esimaion even wih affordable sensors such as Kinec. However, all moion capure sysems suffer o some degree from missing daa. This has creaed a need for mehods for missing marker reconsrucion. The missing marker problem has been radiionally formulaed as a marix compleion ask. Peng e al. [5] solve i by non-negaive marix facorizaion, using he hierarchy of he body o break he moion ino blocks. Wang e al. [6] follow he idea of decomposing he moion and do dicionary learning for each body par. They rain heir sysem separaely for each ype of moion. Burke and Lasenby [4] apply PCA firs and do Kalman smoohing aferwards, in he lower dimensional space. All hose mehods are based on linear algebra. They make srong assumpions abou he daa: each marker is ofen assumed o be presen a leas a one ime-sep in he sequence. Moreover, due o he linear models, hey ofen sruggle o reconsruc complex ypes of moion. To he bes of our knowledge, his paper is he firs aemp o apply Neural Neworks o missing marker reconsrucion. The limiaions discussed above moivae he presened novel approach o he missing marker problem. B. Predicion A highly relaed problem is o predic human moion some ime ino he fuure. Sae-of-he-ar mehods ry o ensure coninuiy eiher by using Recurren Neural Neworks (RNNs) [7], [8] or by feeding many ime-frames a he same ime [9]. While our focus is no on predicion, our neworks archiecures is inspired by hose mehods. The mos relaed o he presen work is Marinez e al. [10], who evaluae exising RNN-based mehods for moion predicion [7], [8] and show ha a primiive baseline (jus aking a previous ime-frame) beas many advanced mehods. They also propose a simple nework archiecure wih jus one layer and analyze i. Since our applicaion is no predicion, our archiecure is slighly differen: we do no have a residual connecion (from he oupu o inpu). Insead, we use LSTM [11] a he oupu layer, hence having a connecion from he previous o he curren oupu. Anoher relaed paper is he work of Büepage e al. [9], who use a sliding window and a Fully Conneced Neural Nework (FCNN) o do moion predicion and classificaion. Again, since our problem is differen, we modify heir nework, using a much shorer window lengh, fewer layers and no boleneck. III. DATASET We evaluae our mehod on he popular benchmark CMU Mocap daase [12]. This daabase conains 2235 mocap sequences of 144 differen subjecs. We use he recordings of 25 Fig. 2. Illusraion of a ypical marker placemen. subjecs, sampled a he rae of 120 Hz, covering a wide range of aciviies, such as boxing, dancing, acrobaics and running. A. Preprocessing We sar preprocessing from ransforming every mocap sequence ino he hips-cener coordinae sysem (ranslaing i o he cener of he hips). Then we normalize he daa ino he range [-1,1] by subracing mean pose over he whole daase and dividing ino he absolue maximal value in he daase. B. Daa Explanaion This daa conains 3D posiions of a se of markers, which were recorded by he mocap sysem a CMU. Example of a marker placemen during he capure can be seen in Fig. 2. All deails can be found in he daase descripion [12]. The human pose a each ime-frame is represened as a vecor of he marker 3D coordinaes: x = [x i,, y i,, z i, ] i=1:n, where n denoes he number of markers used during he mocap collecion. In he CMU daa, n = 41, and he dimensionaliy of a pose is 3n = 123. A sequence of poses is denoed x = [x ] =1:T. C. Missing Markers Missing markers in real life correspond o he failure of a sensor in he moion capure sysem. In our experimens we use mocap daa wihou missing markers. Missing markers are emulaed by nullifying some marker posiions in each imesep. This process can be mahemaically formulaed as a muliplicaion of he mocap frame x by a binary diagonal marix M : ˆx = C 1 (x ) = M x, (1) where M [0, 1] 3nx3n, such ha ha all 3 coordinaes of any marker are eiher missing or presen a he same ime. The percenage of missing values is usually referred o as he missing rae.

3 D. Training, Validaion, and Tes Daa Configuraions Validaion daase conains 2 sequences from each of he following moions: panomime (subjecs 32 and 54), spors (subjec 86 and 127), jumping (subjec 118) and general moions (subjec 143). Tes daase conains he following sequences: (baskeball), (boxing) (jump-urn). Training daase conains all he sequences no used for validaion or esing, from 25 differen folders in he CMU Mocap Daase, such as 6, 14, 32, 40, 141, 143. LSTM LSTM - FC FC - IV. METHOD In his secion he main idea of he proposed mehod is presened. Thereafer, he wo versions of he mehod are explained. Finally, he raining of he model is described. We assume ha he daa has missing markers, as explained in Sec. III. Missing marker reconsrucion is defined in he following way: Given a human moion sequence ˆx corruped by missing markers, he goal is o reconsruc he rue pose x for every frame. A. Missing Marker Reconsrucion as Funcion Approximaion We approach missing markers reconsrucion as a funcion approximaion problem: The goal is o learn a reconsrucion funcion R ha approximaes he inverse of he corrupion funcion C in Eq. (1). This funcion would map he sequence of corruped poses o an approximaion of he rue poses: x = R(ˆx) C 1 (ˆx) (2) The mapping C is under-deermined, so i is no inverible. However, i can be approximaed by learning spaial and emporal correlaions in human moion in general, from a se of oher pose sequences. We propose o use a Neural Nework (NN) approach o learn R, well known for being a powerful ool for funcion approximaion [13]. We employ wo differen ypes of neural nework models, which are described in he following subsecions. They are compared o each oher and o he sae of he ar in missing marker reconsrucion in Sec. V. B. LSTM-Based Neural Nework Archiecure Long-Shor Term Memory (LSTM) [11] is a special ype of Recurren Neural Nework (RNN). I was designed as a soluion o he vanishing gradien problem [14] and has become a defaul choice for many problems ha involve sequence-o-sequence mapping [15], [16], [17]. Our nework is based on LSTM and illusraed in Fig. 3a. The inpu layer is a corruped pose ˆx, and he oupu LSTM layer is he corresponding rue pose x C. Window-based Neural Nework archiecure An alernaive approach is o use a range of previous imeseps explicily, and o rain a regular Fully Conneced Neural Nework (FCNN) wih he curren pose along wih a shor hisory, i.e., a window of poses over ime ( ) :. (a) LSTM-based (b) Window-based Fig. 3. Illusraion of he wo archiecure ypes. (a) LSTM-based archiecure (Sec. IV-B). (b) Window-based archiecure (Sec. IV-C). This nework is illusraed in Fig. 3b. The inpu layer is a window of concaenaed corruped poses [ˆx T,..., ˆxT ] T. The oupu layer is he corresponding window of rue poses [x T,..., xt ] T (Sec. III). In beween, here are zero or more hidden fully conneced layers. This srucure is inspired by he sliding ime windowbased mehod of Büepage e al. [9], bu is adaped o pose reconsrucion. For example, here is no boleneck middle layer and fewer layers in general, o creae a igher coupling beween he corruped and real pose, raher han learning a high-level or holisic mapping of a pose. We also use window lengh T=10, insead of 100, based on he performance on he validaion daase. D. Training For raining purposes, we exrac shor sequences from he daase by sliding window, hen shuffle hem and feed o he nework. The raining was done using Adam opimizer [18]. All hyper-parameers were opimized on he validaion daase. Noe ha for he non-missing markers he exising inpu values were used insead of he nework oupu, so ha only missing values were reconsruced. V. EXPERIMENTS The models presened above are evaluaed in he same seing as any oher random missing marker reconsrucion sysem. A specific amoun of markers (10%, 20%, or 30%) are randomly removed a each ime-frame and each mehod is applied o recover hem. The reconsrucion error is measured. A. Error Measure We use he commonly used [5], [6], [4] Roo Mean Squared Error (RMSE) o measure pose reconsrucion error. This

4 TABLE I COMPARISON OF ARCHITECTURES FOR MISSING MARKER RECONSTRUCTION. RMSE IN MARKER POSITION (CM) FOR LSTM-BASED (SEC. IV-B) AND WINDOW-BASED (SEC. IV-C) ARCHITECTURES. A TRAINING SET COMPRISES ALL ACTIVITIES, 20% OF THE MARKERS (RANDOMLY SELECTED) IN EACH INDATA FRAME ARE MISSING. Archiecure Baskeball Boxing Jump urn LSTM-ba. 1.57± ± ±0.1 Window-based 2.32 ± ± ±0.05 measure calculaes average disance beween a rue and a reconsruced moion according o: x ˆx rmse(x, ˆx) = 2, (3) n e where x is he original moion, ˆx is he reconruced one and n e is he amoun of missing markers. There is randomness in he sysem (in he iniializaion of he nework weighs), so every experimen is repeaed 3 imes and error mean and sandard deviaion are measured. B. Implemenaion Deails All mehods were implemened using Tensorflow[19]. The code is publicly available 1. C. Hyperparameers The hyperparameers for boh archiecures were opimized w.r.. he validaion daase (Sec. III-D) using grid search. Hyperparameers for he LSTM-based nework (Sec. IV-B) were: hidden layer size = 2048, number of hidden layers = 1, dropou = 0.9, bach size = 64, iniial learning rae = , sequence lengh = 32 (number of frames unrolled). These hyperparameers correspond approximaely o hose found in relaed work [8], [10], [7]. Hyperparameers for he window-based nework (Sec. IV-C) were: hidden layer size = 1024, number of hidden layers = 1, dropou = 0.9, bach size = 32, iniial learning rae = , ime window size = 10. D. Archiecure Comparison Table I shows a comparison of differen archiecures on he missing marker problem. The LSTM-based nework performed beer han he window-based one. A probable reason for ha is ha LSTM is beer han window-based FCNN a modeling emporal correlaions. Based on his, he LSTMbased archiecure was chosen for he following sae of he ar comparison. E. Comparison o he Sae of he Ar Tables II provide he comparison of he performance of our sysem wih 3 sae-of-he-ar papers and wih a rivial soluion (subsiue mean values) as a baseline, on 3 acion classes from he CMU Mocap daase. The experimens from [4] were repeaed by us while using he same hyperparameers 1 hps://gihub.com/svio-zar/nn-for-missing-marker-reconsrucion TABLE II COMPARISON TO THE STATE OF THE ART IN MISSING MARKER RECONSTRUCTION. RMSE IN MARKER POSITION (CM) FOR LSTM-BASED (SEC. IV-B) ARCHITECTURE, COMPARED TO THREE RELATED METHODS. THE ERRORS USING THE MEAN POSE OF THE MISSING MARKERS ARE GIVEN AS BASELINE. A TRAINING SET COMPRISES ALL ACTIVITIES. THE NUMBERS FROM [6] WERE EXTRACTED FROM A DIAGRAM. Mehod Baskeball Boxing Jump urn Mean val. 13.2± ± ±0.35 Burke [4] 10.1± ± ±0.7 Wang [6] n.a. Peng [5] n.a. n.a. n.a. LSTM-ba. 1.57± ± ±0.1 (a) 10% of he markers (randomly seleced) in each indaa frame are missing. Mehod Baskeball Boxing Jump urn Mean val. 12.4± ± ±0.34 Burke [4] 12.57± ± ±0.15 Wang [6] 8 4 n.a. Peng [5] n.a LSTM-ba. 1.62± ± ±0.07 (b) 20% of he markers (randomly seleced) in each indaa frame are missing. Mehod Baskeball Boxing Jump urn Mean val ± ± ±0.12 Burke [4] 12.6± ± ±0.4 Wang [6] n.a. Peng [5] n.a LSTM-ba. 1.85± ± ±0.02 (c) 30% of he markers (randomly seleced) in each indaa frame are missing. as in heir original paper. The resuls of he Wang mehod [6] were aken from he diagram in heir paper. Las, he error measures of he Peng mehod [5] were recompued from heir original measure wih averaging he error over all markers, o our measure where he error is compued only over he missing markers. Each sub-able has a differen missing rae (percenage of markers missing a each ime-frame). Our sysem subsanially ouperforms he sae of he ar. The mehod of Burke and Lasenby [4] performs well for repeiive moion if he noise level is low (see Table IIa). However, i requires all he markers o be presen a leas a one ime-sep in he sequence and is unable o reconsruc markers ha failed early on during he mocap, as noed by he auhors. The mehod of Wang e al. [6] performs well for low missing marker rae (see Table IIa), bu degrades seeply wih an increase of missing rae (see Table IIc). In conras, our mehod is robus o an increasing missing rae. The mehod of Peng e al. [5] produces very promising resuls, even if a big fracion of daa is missing. Sill, i is considerably less accurae han our mehod; probably because his mehod relies on linear models, while he dependencies beween differen markers are non-linear in realiy.

5 F. Visualizaion of he Resuls Fig. 4 illusraes he marker reconsrucion in one of he es sequences, jump-urn. The markers have been colored for visibiliy as: head-red, orso-yellow, arms-green, legs-blue. The subjec is lifing heir arms and jumping. The observed marker cloud (Fig. 4b) misses 20% of he markers, in his case, many of he markers on he lower orso, which makes he poses difficul o see. Our reconsrucion resul (Fig. 4c) is visually close o he ground ruh (Fig. 4a), which is also suppored by he resuls in Tables I and II. G. Developmen of Error over Time So far we have been considering a scenario where some percenage of he markers are missing randomly a each imesep. This is a common seing in which o evaluae missing marker reconsrucion algorihms. However, specific markers are ofen missing over exended periods of ime. I is herefore ineresing o simulae his ype of missing daa as well. The following experimens compare our sysem wih he mehod of Burke and Lasenby [4] (for which we have access o he implemenaion) in his seup. We focus on he markers of he hand, because his par of he body is he mos ariculaed and he hardes o reconsruc. I should be noed ha while he mehod of Burke and Lasenby [4] requires a complee sequence, our mehod only uses he pas frames, making i suiable for online esimaion. Resuls for he baskeball es sequence, frames , are illusraed in Fig. 5a. Resuls for he oher moions, frames , are illusraed in Fig. 5b,c,d. We can observe ha he Burke and Lasenby mehod performs much beer for he boxing sequence in comparison o baskeball. The probable reason for ha is ha he baskeball sequence is much longer and hence provides more informaion for inerpolaion. Our sysem performs beer han he one of Burke and Lasenby when jus one marker is missing, bu degrades seeply when he number of missing markers increase. During he raining he daa had random missing markers, so eiher previous imeseps or he neighboring markers were presen for each marker, bu in he es condiion hose markers has neiher informaion abou previous ime-sep, nor abou heir neighbors, so our sysem could no recover hem. (a) Ground ruh markers (b) 20% of markers are missing VI. CONCLUSIONS We propose a daa-driven, Neural Nework-based approach o missing marker reconsrucion. We sugges wo mehods o model spaial and emporal correlaions. The CMU Mocap daase was used o evaluae and benchmark he mehod. Our experimens show ha boh suggesed mehods ouperform he sae of he ar in missing marker reconsrucion; he LSTM-based (Sec. IV-B) mehod can compensae for missing markers a facor 4 beer han he sae of he ar when he rae of missing markers is high and a facor 2 when he missing marker rae is low. (c) Reconsrucion resul Fig. 4. Four keyframes from he boxing es sequence, illusraion of he reconsrucion using he LSTM-based mehod.

6 A. Fuure Work We envision wo fuure direcions for developing he mehod furher. Firsly, o reconsruc he aspecs of he human moion ha are difficul o capure in a full-body scenario, such as fingers, gaze, and facial feaures. Secondly, o use he mehod for human-o-robo mapping of communicaive moion behavior. ACKNOWLEDGMENTS The auhors would like o hank Rika Anonova, Simon Alexanderson, and Hossein Azizpour for useful discussions and commens. This work is funded by Sifelsen för Sraegisk Forskning. REFERENCES [1] M. J. Black and P. Guan, Sysem and mehod for simulaing realisic clohing, Jun , us Paen 9,679,409. [2] J. Marinez, R. Hossain, J. Romero, and J. J. Lile, A simple ye effecive baseline for 3d human pose esimaion, in ICCV, [3] P. X. Nahan, C. J. Cox, F. T. Leiber, M. S. Meadows, and J. S. Mallis, Sysems and mehods for managing a persisen virual avaar wih migraional abiliy, Nov , us Paen App. 11/560,743. [4] M. Burke and J. Lasenby, Esimaing missing marker posiions using low dimensional kalman smoohing, Journal of Biomechanics, vol. 49, no. 9, pp , [5] S.-J. Peng, G.-F. He, X. Liu, and H.-Z. Wang, Hierarchical block-based incomplee human mocap daa recovery using adapive nonnegaive marix facorizaion, Compuers & Graphics, vol. 49, pp , [6] Z. Wang, S. Liu, R. Qian, T. Jiang, X. Yang, and J. J. Zhang, Human moion daa refinemen uniizing srucural sparsiy and spaial-emporal informaion, in IEEE Inernaional Conference on Signal Processing, [7] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik, Recurren nework models for human dynamics, in ICCV, [8] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena, Srucural-rnn: Deep learning on spaio-emporal graphs, in CVPR, [9] J. Büepage, M. Black, D. Kragic, and H. Kjellsröm, Deep represenaion learning for human moion predicion and classificaion, in CVPR, [10] J. Marinez, M. J. Black, and J. Romero, On human moion predicion using recurren neural neworks, in CVPR, [11] S. Hochreier and J. Schmidhuber, Long shor-erm memory, Neural Compuaion, vol. 9, pp , [12] CMU, Carnegie-mellon mocap daabase. [13] K. Hornik, M. Sinchcombe, and H. Whie, Mulilayer feedforward neworks are universal approximaors, Neural neworks, vol. 2, no. 5, pp , [14] S. Hochreier, The vanishing gradien problem during learning recurren neural nes and problem soluions, Inernaional Journal of Uncerainy, Fuzziness and Knowledge-Based Sysems, vol. 6, no. 2, pp , [15] I. Suskever, O. Vinyals, and Q. V. Le, Sequence o sequence learning wih neural neworks, in NIPS, [16] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, Long-erm recurren convoluional neworks for visual recogniion and descripion, in CVPR, [17] B. Shi, X. Bai, and C. Yao, An end-o-end rainable neural nework for image-based sequence recogniion and is applicaion o scene ex recogniion, PAMI, vol. 39, no. 11, pp , [18] D. Kingma and J. Ba, Adam: A mehod for sochasic opimizaion, in ICLR, [19] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Ciro, G. S. Corrado, A. Davis, J. Dean, M. Devin e al., Tensorflow: Large-scale machine learning on heerogeneous disribued sysems, arxiv preprin arxiv: , (a) Reconsrucing baskeball: 1 marker of a hand is missing for all ime frames excep he firs. (b) Reconsrucing baskeball: 3 markers of a hand are missing for all ime frames excep he firs. (c) Reconsrucing boxing: 1 marker of a hand is missing for all ime frames excep he firs. (d) Reconsrucing boxing: 3 marker of a hand is missing for all ime frames excep he firs. Fig. 5. Long-erm missing marker experimen. The plos show he developmen of error over ime when specific markers were missing for an exended period of ime.