Human Skeleton Reconstruction for Optical Motion Capture

Transcription

1 Journal of Computatonal Informaton Systems 9: 0 (013) Avalable at Human Skeleton Reconstructon for Optcal Moton Capture Guanghua TAN, Melan ZHOU, Chunmng GAO College of Informaton Scence and Engneerng, Hunan Unversty, Changsha 41008, Chna Abstract In ths paper, we propose a method of automatcally estmatng the pose of the underlyng human skeleton to avalable n optcal moton capture. These problems are addressed n the method: dentfyng markers over ndvdual frames, predctng the mssng marker locatons and estmatng the jont postons. Frstly, we dentfy markers accordng to local rgdty and space-tme constrants of markers on the same lmb segment n each frame. After that, based on small nter-frame moton, we predct the mssng marker locatons usng the adjacent vsble marker locatons. Fnally, we estmate the jont locatons n two dfferent scenaros, usng a nonlnear optmzaton n the crcumstance of less than three markers on one gven lmb segment. Results show that the approach s able to estmate the jont locatons correctly, and our method s scalable n marker predcton and human pose estmaton. Keywords: Moton Capture; Marker Identfcaton; Mssng Markers Predcton; Jont Locatons Estmaton 1 Introducton Human moton capture technology s the process of recordng the movement of human. It s used extensvely n human anmaton, moton analyss, human engneer research and so on. In the technque, cameras are used to track reflectve markers on a performer s black bodysut, and the body posture of the performer s reconstructed from these marker postons. As so far, the commonly used moton capture systems can be dvded nto mechancal, acoustcs, magnetc and optcal. In ths paper, we focus on passve optcal moton capture system. Human moton data created from optcal moton capture system contan marker poston data, spurous pont data and mssng marker data. Therefore, marker dentfcaton s requred before puttng moton data nto use. Marker dentfcaton s a problem of reconstructng a non-rgd artculated pose from only sparse feature ponts. It s to recognze markers n each sequence frame, there are multple methods for marker dentfcaton. L et al. [1] present a general model-based dynamc pont matchng algorthm to reconstruct freeform non-rgd artculated movements from data presented solely by sparse feature ponts. It contrbutes to the study of self-ntalzng dentfcaton. Krk Correspondng author. Emal address: gcm11@163.com (Chunmng GAO) / Copyrght 013 Bnary Informaton Press DOI: /jcs6766 October 15, 013

2 8074 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) at el. [] usng spectral clusterng to cluster markers nto segment groups, and based on the observaton that portons of an actor s body wll often pass through smlar confguratons at dfferent tmes to determne marker correspondence over frames. In practce, some markers are occluded n the trackng process, mssng markers may exst n capturng moton data. Many papers have focused on methods of predctng the postons of mssng markers. Taylor et al. [3] uses a condtonal restrcted Boltzmann machne wth dscrete hdden states to model human moton. Wang et al. [4] takes a nonparametrc approach to model human moton and propose a Gaussan process dynamcal model, whch ncludes a chan of latent varables and nonlnear mappng from the latent space to observed moton. L et al. [5] ntroduces BoLeRo (Bone length constraned reconstructon for occluson), whch takes nto consderaton bone length constrants. Although ther algorthm results n smooth moton, the method s complex and expensve n terms of computatonal cost. As so far, many dfferent technques have been proposed to calculate the jont locatons. Cha et al. [6] mproves standard forward knematc technques to provde more accurate end-effector postons. Gamage and Lasenby n [7] propose a new method for estmatng the parameters of ball jonts, t does not requre manual adjustment of any optmzaton parameters and procedures closed form solutons. Meanwhle, ths method do not assume strct rgdty but only that the markers mantan a constant dstance from the center or axs of rotaton. Cameron at el [8], propose a method of real-tme jont localsaton of legged skeletons n the presence of mssng data. Ths paper presents an approach for generatng accurate and smooth human moton over tme. Our procedure of skeleton estmaton from moton data n optcal moton capture s as follows: (1) At frst, marker dentfcaton of the frst moton data s carred out usng DSHPM algorthm [1]. After that, we fnsh marker trackng to dentfy moton data. () Based on the assumpton of constant moton parameter between two consecutve frames, we reconstruct the exstng mssng marker locatons usng the postons of adjacent vsble markers. (3) We estmate the underlyng skeleton physcal locatons of the jonts nsde the body of performer n two dfferent cases. In the case of less than three markers on one gven lmb segment, we fnd the jont parameter by solvng for a least squares cost functon wth low ft resduals. The rest of the paper s organzed as follows: In secton, we descrbe the procedure of our methodology n detal; The experment results are showed n secton 3. Secton 4 concludes our work. Human Skeleton Estmaton.1 Marker dentfcaton Marker dentfcaton s to determne whch pont data correspond to each of the markers. It allows non-rgd deformaton wthn a certan lmtaton. Intally, the lmb of performer s stretched. Select a clear pose as a model and dentfy every marker data manually. To realze dentfcaton of frst moton data, we fnd the correspondence between the frst moton data and the model usng DSHPM algorthm [1].

3 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) We defne a human model whch consst of T lmbs P = {P = 1,,, T }. For each lmb P = {e 1,, e,, p,j, f,j j = 1,,, n }, we set a threshold e 1, whch denotes range of moton n two consecutve frames, a threshold e, about non-rgd deformaton, n markers descrbed by ther postons, and ndcator f,j as a value of 1 or 0 accordng to the presence or absence of marker p,j. The am of marker dentfcaton s to fnd the correspondence between dentfed markers R = {p k 1,j j = 1,,, n } n frame k 1 and pont-set Q = {q s s = 1,,, M} n k th frame for segment P, the flow s as follows: (1) Fnd all canddate markers: for each marker p k 1,j n R, calculate the dstance d s = p k 1,j q s ( v refers to -norm), f d s < e 1, s satsfed, lst them n the j th row. The row may contan several canddates or null. Set f,j = 0, f t s null, otherwse f,j = 1. () Take one marker out n each row one by one, and fnd the whole canddate pont-sets. For each canddate pont-set, calculate the average deformaton error. If t meet the requrement of condton (1), record the pont-set Q = {q,j j = 1,,, n }, ē 1 = n 1 n r=1 t=r+1 f,r f,t q,r q,t l r,t n 1 n r=1 t=r+1 f,r f,t < e, (1) where l r,t s the dstance between the r th marker and the t th marker n frame k 1. (3) For each pont-set Q, calculate ts average matchng error accordng to Eq. (). ē = n j=1 f,j q,j (c R p k 1,j + t ) n j=1 f,j () where c,r and t are scalng parameter, rotaton matrx and translaton vector between pont-set R and pont-set Q. We can estmate these parameters usng SVD decomposton [9]. We search the best match by two crtera: average non-rgd deformaton error and average matchng error. Select a pont-set whch has the mnmze error as the best matchng pont-set accordng to Eq. (3), and remove the pont-set from pont-set Q. e = w 1 ē 1 + w ē (3) where w 1 and w s the non-negatve weght assocated wth non-rgd deformaton and pont matchng of markers on a gven lmb segment, and w 1 + w = 1.. Predcton of mssng markers Durng the process of moton, some markers are obscured or occluded by elements of the scene leads to mssng markers. In ths paper, we predct the mssng marker postons based on assumpton that the moton parameters s constant between two consecutve frames on a gven lmb. We consder dfferent scenaros of occluson.

4 8076 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) One vsble markers on a gven lmb We assume the k th marker p s,k on lmb P s s vsble n frame, we estmate the poston of mssng markers accordng to the followng two subsectons. (1) Three or More Markers on a Gven Lmb In the case of three or more markers on a lmb, we estmate the locaton of j th mssng marker p s,j(j = 1,,, n s and j k) accordng to the poston of vsble marker p s,k, p s,j = p s,k c 1 R 1 (p 1 s,k p 1 s,j ) (4) where c 1 and R 1 are scalng parameter and rotaton matrx between frame and frame 1 on the lmb. Fg. 1 shows ths process. In the fgure, the black dot p s,1 s the only one vsble marker n frame. The blue dots represent the estmated mssng markers. 1 p s,1 ps, 1 1 p s, 1 p s,3 ps, ps, 3 Fg. 1: Mssng marker predcton (one vsble marker) ()Two markers on a gven lmb segment Assumng that marker p s,j s mssng n frame. Frstly, we calculate the angle θ 1 between vectors whch from j th marker to k th marker n frame and frame 1, the rotaton matrx R 1 whch rotates around the axs n = (p s,k p s,j ) (p 1 estmate the vector ν = R 1 (p 1 s,k as where l = ( p s,k p 1 p s,j + p 1 s,k p 1 s,j )/. s,k p 1 s,j ) by θ 1 degrees. Secondly, we s,j ) n frame. Fnally, the locaton of marker p s,j s gven p ν s,j = p s,k l ν.. Two or more vsble markers on a lmb segment Assumng marker p s,j s nvsble, vsble markers are p s,1, p s,,, p s,n(n ). We can estmate the locaton of mssng marker p s,j by the followng method. (1) We calculate an ntal poston usng the poston of vsble markers n the same lmb segment, n [p s,k c 1 R 1 (p 1 s,k p 1 s,j )] p k=1 s,j =. (6) n () Arbtrarly choose two markers p s,r and p s,t (1 r, t n), create two spheres S 1 and S wth centers are p s,r and p s,t, radus are r 1 = c 1 p 1 s,r p 1 s,j and r = c 1 p 1 s,t p 1 s,j respectvely. We get the fnal poston p s,j whch s assgned as the closest pont on the ntersecton of two sphere to p s,j. We solve t by Lagrange multplers method, the procedure can be regard as a smple optmzaton wth constrants, mn p s,j p s,j (5)

5 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) (ps,t ps,r ) ps,j = r1 r + ps,t ps,r s.t. We construct a Lagrangan functon, the target poston ps,j can be obtanable by dfferentaton to multpler λ and ps,j, ps,j = p s,j λ(ps,t ps,r ) where λ = (ps,t ps,r ) p s,j r1 +r ps,t + ps,r. Fg. llustrates the above. ps,t ps,r ps,11 ps,1 ps, ps, 1 (7) ps,3 ps,31 Fg. : Mssng marker predcton (at least two markers vsble).3 The jont locatons estmaton After dentficaton on moton data, we estmate the locaton of jont between two adjacent lmb segments Ps and Pr. For markers on the same lmb are rotated around a jont whch s the rotaton nvarant pont. We establsh local coordnates usng markers on lmb segments Ps and Pr, whle the locaton of jont s constant n these two local coordnates over tme. Ths artcle consder the followng two possble subcases..3.1 At least three markers on both lmb segment If the number of markers on a lmb segment s more than 3, select three markers whch are not n a lne. We estmate the 3D poston of jonts usng the approach n [10]. The jont poston Ys,r can be expressed as Ys,r = x1 (ps, ps,1 ) + y1 (ps,3 ps,1 ) + z1 (ps, ps,1 ) (ps,3 ps,1 ) + ps,1 (8) Ys,r = x (pr, pr,1 ) + y (pr,3 pr,1 ) + z (pr, pr,1 ) (pr,3 pr,1 ) + pr,1 (9) s the poston of the jont whch connect two adjacent lmb segments, ps,j and pr,j where Ys,r th are the j marker on two lmbs, [x1 y1 z1 ]T and [x y z ]T are the locaton of the jont n two local coordnates. It s obtaned by solvng an overdetermned equaton system. Fg. 3(left) demonstrates the process. ps,1 ps, ps,1 ps,3 Ys,r p r,1 p r, pr,3 ps, ps,3 Ys,r Fg. 3: Jont locatons estmaton pr,1

6 8078 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) Three markers on one lmb, whle less than three markers on adjacent lmb If more than three markers on a lmb, we choose three non-collnear markers. Assumng that there are three markers on lmb P s and the number of markers on lmb P r s less than 3. Dstance between the jont and one marker s assumed constant throughout the moton, we have the followng formula: Ys,r p r,j = rj, where r j (j = 1,, m, m < 3) s the dstance between jont Ys,r and marker p r,j. Now a cost functon can be gven as C = n =1 m j=1 ( Y s,r p r,j r j ) (10) where n s the total number of frames and m s the total number of markers on lmb P r. To get the poston of the jont, we calculate the ntal values of c = [x 1 y 1 z 1 ] T and r j. We calculate centrods Ms = j p s,j/3 and Mr = j p r,j/m of markers on two lmb segments. The estmated poston of the jont s Ys,r = (Ms + Mr)/. We get the value of parameter c = [x 1 y 1 z 1 ] T by solvng Eq. (8). Therefore the ntal value of parameter s c = c /n. Dstance r j can be obtaned by solvng the frst order partal dervatves of cost functon C wth respect to r j. The ntal value of r j s gven by r j = (Y s,r p r,j )/n. Fnal parameters c and r j are obtaned by mprovng on estmaton of parameters c and r j usng smarquardt nonlnear least square optmzaton algorthm [11]. We fnd the jont poston usng Eq. (8). Fg. 3(rght) llustrates the above. 3 Experments and Results We have mplemented the method on estmatng the skeleton of human n MATLAB. In our experments, all 3D poston of markers are acqured va OptTrack moton capture system, usng 1 cameras wth a sample rate of 60 Hz. We show three results of marker regstraton n Fg. 4, 41 markers are attached to black bodysut on the performer. The movement ncludng walkng, boxng and jumpng. Markers on the same lmb segment are dentfed by dfferent colors. Fg. 4: Marker dentfcaton

7 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) Fg. 5 shows an example of mssng markers predcton. In the demonstraton, we choose occluded markers randomly, black dots represent orgnal markers and red dots are the estmated postons of mssng markers. In our experments, we tested the method usng runnng moton (performer #.07) n a publc dataset from CMU mocap database [1]. We delete one and two markers on foot from frame 50 to 150. Fg. 6 demonstrates the predcton errors of mssng markers by two methods, and our method can acheve better recovery of mssng data than UKF-VTM [8] method. Fg. 5: Mssng markers predcton Error varaton of markers over tme (only one vsble marker) UKF VTM Our method Error varaton of markers over tme (at least two vsble markers) UKF VTM Our method Error/mm Error/mm Number of frames Number of frames Fg. 6: Predcton error of markers aganst the orgnal moton To evaluate the robustness and effcency of our method n the jont locatons estmaton, we select a tral set wth 3 motons representng a varety of moton types whch ncludng swordplayng, washng and bendng. Fg. 7 shows the results of the jont locatons estmaton, red dots represent the jont postons, black dots ndcate the marker postons. In our method, we take the number of markers on a lmb nto consderaton. In the case that number of markers on a lmb s less than 3, the reconstructon errors of the jont postons aganst the orgnal moton s gven n Fg Concluson and Future Researches Ths paper addresses the problem of automatcally establshng the skeletons from moton capture data. Our method s able to dentfy markers n each frame, predct the mssng marker postons and estmate the jont locatons. Experment results demonstrate our method acheve better recovery of occlusons and skeleton moton. Whlst, our method s avalable n markers on a lmb segment are mssng for an extended perod of tme. Future work wll focus on the problem of the jont locatons estmaton n case of less than 3 markers on both adjacent lmbs.

8 8080 G. Tan et al. /Journal of Computatonal Informaton Systems 9: 0 (013) One marker on a lmb Two markers on a lmb Error Varaton (Jonts) Error /mm Number of frames Fg. 7: Jont locatons estmaton (left) and reconstructon error of jonts (rght) Acknowledgement Project s supported by Guangdong Provnce s Mnstry of Educaton projects for Industry- Academy-Research cooperaton (No. 011B ) and Hunan Provnce s Scence and Technology Plan (No. 013SK3310). References [1] B. L, Q. Meng and H. Holsten, Artculated moton recconstructon from feature ponts, Pattern Recognton 41 (008) [] A. G. Krk, J. F. Bren and D. A. Forsyth, Skeletal parameter estmaton from optcal moton capture data, n: Proc. of IEEE conf. on Computer Vson and Pattern Recognton, Jun. 005, pp [3] G. W. Taylor, G. E. Hnton and S.T. Rowes, Modelng human moton usng bnary latent varables, n: Neural Informaton Processng Systems, 006, pp [4] J. M. Wang, D. J. Fleet, Gaussan process dynamcal models for human moton, IEEE Trans. on Pattern Analyss and Machne Intellgence 30 (008) [5] L. L, J. McCann, N. Pollard and C. Faloutsos, BoLeRO: a prncple technque for ncludng bone length constrants n moton capture occluson fllng, n: Proc. ACM SIGGRAPH/Eurographcs Symposum on Computer Anmaton, Span, 010, pp [6] J. Cha and J. K. Hodgns, Performance anmaton from low dmensonal control-sgnals, Journal ACM Transacton on Graphcs 4 (005) [7] S. H. U. Gamage, J. Lasenby, New least squares solutons for estmatng the average centre of rotaton and the axs of rotaton, Journal of Bomechancs 35 (00) pp [8] A. Arstdou and J. Lasenby, Real-tme marker predcton and CoR estmaton n optcal moton capture, Vsual Computer 9 (01) 7-6. [9] S. Umeyama, Least-squares estmaton of transformaton parameters between two pont patterns, IEEE Transacton on Pattern Analyss and Intellgence 13 (1991) [10] G. Wen, Research on 3D human moton data processng, Ph.D. Thess, Department of Computer Applcaton Technology, Chnese Academy of Scences, March 007. [11] K. Madsen, H. B. Nelsen and O. Tngleff, Methods for non-lnear least squares problems, n: Informatcs and Mathematcal Modellng, Techncal Unversty of Denmark, 004. [1] CMU Graphcs Lab Moton Capture Database: