Articulated Motion Capture from Visual Hulls in High Dimensional Configuration Spaces

Transcription

1 Semnar Presentaton June 18th, 010 Artculated Moton Capture from Vsual Hulls n Hgh Dmensonal Confguraton Spaces Azawa Yamasak Lab D, , 羅衛蘭 ABSTRACT In ths paper, we propose a novel approach for trackng artculated body moton n the hgh-dmensonal confguraton spaces. A modfed annealed partcle flterng s presented to estmate the suted knematc postures for a volume sequence, whch seeks for the globally optmal soluton on the bass of local constrants. Then a segmentaton algorthm s performed on the volumetrc models and the error of each body segment s estmated for the local refnement. We develop a herarchcal artculated Iteratve Closest Pont regstraton to refne the 3D poses locally. We assgn the artculated models 4 degrees of freedom and the results show that the mean absolute error s about 10mm on average. Keywords Artculated model, Moton trackng, Segmentaton, Vsual hull. twsts 1. INTRODUCTION Knematc body moton capture or 3D spato-temporal surfaces reconstructon from synchronous mult-camera or mult-vew vdeo sequences s stll a challengng and fundamental problem for many applcatons, ncludng 3D anmaton moves and games, medcal dagnostcs moton analyss, or robot moton smulaton. Marker-based moton capture system s capable to provde moton capture data of hgh accuracy quckly. However, t requres people to wear skn-tght clothng wth markers and specal capture hardware s needed. In the past years marker-less moton capture has receved more and more attenton and been used n commercal felds varyng from survellance to character anmaton and 3D move dsplay. Annealed partcle flterng [1] s shown to be effectve n vsual moton trackng of artculated body smplfed by cones wth ellptcal cross-secton wth 9 degrees of freedom. However, the oversmplfed models are too crude that t s hard to recover complcated shape and moton precsely. Therefore, we employee the volumetrc models [] generated from smultaneous multple mages drectly for 3D pose estmaton. We select 5% volume data for moton trackng to decrease the computatonal cost as the number of each model s beyond 100 thousands. It should be mentoned that the APF algorthm focuses on seekng for the globally optmal soluton so that t wll cause the locally msleadng problem. We develop the stochastc search approach wth memorzaton of local optmzaton to avod t effcently. Furthermore, our proposed modfed APF s robust and effcent for trackng of quck movement or human wth general apparel n hgh dmensonal spaces. Local refnement of the extracted pose s stll necessary as nosy data n the vsual hulls and errors together wth the modfed APF method exst. We present a herarchcal artculated ICP regstraton for local pose refnement as the ICP method [3] s robust wth nosy data and the modfed APF has provded good ntalzaton. We calculate the msmatchng rates and the mean absolute errors of all the body segments and pck up the parts to be realgned. We assume the volumetrc model of the frst frame as the template body and segment t nto 15 body segments. The correspondng skeleton model s also gven. In our work, we frst reconstruct the mesh surface by cubes marchng [4] and extract the artculated skeleton model from t. Then we segment the mesh surface nto 15 parts based on the skeleton and geodesc dstances. Fnally, each voxel data n the template model s labeled wth the segment ndex of the nearest vertex n the segmented mesh surface.. RELATED WORK Marker-less human moton trackng has been a challengng problem n felds of computer graphcs and computer vson for years. It s ntutve to represent knematc postures by artculated skeleton models. Therefore, several smply general geometrc representaton methods are used to replace the body segments of a human. Deutscher and Red [1] [5] reconstruct the subject body shape by cones wth ellptcal cross-sectons wth 9 degrees of freedom. They develop conventonal partcle flters by layerng the search space based on annealng to estmate artculated body moton the hgh dmensonal spaces. However, n generally such models are too smple to recover sophstcated shape and moton accurately. Plänkers and Fua [6] develop a method to reconstruct artculated deformable objects based on metaballs and estmate human moton usng the Levenberg-Marquart least squares estmator. In addton, Corazza and Gambaretto [7] propose an automatc generaton method of a subject- 1

2 Template model Global optmzaton Local refnement Input data Segmentaton Fgure 1. Flow chart about the moton capture algorthm. specfc model wth jont center locatons. A marker-less moton trackng takng advantage of vsual hulls, subject specfc model and artculated usng dfferent number of cameras and dataset s present n [8]. Vlasc [9] and Gall [10] estmate human moton by fttng a template skeleton model nto the vsual hull. Vlasc et al. extract the skeleton pose for each frame to mantan the temporal smoothness. Gall et al. try to recover knematc skeleton chans solvng a local optmzaton problem to algn models to mages usng slhouettes and texture features. Msalgned lmbs are detected and the pose s refned usng partcle flter method. We present a model-based trackng algorthm that conssts of a stochastc search wth global optmzaton and local refnement. Our processng ppelne of the pose estmaton algorthm conssts of three steps, as descrbed n Fgure 1. Stochastc approaches are wdely used n the feld for moton trackng wth global optmzaton. So we frst present the modfed APF to estmate the artculated postures usng vsual hulls nstead of slhouettes and rm features. Sample data from the template are chosen and deformed to match the volumetrc model to be tracked. The soluton converges to the global mnmum. Then the msmatchng rates of the whole sample model and each body segment are calculated n order to fnd out locally msalgned body segments. The volumetrc model s segmented nto 15 parts based on the segmented template body and the estmated moton data. In addton, we check the mean absolute dstance error of each segment and refne the pose by a herarchcal artculated ICP regstraton. The global transformaton wth 6 degrees of freedom s calculated for each msalgned body segment by ICP regstraton. We estmate the correspondng artculated moton wth 3 degrees of freedom by solvng a lnear equaton. The msalgned body segments are searched and refned herarchcally by our proposed artculated ICP regstraton. 3. MODEL-BASED POSE ESTIMATION A sequence of vsual hulls s utlzed to estmate 3D artculated postures. Snce vsual hulls provde not only surface features but also nner detals, our proposed method tends to estmate knematc models n two aspects correspondngly. In the frst phase, the artculated moton of the whole human body s captured wth the global optmzaton by deformng the segmented template model to ft the model to be tracked by the modfed APF algorthm. It s demonstrated that modfed APF s a good optmzaton tool for not only globally soluton bust also local mnmzaton n some aspects. The global msmatchng rates are about 5% and the maxmum values of all body segments are about 8% n our expermental results. However, t stll suffer from loss of local optmzaton of some segments as nose exsts n vsual hull and just sample data are utlzed for trackng prevously by our modfed APF method. Therefore, we expect to adjust 3D pose locally to enforce accuracy of the knematc postures estmaton. Then the current vsual hull s segmented by the global optmal posture and the template labeled model. We detect msalgned segments by evaluatng the mean absolute error dstance of each body segment. Fnally, the segmented parts labeled msalgned are herarchcally detected and refned locally by artculated ICP regstraton. The 3D poses wth global and local optmzaton are estmated.

3 (a) The model to be matched (b) The labeled model Fgure 3. Matchng functon. (a) APF (b) Modfed APF (c) Refnement Fgure. Trackng results. 3.1 Global Optmzaton In ths secton, a developed artculated APF approach s used for rgd transformaton n hgh dmensonal confguraton space. Twsts representaton and exponental coordnates as gven n [11] are employed for rgd moton. We construct an artculated human model wth 4 degrees of freedom. Degrees of freedom of the global translaton and rotaton are treated as 6. Wrst, knee and ankle jonts are defned wth degrees of freedom. Shoulder, hp, neck and upper body jonts are gven 3 degrees of freedom. We choose sample data randomly from the prevous model for moton estmaton. The number of a volumetrc model s about 100 thousands and we select 5 percent of the volume data. In addton, we let the number of the sample data of each body segment be same to ensure the same sgnfcance of each lmbs whle trackng. We seek for locally optcal solutons for the 15 body segments n spte of the stochastc search wll converge to a globally soluton fnally. Deutscher [1][5] developed APF method by usng crossover operator mtatng genetc algorthm and herarchcal parttonng. Our proposed method s also lke to smulate the breedng of ndvduals as the conventonal genetc algorthms by takng local errors mnmzaton nto consderaton. The weghtng functon between two volumetrc models are shown as followng 15 p X, Z w X, Z exp (1) 1 N where X s the segmented volumetrc model and N s the number of volume data labeled as and Z s the model to be matched. p (X, Z) s the number of matchng ponts of the segment to the model X. We provde a fast method for computer p (X, Z). We computer the bound box of the volumetrc model Z as shown n Fgure 3(a), and represent the volume data n the bound box by a 1D boolean table. For each volume data n the labeled model (b), we compute the correspondng ndex, f t s n the bound box and the value n the boolean table s 1, the data s consdered as a matchng voxel data. The cost functon s O(n). We can compare the dfferences of these selected partcles wth the model to be tracked quckly by usng ths fast algorthm. Our proposed globally trackng algorthm s conducted accordng to the followng steps: 1) Start an annealng run at layer M, wth m = M. ) Deform the sample model to construct N un-weghted partcles. 3) Assgn a normalzed global weght to each partcle as descrbed n [1]. The local weghts for all body segments are able to be calculated correspondng. 4) Sort the partcles by local weghts and combne them. ( ) ( ) ( ) Assume 15 partcles s x, x, x selected and, 1,, 15 are 1, s fts well to the body segment. We construct new partcles s from s by choosng the values affect the transformaton of the segment and set others 0. We set n k 0, k 1,, 4. If the kth data n s s nonzero, n k n k 1. Set s s x1, x,, x4.then the 15 selected partcles are combned to form the new partcle new s where s new x, x n () 1 n1 x n,, 5) Recalculate the normalzed global weght for each partcle and draw N new partcles randomly accordng to the weghts. 6) The selected partcles are used to ntalze layer m-1. The process s repeated untl we arrve at layer 0. 7) The optmal soluton s estmated by combnng the partcles of layer 0 accordng to the normalzed weghts

4 t = 11 t = 16 t = 5 (a) A volume sequence Fgure 4. Trackng errors of each body segment and the whole model by APF and modfed APF. In our experments, t was found that settng the layer number M = 10 wth partcle number N = 300 worked well for human moton trackng. As seen n Fgure, the extracted artculated skeleton by our proposed method fts well to the volumetrc model as shown n the top left plot than APF method. The correspondng msmatchng rates for all body segments and the whole body are shown n Fgure 4. For nstance, the segment 4 represents the rght foot. Our method lmted the msmatchng rate of ths lmb from 35% to 7%. Also t s obvous as shown n Fgure that our method has located the rght foot more accurately than APF method. We also can see from Fgure that the orgnal msmatchng rate of the rght hand s 100%. The APF method and our method decrease t to 98% and 7% respectvely as shown n Fgure 4. We respect the error of the pose of the rght hand s able to be decreased below 50%. In addton, f we do nothng to msalgned the msmatchng body segment locally, the errors wll be accumulated. Therefore, t s mportant to do the local refnement work n each step. It s shown n Fgure 5 that n the frst column of the deformed model and the extracted posture matches well to the orgnal vsual hull. As tme ncreases, the matchng errors are accumulated and t wll cause problems. 3. Segmentaton It s ntutve that the regstraton algorthm wll be tmeconsumng and hard to acheve accurate results f we regster part of the human body such as the rght hand to the whole model. So we ntend to dvde the model nto several parts that the regstraton algorthm wll be appled to match part to part for each body segment. As our modfed annealed partcle flter approach s able to estmate the approxmated artculated pose wth global (b) Deformed models (c) Artculated skeleton models Fgure 5. Accumulated errors. optmzaton, the observed volumetrc model s parttoned nto several lmbs n accordance wth the mnmum Eucldean dstances to the deformed template vsual hull. A rapd and effcent mnmum dstance calculaton method s provded. The bound box of the segmented model s gven. Then ths volumetrc model s able to be represented by boolean values. We detect the neghborng voxels of each data n the vsual hull to be segmented n turn. As the results of moton trackng by modfed APF shown that the mean absolute dstance s about 10mm as shown n Fgure 8, the cost functon s about O(n). Whle the model to be tracked s segmented nto 15 parts respectvely, the mean absolute dstance between the deformed model and the model to be tracked s able to be estmated and utlzed for local refnement. 3.3 Local Refnement In global optmal scheme, trackng of knematc models by matchng volume data s effcent and reasonable for computng. However, nose exsts n vsual hull generaton by volume ntersecton methodologes []. Subsequently, surface shape regstraton should be taken nto account whle our artculated annealed partcle flterng method only captures global optmal pose fttng to vsual hull. So we 4

5 Fgure 6. Mean absolute dstances of each body segment and the whole model wth/wthout local refnement. employee the surface geometrc nformaton of each body segment, herarchcally search the msalgned part and refne t by artculated ICP regstraton. An artculated ICP regstraton has been presented n Demrdjan [1], enforcng artculated constrants on moton transformatons for all body segments. Ths method s also utlzed to extract human moton through vsual hull n Corazza [7]. They seek for a globally optmal soluton usng the artculated human moton constrants. So t may cause locally msalgned problem. In addton, they assume the rotaton angle s small enough n order to convert the problem to the lnear soluton equatons. If we expect to estmate quck moton t s hard to extract the 3D artculated posture accurately. It s clear that we are able to avod ths problem as we have estmate the ntal pose for the model by our proposed APF algorthm prevously. Furthermore, we refne the pose herarchcally to enforce local optmal soluton for each body segment. As we generate new partcles by usng crossover operator method accordng to the local ftness of each body segment, local optmzaton s enforced although the purpose of the stochastc search method s to chase a globally optmal soluton. Therefore, the result provdes good ntal transformaton for ICP regstraton and the rotaton angle won t be large. We detect the msalgned segment from the root part to leaf parts. If segment needs to be regstered, all body segments subject to t wll be realgned. Let T be the unconstraned transformaton for the body segment to be realgned and q be the jont. An artculated rgd moton s parameterzed by a twst [11][1][13]. Then the 4 4 matrx ˆ s defned to be ˆ wˆ 0 w q 1 (3) Where w ponts n the drecton of the rotaton axs and ŵ s the skew-symmetrc matrx assocated wth w. The rgd transformaton assocated wth the twst can be represented by G where G exp( ˆ) I ˆ because the body segment perform small moton after global optmzaton. We search for G that mnmzes the dstance: Fgure 7. The mean msmatchng rate. E (4) G V TV (5) where V represent vertces of body segment and t s a lnear problem. The results as shown n Fgure 6 are the mean absolute errors of deformed models n Fgure, whch demonstrate that artculated posture s able to be refned by ICP regstraton. The mean absolute error decreases from 10mm to 6.5mm. In addton, segment (the rght hand) mantaned better regstraton after local adjustment. 4. EXPERIMENTAL RESULTS We use the publc datasets provded by Gall [10] to test our algorthm at frst. Our approach s demonstrated to be robust and accurate for human moton trackng. Artculated knematc chans are tracked by comparng the tme-varyng vsual hulls. In addton, the property of bnary representaton of volumetrc model makes t easy for comparng models or dstance calculaton. In Fgure 7, artculated poses are able to be recovered from vsual hulls wth robustness and accuracy even for poor 3D representaton n general. The global msmatchng rates between the deformed template model and the ones to be tracked are below 10%. Furthermore, the mean dstance errors are lmted blow 10mm as shown n Fgure 8. 5

6 t = 11 t = 30 t = 35 Fgure 9. Model segmentaton. E D V Vox (9) Fgure 8. The mean dstance error. However, t should be menton that the segmentaton results wll affect on the moton trackng data. We can see the expermental results from Fgure 7 and Fgure 8 that the errors tend to ncrease. The man reason s that the errors of the model segmentaton are accumulated as shown n Fgure 9. We ntend to search for more robust model segmentaton method to mprove the accuracy and robustness. The volume data should be analyzed frst or teratve methods for model segmentaton are preferred. 5. FUTURE WORK The extracted knematc postures are able to be utlzed for 3D tme-vary surfaces reconstructon. Blend sknnng approaches are wdely used n shape generaton whle 3D skeleton poses are provded. However, these methods suffer from recoverng from detaled deformaton as they couple of vertces to underlyng relatve bones. We have present a robust method for extractng the artculated postures and correspondng transformatons for each body segment are also estmated. Then the tme-varyng surfaces are able to be generated based on the template mesh and captured moton data. We obtan robust rgd moton by global trackng and local refnement processng over the vsual hulls. Non-rgd transformaton should also be consdered. Snce we have segmented vsual hulls and template mesh surface nto 15 parts and estmated rgd transformatons for all segments, we deform t to construct tme-varyng surfaces drectly from 3D ponts. The refned surface are generated by solvng the least-squares problem where v arg mn E E E (6) E S T E S LV (7) N V T TV prevous 1 D (8) Durng the surface reconstructon process, we seek for smoothness by energy functon ES where L s the Laplacan matrx, V are the vertces and δ are the dfferental coordnates of current mesh shape translated by rgd transformaton. ET enforces surface deformaton affected by estmated pose. Vox are a set of nearest volume data n vsual hull from deformed mesh. Nearest voxel data means that s the nearest one not only n terms of Eucldean dstance but also n terms of normal drecton. We enforce that the dot product of normalzed drectons between the vertex on the surface and nearest voxel s greater than 0, f better, close to 1. We show some smple expermental results about 3D surface reconstructon n Fgure 10. In fact the current results are not so good and more mplemented work s needed. Unfortunately, nose exsts n vsual hulls that deformng surfaces rely on the nearest volume data causes problem as shown n Fgure 10 because the nearest voxel data n the surface of generated vsual hulls s usually smaller than real models. 6. CONCLUSION We propose a model-based moton trackng method capable of extractng robust 3D artculated postures wth 4 degrees of freedom through a sequence of vsual hulls. An ntal pose preservng global optmzaton s evaluated by takng advantage of the modfed artculated APF method. Then we segment the vsual hull to be tracked nto 15 parts and calculated the surface errors to search msalgned lmbs for local refnement. We refne the rgd posture n order to enforce local optmzaton and avod errors caused by noses. The rgd transformaton for the msalgned lmb s estmated separately usng ICP regstraton. Then, we solve a lnear equaton to enforce artculated constrants on conventonal ICP regstraton method. Our proposed approach s effcent for moton trackng n hgh dmensonal confguraton spaces and t can be easly convert to extract moton from tme-vary mesh surfaces. 6

7 Fgure 10. From top to bottom, Vsual hulls to be tracked, deformed mesh surfaces accordng to the pose transformaton estmated by our modfed artculate APF, refned ones usng ICP regstraton and surface estmaton by matchng 3D ponts to correspondng vsual hull. 7. REFERENCES [1] J. Deutscher and I. Red, Artculated body moton capture by stochastc search. Int. J. of Computer Vson, vol.61, no., pp , February 005. [] I. Mkć, M. Trved, E. Hunter, and P. Cosman, Human Body Model Acquston and Trackng Usng Voxel Data, Int. J. Comput. Vson, Vol.53, No.3, pp.199 3,

8 [3] P. Besl, and N. McKay, A Method for Regstraton of 3-d Shapes, In IEEE Trans. on Pattern Analyss and Machne Intellgence, vol.14, no., pp.39 56, February 199. [4] Lorensen, W. E. and Clne, H. E Marchng cubes: A hgh resoluton 3D surface constructon algorthm. ACM Transactons on Graphcs (TOG), v.1, no.4, pp , August [5] J. Deutscher, A. Davson and I. Red, Automatc parttonng of hgh dmensonal search spaces assocated wth artculated body moton capture. In Proceedngs of the IEEE Conf. on Computer Vson and Pattern Recognton, v., pp , 001. [6] R. Plankers and P. Fua, Artculated soft objects for multvew shape and moton capture, IEEE Trans. on Pattern Analyss and Machne Intellgence, 5(9): , 003. [7] S. Corazza, L. Mündermann, and T. P. Andracch, Automatc generaton of a subject specfc model for accurate markerless moton capture and bomechancal applcatons, IEEE Trans. on Bomedcal Engneerng, n press. [8] S. Corazza, L. Mündermann, E. Gambaretto, G. Ferrgno, and T. P. Andracch, Markerless moton capture through vsual hull, artculated ICP and subject specfc model generaton. Int. J. of Computer Vson, vol.87, no.1-, pp , March 010. [9] D. Vlasc, I. Baran, W. Matusk, and J. Popovć, Artculated mesh anmaton from mult-vew slhouettes, ACM Transactons on Graphcs (TOG), v.7, no.3, pp.1-9, August 009. [10] J. Gall, C. Stoll, E. de Aguar, C. Theobalt, B. Rosenhahn, and H.-P. Sedel, Moton capture usng jont skeleton trackng and surface estmaton, IEEE Conf. Computer Vson and Pattern Recognton, pp , June 009. [11] C. Bregler, J. Malk, and K. Pullen, Twst based acquston and trackng of anmal and human knematcs. Int. J. of Computer Vson, vol.56, no.3, pp , February-March 004. [1] C. Bregler and J. Malk, Trackng people wth twsts and exponental maps. In CVPR 98, [13] R. Murray, Z, L and S. Sastry, A mathematcal ntroducton to robotc manpulaton. CRC Press, [14] J. Starck and A. Hlton, Model-based multple vew reconstructon of people, In Int. Conf. on Computer Vson, October 003. [15] K. Varanas, A. Zaharescu. E. Boyer, and R. Horaud, Temporal surface trackng usng mesh evoluton, In European Conf. on Computer Vson, pp.30-43, October 008. [16] E. de Aguar, C. Theobalt, C. Stoll, and H.-P. Sedel, Marker-less 3D feature trackng for mesh-based human moton trackng, In Proc. ICCV HUMO07, pp.1 15, 007. [17] E. de Aguar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Sedel, and S. Thrun, Performance capture from sparse mult-vew vdeo, ACM Transactons on Graphcs (TOG), vol.7, no.3, pp.1-10, August 008. [18] E. de Aguar, C. Theobalt, C. Stoll, and H.-P. Sedel, Markerless deformable mesh trackng for human shape and moton capture, IEEE Conf. Computer Vson and Pattern Recognton, 007. [19] I. Baran, D. Vlasc, E. Grnspun, J. Popovć, Semantc deformaton transfer, ACM Transactons on Graphcs (TOG), v.8, no.3, pp.1-6, August 009. [0] J. Gall, B. Rosenhahn, T. Brox, and H.-P Sedel, Optmzaton and flterng for human moton capture, Int. J. of Computer Vson, vol.87, no.1-, pp.75-9, March 010. [1] K. Tomyama, Y. Orhara, M. Katayama, and Y. Iwadate, Algorthm for Dynamc 3D Object Generaton from Mult-Vewpont Images, Proc. of SPIE, vol.5599, pp , 004. [] D. Demrdjan, Combnng geometrc- and vew-based approaches for artculated pose, In European Conf. on Computer Vson, pp , 004. [3] M. Botsch and O. Sorkne, On lnear varatonal surface deformaton methods, IEEE Transactons on Vsualzaton and Computer Graphcs, vol.14, no.1, pp , January 008. [4] P. Peursum, S. Venkatesh, G. West, A study on smoothng for partcle-fltered 3D human body, Int. J. of Computer Vson, vol.87, no.1-, pp.53-74, March 010. [5] R. M. Neal, Annealed mportance samplng, Statstcs and Computng, vol.11, no., pp , Aprl