3D Reconstruction of Human Bodies with Clothes from Un-calibrated Monocular Video Images presented by Tran Cong Thien Qui PhD Candidate School of Computer Engineering & Institute for Media Innovation Supervisor: Co-Supervisor: Prof. Seah Hock Soon Prof. Nadia Magnenat Thalmann 9 November 2010
Outline Motivation Research Objectives Literature Review Proposed Research Method Current Work Further Work
Motivation Ability to clone human body is necessary for many different applications Animation & movies Avatars for games Crowd simulation Medical applications Sport Analysis Fashion (Beowulf 2007) (Crowd Simulation -VRlab, EPFL)
Motivation Traditional Laser Scanners Incomplete surface data Tedious, manual process Do not have skeleton information Expensive Multi-camera setup (Cyberware Scanner) Difficult to setup Calibrate and synchronize Lighting condition Limit its applications (D. Vlasic et. al. Siggraph Asia 09)
Research Objectives 3D Reconstruction of Human Bodies with Clothes from Un-calibrated Monocular Video Images Video Images from A Single Camera 3D Geometry of human body (Animatable) 3D Geometry of Clothes
Literature Review Human Body Reconstruction Has been an open research for a long time Can be categorized into 3 groups Approaches with no body template Shape-from-silhouette Approaches with a human model template Sample-based approaches
Literature Review Reconstruction with a human model template VRML H-Anim 1.1 [W. Lee, J. Gu and N.M. Thalmann 2000] VRML model [Hilton 2000] Soft models using metaball representation [Fua 2003] Generic template [D. Grest 2005] Generic template with B-spline curves [Theobalt 2005] ( J. Shen and D. Thalmann 1995) ( D. Grest 2005) (W. Lee, J. Gu and N.M. Thalmann 2000 )
Literature Review Sample-based Reconstruction Using many 3D body scans to learn the space of human body shape [H. Seo and N.M. Thalmann 2003] (57 females subjects in one posture) [Allen and Popovic 2003] (250 male and female subjects in one posture) [Anguelov 2005] (37 subjects in A pose + 1 subject in 70 postures) [Hasler 2009] (114 male and female subjects in 34 postures)
Literature Review 3D Reconstruction of human body from a single camera P. Guan, A. Weiss, A. O. Bălan, M. J. Black, Estimating Human Shape and Pose from a Single Image, ICCV 2009 Limited to naked-like subjects Only applied to one single image N. Hasler, H. Ackermann, B. Rosenhahn, T. Thormaḧlen, H. P. Seidel,: Multilinear pose and body shape estimation of dressed subjects from image sets, CVPR 2010 Manual input is needed for each image to detect posture Only uses image silhouette Loose clothes have not been fully addressed (Guan - ICCV 09) (Hasler- CVPR 10)
Literature Review 3D Reconstruction of human body from a single camera N. Hasler, H. Ackermann, B. Rosenhahn, T. Thormaḧlen, H. P. Seidel,: Multilinear pose and body shape estimation of dressed subjects from image sets, CVPR 2010 Loose clothes have not been fully addressed
Proposed Research Method Using video as input Increase the accuracy Help in separating human body and clothes Cloth deformation by motion can be used to verify the body geometry How to determine postures for each image frame? User will annotate some markers on keyframe images In-between posture will be interpolated
Proposed Research Method Manual Input in key frames Body Database Video Image s Keyframe Annotation Postures Fitting Body Model Estimated Body Geometry Silhouette Segmentation Silhouette Body Geometry Extract Garments Garment Database Bundle adjustment across the video sequence Cloth Geometry Body Geometry Cloth Geometry
Proposed Research Method Bundle Adjustment to adjust all the parameters Parameters: Postures, Body Sizes, Cloth Sizes Silhouette Error Function: the differences between silhouettes and the projection images of the model ( pos, bodysizes, clothsizes) Arg min ( f ( pos, bodysizes, clothsizes) sil ) pos : 3D positions, joint angles in frame i i i i i i bodysizes: sizing parameters of human body clothsizes: sizing parameters of clothes sil : silhouettes extracted from frame i i 2
Current Work Silhouette Segmentation C. Rother, V. Kolmogorov, A. Blake. - SIGGRAPH 04 GrabCut: Interactive Foreground Extraction using Iterated Graph Cuts.
Further Work Keyframe annotation and interpolation Resizing human model to fit the extracted silhouette Extracting garments
Thank you!