Human Motion Reconstruction and Animation from Video Sequences

Transcription

1 Human Motion Reconstruction and Animation from Video Sequences Fabio Remondino Institute for Geodesy and Photogrammetry ETH Zurich, Switzerland Web: Abstract In this paper we present a new framework for the modeling and animation of human characters from monocular videos. The modeling is performed semi-automatically using a polygonal shape model obtained from laser scanner measurements. Most of the image-based techniques use probabilistic approaches to derive the poses of a character from an image sequences; on the other hand we use a determinist approach, recovering character s poses and movement through a camera model. The recovered human models can be used for visualization purposes, to generate new virtual scenes or for gait analysis. Key words: 3D Reconstruction, Animation, Human Body Modeling 1. Introduction The realistic modeling and animation of human characters is one of the most difficult tasks in the vision and graphic community. In particular, body modeling from video sequences is a challenging problem that has been investigated a lot in the last decade. Recently the demand of 3D human models is drastically increased for applications like movies, video games, ergonomic, e-commerce, virtual environments and medicine. A complete human model consists of the 3D shape and the movements of the body; some approaches consider these two modeling procedures as separate even if they are very closed. The issues involved in creating virtual humans are the acquisition of the body shape, the modeling of the data and the acquisition of the information for the animation. Different representations of virtual characters are present in the literature. One of the most used articulated 3D representation for animated Andreas Roditakis Institute for Geodesy and Photogrammetry ETH Zurich, Switzerland roditak@antrikos.net Web: character is the H-Anim (Figure 1). It is an International Standard that defines the H-Anim as an abstract representation (in VRML97 language) for modeling 3D human figures; it describes a standard way to represent and animate humanoids and it is a simple design for 3D Internet applications [28]. Figure 1: Different representation of virtual characters (avatars) with articulated joints according to the ISO H-Anim [28]. 1.1 Related work on static shape capture A standard approach to capture the static 3D shape (and colour) of an entire human body uses the laser scanner technology [8][27][32]: it is quite expensive but it can generate a whole body geometry in ca 20 seconds. Afterwards the 3D shape can be animated (Figure 2), articulating the model and changing its posture [1][16][20] or dressed for garment models [2][3]. Figure 2: 3D human shape recovered with laser scanner and then animated. Sources (from left to right): [1][16][20]. Concerning image-based approaches, they require different acquisitions around a static person: then they can deform a 3D template model to the extracted human silhouette [7][10][22] or they can recover a 3D human model through a camera model and with

2 template matching [12]. Finally, computer animation software [26][29][30][31] allows the generation of virtual humans using splines and smoothing simple polygonal elements. 1.2 Related work on movement reconstruction Precise information related to character movements is generally acquired with motion capture techniques: they involve a network of cameras and prove an effective and successfully mean to replicate human movements [23][24][25]. Other approaches instead rely on video sequences as primary input. Single- or multi-stations videogrammetry offers in fact an attractive alternative technique, requiring cheap sensors, allowing markerless tracking and providing, at the same time, for 3D shapes and movements information. The analysis of existing videos can moreover allow the generation of 3D models of characters who may be long dead or unavailable for common modeling techniques. In [11][17][18][21] computer vision techniques, image cues, background segmentation, prior knowledge about human motion, pre-defined articulated body models and no camera model are used to recover motions and 3D information from monocular sequences (Figure 3). Multi-cameras approaches [4][5][19] are instead employed to increase reliability, accuracy and avoid problems with self-occlusions (Figure 4). the difficulties in recovering the camera parameters and because of occlusions of the body parts. The framework that we propose is an image-based deterministic process to recover shape and motion of a moving person from monocular videos ( Figure 5). First the camera parameters are determined and afterwards the human s poses and movements are recovered and modeled using a polygonal body model. 2. Sequence analysis for 3D pose reconstruction A moving character imaged with one moving camera represents the most difficult case for the deterministic 3D reconstruction of its poses. We analyse existing sport videos, to avoid copyright problems. They are usually acquired with a stationary but freely rotating camera and with a very short baseline between the frames. Therefore, due to the movements of the character, a standard perspective approach cannot be used, in particular for the 3D modeling of the character. : fully automated : semi-automated = manual Figure 3: Monocular approach [11]: original frame and skeleton reconstruction with a gaussian probability model. Figure 4: Multi-camera approach [4]: original frames, silhouette tracking and fitting process. In [4] smooth implicit surfaces are fitted to the recovered 3D data while in [5] tapered superquadratic are fitted onto the extracted human poses. Usually model-based approaches are more common, in particular with monocular video streams, while deterministic approaches are almost neglected, often due to Figure 5: Workflow for character modeling and animation from video sequences 2.1 Camera calibration and image orientation The photogrammetric calibration and orientation of the video is performed with a bundle adjustment [14]. The procedure is required to achieve the camera parameters necessary for the determination of the scene s metric information and for the human s poses estimation.

3 2.2 Body Poses estimation For man-made objects (e.g. buildings), geometric constraints on the object (e.g. perpendicularity and orthogonality) can be used to solve the ill-posed problem of the 3D reconstruction from a monocular image. In case of free-form objects (e.g. the human body), we could use probabilistic either modelbased approaches or other assumptions must be provided [13]: the perspective collinearity model is simplified into a scaled orthographic projection; the human body is represented in a skeleton form, with a series of joints connected with segments of known relative lengths (Figure 6, left); further constraints on joints depth and segment s perpendicularity are applied to obtained more accurate and reliable 3D models. This reconstruction algorithm (Figure 6, right) is applied to the analyzed frames of the sequence, given the image coordinates of some human joints and the relative lengths of the skeleton segments. The image points of the joints cannot be automatically tracked along the sequence, due to occlusion problems and interlaced video; therefore the points are measured semi-automatically with a least squares matching (LSM) algorithm [6]. For each image, a 3D human model is generated and afterwards transformed to the absolute reference system with a 3D-conformal transformation. Finally the 3D coordinates are refined using the camera parameters recovered in the orientation process [15]. Figure 6: The human body simplified and represented with a skeleton of 13 joints plus the head (left). The process to recover 3D human models from single images or monocular videos (right). 3. Human character modeling and animation The recovered character s poses are used to reconstruct the 3D model of the human skeleton. To improve the visual quality and realism of the model, a pre-defined polygonal model [27] is afterwards fitted to the recovered photogrammetric 3D data. The scanned model will be the skin of our 3D virtual character. The fitting and animation processes are performed with the animation features of Maya software [30]. Other approaches where a whole body scanned data is deformed to generate animatable human bodies are described in [1][16][20]. 3.1 Skeleton reconstruction and visualization The recovered 3D coordinates of the human joints are given to a procedure that uses VRML language to automatically visualize the simplified human model. This step is mainly used to check if the recovered 3D model is correct. A simple skeleton is created, where all the human joints are represented with spheres connected with cylinders or tapered ellipsoids in case we want to model the muscles (Figure 7). Figure 7: 3D model of the human skeleton shown in VRML using spheres, simple cylinders or different shapes for torso arms and legs [13]. 3.2 Polygonal model fitting The first step in fitting a polygonal model (Figure 8) to the extracted poses is the construction of a new native skeleton, whose joints are running through the fourteen characteristic points of the input skeleton; it will be the structure on which the skin fitting and animation will be based. The new skeleton is then moved inside the polygonal body, translating, scaling and rotating its limbs so that they lie inside the body. Small disagreements can be present because of errors in estimating the lengths of the limbs from the image data.

4 Figure 8: The polygonal model obtained with laser scanner [27], consisted of approximately polygons (left). Two closed views of the wireframe model in areas subordinated to strong deformations during the movements. 3.3 Binding the skin with the skeleton The basic idea behind skeleton and body binding is that the skeleton controls the body movements and deformations. Thus the body takes the poses of the skeleton limbs. As we want to represent any possible human pose, all the deformations that occur when the body parts are moving must be taken into consideration. The Maya animation platform, as most of the animation software on the market, supplies two solutions for simulating the interaction of a skeleton with its skin: (1) soft bind, that allows two or more skeleton joints to influence their surrounding geometry and works very well in soft object simulation; (2) rigid bind, that allows the influence of only one joint per body segment and performs better in solid body modeling. During the movements, while some human body parts remain solid, there are soft body deformations in areas such as the arms and knees, which makes the application of any of the above methods unrealistic. Moreover, in soft binding, if we want to modify the weight values of a joint, we need to do this on every part of the influenced body surface. And the process is time-consuming if we have to fine-tune all the influences on a soft bind skin. Therefore, a combination of rigid body modeling with 3D deformation lattices is required. Lattices are applied on the sensitive areas and they are one of the best approximations of the physical body behaviour. Deformation lattices are placed in the body s areas subordinated to strong deformation during the movements, e.g. shoulders and feet ankles of the polygonal human model (Figure 9). The basic idea is that the skeleton should deform the lattice, which consequently influences the position of the affected points of the polygonal model. In this way the deformations follow the skeleton movements. The number of grid cells of the lattices depends on the average number of points that every cell should contain, so it depends on the spatial resolution of the polygonal body. Lattices with dimensions larger than 10 in all three axes are proven computationally expensive, while dimensions smaller than 3 are too coarse. Figure 9: The lattices set in areas of the polygonal model subordinated to strong deformation during the movements. For joints like elbows and knees, an additional and faster deformation procedure can be used. The flexor deformers are specially designed deformers for human body animation that have automated the task of smooth skin simulation in such areas. Because of the simplicity of their geometry, they are used in areas that are not stressed too much by the movements deformations. 3.4 Reverse Feet mechanism The animation of some human s parts (like the foot) is not a trivial issue. In fact, when animating a walking biped, the feet should stay firmly on the ground. In addition to this, deformations occur in the ankle area and the angle between the foot and the leg is changing, leading to high modification of the polygonal area. In order to model correctly the movements of the human feet and to simulate their deformations in a realistic way, the reverse foot mechanism is used [9]. A new additional skeleton from the ankle to the toe is created (control skeleton) and an extra joint is added (Figure 10). Figure 10: The control skeleton defined to animate and model correctly the movements of the feet.

5 This last joint (control joint) is the root of the duplicate skeleton and is not connected to the hierarchy of the main skeleton. Then, using Point and Orientation constraints, the feet on the main skeleton is constrained to follow the position and orientation of the corresponding control skeleton. This does not belong to the main skeleton and can be key-framed separately. 3.5 Animation (IK handles) The movement of the human limbs is controlled by automated procedures called Kinematics. They are also known as handles and work as a chain running through all the joints. They control the rotations and translations among sequential skeleton joints that build up a limb. In this way complex movements on multiple joints can be performed by only adjusting the chain s endpoint (effector) position and rotation. In order to achieve this effect, there are two different types of kinematics: Inverse Kinematics and Forward Kinematics. In the first case the position and rotation of the effector is enough to position all the intermediate joints; the latter demands that the translation and rotation of every joint is given, from top to bottom in the hierarchy, until the desired pose is achieved. Because of their simple and intuitive control, inverse kinematics handles are preferred. 3.6 Movement constraints Although the handles require less input from the animator, the presence of more than one possible correct solution for the intermediate joints makes the use of skeleton constraints necessary. The physical movement limits of the human skeleton are therefore defined with angular constraints, in order to avoid false positions or impossible twists and get the correct poses of the limbs. 3.7 Performance Once the human skeleton is constructed and imported in Maya, the maximum time spent for bringing the polygonal body into the correct pose is approximately 2 minutes. The lattices definition take ca 3 minutes, as more attention must be given to their positioning and to include the correct points that must be affected. For the rendering, due to the interpolation functions, and only some key-frames are imported and analyzed. We used the native Maya renderer and for every frame (PAL resolution) approximately 1 minute is required. The animation of a sequence composed of 200 frames required a little more than 3 hours to be completed. 4. Results A sequence of 60 images was digitized from an old videotape, using a Matrox DigiSuite grabber, with a resolution of 720x576 pixels (Figure 11). Figure 11: Some original frames of a moving character. After camera calibration and image orientation, the human joints are tracked all over the sequence and afterwards the 3D coordinates are computed. A VRML model of the imaged scene, with some camera positions and the moving human skeleton is presented in Figure 12. Figure 12: The camera poses as well as the 3D reconstruction of the basketball court and the moving character.

6 The polygonal human model is then fitted to the skeleton and animated using the recovered poses and following to the presented methodology (Figure 13). Figure 13: Results of the fitting process onto the photogrammetric 3D data. In this case also the ball is modeled. Another sequence, presented in Figure 15, was analyzed. The camera, mounted far away from the scene and probably on a tripod, is rotating and zooming to follow the moving character. The calibration and orientation process, performed with a self-calibrating bundle adjustment with frame-invariant APs sets, recovered a constant increasing of the camera focal length [14]. The 3D reconstruction of the moving character is afterwards performed and the results are shown in Figure 16. A new virtual scene of the analyzed sequence is also presented. Critical polygonal areas subordinated to strong deformations are quite well modeled as shown in Figure 14. Figure 16: Results of the 3D modeling of the moving character visualized in skeleton form. Figure 14: Two examples of areas where the movements create high deformations of the polygons. The lattices helped to create a realistic result. To improve the visual quality and the realism of the reconstructed 3D human skeleton, the fitting is afterwards performed. The semiautomatic modeling process produced satisfactory results, as shown in Figure 17. The camera viewpoint is also changed, to generate new virtual views of the scene. In this example we chose lattices with 6 cells in all dimensions for the shoulders and 4 cells for the feet ankles, which gave a balance between performance and quality. Figure 15: Some frames of a basketball video sequence. Figure 17: Two examples showing the results of the fitting and the modeling process. Five frames of the sequence (upper image) and the middle frame seen from 2 different viewpoints (lower images).

7 Despite of the complex movements and strong deformations, the critical areas like feet and shoulders are realistically rendered (Figure 18). Figure 18: The feet and the shoulders of the character modeled using the deformation lattices. 5. Conclusions The analysis of monocular video sequences and the generation of 3D human models were presented. After the sequence orientation and calibration, the positions of some human joints are recovered throughout the frames. This 3D information is afterwards used for further modeling, where a whole body scanned model is fitted and animated using photogrammetric data. The use of existing and old videos showed the capability of videogrammetry to provide for virtual characters useful for augmented reality applications, persons identification and to generate new scenes involving models of characters who are dead or unavailable for common modeling systems. We also demonstrated that in combination with computer animation software we could achieve realistic reconstruction from any desired viewing angle and produce realistic realitybased animations. The proposed framework can become a valuable tool for supplying input motion data for avatars in virtual reality environments. Currently we are trying to improve the polygonal model, in particular in the hands area. Furthermore we will try to back-project the fitted polygonal model onto the images, using the recovered camera parameters, to deeply check the quality and accuracy of the reconstruction and fitting process. Finally, we plan to fit the image-based results with a simpler human model, like. a H-Anim VRML model. References [1] B. Allen, B. Curless and Z. Popovic. Articulated body deformation from range scan data. Proc. SIGGRAPH, pp , 2002 [2] F. Cordier, N. Magnemat-Thalman. Realtime animation of dressed virtual humans. Computer Graphics Forum, Blackwell, Vol.21(3), 2002 [3] F. Cordier, H. Seo and N. Magnenat- Thalmann. Made-to-Measure technologies for online clothing store. IEEE CG&A special issue on Web Graphics, pp.38-48, 2003 [4] N. D'Apuzzo, R. Plänkers, P. Fua, A. Gruen and D. Thalmann. Modeling human bodies from video sequences. In El-Hakim/Gruen (Eds.), Videometrics VI, Proc. of SPIE, Vol. 3461, pp , 1999 [5] D.M. Gavrila and L. Davis. 3D modelbased tracking of humans in action: a multi-view approach. IEEE CVPR Proc. pp , 1996 [6] A. Gruen. Adaptive least squares correlation: a powerful image matching technique. South African Journal of Photogrammetry, Remote Sensing and Cartography, Vol. 14(3), pp , 1985 [7] A. Hilton, D. Beresfors, T. Gentils, R. Smith, W. Sun and J. Illingworth. Wholebody modeling of people from multiview images to populate virtual worlds. The Visual Computer, Vol. 16, pp , Springer-Verlag, 2000 [8] Horiguchi. Body Line Scanner. The development of a new 3-D measurement and Reconstruction system. International Archives of Photogrammetry and Remote Sensing, Vol.32(5), pp , 1998 [9] Learning Maya 5 - Foundation, Alias Wavefront [10] W. Lee, J. Gu and N. Magnenat- Thalmann. Generating animatable 3D virtual humans from photographs. Eurographics, Vol. 19, No. 3, [11] M. Leventon, W. Freeman. Bayesian estimation of 3D human motion from an image sequence. TR-98-06, MERL, 1998 [12] F. Remondino. 3D reconstruction of static human body with a digital camera. SPIE Proc, Vol. 5013, Videometrics VII, pp , 2003.

8 [13] F. Remondino and A. Roditakis. Human Figures Reconstruction and Modeling from Single images or Monocular Video Sequences. IEEE International 3DIM Conference, pp , 2003 [14] F. Remondino and N. Boerlin. Photogrammetric calibration of sequences acquired with a rotating camera. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIV, part 5/W16, 2004 [15] F. Remondino. Character reconstruction and animation from monocular sequence of images. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol.XXXV/5, in press [16] H. Seo and N. Magnenat-Thalmann. An Automatic Modeling of Human Bodies from Sizing Parameters", ACM SIGGRAPH Symposium on Interactive 3D Graphics, pp , 2003 [17] H. Sidenbladh, M. Black, and D. Fleet. Stochastic Tracking of 3D Human Figures Using 2D Image Motion. ECCV, D. Vernon (Ed.), Springer Verlag, LNCS 1843, pp , 2000 [18] C. Sminchisescu. Three Dimensional Human Modeling and Motion Reconstruction in Monocular Video Sequences Ph.D. Dissertation, INRIA Grenoble, 2002 [19] S. Vedul and S. Baker. Three Dimensional Scene Flow. ICCV '99, Vol. 2, pp [20] X. Ju, N. Werghi and J.P. Siebert. Automatic Segmentation of 3D Human Body Scans. International Conference on Computer Graphics and Imaging (CGIM 2000), pp , 2000 [21] M. Yamamoto, A. Sato, S. Kawada, T. Kondo and Y. Osaki. Incremental Tracking of Human Actions from Multiple Views. IEEE CVPR Proc., 1998 [22] J.Y. Zheng. Acquiring 3D models from sequences of contours. IEEE Transaction on Pattern Analysis and Machine Intelligence 16(2), pp , [23] Ascension: [24] Vicon: [Accessed April 2004] [25] Motion Analysis: [26] 3D Studio Max 3.1: / [27] Cyberware: [28] H-Anim: [29] Lightwave: [30] Maya: [31] Poser, Curious Labs: [Accessed April 2004] [32] Vitus: