3D Modeling for Capturing Human Motion from Monocular Video

Transcription

1 3D Modeling for Capturing Human Motion from Monocular Video 1 Weilun Lao, 1 Jungong Han 1,2 Peter H.N. de With 1 Eindhoven University of Technology 2 LogicaCMG Netherlands P.O. Box 513 P.O. Box MB Eindhoven 5605JB Eindhoven The Netherlands The Netherlands w.lao@tue.nl P.H.N.de.With@tue.nl Abstract This paper presents an approach to build a 3D human-body model from uncalibrated monocular video sequences. Using an orthographic projection model for the camera and a generic human body model, we constitute a 2D-3D conversion scheme to generate a 3D skeletal model from the 2D image domain. The complete 3D geometry and kinematics information of a human being can be extracted without any a-priori information about the camera system or the person involved. Our promising experimental results can accurately reconstruct the human motion trajectory. The system can be utilized for an effective human-motion capture/analysis and a 3D skeleton reconstruction of deformable objects. 1 Introduction Recent advances in human-body modeling promise to create a wide variety of new applications, particularly those requiring 3D human models. Examples of such applications [1] include fitting of virtual clothes, anatomical medical diagnosis, virtual characters in film and video, human computer interactions and many others. In computer graphics and computer vision areas there is a relentless pursuit of more realistic modeling of human body geometries and human motions for applications like gaming, virtual reality and computer animation that demand highly realistic Human Body Models (HBMs). In summary, the different approaches to 3D human modeling are essentially based on two main modes of data capture: computer-graphics based systems and image-based systems. Computer-graphics based computer animation software, such as Maya, 3D Studio Max and Autocat that implement the modeling of a wide range of objects including the human body, is widely available. A full suite of polygons, NURBS and subdivision surface modeling tools [2] can be utilized to obtain high-resolution HBMs. Smooth 3D meshes and the various underlying human skeleton models empower these software tools to create, edit, render and animate human body models. However, as most systems do not directly integrate information acquired from actual objects or individuals, the generated models lack realism. At present, the process of generating realistic human models is still very human labour intensive and so their application is therefore currently limited to the movie industry where HBMs movements are predefined, well studied

2 and painstakingly manually produced. Fully automatic rendition of highly realistic and fully configurable HBMs is still an open research problem. A major constraint involved is the computational complexity to produce realistic models with natural behaviors. Lately, computer-vision based approaches [3][4][5] are increasingly being used for automatic generation of HBMs, processing video-captured image sequences by incorporating and exploiting prior knowledge of human appearance. In contrast to computergraphics systems, computer-vision approaches concentrate more on efficient rather than accurate models for human-body modeling. Also, they are typically designed to generate novel views of a real scene from camera-captured input images of a particular view. Different 3D representations and mathematical formalisms have been proposed to model both the structure and movements of a human body. An HBM can be generally represented as a chain of rigid bodies, called links, interconnected to one another by joints. Links are typically represented as sticks, polyhedrons, generalized cylinders or superquadrics. A joint interconnects two links by means of rotational motions about the axes of rotation. The number of independent rotation parameters will define the degrees of freedom (DOF) associated with a given joint. But most of the computer-vision based approaches require more than one camera and highly accurate camera-calibration parameters. Unfortunately, it is not feasible for surveillance and e.g. low-cost consumer multimedia applications. The fast and accurate generation of 3D HBMs from uncalibrated video sequences remains a challenging problem although previous work was done on single images [6]. An area of particular interest is the automatic modeling of real human individuals from monocular video sequences. We propose a method for 3D reconstruction of moving human bodies, employing human body knowledge. The method involves the following steps: prior human-body model design, camera calibration, temporal filtering and geometry/kinematics model generation. which effectively contribute to reconstructing a 3D skeletal human model. Our contributions to the 3D modeling of human-motion capture are twofold. First, we propose a simple but effective system to generate realistic 3D models of moving human bodies from only one uncalibrated camera. No a-priori information about the camera or the person involved is required. It captures the human motion and generate a 3D body model which can be easily exported to create a realistic surface model of an individual using 3D processing software. Second, despite the intelligence of the model, we have employed a simple camera-calibration model for mapping the data to the real-world domain, followed by a temporal filter to smooth the trajectory of every estimated 3D point. The calibration model is omitting the camera-intrinsic parameters and is based on scaling only. Our methodology differs from current state-ofthe-art work in the simplicity of its operations while preserving the overall quality of the final reconstruction. The presented work is intended to contribute to the object/scene analysis and behavior modeling of deformable objects.

3 Figure 1: Block diagram of our human-motion modeling system. 2 System Framework The top-down block diagram of our 3D human-motion modeling system is shown in Figure 1. It consists of three different modules. First, at the pre-processing module, each image covering an individual body is segmented to extract the human silhouette and each joint location is detected afterwards. Secondly, at the modeling stage, we use a generic 3D human model to represent a person. Knowledge about the human body and its motion constraints, such as its symmetry and validity, plays an important role in tracking of the body segments. Meanwhile, the orthographic projection model is applied to map the camera signal to the real-world domain. This simplifies the mapping between 3D world space and image space. Then the 2D-3D human model projection step produces both 3D human geometry and 3D kinematics represented by a skeletal model. Also, we use a temporal filter to smooth the location of each joint in the human-motion model. Finally, the outcome of modeling module is fed into the semantics module for analysis related to a specific application. For example, the geometry/motion information of the reconstructed 3D model is essential for tennissports performance analysis. The players and coaches can benefit from the video-based performance analysis to improve their training. Our work concentrates on the modeling module of the whole system in this paper. Each component involved in this module is described in more detail in Section 3. The remaining modules of the system are developed in ongoing work.

4 3 Methodology Figure 2: 3D human-body model design D-3D human-body model projection The 2D-3D human model projection is implemented based on two key components: prior human-body model design and camera calibration D human-body model design The 3D body model we use in the scheme is shown in Figure 2. It is a generic model consisting of an arrangement of 14 body segments (head-neck, upper arm, torso etc.). The length of each segment is estimated from the measurements provided by [7]. It can also adaptively align with the actual person by exploiting a sophisticated approach [3] to the video sequences. The model structure is visualized in Figure 2 via a skeleton consisting of 15 joints (dots) connecting 14 bone segments (lines between the dots) Camera calibration If the depth of objects (human body in our case) is small compared to the distance between them and the camera, a scaled orthographic projection model can be applied for camera calibration [8]. Under this assumption, the camera calibration is simplified and the intrinsic/extrinsic parameters of the camera are not required. Let the 3D coordinates and its corresponding 2D image coordinates of the joint i be represented by P i = [X, Y, Z] T and p i = [u, v] T, respectively. Using the scaled orthographic projection with a scalar factor s, we obtain ( u v ) = s ( ) X Y Z. (1) Segments of the human model shown in Figure 2 are denoted by Seg=(Seg 1,..., Seg 14 ). For each segment with index i we have its 3D end-points Seg i,1 and Seg i,2. We can calculate the length of the segment i from the Euclidian distance length i = Seg i,1 Seg i,2 between its end-points. With the constraints imposed by Equation (1), we compute the relative depth between two end-points for the segment i by

5 [Z(Seg i,1 ) Z(Seg i,2 )] 2 = length 2 i [u(seg i,1) u(seg i,2 )] 2 + [v(seg i,1 ) v(seg i,2 )] 2 (2) where Z(.) represents the z-coordinate of a specific 3D point and u(.), v(.) are its corresponding 2D image coordinates. Since the left-hand side of Equation (2) is always positive, we find an important constraint on the scalar factor s i, which is s i [u(segi,1 ) u(seg i,2 )] 2 + [v(seg i,1 ) v(seg i,2 )] 2 ) length i. (3) We define the four joints of neck, shoulders, waist to be part of the same plane and choose eight segments for mapping the arms and legs, in order to calculate the individual results s i (i = 1, 2,..., 8) from the right-hand side of Equation (3). Here we choose these segments instead of others because arms and legs are normally bent for poses allowing for easier detection of elbow and knee locations. Afterwards, we choose the median value of s i to determine the scalar factor by s = Median(s 1, s 2,..., s 8 ). (4) As the left-hand side of Equation (2) yields two solutions, we need to find out which end-point of the segment is closer to the camera. It can be solved by manual input, estimation from previous frames or with a learning-based approach [9]. Finally, we combine Equations (1)-(5) prior to extracting the 3D coordinates of each joint in the human model D geometry & kinematics model generation When the estimated locations of each joint at each frame are available, we are able to generate both a 3D geometry (skeleton) and a kinematics (motion) model. More specifically, two components simultaneously impose constraints on smoothing human segments trajectory. First, in the spatial domain, knowledge about the human-body structure and its motion statistics, such as its symmetry and validity, plays an important role in the tracking of body segments. Second, in the temporal domain, we use a temporal filter to smooth the location of each joint in the model along the human motion. The temporal filter is defined as X k Y k Z k = X i Y i Z i + k i j i X j X i Y j Y i Z j Z i s 2 i, (5) where the estimated 3D coordinates of the specific points P i = [X i, Y i, Z i ] T and P j = [X j, Y j, Z j ] T at the key frames i and j provide the input for calculating the corresponding 3D coordinates P k = [X k, Y k, Z k ] T at the frame k with (i < k < j).

6 Figure 3: Reconstructed 3D model of a walking person (left column: two different frames from the original monocular video sequence; middle & right columns: the recovered 3D human skeletal model visualized from two different viewpoints.). 4 Experimental Results In our experiment, the CMU MoBo Database [10] was applied to capture humanmotion sequences. In the test video, a subject (one person) is walking on a treadmill, positioned at the center of the image. The test color images have a resolution of samples. The sequence is 1.2 seconds long, recorded at 30 frames/s. The locations of the joints were manually marked. We investigate the pre-processing steps of the automatic silhouette extraction. One key frame is selected for every group of 10 frames for the processing of the 2D-3D model projection. An example of our experimental results is illustrated in Figure 3. The reconstructed 3D model of the moving person can be visualized from different viewpoints at every time instant. The recovered model shows the satisfactory accuracy and reliability of our proposed scheme. 5 Conclusions and Future Work This paper presents an effective 2D-3D conversion scheme to reconstruct 3D skeletal models from uncalibrated monocular video sequences. A primary assumption is that the camera capturing process is according to the orthographic projection model. Both geometry and kinematics information are obtained during the modeling procedure. A temporal filter is therefore used for smoothing extracted motion sequence. First experiments have shown promising results and confirm that our scheme is of satisfactory accuracy. This scheme can also apply to the case of fairly complicated human motion. In the near future, work will focus on the pre-processing module and semantics module

7 as depicted in the system block diagram. A future improvement is the automatic finding of model joints at the initialization stage in the pre-processing module, because at present annotation is still required in the key frames. When automating this process, we need to consider the special singular cases of body parts resulting from the projection. The segmentation step can be automatically implemented by means of further contour analysis. Meanwhile, the location of joints from silhouette may be conducted automatically and robustly. At the semantic-analysis level, our system will be utilized for potential applications including surveillance, personalized training in golf, tennis, etc. and clinical studies. References [1] T.B. Moeslund and E. Granum, A Survey of Computer Vision-Based Human Motion Capture, Computer Vision and Image Understanding, Vol. 81, pp , 1. [2] P. Ratner, 3-D Human Modeling and Animation, John Wiley & Sons, [3] I. Mikic, M. Trivedi, E. Hunter and P. Cosman, Human Body Model Acquisition and Tracking Using Voxel Data, Int. Journal of Computer Vision, Vol. 53, pp , 3. [4] R. Plankers and P. Fua, Tracking and Modeling People in Video Sequences, Computer Vision and Image Understanding, Vol. 81, pp , 1. [5] J. Carranza, C. Theobalt, M.A. Magnor and H.P. Seidel, Free-Viewpoint Video of Human Actors, ACM Transactions on Graphics, Vol. 22(3), pp , 3. [6] C.J. Taylor, Reconstruction of Articulated Objects from Point Correspondences in a Single Uncalibrated Image, Computer Vision and Image Understanding, Vol. 80, pp , 0. [7] A.R. Tilley, The Measure of Man and Woman: Human Factors in Design, H. D. Associates, NY, [8] R.I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 0. [9] C. Orrite, J. Martinex, J. Elias and G. Rogez, 2D Silhouette and 3D Skeletal Models for Human Detection and Tracking, IEEE Int. Conf. on Pattern Recognition, Cambridge, UK, 4. [10] J. Shi and R. Gross. The CMU Motion of Body (MoBo) Database, Technical Report CMU-RI-TR-01-18, Robotics Institute, Carnegie Mellon University, June 1.