Human Body Shape Deformation from. Front and Side Images

Human Body Shape Deformation from Front and Side Images Yueh-Ling Lin 1 and Mao-Jiun J. Wang 2 Department of Industrial Engineering and Engineering Management, National Tsing Hua University, Hsinchu, Taiwan Email: d943845@oz.nthu.edu.tw 1 mjwang@ie.nthu.edu.tw 2 Abstract. This paper describes a method of shape deformation using front and side images of a human body. Through image processing, the edges of human body contour are detected, and many features are extracted. For the construction of digital human models, the human body shape can be varied based on the measurements taken from the 2D images. By using two-dimensional, image-based anthropometric measurements, standard 3D human model of man and woman to approximate both male and female body shapes can be obtained. The system then moves some points defined around the human model to modify the body shape. Realistic appearance is achieved by color texture mapping from two specific views. Four subjects have been tested by constructing models of two males and two females from images. For the approximation shape-variation, the resulting models present good matches to the given images. The results reveal that the visualization of human body shapes can be greatly improved and the applications of the virtual human modeling can be well-constructed. Keywords: Front and side images, feature extraction, virtual human model, shape variation. 1. INTRODUCTION 1.1 Feature Extraction from Images Modeling of human body shape, size, and appearance have attracted great interests in recent years (Magnenat- Thalmann et al. 2004). Many researches are using the image-based approach to get the human body contours from photographs. In order to extract the features on the human contours, Seo et al. (2006) adopted color key for silhouette detection. After the contours of the human shapes are extracted from photographs, the features are extracted by human morphology rules (Wang et al. 2003). Then, the shape curves extracted from the body contours provide rich set of human body features. Accordingly, feature points can be extracted on the edges of the human body contours. And, the distances between feature points on the human contour were measured for characterizing human body shape (Meunier and Yin 2000). For the generation of virtual human models from photographs, Hilton et al. (2000) introduced a technique to generate a 3D model of an individual by cameras. The features extracted from silhouettes with photographs of people wearing tight clothes are the input of the deformation. The shape data from 2D images of a person are obtained for reconstructing a 3D human body (Stylios et al. 2001). Lee et al. (2000) also presented a method for full body-cloning utilizing photos taken from multi-view images. By using information extracted from silhouettes, the template human model was deformed with reference to the correspondence between the silhouettes of the template human model and the silhouettes of the human from photographs (Wang et al. 2003). A new human model can be constructed based on the template model with the image-based reconstruction. 1.2 Shape Variation For body shape variation, Mochimaru and Kouchi (1998) used the Free-form deformation (FFD) technique to analyze human body shape. The FFD technique is a 3D morphing technique in computer graphics, that a 3D shape is deformed freely by moving control lattice points defined around the shape. In the FFD, the movement of control points is calculated and the shape variation is well represented by transformation grids. Here, the featuredeformation based on the FFD is used. The modeling of : Corresponding Author 748

silhouettes of human body from the front and side images. With feature extraction, the body characteristics can be extracted. Subsequently, the measurement data can be used to determine the spatial relation between the 2D contours and 3D shapes. The shape of the 3D model can be constructed with control points defined around the body surface. According to the modification in two specific views, this study applied FFD method to vary the shape of human body. With the procedure, the human body can be smoothly deformed and well represented. Finally, the resulting human model mapped with color information can show realistic appearance and provide applications in computer world. 2. METHOD Figure 1: The flowchart of the proposed method. digital human uses the FFD to deform the shape of human body (Wang et al. 2002). With the FFD technique, the continuity and the deformation properties of the human model are kept. Besides applying FFD for shape modification, the original shape of the human model is modified to create new deformation (Lee 2000). The 3D human model has a compact representation after the view-based deformation. And the visualization of the result shows a rapid and lowcost approach of the 3D digital human construction. 1.3 3D Human Model A model built from the standard human body is used to generate an individualized virtual human. Based on the shape reconstruction from silhouettes, the feature points and body measurements are used to approximate the person s shape and anatomical structure (Wang et al. 2003). Lee et al. (2000) used the extracted features from the front and side images and applied FFD for shape modification. The new 3D human model is deformed for approximating realistic appearance by a template human model. Based on the template mesh, different body shapes can be vastly created (Allen 2005). In addition, a common template model is fitted and used to analyze the variation in body shape (Zhang 2005). Then, a realistic human body shape can be generated and represented in a virtual world. This study presents a systematic method to detect feature points and obtain body measurements on the Figure 1 illustrates the proposed method of human model shape deformation. The method begins with silhouette detection from front and side images. Given an input photograph, the Canny edge detector (Canny 1986) is applied for edge detection, and the silhouette of a body s contour is collected. With feature extraction, several features including anatomical points and body dimensions can be obtained automatically from the edge of human body contour. In order to approximate photo-realistic shape, the correlation of human body shape variation was found. The human body shape can be well represented according to the features taken from 2D images. Subsequently, the FFD technique is applied to the 3D human model to perform the shape deformation. To acquire more detailed match between the 2D image and the template model, the approximation variation is calculated. After accomplishing color mapping from image texture, a visualization of virtual 3D cloning can be successfully achieved as shown in Figure 2. 2.1 Silhouette Detection After acquiring the input images, a silhouette detection method is used to extract silhouettes from photographs. Through image processing, each image pixel was converted to the color space to differentiate the difference between background and foreground. Then, Canny edge detector was applied to trace the contour of the binary image. With the morphological operations on the binary image, the output image can provide visual feedback about the silhouette curve of a body shape. The pixels on the edge of a boundary are linked into a closed silhouette contour. Subsequently, the silhouette curve can be represented by 8- connected chain codes. Thus, the location and orientation of the silhouette s coordinate system can be specified in the front and side images. 749

Figure 2: The framework of the image-based human modeling.. 750

2.2 Feature Extraction This study applied the conventional eight-directional code (Freeman 1961) to represent human body shape from input images. In a series of chain codes on human silhouette contour, most of the codes are similar to the neighboring codes. When there is a change between two connected chain codes, a coding algorithm can be used to find the feature-threshold points on a closed silhouette contour. The proposed method encodes the adjacent codes and extracts the feature characteristics in the coding sequence. From the chain codes of a body shape, the body feature characteristics can be extracted. The identification of these particular landmarks can be made by following the procedure of feature points encoding. Finally, the feature landmarking system is capable of identifying landmarks for a set of anthropometric variables. The feature points extracted from the front and side images can be used to obtain the feature dimensions. By connecting a line through certain feature points extracted from the silhouette to generate several feature lines, the body dimensions can be obtained. Two-dimensional, image-based anthropometric measurements are conducted by referring the feature points. Thus, the dimensions between two appropriate landmarks can be determined by computing the linear distance between the corresponding points. With the body features detected in the coded images, the feature points and the measurements can be efficiently extracted from the body silhouettes. 2.3 2D Image to 3D Model 2D Images provide an easy representation of human body figure. In recent years, many features on the silhouette of a human contour are used to describe the characteristics of human body shape. By referring to some related studies, it appears that the detected body features can be used to deform the 3D human model for representing photo-identical shape. In order to make a consistent representation between the 2D images and the 3D template model, establishing the relationship between the 2D contour image and the projected image of the 3D template model is required. The 2D projected images of the 3D template model are obtained by performing a perspective projection. Then, the contours of each projected image are extracted. By projecting the 3D model to 2D image, certain landmarks on the silhouette contour are mapped as features. The visualization presents a good basis in finding a consistent relationship between the 2D and 3D images. Each feature point extracted from the silhouette of a human body has a corresponding point extracted from the silhouette of the template human model. A consistent representation is built for each silhouette by contour tracking. After establishing consistent representation on the 2D contour, the correspondence relationship is constructed for the 3D template model and the 2D contour images. The correspondence relationship can be used to offer the template human model to approximate realistic appearance from the front and side images. Based on the features extracted from photographs, the template human model can then modify its body shape, thereby constructing a 3D virtual model of the input images. 2.4 View-dependent Shape Deformation In the shape deformation approach, the 3D template human model modifies its shape based on the features extracted from the front-view and side-view images. The correlation between the feature points and the shape curves can be determined with FFD technique. Using the template model from the model to be deformed, and construct the lattice points for each axis to cover the whole body. The deformation mesh is located near the surface of the template model with arbitrary topology as illustrated in Figure 3. Then, the system moves the vertical points defined on the template model to modify the body shape. The movement depends on the calculation of the control points and is perpendicular to the view direction. Accordingly, the body shape is deformed along three directions: vertical, horizontal, and diagonal. Then, the deformation represents a shape change in the orthogonal view. The desired shape is obtained by moving certain points discovered around the mesh surface. Obviously, the deformed positions of all the points on the template human model have changed to be close to the silhouettes extracted (a) (b) (c) (d) Figure 3: Shape deformation (a) The 3D human model (b) FFD technique (c) The deformed 3D model (d) The desired 3D model. 751

Table 1: Silhouette Variation. Ea (%) Front Ea (%) Side Ea (%) Prior photo of a male subject is calculated and compared in Table 1. The silhouette variation, E a, which refers to acquire detailed match of the template model to the image (Seo et al. 2006) is equivalent to the notion of non-overlapping area by Sand et al. (2003). The silhouette variation is measured by comparing the non-overlapping area between the projected model and the observed silhouettes. 52.4 % 41.7 % E a = ( T ( i, D( i, ) T ( i, + ( T ( i, D( i, ) D( i, T ( i, and T ( i, are the values of the pixel at location ( i, which is inside and outside of the template model. D ( i, and ( i which is foreground and background. (1) D, are the values of the pixel located at ( i, to deformation After deformation from the images. The spatial relation becomes identical to the human silhouettes from two specific views. So, the body shape of the constructed model and that of the imagebased human can make a good representation. It is important to obtain an ideal 3D body shape and appearance from 2D photos. Finally, both the male and female 3D template models can be deformed according to the movements of the control points defined around the mesh surface. Figure 3 shows an example of the deformation of the male 3D template human model. 2.5 Approximation Variation 10.5 % 7.2 % The body features extracted from the silhouette of the human images have some correlations with that of the template human model. According to the features extracted from the silhouettes of two orthogonal views, the template human model modifies its body to approximate the real body shape. The results show the constructed model for representing shape deformation to visualize the body shape extracted from images. The approximation variation is used to compare the shape feature difference between the deformed model and the original human. In order to assess the match between the 2D image and the 3D human model for shape deformation, a silhouette variation comparison method mentioned by Seo et al. (2006) and Gu et al. (1999) are adopted. The number of pixels between the projected template model and the front 2.6 3D Model Application The technique of the 3D human model construction described in this paper uses the information extracted from the image-silhouettes to modify the shape of the 3D template model with the features defined. The system can present the constructed human model with the color texture. The color information is obtained from the front and side images. Identical appearance can be finally realized by color texture mapping. In addition, an interactive interface design which enables the users to adjust the measurement values to human body with various changes in shapes is developed, as shown in Figure 4. The interface also allows the users to access the results of the 3D human model shape deformation. The resulting human model can further perform animation in a virtual world. 3. RESULTS The proposed methods are coded in 3ds MAXScript format. From only two images, the system can extract the features from photographs and use the data obtained as the parameter of shape deformation. Constructing the human body shapes with images has gained many applications in practice because the resulting models can be used in computer animation and virtual reality. Since the 3D model has three dimensions higher than 2D contour, it is important to determine the corresponding spatial relation between the projected 3D shape and the given images. In order to verify the proposed method, two male subjects and two female subjects were made an approximation-variation comparison. Table 1 shows the silhouette comparison between the reconstructed model and the input images. 752

After deformation, the reconstructed model is mapped to the input images and the silhouette variation of the human model to the image is decreased. The resullts present a much better match to the data mentioned by Seo et al. (2006). In addition, the resulting human model is mapped with color texture and animated in a virtual world. Thus, the reconstruction of 3D human model from 2D images is efficient and effective. 4. CONCLUSION The paper presents a method of human body shape deformation from front and side images. Based on the features extracted from the silhouettes of the human body, the feature points and body dimensions that can describe 3D shape are identified. For the shape deformation, this study applies FFD technique to modify the human shape by moving the control points defined around the body parts. With the movement in two orthogonal directions, the human body shape can be smoothly deformed and well represented. Following this procedure, the virtual human model was fast-constructed using two specific images. In order to verify the effectiveness of the method, four subjects including two males and two females aged from 18-25 were tested. Comparing the resulting 3D shapes with the original 2D contours, a much better match can be obtained with the view-dependent shape deformation. The reconstruction of 3D model from photographs is inexpensive and practical in photo-realistic body cloning. Further, the system also provides an interface which allows users to adjust the human body shape. Finally, by color texture mapping, the realistic appearance of 3D model can be achieved and the animation of 3D motion can be simulated in a virtual world. REFERENCES Allen, B. (2005) Learning Body Shape Models from Real-World Data. PhD thesis, Department of Computer Science and Engineering, Washington University. Canny, J. (1986) A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, 679-698. Freeman, H. (1961) On the encoding of arbitrary geometric configuration. IRE Transactions on Electronics Computers, vol. 10, 264-268. Gu, X., Gortler, S. J., Hoppe, H., Mcmillan, L., Brown, B. J., and Stone, A. D. (1999) Silhouette mapping. Tech. Rep. TR-1-99, Harvard. Hilton, A., Beresford, D., Gentils, T., Smith, R., Sun, W., and Illingworth, J. (2000) Whole-body modelling of people from multiview images to populate virtual worlds. The Visual Computer, vol. 16, no. 7, 411-436. Lee, W., Gu, J., and Magnenat-Thalmann, N. (2000) Generating Animatable 3D Virtual Humans from Photographs. Eurographics, Computer Graphics Forum, vol. 19, no. 3, Blackwell publisher. Lee, W. (2000) Feature-Based Approach on Animatable Virtual Human Cloning. PhD thesis, in MIRALab, University of Geneva, Switzerland. Magnenat-Thalmann, N., Seo, H., and Cordier, F. (2004) Automatic Modeling of Virtual Humans and Body Clothing. Journal of Computer Science and Technology, vol. 19, no. 5, 575-584. Meunier, P., and Yin, S. (2000) Performance of a 2D image-based anthropometric measurement and clothing sizing system. Applied Ergonomics, vol. 31, issue 5, 445-451. Mochimaru, M., and Kouchi, M. (1998) A new method for classification and averaging of 3D human body shape based on the FFD technique. International Archives of Photogrammetry and Remote Sensing, vol. XXXII, 888-893. Seo, H., Yeo, Y., and Wohn, K. (2006) 3D Body Reconstruction from Photos Based on Range Scan. Lecture Notes in Computer Science, vol. 3942, 849-860. (a) (b) (c) (d) Figure 4: Application-the user interface for shape deformation (a) Images Input (b) Silhouette detection (c) Feature extraction (d) Body dimensions. 753

Sand, P., McMillan, L., and Popovic, J. (2003) Continuous Capture of Skin Deformation. In Processing of ACM SIGGRAPH, 578-586. Stylios, G.K., Han, F., and Wan, T. R. (2001) A Remote On-line 3D Human Measurement and Reconstruction Approach for Virtual Wearer Trials in Global Retailing. International Journal of Clothing Science and Technology, vol. 13, no.1, 65-75. Wang, C. C.L., Wang, Y., Chang, T. K.K., and Yuen, M. M.F. (2003) Virtual human modeling from photographs for garment industry. Computer-Aided Design, vol. 35, no. 6, 577-589. Wang, C. C.L., Wang, Y., and Yuen, M. M.F. (2002) Feature based 3D garment design through 2D sketches. Computer-Aided Design, vol. 35, 659-672. Zhang, X. (2005) Data-driven human body morphing. Master's thesis, Department of Visualization Sciences, Texas A&M University. AUTHOR BIOGRAPHIES Mao-Jiun J. Wang is a Chair Professor of Industrial Engineering and Engineering Management at National Tsing Hua University, Hsinchu, Taiwan. He received his PhD in Industrial Engineering from State University of New York at Buffalo in 1986. He is a Fellow of IIE and IEA. His research interests include digital human modeling, applied ergonomics, computer vision and applications, and fuzzy set theory. His email address is <mjwang@ie.nthu.edu.tw> Yueh-Ling Lin is a PhD candidate of Industrial Engineering and Industrial Management at National Tsing Hua University, Hsinchu, Taiwan. She received her bachelor's degree in Industrial Education from National Taiwan Normal University in 2005. Her research interests include feature extraction from photographs and digital human modeling. Her email address is < d943845@oz.nthu.edu.tw > 754