Deliverable 5.2 Creation of Population

Transcription

1 ABSTRACT Deliverable 5.2 Creation of Population Population of virtual humans is an essential keystone in any performance that involve a number of virtual humans cohabiting within the same virtual environment and the public. This deliverable describes a method for generating population. We first discuss cloning faces from two orthogonal pictures and then for generating populations from a small number of these clones. The problem of generating various body shapes is also discussed. An efficient method for reconstructing 3D heads suitable for animation from pictures has shape modification part and texture mapping part. The several individualised head serve either to statistically infer the parameters of the multivariate probability distribution characterising a hypothetical population of heads or to a dynamic system for 3D morphing with 3D spatial interpolation and powerful 2D textureimage metamorphosis. To realise a full body shape, we construct a database of several body sizes for men and women. Depending on a given face, we select a suitable body and connect individualised face to it and animate a full face and body in a virtual world. Various examples of different bodies with different heads are provided. Document ID Type erena D5.2 Deliverable report with files and video material Status Final Version 1.0 Date Aug Author(s) Pierre Beylot, WonSook Lee, Jean-Claude Moussaly, HyeWon Seo, Christian Zanardi, MIRALab, University of Geneva. Director of Professor Nadia Magnenat-Thalmann, MIRALab, the research University of Geneva Task 5.2

2 Table of Contents 1. Introduction and problematic of representing real looking humans Face reconstruction Generation of face population Body reconstruction Outline Capturing data out of photographs: head reconstruction6 using a generic model & features points Feature detection Modification of a generic model Automatic texture mapping Texture generation Image deformation Multiresolution image mosaic Texture fitting Cloning Results Real-time animation Statistical methods to generate a large population Constructing and sampling from an hypothetical population Inference Sampling D morphing for creating various individuals Texture morphing Image metamorphosis based on triangulation Results of large population Generate various body shapes out of a body database Creation of a body How to simply create diverse shapes Connecting the head to the body Changing texture Examples of complete virtual humans Conclusion Link with erena Project References ESPRIT Project 25379

3 1. Introduction and problematic of representing real looking humans Population of virtual humans is an essential keystone in any performance that involve a number of virtual humans cohabiting within the same virtual environment and the public. This deliverable aims at presenting the method we have developed within the erena project in order to provide huge crowds of realistic looking virtual humans that may be used within performance. Before constructing virtual groups, crowds or populations, we must be able to synthesise realistic figures and move them about in a convincing and efficient way. Animators agree that the most difficult subjects to model and animate realistically are humans and particularly human faces. The explanation resides in the universally shared (with some cultural differences) processes and criteria not only for recognising people in general, but also for identifying individuals, expressions of emotion and other facial communicative signals, based on the co-variation of a large number of partially correlated shape parameters within narrowly constrained ranges. Though some of this may be amusingly conveyed in a schematic manner by various 2D or 3D animation technologies, the results are easily identified as cartoons by the most naive observer, while the complex details and subtle nuances of truly realistic representation remain a daunting challenge for the field. Once we have a way of modelling specific individuals, how can we make use of a limited number of reconstructed faces in the automatic generation of a large population of distinct individuals of the same general characteristics? Simple approaches, such as the random mix and match of features, do not take into account local and global structural correlation among facial sizes and structures, distances between features, their dimensions, shapes, and dispositions, and skin complexion and textures. In this report, we describe an efficient and robust method for individualised face modelling, followed by techniques for generating animation-ready populations in a structurally principled way, based on a number of existing head models and a database of existing body models. We also introduce a rapid dynamic system, which enables 3D polymorphing with smooth texture variation under intuitive control. The problematic for the creation of realistic human bodies will be also described. 1.1 Face reconstruction Approaches to the realistic reconstruction of individuals, some of them with a view to animation, include the use of a laser scanner [7], a stereoscopic camera [4], or an active light stripper [8]. There is increasing interest in utilising a video stream [13] to reconstruct heads and natural expressions. These methods are not developed enough; however, to be commercialised (such as in a camera-like device) in the near future, in terms of direct input of data for reconstruction and final animation as output. The animation results by Guenter et al.[2] are impressive, but input and output processes are not practical for real-time animation. The term real-time animation refers here to the ability of having one or more virtual human that is deformed and according to facial or body parameters and displayed on the computer screen at more than 10 frames per second in its virtual environment. Real-time is important for animation but it is not necessary that the face reconstruction is also in real-time. This process can be done completely offline and then loaded later in the virtual environment. Pighin et al.[3] get naturalistic results for animation using several views as input. Their techniques have elements in common with our methods, but the aims are different. For example, we are primarily interested here in morphing between a number of different individuals, while their application is to interpolate between different stages of a single facial model. Modelling has also been done from picture data [6][9], detecting features, modifying a given generic model and then mapping texture on it. Not all of these, however, combine sophisticated and reliable shape deformation methods with seamless, high-resolution texture generation and mapping ESPRIT Project 25379

4 1.2 Generation of face population Techniques for generating virtual populations have been investigated by DeCarlo et al.[1]. They vary a geometric human face model using random distances between facial points sampled from distributions of anthropometric statistics, but they do not consider a real-life face data as an input and realistic texture variation. The term morph, short for metamorphosis, has been applied to various computer graphics methods for smoothly transforming geometric models or images. Morphing techniques can be classified into image based methods and object-space methods. In most cases, the imagebased methods are used for 2D morphing and the object-space methods for 3D morphing. Image morphing has three terms to be considered, which are feature specification, warp generation methods, and transition control[20]. These areas relate to the ease of use and quality of results. Feature specification is the most tedious aspect of morphing. Although the choice of allowable primitives may vary, all morphing approaches require careful attention to the precise placement of primitives. Given feature correspondence constraints between both images, a warp function over the whole image plane must be derived. This process, which we refer to as Warp Generation, is essentially an interpolation problem. Another important problem in image morphing is transition control. If transition rates are allowed to vary locally across in-between images, more interesting animations are possible. The main methods are based on mesh warping [21] field morphing [22], radial based functions [23], thin plate splines [24], energy minimisation [25], and multilevel free-form deformations [12]. A tradeoff exists between the complexity of feature specification and warp generation. As feature specification becomes more convenient, warp generation becomes more formidable. Some methods do not guarantee the one-to-one property of the generated warp functions while others derive one-to-one warp functions, but the performance is hampered by its high computational cost. Most of the morphing methods and applications require manual user intervention to specify matches between given objects. In many cases, significant user involvement is required to achieve a pleasing visual result. It is desirable to eliminate, or at least to limit, this involvement in morphing design, and to have a more automatic process. A work minimisation approach by Gao and Sederberg [26] to image morphing is capable of automatically creating good image morphs in many case where the two images are sufficiently similar, where good results can be obtained in less than 10s for 256x256 images. The traditional formulation for image morphing considers only two input images at a time, i.e. the source and target images. Morphing among multiple images, referred as polymorph, is understood to mean a seamless blend of several images at once. A metamorphosis or a 3D morphing, for example topology independent morphing in a short animation film Galaxy Sweetheart [28] in late 80, is the process of continuously transforming one object into another[10][11]. Since there is no intrinsic solution to the morphing problem, user interaction can be a key component of morphing software if two objects do not share the same structure. Most efforts have been dedicated to the correspondence problem, and we still lack intuitive solutions to control the shape interpolation. The idea of combining 2D-image morphing and 3D-shape morphing has been little exploited. Nevertheless, this seems a natural approach to better results with less effort. Furthermore, were this to be combined with face cloning methods, many user interactions could be avoided. 1.3 Body reconstruction When it comes to the modelling and deformation of human body shapes, this is an important but difficult problem. The human form is very complex; it comes in a variety of sizes and has two main types (or gender); it includes a skeletal frame that supports muscle, fat and fleshes; all enclosed by a skin that can slide, stretch, and fold over this volume. The torso, head, arms, and legs are relatively large, roughly symmetrical collections of articulated components which vary with age, gender and race factors. Attempting to model and animate such a structure is one of the most difficult and challenging problems in computer graphics. Moreover, since our eyes are especially sensitive to the human Fig., computer generated images must be extremely convincing to satisfy our demands for realism ESPRIT Project 25379

5 Several approaches[31][32][33] have been proposed but they are not convenient for our problem of producing realistic multiple humans. We are currently using the approach based on implicit surfaces. An implicit surface is a surface consisting of those points p that satisfy the arbitrary implicit function f, f(p) = 0. Because implicit defined surfaces possess some unique and useful attributes for modelling, such as blending and constraint properties, they have received increasingly attention in the design of 3D objects. A particular subset of implicit surfaces, called soft objects (or metaballs, blobs) is becoming hot topic in computer graphics. Because metaballs joint smoothly and gradually, it gives shape to realistic, organiclooking creations, suitable for modelling human bodies, animals and other organic figures, which are very hard to model using traditional geometric methods. 1.4 Outline In this report, we describe our methods to create individualised faces just from a few photos and generate a population based on existing virtual humans. Then body modelling and how to build body database are explained. Fig. 1 contains a flow chart for the generation of photorealistic virtual population from photos and database of body. Orthogonal Picture of faces Feature Points Texture Generation A set of Feature Points Texture fitting Principal Component Space Sampling Texture Morphing Body construction sets of Feature Points Variation of body Shape Modification Generic Model Texture Mapping Data base for various body shapes and textures Dynamic system for 3D morphing for faces Individualized virtual human Fig. 1: Generation of populations from picture input: from the two orthogonal photos of an individual feature points are semi-automatically extracted. The feature points are used both for the texturing process of the virtual face and with the geometrical deformation of the generic face model. The 3D-morphing system is then used on the database of all available virtual faces to automatically create completely new faces by dynamic morphing. The body is created using a generic database of bodies and simple available transformation. Association of one head to one body creates the final virtual human ESPRIT Project 25379

6 In Section 0 of this report, we describe a fast and robust method for head-shape modelling based on the extraction of feature points from two orthogonal views. Section 0 shows fully automatic and robust texture mapping method for realistic looking virtual human. Based on a representation in terms of a large number of vectors between feature points, principal component analysis is used to discover the statistical structure of an input sample of heads, namely a representation of the data in terms of a reduced number of significant (and independent) dimensions of variability in Section 0. Section 0 is devoted to a dynamic system for generating heads, including shape and texture details that interpolate several input heads. This enables the user to visually control mixing ratios with close to real-time calculation. Some experimental results are presented in Section 0. In the last section of this report we will present different examples of virtual human created out of cloned heads and from a database of virtual human bodies. 2. Capturing data out of photographs: head reconstruction using a generic model & features points 2.1 Feature detection 2D photos offer cues to the 3D shape of an object. It is not feasible; however, to get 3D coordinates for points densely distributed on the head. In most cases, we know the location of only a few visible features such as eyes, lips and silhouettes, key characteristics for recognising people. We call them as feature points whose location can be detected either automatically or at least interactively. Our process is semi-automatic. It means that most of the tedious processing is done automatically and that only minor manual adjustment are necessary to complete the models To reconstruct a photographically realistic head, ready for animation, we detect corresponding feature points on both of two orthogonal images front and side and deduce their 3D positions. This information is used to modify a generic model through a geometrical deformation. Feature detection is processed in a semiautomatic way using the structured snake method [9] with some anchor functionality. Fig. 2 depicts an orthogonal pair of normalised images, showing the features detected. Here normalisation signifies locating images in the feature point space, so that the front and side views of a head have the same height. The feature points are overlaid on the images even though they are located in spaces with different origins and scales from the images. The two 2D sets of position co-ordinates, from front and side views, i.e., the (x, y) and the (z, y) planes, are combined to give a single set of 3D points. To get perfectly aligned and orthogonal views is almost impossible, and this leads to difficulties in determining the (x, y, z) co-ordinates of a point from the (x, y) on the front image and the (y, z) on the side image. Taking the average of the two points often results in an unnatural face shape. Thus we rely mainly on the front y co-ordinate, using the side y only when we do not have front one. This convention is very effective when applied to almost orthogonal pairs of images. In addition, for asymmetrical faces, this convention allows for retention of the asymmetry with regard to the most salient features, even though a single side image is used in reconstructing both the right and left aspects of the face. A global transformation moves the 3D feature points to the space containing the generic head, which is a normalisation of every head. 2.2 Modification of a generic model We have a certain set of 3D feature points, which has about 160 points. The problem is how to deform a generic model, which has more than a thousand points to make an individualised smooth surface. One solution is to use 3D feature points as a set of control points for a deformation. Then the deformation of a surface can be seen as an interpolation of the displacements of the control points. Free-Form Deformations (FFD)[14] belong to a wider class of geometric deformation tools. However FFD has a serious constraint in that control points boxes have to be rectangular, which limits the expression of any point of the surface to deform relative to the control points box. Farin [17] extends the natural neighbours interpolant based on the natural neighbours ESPRIT Project 25379

7 co-ordinates, the Sibson co-ordinate system[15] based on Voronoi/Dirichlet and Delaunay diagrams [16] for a scattered data interpolant, using the support for a multivariate Bezier simplex [18]. He defines a new type of surfaces with this extended interpolant called Dirichlet surfaces. Combining FFD s and Dirichlet surfaces leads to a generalized model of FFD s: Dirichlet FFD s or DFFD s. In the Dirichlet-based FFD approach, any point of the surface to deform located in the convex hull of a set of control points in general position, is expressed relative to a subset of the control points set with the Sibson co-ordinate system. One major advantage of this technique is that it removes any constraint on the position and topology of control points. It also removes the need to specify the control lattice topology [19]. One control point is defined at the position of each surface point, so that any displacement applied to the control point will also be applied to the surface point. Therefore DFFD is used to get new geometrical co-ordinates for a modification of the generic head on which are situated the newly detected feature points. All points on this head are located utilising the feature points as constraint control points for DFFD. Since our feature points includes the front and side silhouette, the convex hull of control points contains most points on a 3D head, but there can be some missing points outside the convex hull. So we add 27 extra white points surrounding the 3D head in Fig. 2. The correct shapes of the eyes and teeth are assured through translation and scaling appropriate to new head. As shown in Fig. 2, this is a rough matching method; it does not attempt to locate all points on the head exactly, in contrast to range data from laser scanners or stereoscopic cameras, which create a enormous number of points. Considering the input data (pictures from only two views), the result is quite respectable. Most important, it greatly limits the size of the data set associated with an individual head, as is necessary to accelerate animation speed. The problem of how to reduce the size of the data from range data equipment for animation purposes has not been satisfactorily solved. To improve realism, we make use of automatic texture mapping, as described in the next section. Feature points in 2D Features in 3D A generic model An individualized head DFFD Fig. 2: Modification of a generic head according to feature points detected on pictures.points on a 3D head are control points for DFFD. 3. Automatic texture mapping Texture mapping serves not only to disguise the roughness of shape matching determined by feature point matching only, but also to imbue the face with more realistic complexion and tint. We use information from the set of feature points detected to generate texture automatically, based on the two views. We then obtain appropriate texture co-ordinates for every point on the head using the same image transformation. 3.1 Texture generation The main criterion is to obtain the highest resolution possible for most detailed portions. We first connect two pictures along predefined feature lines using geometrical deformations and a multiresolution technique to avoid visible boundary effects ESPRIT Project 25379

8 3.1.1 Image deformation We use the front view preponderantly, since it provides the highest resolution for the features. The side view is deformed to be connected to front view along certain defined feature points lines on the left hand side and, flipped over, on the right hand side. The feature lines are drawn on the front image as shown Fig. 3 by the red lines. A corresponding feature line is defined for the side images. We deform the side image to transform the feature line to coincide with the one on the front view. Image pixels on the right side of the feature line are transformed by the same transform as the line transform. To get the right part of the image, we utilise the side image as it is and deform it according to the right-hand red feature line on the front image. For a left part of the image, we flip the side image and deform it according to the left-hand red feature line on the front image. The resulting three images are illustrated in Fig. 4 (a). Here we use a piecewise linear transformation using piecewise feature lines, but higher degree feature curves can perform a smoother deformation for side images Multiresolution image mosaic The three images resulting from the deformation are merged using a pyramid decomposition [5] based on the Gaussian operator. We utilise common REDUCE (also called shrinking) and EXPAND operators to obtain G k (the Gaussian image) and L k (the Laplacian image) and connect three L k images at each level on any given curve; here they are the feature lines described above. Then the connected image P k is augmented to obtain S k, which is the combination of P k and S k+1. The final image is S 0. This multiresolution technique is effective in removing the boundaries between the three images. No matter how carefully the picturetaking environment is controlled, in practice boundaries are always visible. As in Fig. 4 (a), skin colours are not continuous when the multiresolution technique is not applied. The image in (b) shows the results with the technique, which has smoothed the connection between the images without visible boundaries. Front Side ( right, left ) Deformed side ( right, left ) (a) (b) (c) Fig. 3: (a) Red lines are feature lines. (b) Feature lines on three images. (c) Feature lines on a 3D head. (a) Fig. 4: Combining the texture images generated from the three (front, right and left) images without multiresolution techniques, in (a) and with the technique in (b). Eye and teeth images are superimposed automatically on the image. These are important for animation of eye and mouth regions. 3.2 Texture fitting To find suitable co-ordinates on a combined image for every point on a head, we first project an individualised 3D head onto three planes as shown in Fig. 5 (a). Guided by the feature lines used for image merging in above section, we decide to which plane a point on a (b) ESPRIT Project 25379

9 3D head is to be projected. The points projected on one of three planes are then transferred to either the front or the side 2D-feature point space. Finally, one more transform on the image space is processed to obtain the texture co-ordinates and the final mapping of points on a texture image is generated. The origins of each space are also shown in Fig. 5 (a). The 3D head space is the space for the 3D head model, the 2D feature point space is the one for feature points used for feature detection, the 2D image space is the one for the orthogonal images used for input, and the 2D texture image space is for the generated image space. The 2D-feature point space is different from the 2D-image space even though they are displayed together in Fig. 5 (a). The final texture fitting on a texture image is shown in Fig. 5 (b). The final texture mapping results in smoothly connected images inside triangles of texture co-ordinate points, which are accurately positioned. The eye and teeth fitting process are done with predefined co-ordinates and transformations related to the texture image size. After doing this once for a generic model, it is fully automated. The brighter points in Fig. 5 (b) correspond to feature points and the triangles are a projection of triangular faces on a 3D head. Since our generic model is endowed with a triangular mesh, the texture mapping benefits from an efficient triangulation of texture image containing finer triangles over the highly curved and/or highly articulated regions of the face and larger triangles elsewhere, as in the generic model. This triangulation is also used for the 3D-image morphing in Section 0. Top view of a 3D head front front Texture image right left (a) front 3D Head space side 2D Feature Points space side 2D Image space (input pictures) 2D texture Image space (generated image) Feature lines Fig. 5: (a) Texture fitting giving a texture co-ordinate on an image for each point on a head. (b) Texture co-ordinates overlaid on a texture image. (B) 3.3 Cloning Results Fig. 6 shows several views of the head reconstructed from the two pictures in Fig. 2. Fig. 6: Snapshots of a reconstructed head in several views. Other examples covering wide range of age and many kinds of ethnic group are shown in Fig. 7. Every face in this report is modified from the SAME generic model shown in Fig. 5. How individualised the representations are depends on how many feature points are identified on the input pictures. Our generic model currently does not have many points on the hairdo, so it is not easy to vary hair length, for example ESPRIT Project 25379

10 Fig. 7: Examples of reconstructed heads from pictures. Two views of each virtual human are shown just after two photos of the corresponding real person. These are ready for immediate animation in a virtual world. As an example of the cloning capabilities we have asked the member of the erena consortium to send us photos in order to clone them and used their face in the future for the creation of the population. Results from cloning are presented in Fig ESPRIT Project 25379

11 Fig. 8 Examples of cloned head of erena participants 3.4 Real-time animation Real-time animation means that the virtual human can be animated according to facial or body parameters and that it will be displayed on the computer screen at more than 10 frames per second in its virtual environment. Real-time is important for animation but the cloning and population creation can be done completely offline and then loaded later in the virtual environment. The predefined regions [27] of the generic model are associated with animation information, which can be directly transported to the heads constructed from it by geometrical modification in Section 0. Fig. 9 shows several expressions on a head reconstructed from three pictures. Fig. 9: Examples of a reconstructed head from three photos and several expressions. 4. Statistical methods to generate a large population The statistical methods provide efficient tools for automatically generate a large population. The ideas and objectives of this method are similar to the method that will be presented in the next chapter but they rely heavily however on specific biostatistical notions. This notion are related specifically to face features and when extracted provide an efficient way of basically representing a type of person. The dynamic morphing system is not considering those ESPRIT Project 25379

12 aspects or more precisely is not based upon those basic principles but may have the similar results, as it will be explained in the next chapter. Those two methods are therefore complementary rather that opposed. 4.1 Constructing and sampling from an hypothetical population Our approach to generating populations is based on biostatistical notions of morphological variation within a community. The underlying hypothesis is that if we determine a large number of facial measurements, these will be approximately distributed in the population according to a multivariate normal distribution, where most of the variation can be located in a few orthogonal dimensions. These dimensions can be inferred by principal component analysis [37] applied to measurements of relatively few input heads. A random sample from the reduced distribution over the space spanned by the principal component yields the facial measurements of a new head, typical of the population. An input head is described by its set of feature points obtained interactively from photographs as explained in Section 0. The principal component analysis will generate new sets of feature points having original positions and therefore describing the geometry of new heads. These new samples will be used to modify the shape of a generic model with the same way as in Section 0. Fig. 1 shows how principal component analysis is used inside the population generation process. For a realistic morphological variation, it is better to use a large number of heads. We currently use 30 heads representatives of an average population (15 males / 15 females, 15 Asian / 15 Caucasian) but we are continuously increasing this number using the face of cloned visiting persons to our lab or during events such as the CEBIT fair in Hanover (1999) Inference The feature points are divided into subset according to the head region where they are defined (mouth, eye, nose, cheek, and etc.). Two sets of pre-specified 3D vectors representing distances between feature points are calculated for each head. The first set reflects the overall shape parameters and consists of distances between a central point situated on the nose and local points, each of them belonging to a different region. The second set represents local relationships and corresponds to distances defined between a local point and the points being in the same region as shown in Fig. 10. Denote by n the total number of measurements represented by the central point co-ordinates and distance vectors co-ordinates and denote by H the number of input heads. The measurements for the H heads are each standardised to Normal [0,1] and entered into an H x n matrix M. The principal components of variation are found using standard procedures, involving the decomposition XLX t =MM t, where X is orthonormal and L contains the eigenvalues in decreasing order. Only those dimensions of X with non-negligible eigenvalues, i.e. the principal components, are retained Sampling For each head in the population being constructed, independent samples are drawn using a one-dimensional sampling distribution (N[0,1]) on each principal component axis. The i-th component is multiplied by sqrt(l i ), where L i is the i-th eigenvalue of the i-th axis, and the feature point distance vectors are then constructed by inverting the transformation to X from the original measurement co-ordinates in n-dimensional space, and then inverting the standardisation process for each measurement. It is then straightforward to recover all the new feature points from the n measurements, starting with the central point on the nose. Sampling from the principal component space is a rapid method for generating any number of feature point sets. We currently use 160 feature points. The matrix MM t has a dimension of 160 x 3. It is decomposed in less than one minute on a computer with at today s performance ESPRIT Project 25379

13 Fig. 10: 3D vectors representing distances between feature points. Arrows from nose tip to facial region centre (forehead, cheek, and mouth on the Fig.) represent global distances. Arrows from region centre to local feature points represent local distances. 5. 3D morphing for creating various individuals Morphing technology has seen a great deal of development, both in 2D and 3D. Very high quality morphing between people or animals is available in 2D, while in 3D, it has been applied mainly to pairs of objects with differing topology. In this report, we vary aspects of both 2D and 3D representations in creating new virtual faces. When we morph one person to another in 3D, we must deal with alteration in both shape and texture. Since every face created through the methods described here shares the same topology, it is relatively easy to vary head shapes in 3D just using a simple linear interpolation of 3D co-ordinates. On the other hand, it is less straightforward to carry out 2D morphing of textures since this requires some feature information specifying correspondences between specific points in the two images. Here we use a surprisingly simple method, drawing on 3D information to facilitate 2D morphing among texture images. 5.1 Texture morphing For the heads newly created through sampling in the principal component space, we do not have source images to generate texture mapping. For this case we use texture polymorph with a given set of cloned heads with textures. Image morphing normally considers feature specification, warp generation, and transition control. When we, however, use texture image with 3D head information, feature specification and warp generations are automatically given by 3D head topology and only a simple calculation based on triangulation with given ratios among input texture images is performed. Two steps are necessary to obtain texture morphing. First, texture co-ordinates are interpolated, and then image morphing is carried out. We use simple linear interpolation among several texture co-ordinate sets ESPRIT Project 25379

14 5.1.1 Image metamorphosis based on triangulation We morph several images with given ratios using the information of texture co-ordinates and the triangulation information of the texture mapping. We first interpolate every 3D vertex on the several heads. Then to generate a new intermediate texture image, we morph triangle by triangle the entire 3D head. The parts of the image used for texture mapping are triangulated by projection of triangular faces of 3D heads since the generic head is a triangular mesh as seen in Fig. 5 (b). With this information for triangles, barycentric coordinate interpolation can be employed for image morphing. First, the three vertexes of each triangle are interpolated. Then pixel values inside triangles can be obtained from interpolation between corresponding pixels in several triangles with the same barycentric co-ordinate. Smoothing of the image pixels is achieved through bilinear interpolation among four neighbouring pixels. Given two texture images, for instance, we find, the barycentric coordinate (u, v, w) for each pixel (x, y) in an intermediate images inside a triangle P 1 P 2 P 3. The corresponding triangles in the two input images are P 1L P 2L P 3L and P 1R P 2R P 3R, respectively. Then up 1L + vp 2L + wp 3L is the corresponding pixel in the first image and up 1R + vp 2R + wp 3R in the second. The colour value M(x, y) of a given pixel (x, y) is found from linear n interpolation, with given ratios r i where r = 1 and n=2, between the colour value M L (x, y) i i= 1 of the first image and M R (x, y) of the second. Fig. 11 illustrates intermediate texture images between two persons. Note that pixels in the eyes, mouth and part of the hair region, which are not covered by triangles, are not treated in the same way to get intermediate values. Indeed, they are not used for texture mapping at all. If morphed heads have different eye colours, our procedure also interpolates these. The resulting head manifests very smooth images without any gaps in the textured parts. Fig. 12 shows the final result of 3D morphing. It depicts the results of interpolating the shapes, the skin and the hair colours between an older Caucasian male and a younger Asian female, whose input pictures are shown in Fig. 7. Fig. 11: Intermediate images mixed for (Caucasian male, Asian female) Fig. 12: Morphing steps for (Caucasian male, Asian female). This approach to 3D morphing has the advantage of extending to several people in a natural way. Fig. 13 illustrates a dynamic system for 3D morphing among a given number of people, here three females, Caucasian, Asian and Indian, respectively, and an African male with a nonneutral expression on his face. The interface has two windows, the left one for controlling input ratio and the upper right one containing the result. Two options are provided, one-time morphing and continuous morphing. One-time morphing provides a convenient way to select an input ratio in an n-polygon in the left window for n given virtual persons. The result appears immediately. For continuous morphing, the result varies according to the ratio points, which are chosen along a line or curve in the polygon. Since only barycentric calculation for ESPRIT Project 25379

15 texture image pixels is required, the calculation is very fast. There is also an option to set on/off for shape and image variation rate separately. Fig. 13: A dynamic system for 3D morphing. Here four persons are used for continuous morphing following a user-specified curve. A key to the efficiency of this system is the restriction that all the heads share the same structure. A benefit is that the user can experiment with the ratios for mixing two or more heads in real-time. 6. Results of large population We report here on experiments with cloning and the creation of new heads out of cloned one. We used seven orthogonal photo sets as inputs. Fig. 7 shows the input photos and output reconstructed virtual faces. This includes four Caucasians of various ages, an Indian, an Asian, and an African. There are three adult females, three adult males and a child. The creation of a population is illustrated in Fig. 14. The 42 faces, some of whom have nonneutral facial expressions, are generated from the H=7 sets of orthogonal photos in Fig. 7 according to the steps schematised in Fig. 1. All faces are ready to be animated. The size of the texture image for each person is 256x256, which is less than 10 KB. The total amount of data for the heads in OpenInventor format is rather small considering their realistic appearance. The size of Inventor format (corresponding to VRML format) is about 200 KB. The texture image is stored in JPEG format and has sized about 5~50 KB depending on the quality of pictures; all examples shown in this report have size less than 45 KB. The number of points on a head is 1257, where 192 of them are for teeth and 282 of them are used for the two eyes. This leaves only 783 points for the individualisation of the face surface aside from eyes and teeth. The small size of data used in the face geometry and texture gives us the possibility of animating several heads together in real time. Fig. 15 shows a snapshot of an OpenGL-based interface allowing the animation of a few heads. Every head has its own location and script file, which defines several continuous expressions (like eye, eyebrow, mouth movement or head rotation) with a given range of time. The expressions repeat with different frequency for each head.. In this interface, up to 5 heads can be animated in real-time. With smaller size of texture images and fewer points on heads, we will certainly increase number of heads for real-time animation ESPRIT Project 25379

16 Fig. 14: Forty-two virtual faces created from seven faces reconstructed from images ESPRIT Project 25379

17 Fig. 15: Animation of several people together. The snapshot with 7 heads is from cloned persons while the one with 9 heads from created persons. 7. Generate various body shapes out of a body database This section describes the methods used to create diverse body shapes. The first section explains the creation of generic body shapes. The second section deals with the deformation that can be applied to the generic bodies to create a multitude of different bodies. Once the body has been created, the head may be attached to obtain the complete shape of the virtual human as explained in Section 0. Finally, with the flexibility that is offered by the texturing method of our virtual human, the realism of the population can be further improved by appropriately changing the texture applied to the virtual human bodies. 7.1 Creation of a body Our goal with the population is to make realistic and efficient human modelling and deformation capabilities for many different bodies without the need of physical prototypes or scanning devices. These capabilities will make possible the automatic creation and animation of a rich variety of human shapes. Fig. 16 describes the four steps necessary to obtain a complete realistic human body: 1) the skeleton, 2) the deformation model, 3) the skin model, and 4) the realistic texturing model. Fig. 17 presents the interface of the BodyBuilder software used for the creation of body. Fig. 16 The 4 steps towards realistic human bodies ESPRIT Project 25379

18 BodyBuilder is a software package developed for the creation of human body envelopes. It uses a multi-layered approach for the design of human bodies [28][29]. The first layer is an underlying articulated skeleton hierarchy. This skeleton is close to a schematic real-human skeleton. All the human postures can be defined using this skeleton. The proportions of the virtual human are designed at this stage. The second layer is composed of grouped volume primitive. The volume primitives are metaballs, which fall into two categories: blendable and unblendable. Because metaballs can joint smoothly and gradually, they give shape to realistic organic-looking creations, suitable for modelling human bodies. They are attached to the proximal joint of the skeleton and can be transformed and deformed interactively. This way, a designer can define the 3D shape of a virtual human. Designers can start from a rough shape, made with just a few metaballs, and then add details by increasing the number of metaballs, while using editing tools (add, delete, transform, adjust the parameters). The human form is very complex, and modelling is a tedious task since the human eye is very sensitive to human Fig. It is even more challenging because the metaballs simulate the shape and behaviour of muscles, so designing requires good anatomical knowledge. The third layer is the equivalent of the human skin. We define the envelope of the body with spline surfaces using a ray-casting method. This way, metaballs have observable effects on the surface shape. We use this approach because human limbs exhibit a cylindrical topology and the underlying skeleton provides a natural centric axis upon which a number of cross-sections can be defined. The metaball technique is inherent to interactive design. One can start with a rough shape consisting of just a few metaballs. Then adding details by simply editing metaballs: add, delete, transform, and adjust the parameters. However, the metaball technique suffers from two serious drawbacks: first, it requires considerable skill to construct complex objects by properly generating numerous overlapping functions. Second, interactive shape design demands quick feedback when the designer is editing blobs. Unfortunately polygonalization or ray tracing is usually required to visualise a soft object which is very expensive. Fig. 17 Interface for the creation of bodies ESPRIT Project 25379

19 In order to enhance the modelling capability of the metaball technique, we devised some ways to reduce the number of metaballs needed for a desired detail shape and to allow the designer to manipulate body surface at interactive speed. A cross-sectional based isosurfacesampling method, combined with the B-spline blending, enable us to achieve those two goals. From a practical point-of-view, an interactive metaball editor for shape designers is written. We start the shape design by first creating a skeleton for the organic body to be modelled. Metaballs are used to approximate the shape of internal structures, which have observable effects on the surface shape. Each metaball is attached to its proximal joint, defined in the joint local co-ordinate system of the underlying skeleton which offers a convenient reference frame for position and editing metaball primitives, because the relative proportion, orientation, and size of different body parts is already well-defined. In our editor, the workstation can display shaded or wireframe colour metaballs and high resolution body envelope near real time. Metaballs are displayed as ellipsoids either with effective radius or threshold radius. The threshold mode shows the visible size of metaballs, while the effective mode shows how far the metaball will influence. Some widget panels are used to interactively adjust the size, weight, position, and orientation of metaballs. SpaceBall or trackball enables the user to rotate models around in space for different viewing. By turning on/off various display entities of different layers, the designer can selectively check skeleton, metaballs, cross-section contours, and skin envelope simultaneously. The designer can interactively create, delete, pick, and joint attach/detach metaballs. A file format is designed which can store both joint hierarchy and metaball information. Models can be saved to a file and load in later for successive sculpting. The designer can get quick feedback of the resulting skin form. Fig. 18 presents two examples of created man and woman bodies. In Fig. 19, the examples of small and large virtual humans are presented. Fig. 18 Examples of man and woman bodies ESPRIT Project 25379

20 Fig. 19 Various shapes of body small/large A further enhancement of the body shape and its realism is possible by appropriately using the metaball techniques to simulate the clothes of a person. This is done mostly by using typical scaling techniques on specific metaballs. This technique can be used to create a jacket or pants but it is not suitable for some clothes such as skirts for instance. The main advantage of this technique is that the body thus created has much more realism due to the added clothing and it can be used for real-time animation. Fig. 20 present an example of a pant and a costume created out of metaballs. Fig. 20 Metaballs-based virtual clothes ESPRIT Project 25379

21 7.2 How to simply create diverse shapes The body creation interface includes also various transformations that can be easily applied to an already constructed body. Fig. 21 presents the most typical transformations. The first and most important transformation is the global scaling that allows the creation of child body to normal height human bodies. Using the frontal and lateral scaling, we can easily create virtual humans with normal characteristic towards much stronger built humans. Finally the origin of the spine may be changed to create different ratio between the torso and the legs. Extreme examples are displayed in Fig. 21. Fig. 21 Available transformation to change the body shape 7.3 Connecting the head to the body In the previous sections, we have explained how we could obtain various heads and body shapes. In this section we will explain how we can position those heads and attached them to the bodies. Because of the disparity between bodies and heads, we do not include the neck with any of them. Therefore the neck must be created automatically according to the relative position of the head and body. The first step is therefore to position the head relatively to the body using a suitable matrix of transformation. This matrix is usually available when the body is created but it may be changed if necessary. The automatic program for the creation of the neck proceeds then first by identifying the vertex on the head where the neck will be attached. Because the number of vertices on the body and on the head may vary with from one instance to another, a neighbouring technique is used to find the ordered n-to-m relationship between the neck vertexes on the body and on the head. This relationship is finally used to generate the neck mesh that links the head to the body. The mesh thus created has a generic procedure for its deformation according to the value of the head rotation. Fig. 22 displays an example of a created neck that connects the body to the head. Fig. 22 Neck connection between the body and the head ESPRIT Project 25379

22 7.4 Changing texture In previous section, we have shown how multiple bodies and faces could be created and how they can be assembled to create a population of virtual humans. In this section, we will explain the texturing method used for our virtual human and show that this final feature can further improve the variety of virtual human inside the population. Fig. 23 presents the different textures that are applied to the body parts of the virtual human. The head texturing technique was fully described previously. A different texture can be applied for left and right upper and lower leg, left and right arm and forearm, hip, neck and front and back torso. In total 12 textures can be applied to the body of a virtual human. This leads to a simple way to create multiple population of virtual human by simply changing the appropriate texture. Right column in Fig. 23 present s two simple examples of two different sets of texture applied to the same body. Fig. 23 Textures for Virtual Humans and a body with two different textures ESPRIT Project 25379

23 7.5 Examples of complete virtual humans Fig. 24 Example of bodies with Heads In Fig. 24, we present several examples of various bodies (male and female) with several heads placed on top of them. These complete models are ready to be animated because they are based upon generic models and that the animation parameters are embedded within the definition of the body and head files. Depending upon the kind of population that we would ESPRIT Project 25379