Multilevel Deformation Model Applied to Hand Simulation for Virtual Actors

Transcription

1 MIRALab Copyright Information 1998 Multilevel Deformation Model Applied to Hand Simulation for Virtual Actors Laurent Moccozet, Nadia Magnenat Thalmann MIRALab, University of Geneva Abstract Until now, real-time environments have focused on optimizing the speed of the display, mainly by managing various levels of details of the scene's objects. These approaches are generally aimed at virtual scenes with rigid objects. If we consider that objects can be deformed trough time, another process has to be considered in the scene building pipeline: each object must be deformed prior to its display in a given frame. Similarly to other processes of the pipeline, the control of the display speed must be made by balancing between visual quality and time cost. This requires either to combine various deformable models, sorted by their cost in time, or to use a single deformable model able to work at various levels of details. Our purpose is to propose such a multilevel deformable model that we illustrate by its application to human hands simulation. 1. Introduction and background In VR environments, the main objective in rendering a virtual scene is to reach a pseudo real time display speed (as near as possible to the 25 frames per second), and to maintain a constant frame rate. In such environments, objects are modelled using triangular meshes. The main reason for that is that graphics hardware have the best performance with that type of geometric representations. Level of details (LODs) modelling is one of the main approaches developed to optimize the display of VR huge and complex scenes.this technique consists in providing each object of the scene with a set of geometric representation sorted by their number of vertices. The underlying idea is that depending on the relative situation of a given object in the scene, its visual aspect can be depreciated against saving computation time. Such environments are focused on rigid objects.the next challenging step in VR environment is to involve deformable objects[shad96], such as virtual humans and human-like avatars[pand97, CAPI97]. Introducing deformable components into real time environments requires to take them into account with the scene rendering pipeline. The scene rendering pipeline involves all activities needed to construct a 2 dimensional view of the scene updated at a given frame. In order to incorporate the deformation process, we will rather talk about the scene building pipeline. By scene building pipeline, we mean all the processes that are involved in generating a 2 dimensional display of the scene at a given frame from either a default scene or from the scene at the previous frame situation. With such a definition, the deformation and rendering processes are parts of the building pipeline. In [FUNK93], Funkhauser et al. only consider the visual rendering pipeline and subdivide it into two steps: first is the per-primitive process, including applying

2 coordinate transformations, lighting calculations, etc.; then comes the per-pixel process, including z-buffering, texture mapping, etc... In figure 1 which describes the scene building pipeline, both are combined into the display process, which occurs after the deformation and transformation process. This process is aimed at determining the global transformation (translation, rotation, etc.) to be applied to the objects, either rigid or deformable, and at determining and applying the deformations to the deformable objects. Whereas global transformations can be stored in a single transformation matrix for each object, deformations have to be applied to each vertex of the deformed object, as the deformation function is changing for each vertex. Figure 1 shows the whole scene building pipeline. The scene database can describe the scene either at frame i-1 or at a default status. Fig. 1: scene building pipeline Current researches in geometric LODs involves two aspects. The first one is focused on getting LODs for a given representation of an object[schr92, TURK92, HOPP93, ROSS93, ECK95, HOPP96]. The second one is focused on proposing strategies to choose the LOD to display for a given object at a given frame, and handle the swap from a given LOD to another one[funk93,rohl94]. Similarly to the geometric LODs, Funkhauser and Sequin[FUNK93]propose to parametrize the choice of the rendering algorithm for an object. The various rendering algorithms are sorted by time cost. The set of sorted rendering algorithms can be considered as defining rendering LODs. The strategy will then choose, for a given frame, the best geometric LOD with the best rendering LOD. In this context best means the best visual quality for a given time constraint. We propose here to define deformation LODs in order to further incorporate them inside LODs choice strategies that would control geometric, rendering and deformation LODs. The rest of the paper is articulated into 2 main sections. We first describe the basic approach used to model and simulate virtual human hands. The model is based on a 3-layers model where muscles and hands lines are modelled using a generalized Free-Form Deformation model. The next section is dedicated to the description of the deformation LODs we have defined from the basic model. We describe how the deformation process can be optimized, and at which levels. 2. Hand modelling approach Deformable models for articulated characters are traditionally based on a multi-layer approach[chad88]. We propose here a 3-layer model where the intermediate muscular layer between the skeleton and skin is modelled using a generalized Free-Form Deformation[SEDER86]. 2.1 Dirichlet based FFD model Free-Form Deformations(FFD)[SEDER86] belong to a wider class of geometric deformation tools. An extended survey of existing models with comparisons included can be found in

3 [BECH95]. We propose a generalized Free-Form Deformation model. Our main goal is to overcome their basic limitations concerning the design of muscle layers in a multi-layer realistic animation model. These limitations are partially addressed by further extensions : EFFDs by Coquillart in [COQU90], Direct FFD manipulation by Hsu in [HSU92] and NFFDs by Lamousin in [LAMO94]. The main cause for these limitations is the use of rectangular local coordinates to express any point of the surface to deform relatively to the control points box. The main resulting constraint is that control points boxes have to be rectangular. In [COQU90], Coquillart et al. propose an Extended FFD model that allows other shapes of control boxes, but as they do not reconsider the problem of the local coordinates, restrictions still exist. Even in [MACC96] the model proposed by MacCracken and Joy, where the deformation approach is based on a recursive subdivision of the control lattice, the model still requires the explicit construction of the topology of the control lattice. The topology construction can be quite tedious for a complex control lattice, and it is almost impossible to evaluate the influence of a given topology on the resulting deformations. We may conclude that, most FFDs extensions aim to overcome limitations, but instead of reconsidering the geometric tools on which FFDs are based, they keep the initial FFD method and extend its possibilities by using approximation methods. This way, each extension solves only one specific limitation, but does not generalize the FFD concept. Local coordinates is a general problem extensively addressed in visualization [WATS92], especially in the area of data interpolation. FFDs are close to data interpolation if viewed as the interpolation of displacements of given control points at given locations. The most general of all the systems is the natural neighbors or Sibson coordinates system[sibs80], from which Sibson derived a scattered data interpolant. Sibson coordinates computation is based on Voronoi/Dirichlet and Delaunay diagrams [AURE91] In [FARI90], Farin extends the natural neighbors interpolant based on the natural neighbors coordinates, by using the natural neighbors as the support for a multivariate Bézier simplex[dero88], in which any point can be expressed relative to a set of data nodes with a similar relation as the deformation function in FFD's. Farin defines a new type of surfaces defined with this extended interpolant called Dirichlet surfaces. Combining FFD's and Dirichlet surfaces leads to a generalized model of FFD's : Dirichlet FFD's or DFFD's. The deformation function also can be seen as an interpolation of the displacements of the control points. In the Dirichlet based FFD approach, any point of the surface to deform located in the convex hull of a set of control points in general position, is expressed relative to a subset of the control points set with the Sibson coordinate system. Control points of this subset are called Sibson or influencing control points. One major advantage of this technique is that it removes any constraint on the position and topology of control points. It also removes the need to specify the control lattice topology. The deformation process is similar to the one developed by Coquillart for EFFD in [COQU90]: the control points set can be edited, then frozen on the top of a surface by computing the Sibson coordinates of each point of the surface relative to the control points set. The deformation function can then transmit the displacements applied to the control points to the surface. The complete deformation model is extensively described in [MOCC96]. 2.2 DFFD extensions Most of standard FFD extensions can be applied to DFFD. We review here the main ones, that are used in the hand deformation model: * Rational DFFD: As in [KARL92], we can assign weights to the control points. DFFDs are then

4 special cases of rational DFFDs where the weights are all given a value of 1.0. By varying the weight, deformations can be introduced without moving the control points. When the weight of a control point increases, the deformations behave as if the control point was attracting the surface towards itself. * Direct point manipulation: As in [HSU92], direct manipulation consists in applying FFDs not by moving one or more control points, but by directly moving one or more points on the surface to be deformed. This feature can be directly added to DFFDs by inserting the points to move as control points of the lattice. A control point is defined at the same position as the surface point, so that any displacement applied to the control point will be completely applied to the surface point. This relation can be expressed like the general relation by assigning a Sibson coordinate of 1.0 to the surface point relative to its associated control point. These control points are called constraint control points. We combine the approaches of Chadwick et al. in [CHAD89] and Delingette et al. in [DELI93] to design a multi-layered deformation model for hand animation. The muscle layer will be modeled by a DFFD control point set that will correspond to a simplified surface hand topography. (1) palm (2) upper transversal line (3) lower transversal line (4) thenar eminence (5) thenar line (6) thumb first line (7) thumb second line (8) hypothenar eminence (9) finger first line (10) finger second line (11) finger third line Figure 2: Topography of the hand

5 From the observation of a hand, especially its topography[kapa80], we can derive the following simplified observations : * each joint of the skeleton is associated with a hand line on the surface of the hand. * each hand line can be seen as a closed line, and has almost no change in its shape. * rotation at the joint creates an inflation of the part delimited by the hand line associated with the rotating joint and the hand line just above. * hand lines stop the propagation of the skin inflation. * These observations lead us to define the basic data structure for our hand model, that we call wrinkle, as we want to keep that data structure general enough to be used forthe animation of other human parts: * the wrinkle itself, which is a set of constraint control points, that are generally selected around the joint and form a closed 3D line. We call such points wrinkle control points * two points among the wrinkle control points, that are used to define the axis on which the associated skeleton joint should lie. In this way, a skeleton model can be easily adapted to the hand's skin. This data allows an easy and realistic hand-skeleton mapping by defining an implicit skeleton to which the skeleton can be fitted. * a mixed set of control points and constraint control points that surround the upper part of the hand surface that will be affected by the rotation of the joint associated with the current wrinkle. We call these points influenced wrinkle control points, as they are influenced by the rotation of the wrinkle itself. * one control point, called an inflation control point, which will be used to simulate inflation at the upper limb associated with the joint. For each wrinkle, the muscle layer gets the joint angle variation from the skeleton layer. If the rotation angle is a, the wrinkle itself is rotated atan angle of a/2, and the set of influenced control points is rotated by a. At the rest position, all control points have a weight of 1. When the joint angles vary, the weights of the inflation control points vary accordingly, such that weight(p) = f(a), where P is an inflation control point and a an angle of rotation at the joint. This point is placed on the mesh so that when its weight increases, it attracts the mesh. One unclosed wrinkle exists at the base of each of the four fingers. The union of these four wrinkles defines a closed

6 (a) (b) (c) (d) Figure 4: Hands postures reproductions from famous painting details (a) is a detail from 'La Genèse. Création d'adam' from Michael Angelo, reproduced in (b). (c) is a detail from 'La diseuse de bonne aventure' from Caravage, reproduced in (d).

7 Figure 2 shows the simplified surface hand topography we want to model with the important hand lines associated with the underlying skeleton joints. Figure 3 shows how control points, constraint control points and inflation control points are designed around the surface of the hand to build the control points set and the different wrinkles. Normal control points approximate the shape of the hand, and embed the hand surface in a deformable space. Constraint control points are used to define the hand lines. The resulting hand topography can be compared with figure 2 that represents the 'real' hand topographyfigure 4 shows some hands postures produced with the described approach 3. Multilevel deformable model After the description of our basic approach for hand modelling, we now wantto show how we can define deformation LODs. According to the multilayer approach, there are two levels on which we can work. The first one is the general deformation function involved in the deformable model. We are going to show that DFFDs allow to express deformations at various levels. Another approach is to skip layers in the model. The most direct approach would be to skip directly from the skeleton layer to the skin layer without going through intermediate layers. In the humans simulation context, deformations are constrained by the skeleton's degrees of freedom. Based on this assumption, it is possible not to optimize the muscular layer, but to skip it, using interpolation of extremal hands postures. 3.1 Deformation function level It is possible to define different levels of details at the deformation functions level. The first option is to control the complexity of the deformation function. In our context it can be performed at two levels. The deformation function complexity depends on: - the degree of the function. - the number of Sibson control points involved deformation function degree As for basic FFDs, the deformation function is a cubic function of the local coordinates. The use of Sibson coordinates, and particularly the resulting continuity properties they offer makes it possible to use deformation functions of lower degrees. Figure 5 shows the effect of the different degrees used for the deformation functions. A cylinder is embedded inside a cubic control points set, similar to the one used in FFDs. The control box is then bent. The two bent cylinders at the bottom of the pictures show the resulting deformations using a deformation function of degree 1 (linear) at the bottom left, and the same deformation using a deformation function of degree 3 at the bottom right. For the

8

9 3.2 Deformable model layers level The complete deformable articulated model is defined in multilayers. The complete deformation process is in fact divided into sub-processes. Each sub-process, that we call a procedural layer, has to use the outputs of the previous layer to transform the next structural layer, as shown in figure 8. Our deformable model can be subdivided into two main sub-processes. 1 - pre-process: the control lattice (muscular layer) is updated according to the current skeleton posture. 2 - deformation: the geometric skin is warped according to the current control lattice status. The deformation function only affects the second step, and so it allows to control the second procedural layer (muscle mapping). If we want to optimize the time cost, we must be able to control also the first sub-process. Unfortunately, this sub-process cannot be optimized: all control points have to be moved to mirror the current skeleton posture.the only optimization option is to restrict the number of the skeleton's Degrees of Freedom (DOFs). The number of rotation to apply to control points would then be reduced, and as a consequence, would reduce the time cost of the procedural layer. We can consider for example that some DOFs, such as metacarpi flexion or finger twists,defined in the skeleton model we use, are very rarely activated. There is anyway a minimum number of DOFs to keep and once this number is reached the muscular operators layer cannot be optimized further with this approach. As the minimum number of DOFs is high, there is a strong limitation to the gain we can expect.whatever level of the deformation function we will use, the cost of the preprocess will finally remain constant. As hand movements are constrained by the pre-defined DOFs of the underlying skeleton, we can use interpolation between extreme postures. The interpolation is not performed at the global hand surface level, but by regions.

10 degree linear quadratic cubic cpu time () Table 1: Deformation function time cost for different degrees number of Sibson control points 4 9 unconstrain ed linear quadratic cubic Table 2: Deformation function time cost for different Sibson control points numbers interpolation DFFD (linear, 4 Sibson control points) preprocess deformatio n total

11 Table 3: Total deformation time cost for the model layers optimization All these time cost evaluations have been performed on a Indigo II Impact, R MHZ Time costs are the total CPU time cost and are given in seconds. For each estimation, the deformation process has been applied on one hand with the same skeleton's animation. The hand's geometric skin contains around 1500 vertices. 5. Conclusion We have proposed a multi-level deformation model with the aim of being inserted into VR real-time environments involving deformable objects. Such a model is designed to be used in various types of contexts, from off-line raytraced animation to real-time environments. For the latter, the different levels of deformations can provide a set of Level Of Details for the deformation process. The main motivation is that handling deformable models requires the capacity to control the time cost of the deformation process. Similarly to geometric LODs, widely used in VR environments to control the object display in the scene, deformation LODs can be developed and inserted into the scene building pipeline. This approach is illustrated and applied to a human hand simulation model. Our choice is based on the fact that this part of the body is generally under-treated despite its great importance in terms of visual realism of human-like avatars and of human communication.the hand multi-layers model is based on a general geometric deformation model, that generalizes the traditional FFDs approach: DFFDs. The properties of DFFDs make the model flexible enough to work at various levels, with a minimal loss in the visual results. We have shown at which levels of a multi-layer deformable model optimization can be performed: deformation function, intermediate layers. The resulting multilevel deformation proposes a set of compromises between a low time cost and a high level of deformation quality. The deformation LODs can be further integrated into a LODs management strategies in order to keep a high and constant frame rate for virtual scenes involving rigid and deformable objects. 6. Acknowledgments This work has been funded by the Swiss National Research Foundation. 7. References [AURE91] Aurenhammer F., Voronoi Diagrams - A Survey of a Fundamental Geometric Data Structure, ACM Computing Survey, 23, 3, September [BECH95] Bechmann D., Space Deformation Models Survey, Computers and Graphics, 18, 4, Pergamon, pp , 1995.

12