FACS Based Generating of Facial Expressions

Transcription

1 FACS Based Generating of Facial Expressions A. Wojdeł L.J.M. Rothkrantz Knowledge Based Systems Group, Faculty of Information Technology and Systems, Delft University of Technology, Zuidplantsoen BZ Delft, The Netherlands A.Wojdel@cs.tudelft.nl L.J.M.Rothkrantz@cs.tudelft.nl Keywords: facial animation, 3D graphics, non-verbal communication, user interfaces Abstract In this paper we describe a semiautomatic system for facial animation based on the Facial Action Coding System. The overview of the system highlights its modularization and the distribution of the knowledge about facial expressions that is being used. Further in this article we describe the two last modules in the expression processing part of the system. Both the Facial Model and AUs Blender modules influence in a direct way the visual outcome of the expression being rendered. 1 Introduction A human face is an extremely interesting object. At the first sight, all faces are the same: a pair of eyes, a nose situated in the middle, a mouth in the lower part of the face etc. Yet it is the face that gives us the primary information about the identity of the person. The face provides information about the sex and the age of the subject. Even more; it is the face that communicates the emotions which are an integral part of our daily life. Human face remains very rarely still. Even children learn how to communicate with the facial expressions long before they grasp the language in its verbal representation. The face itself is therefore a very important component of the communication between humans. It provides background information about the mood of the other participant of the conversation. It shows how the person perceives the form and the content of our words. Facial expressions can complement the verbal part of the speech. We often shake the head as a sign of confirmation, we use the gaze direction to stress or specify the verbal description of the spatial dependencies. On the other hand, facial expressions provide a flexible means of controlling the dialogue. Without interfering with the acoustic part of the conversation we use our face to draw the attention of the other person, to signal our readiness to respond or to show that we re waiting for the response. People feel much better if during the course of conversation they can observe the other person. The importance of the face in a human-oriented communication has been noticed by the designers of interactive multi-modal interfaces. Since the early seventies there is an ongoing research in developing a realistic and expressive models of the face. The first one, proposed by F. Parke [1] in 1972 was based on the key-frame modeling. In this approach, for each emotion/expression a new wire-frame has to be generated and stored in the library of facial expressions. An animation is performed by linear interpolation between the two specific masks from the library. This approach, although still used in some cases, is tedious and data expensive. The need to reduce the amount of data being stored and to allow for more subtle facial movements to be reproduced was a driving force in the development of the new facial animation models. At first the so called parametric models appeared [2, 3, 4, 5] and later, as the computation capabilities of the computers grew also the models based on facial anatomy, structure and functionality of facial muscles [6, 7, 8]. Both types of facial models are used in different application fields. The physically based modeling is used in cases when the precision and realism of the generated face is crucial (e.g. in the medical environment). The simpler modeling is used to perform a real-time facial animation. Apart from generating the visually appropriate face image it is also important to have a system that would generate a psychologically proper facial animation in a given context. That means a system

2 Text Text Processing Text with signs Facial Expressions Generator Dictionary of Facial Expressions Knowledge Expressions Processing Expressions Synchronizator Sign(t) sign->set of AUs Translator AU(t) Action Units Blender Combined AUs Model of the Face Face Animator Lips Synchronizator AU(t) 3D Animated Face Figure 1: Design of the system for generating facial expressions with the human face that could be a substitute of a real person in the conversation. Such a system must for example decide which facial expression should be shown, with what intensity and for how long should it last. Most of the systems developed with such a task in mind are rule-based [9, 10]. The sets of rules used in these systems were developed on the basis of psychological research that described the relationships between the textual content, the intonation and the accompanying facial expressions. Such sets are generic in nature, they describe the average responses of a large set of people and disregard the person-specific variations. Each one of us has a very characteristic facial movements used (usually subconsciously) in different situations. Such a personification of the facial animation is next to impossible to be obtained in the automatic generating of the facial expressions. For that reason we decided to concentrate our research on a semiautomatic generating of the facial animation. Our goal is to provide the user with a simple tool for designing facial animations. Such animations when designed by human can incorporate a person specific behavior. It is up to the user, which facial expression would be appropriate in a given context. The system will however support the user on various levels of the design process, so that the obtained animation will come as close to being realistic as possible. 2 Overview of the system At the time of designing the generic form of the system we assumed that it should be based on the already established standards instead of introducing new ones. We decided to use the Facial Action Coding System (FACS) developed by Ekman and Friesen [11] which is a standard way of describing facial expressions in both psychology and computer animation. FACS is based on 44 Action Units (AUs) that represent facial movements that cannot be decomposed into smaller ones. It is argued by Ekman and Friesen that all possible facial expressions can be described as a combinations of those AUs. Our facial animation system is directly based on them. The user of our system does not however have to be an expert in FACS in order to use the system itself. For the user we designed a facial expressions script language that wraps up the AUs in more intuitive terms. It allows the user to chose from the predefined facial expressions that are accompanied by their verbal description and usage examples. The dictionary of the proposed language contains also the information about such facts as for example: when people use a given expression and what they usually communicate through it. More on the script language itself can be found in [12]. In this way, the user can generate a facial animation by putting the predefined emblems in freely chosen places of the text. Those places determine the time at which user wishes the given expression to be shown with a full intensity. The schematic diagram of the system is presented in Fig. 1. The system consists of two independent parts, the first of which concerns the text processing. The second one deals solely with the generating facial expressions. In our system, the text processing part is relatively simple. It contains only one module (Facial Expression Generator) that bases its output on the interaction with a user. It is here that the user uses the facial expression script language and decides at which times what kind of facial expressions should appear. The output of this module consisting of the text accompanied with the emblematic facial expressions forms the input for the second part of the system. The expression processing part contains the 6 modules that automatically process the incoming data. The first step in facial expression processing

3 deals with the time synchronization of the expressions. It is done by two modules: Expressions Synchronizator and Lips Synchronizator. The Lips Synchronizator is responsible for producing the appropriate lip movements according to the text. The Expressions Synchronizator determines the time characteristics of the expressions provided by the user. It decides when the given expression should start, how it should intensify and when it should cease to influence the face. In the next step facial expressions have to be translated into sets of AUs with their appropriate activations. The task of the AUs Translator is to produce the set of AU activations for each of the incoming facial expressions at a given expression intensity. The output of this module is defined as a set of AUs, their timing and intensities. The same type of output is generated by Lips Synchronizator and both must be provided further to the AUs Blender. It is worth noting that the information about the AU activations may contain redundant or conflicting information at this stage. For example it may happen that the Lips Synchronization module offers a face with closed lips at the same time at which the user chose to show a facial expression that describes the mouth as being open. It is the task of AUs Blender to combine those AUs in such a way that they can be shown on the synthetic face. The AUs Blender may decide to appropriately change the intensity, the timing or even the fact of occurrence of a specific AU in order to provide the consistency of the expression (see section 4). At the end of the processing, the prepared AUs are sent to the Facial Model module where they are properly interpreted and finally transformed into the 3D animation of the synthetic face. It is worth noticing that in our design, each of the modules contains some independent fragment of the knowledge about the relationships between facial expressions and AUs. Such a structure allows for an incremental approach in constructing the system and at each stage gives increasing support for the user in designing the animation. For example, the Facial Model itself takes the burden of modifying the wire-frame at the vertex level off the user s shoulders. Going one step earlier, the AUs Blender module takes care of resolving conflicts and inconsistencies at the AU level. The AUs Translator introduces the higher level of the representation and provides the means of using a more intuitive language of facial expressions instead of AUs. This scheme can be prolonged further even to the text processing part of the system in which case we would have a fully automated system for generating facial animation. Moreover the modularity of the system and the independence of the knowledge used in different modules allows for an easy way of improving or modifying specific aspects of the system. 3 Facial Model Our facial model is performance based and at the same time a parametric one. It is based on AUs which means that for each AU we have defined a set of parameters that describe in which way the wire-frame vertices must be displaced on the synthetic face so that it shows the appropriate AU. Moreover, as the typical facial expression incorporates multiple AUs, we have defined also the procedures to accumulate the displacements together. We can divide the AUs in three categories based on their area of influence and on which facial objects they act. Those categories differ not only in definition but also in the implementation details. The categories are namely: Single Object AUs, Sub-object AUs and Multiple Object AUs. We will further describe the implementation of the first type of AUs and highlight the differences between this generic model and the other categories of AUs. 3.1 Single Object AUs The biggest group of AUs are the Single Object AUs. Their implementation provides therefore the basic framework for implementing facial movements. Each of the AUs is represented as the intensity value and the two functions that form together the description of the displacement. For each AU we must define therefore: ³ Úµ The density function. It represents the area at which the AU influences the face and the displacement amount with the AU activated in 100%. Úµ The direction function. This function defines the direction of the displacement caused by a given AU. ¾ ¼ ½ The intensity value. When the AU is not activated at all ¼ and in case of full activation ½. For most of the AUs we can define the displacement of the face at a given point Ú as: Ú Úµ³ Úµ (1) The above equation can be used in all cases when we can assume the linear dependency between the AU intensity and the displacement. There are however some AUs that do not behave in this way. Such AUs are for example the rotations of the eyes or the movements of the head as a whole. For those AUs we use a more generic displacement calculation: Ú Ú ³ Úµ µ (2) where the direction of the displacement depends not only on the initial position of the point but also on

4 a) b) Figure 2: Landmark points in a neutral face (a) and a face showing AU15 - Lip Corner Depressor (b) the intensity of the activated AU and the value of the density function in this point. The implementation of this generic model of an AU is done by finding the appropriate forms of both of the functions for each of the AUs. The functions are customized to a specific person in the two stages. As it was stated in the beginning of this section, our model is performance based and therefore the first stage consists of measuring the facial movements of the person being modeled. We need here the 3D measurements of a real face as deformed when showing the pure and fully activated AUs. The measurements are not constrained by the choice of the wire-frame that will later be modeled. The only constraint on the measurements is that they describe the facial movements in a complete way. That means that all of the significant changes on the face must be reflected in the measured data. In our experiments we used 36 landmark points placed on one side of the face assuming that both sides are equivalent to each other (see Fig. 2). We used further also the visible contours of the eyes, eyebrows and lips as the natural landmarks. The second stage of implementing the AU relies on fitting the parameters of the density and direction functions on the measured data. Thanks to the form in which the displacement is being calculated (Eq. 1) for most of the AUs both functions can be optimized independently, which simplifies the fitting process greatly. A more in-depth description of such an optimization can be found in [13] together with the validation of the fitting results. 3.2 Sub-Object AUs AUs from this group are characterized by the fact, that their activation effects in separation of the upper and lower lip. Separation of the lips induces rapid changes in the density as well as in the direction of the movement on a relatively small area of the face. Therefore in order to obtain a better optimization we decided to divide the wireframe representing the surface of the face into two parts. This division is de- Figure 3: Division of the wireframe into two parts for sub object AUs fined by the topology of the facial surface and can therefore be determined independently from the wireframe configuration. For the sake of simplicity, in our implementation we use a single plane that intersects with the face in positions corresponding to the mouth corners (see Figure 3). When fitting the sub-object AU on the measured data we need to consider that there are now in total four functions. There are two independent density functions and two independent direction functions. Fortunately they act on the separate parts of the wireframe in pairs and do not influence each other. Obviously, there is still only one intensity value that is used together with either pair of the functions depending on the initial position of the point being displaced. 3.3 Multiple Object AUs A model of the face can be built from a few objects as e.g. face, eyes, teeth. Usually a specific AU modifies only one object. For example moving the eyes influences only the eyes and does not change the face around them. On the another hand, closing the eyes acts only on the face and does not have any influence on the eyes. Although closing the eyelids obscures the eyeballs, their geometry is not influenced by this movement. However the activation of some AUs can result in deformation of more than one object. Such AUs include e.g. all AUs related to the movement of the whole head; when we rotate the head, the rotation acts on the facial surface as well as the eyes and the teeth even though eyes and teeth are not necessarily visible. Another example can be AU26 Jaw Drop. Although showing this single AU the mouth is closed, we should remember about the teeth. When we for example combine AU26 with AU25 Lips Part the teeth can become visible and they should be appropriately moved. Therefore while implementing multiple object

5 a) b) Figure 4: Displacement vectors calculation in additive mixer (a) and successive mixer (b) AUs we have to remember about defining appropriate AU components for all of the objects from a facial model that a given AU can affect. 3.4 Combining AUs As we know, the facial expressions used in a reallife rarely contain only a single AU activation. A typical facial expression consists of three or more AUs. Therefore the definition of interactions between the AUs on the geometrical level must be the integral part of our model. We use here two different types of AU mixers; additive and successive. In an additive mixer, the composite vectors of the movement are calculated separately for each of the AUs. Then the resulting vector of the movement is a summation of the composite vectors and can be applied on the original model. In this way the result of the rendering does not depend on the order in which the AUs are modeled (see Figure 4a). In case of successive mixing of AUs, the original wireframe is adapted through the successive AU modifiers in a specific order. The wireframe vertices change their positions while applying one AU after another (see Figure 4b). Which one of the above defined mixers is used in a given combination depends on the types of the AUs that take part in the expression. For example, most of the single object AUs can be combined using additive mixing. The only exception here are the nonlinear AUs (described by Eq. 2), which by their nature must be combined in a successive way. Also the sub-object AUs are usually better combined with the additive mixer. This kind of AUs relates to the rapid changes in some small areas of the face and therefore using the successive mixing may produce unrealistic and unexpected facial expressions (see Figure 5). The multiple object AUs present a special case. Generally we must apply a successive mixer with regards to all of the secondary objects. That means, that at first we have to modify a secondary object according to those AUs from a combination that act only on it and later on take into consideration the influence of the multiple object AUs. For example, in order to move a head we have to activate all of the AUs for the face, the eyes and the teeth first and then the AUs responsible for the movement of the whole head should be applied on all modified objects. a) b) Figure 5: Combination of AU12 and AU25 using an additive mixer (a) successive mixer (b) 4 AUs Blender The model of the face itself does not contain any information about the dependencies between specific AUs. It contains only a geometrical description of the way in which the AUs are combined. Whether there is or not a physiological sense in combining those AUs is of no concern in that module. It is the task of AUs Blender to prepare the set of activated AUs in such a way that it can be directly rendered by the facial model. There are several restrictions on the possible combinations of AUs. Some of the AUs are contradictory to each other (such as opening and closing the eyes), and some cannot be activated together at the same time even though they don t seem contradicting at the first sight. According to Ekman there are 5 distinct ways in which the AUs combine and influence each other. First of all the combination of the AUs can be additive. In such a case they operate as if they were activated separately and the resulting facial movement is a plain summation of the separate displacements. The AUs can combine also in such a way that one dominates over the other, diminishing therefore the results of the activation of the latter AU. In case when the AUs are contradictory to each other, they combine in an alternative way. There is also a possibility of substitution in case when the occurrence of two AUs at the same time is equivalent to activation of the third AU alone. In the end all of the exceptions that cannot be modeled in the above mentioned ways fall into a group of different ways of combining AUs. The AUs Blender does not implement the above description in a direct way. Its task is not only to make sure that the activations of the AUs are appropriate, but it also has to support the user in using the AUs for animation. Therefore, instead of blocking the activation of the contradictory AUs, the blender will try to infer the activation value for just one of them so that it corresponds in the best way to the intent of the

6 user. For example, a 100% activation of AU51 (Head Turn Left) with 60% activation of AU52 (Head Turn Right) will result in the 40% activation of the AU51 alone. In the similar way the domination of a given AU over another one cannot be done by simply removing the activation of the latter one. We need to make sure that the smoothness of the changes in the face is preserved. Therefore the resulting activations of the AUs have to be recalculated so that both the smoothness and the dominating role of one of the AUs are properly represented. Moreover, there are also some constraints on the activation intensities even if the AUs do not influence each other directly. As an example here, the AU51 (Head Turn Left) and AU53 (Head Up) are simply additive. This is entirely true however only when scoring their appearance in the binary scale of the FACS system. Both of them cannot be activated to the full extent at the same time. The AUs Blender normalizes their activations so that their sum does not exceed 100% in this case. All of the AUs Blender s actions can be encapsulated in a set of simple if-then rules and some arithmetics. The end result however is a complex model of the AUs combinations that is compatible with the Ekman s definitions of the facial movements. 5 Conclusions The proposed system design in its current highly modular form allows for the high flexibility on both the user and the developer side. For the user the system provides the varying levels of the automatization. Depending on the user experience and abilities, there are ways to influence the facial animation at all levels of the facial expression processing. On the other hand, the knowledge about the facial animation and facial expressions is nicely encapsulated in a small independent chunks so that it can be easily extended and/or modified to a specific application. Our facial model and the AUs Blender are based on the well established work of P. Ekman. This allows us to use the existing expertise and data in developing the system further. References [1] Frederic I. Parke. Computer generated animation of faces. In Proceedings of the ACM National Conference, pages , [2] Frederic I. Parke. Parametrized models for facial animation. IEEE Computer Graphics, 2(9):61 68, [3] Keith Waters. A muscle model for animating three-dimentional facial epressions. Computer Graphics (SIGGRAPH 87), 21(4):17 24, July [4] Nadia Magnenat-Thalmann, E. Prmeau, and Daniel Thalmann. Abstract muscle action procedures for human face animation. The Visual Computer, 3(5): , [5] Prem Kalra, Angelo Mangili, Nadia Magnenat- Thalmann, and Daniel Thalmann. Simulation of facial muscle actions using rational freeform deformations. In A. Kilgour and L. Kjelldahl, editors, Proceedings of Eurographics 92 Computer Graphics Forum, volume 11, pages 59 69, Oxford, UK, NCC Blackwell. [6] Demetri Terzopoulos and Keith Waters. Analisis and synthesis of facial image sequences using physical and anatomical models. IEEE Transaction on Pattern Analysis and Machine Intelligence, 15(6): , [7] Yuencheng Lee, Demetri Terzopoulos, and Keith Waters. Realistic modeling for facial animation. In Computer Graphics Proceedings, Annual Conference Series, pages 55 61, [8] Rolf M. Koch, Markus Hans Gross, and Albert A. Bosshard. Emotion editing using finite elements. Technical Report 281, ETH Zürich, Institut of Scientific Computing, January [9] Justine Cassell, Catherine Pelachaud, Norman I. Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Skott Prevost, and Matthew Stone. Animated conversation: Rulebased generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Proceedings of ACM SIG- GRAPH, pages , Orlando (FL.), [10] Catherine Pelachaud, Norman I. Badler, and Mark Steedman. Generating facial expressions for speech. Cognitive Science, 20(1):1 46, [11] Paul Ekman and Wallace F. Friesen. Unmasking the Face. Prentice-Hall, Inc., Englewood Cliffs, New Jersey, USA, [12] Anna Wojdel and Leon J. M. Rothkrantz. Intelligent system for semiautomatic facial animation. In Proceedings of Euromedia 2000, pages , May [13] Anna Wojdel and Leon J. M. Rothkrantz. A performance based parametric model for facial animation. In Proceedings of IEEE International Conference on Multimedia and Expo 2000, New York, NY USA, July-August 2000.