Real-Time Speech-Driven Face Animation. Pengyu Hong, Zhen Wen, Tom Huang. Beckman Institute for Advanced Science and Technology

Transcription

1 Real-Time Speech-Diven Face Animation Pengyu Hong, Zhen Wen, Tom Huang Beckman Institute fo Advanced Science and Technology Univesity of Illinois at Ubana-Champaign, Ubana, IL 61801, USA Abstact This chapte pesents ou eseach on eal-time speech-diven face animation. Fist, a visual epesentation, called Motion Unit (MU), fo facial defomation is leaned fom a set of labeled face defomation data. A facial defomation can be appoximated by a linea combination of MUs weighted by the coesponding MU paametes (MUPs), which ae used as the visual featues of facial defomations. MUs exploe the coelation among those facial featue points used by the MPEG-4 face animation (FA) to descibe facial defomations. MU-based FA is compatible with MPEG-4 FA. We then collect a set of audio-visual (AV) taining database and use the taining database to tain a eal-time audio-to-visual mapping (AVM). 1. Intoduction Speech-diven face animation takes advantage of the coelation between speech and facial coaticulation. It takes speech steam as input and outputs coesponding face animation sequences. Theefoe, speech-diven face animation only equies vey low bandwidth fo face-to-face communications. The AVM is the main eseach issue of speechdiven face animation. Fist, the audio featues of the aw speech signals ae calculated. Then, the AVM maps the audio featues to the visual featues that descibe how the face model should be defomed.

2 Some speech-diven face animation appoaches use phonemes o wods as intemediate epesentations. Lewis [14] used linea pediction to ecognize phonemes. The ecognized phonemes ae associated with mouth shapes which povide keyfames fo face animation. Video Rewite [2] tains hidden Makov models (HMMs) [18] to automatically label phonemes in both the taining audio tacks and the new audio tacks. It models shot-tem mouth co-aticulation within the duation of tiphones. The mouth image sequence of a new audio tack is geneated by eodeing the mouth images selected fom the taining footage. Video Rewite is an offline appoach. It equies a vey lage taining database to cove all possible cases of tiphones and needs lage computational esouces. Chen and Rao [3] tain HMMs to pase the audio featue vecto sequences of isolated wods into state sequences. The state pobability fo each audio fame is evaluated by the tained HMMs. A visual featue is estimated fo evey possible state of each audio fame. The estimated visual featues of all states ae then weighted by the coesponding pobabilities to obtain the final visual featues, which ae used fo lip animation. Voice Puppety [1] tains HMMs fo modeling the pobability distibution ove the manifold of possible facial motions fom audio steams. This appoach fist estimates the pobabilities of the visual state sequence fo a new speech steam. A closed-fom solution fo the optimal esult is deived to detemine the most pobable seies of facial contol paametes, given the bounday (the beginning and ending fames) values of the paametes and the visual pobabilities. An advantage of this appoach is that it does not equie ecognizing speech into high-level meaningful symbols (e.g., phonemes, wods), which is vey difficult to obtain a high ecognition ate. Howeve, the speech-diven face animation appoaches in [1], [2] and [3] have elative long time delays.

3 Some appoaches attempt to geneate the lip shapes using one audio fame via vecto quantization [16], affine tansfomation [21], Gaussian mixtue model [20], o atificial neual netwoks [17], [11]. Vecto quantization [16] fist classifies the audio featue into one of a numbe of classes. Each class is then mapped to a coesponding visual featue. Though it is computationally efficient, the vecto quantization appoach often leads to discontinuous mapping esults. The affine tansfomation appoach [21] maps an audio featue to a visual featue by a simple linea matix opeation. The Gaussian mixtue appoach [20] models the joint pobability distibution of the audio-visual vectos as a Gaussian mixtue. Each Gaussian mixtue component geneates an estimation of the visual featue fo an audio featue. The estimations of all the mixtue components ae then weighted to poduce the final estimation of the visual featue. The Gaussian mixtue appoach poduces smoothe esults than the vecto quantization appoach does. In [17], Moishima and Haashima tained a multilaye pecepton (MLP) to map the LPC Cepstum coefficients of each speech fame to the mouth-shape paametes of five vowels. Kshisaga and Magnenat-Thalmann [11] tained a MLP to classify each speech segment into the classes of vowels. Each vowel is associated with a mouth shape. The aveage enegy of the speech segment is then used to modulate the lip shapes of the ecognized vowels. Howeve, those appoaches poposed in [16], [21], [20], [17], and [11] do not conside the audio context infomation, which is vey impotant fo modeling mouth coaticulation duing speech poducing. Many appoaches have been poposed to tain neual netwoks as AVMs while taking into account the audio contextual infomation. Massao et al. [15] tained a MLP as the AVM. They modelled the mouth coaticulation by consideing the

4 speech context infomation of eleven consecutive speech fames (five backwad, cuent, and five fowad fames). Lavagetto [12] and Cuinga et al. [5] tain time delay neual netwoks (TDNNs) to map the LPC cepstal coefficients of speech signals to lip animation paametes. TDNN is a special case of MLP and it consides the contextual infomation by imposing odinay time delay on the infomation units. Nevetheless, the neual netwoks used in [15], [12], and [5] have a lage numbe of hidden units in ode to handle lage vocabulay. Theefoe, thei taining phases face vey lage seaching space and have vey high computational complexity. 2. Motion Units The Visual Repesentation MPEG-4 FA standad defines 68 MPEG-4 FAPs. Among them, two ae high-level paametes, which specify visemes and expessions. The othes ae low-level paametes that descibe the movements of spase featue points defined on head, tongue, eyes, mouth, and eas. MPEG-4 FAPs do not specify detail spatial infomation of facial defomation. The use needs to define the method to animate the est of the face model. MPEG-4 FAPs do not encode the infomation about the coelation among facial featue points. The use may assign some values to the MPEG-4 FAPs that do not coespond to natual facial defomations. We ae inteested in investigating natual facial movements caused by speech poducing as well as the elations among those facial featue points in MPEG-4 standad. We fist lean a set of MUs fom eal facial defomations to chaacteize natual facial defomations duing speech poducing. We assume that any facial defomation can be appoximated by a linea combination of MUs. Pincipal Component Analysis (PCA) [10] is ap-

5 plied to leaning the significant chaacteistics of the facial defomation samples. Motion Units ae elated to the woks in [4], [7]. We put 62 makes in the lowe face of the subject (see Figue 1). Those makes cove the facial featue points that ae defined by the MPEG-4 FA standad to descibe the movements of the cheeks and the lips. The numbe of the makes decides the epesentation capacity of the MUs. Moe makes enable the MUs to encode moe detailed infomation. Depending on the need of the system, the use can flexibly decide the numbe of the makes. Hee, we only focus on the lowe face because the movements of the uppe face ae not closely elated to speech poducing. Cuently, we only deal with 2D defomations of the lowe face. Howeve, the method descibed in this chapte can be applied to the whole face as well as the 3D facial movements if the taining data of 3D facial defomations ae available. To handle the global movement of the face, we add thee additional makes. Two of them ae on the glasses of the subject. The est one is on the nose. Those thee makes mainly have igid movements and we can use them to align the data. A mesh is ceated accoding to those makes to visualize facial defomations. The mesh is shown to ovelap with the makes in Figue 1. Figue 1. The makes and the mesh.

6 We captue the font view of the subject while he is ponouncing all English phonemes. The subject is asked to stabilize his head as much as possible. The video is digitized at 30 fame-pe-second. Hence, we have moe than 1000 image fames. The makes ae automatically tacked by template matching. A gaphic inteactive inteface is developed fo manually coecting the positions of tackes using the mouse when the template matching fails due to lage facial motions. To achieve a balanced epesentation on facial defomations, we manually select facial shapes fom those moe than 1000 samples so that each viseme and the tansitions among each pai of visemes ae nealy evenly epesented. To compensate the global face motion, the tacking esults ae aligned by affine tansfomations defined by those thee additional makes. Afte nomalization, we calculate the defomations of the makes with espect to positions of the makes in the neutal face. The defomations of the makes at each time fame ae concatenated to fom a vecto. PCA is applied to the selected facial defomation data. The mean facial defomation and the fist seven eigenvectos of the PCA esults, which coespond to the lagest seven eigenvalues, ae selected as the MUs in ou expeiments. The MUs ae epesented as M { mi} i= 0. Hence, we have s = m M 0 + cimi + s0 i= 1 (1) whee s 0 is the neutal facial shape and M ck } k 1 { = is the MUP set. The fist fou MUs ae shown in Figue 2. They espectively epesent the mean defomation and the local defomations aound cheeks, lips, and mouth cones.

7 (a) s + 0 m (b) s0 + km (c) 0 1 s + 0 km (d) s km 3 Figue 2. Motion Units. k = m0. MUs ae also used to deive obust face and facial motion tacking algoithms [9]. In this chapte, we ae only inteested in speech-diven face animation. 3. MUPs and MPEG-4 FAPs It can be shown that the convesion between the MUPs and the low-level MPEG-4 FAPs is linea. If the values of the MUPs ae known, the facial defomation can be calculated using eq. (1). Consequently, the movements of facial featues in the lowe face used by MPEG-4 FAPs can be calculated because MUs cove the featue points in the lowe face defined by the MPEG-4 standad. It is then staightfowad to calculate the values of MPEG-4 FAPs. If the values of MPEG-4 FAPs ae known, we can calculate the MUPs in the following way. Fist, the movements of the facial featues ae calculated. The concatenation of the facial featue movements foms a vecto p. Then, we can fom a set of vectos, say { f 0, f 1,, f M }, by extacting the elements that coespond to those facial featues fom the MU set { m 0, m 1,, m M }. The vecto elements of { f 0, f 1,, f M } and those of p ae aanged so that the infomation about the defomations of the facial featue points is epesented in the same ode. The MUPs can be then calculated by

8 whee F f f f = L ]. [ 1 2 M c1 M cm T 1 T = ( F F) F ( p f0) (2) The low-level paametes of MPEG-4 FAPs only descibe the movements of the facial featues and lack detailed spatial infomation to animate the whole face model. MUs ae leaned fom eal facial defomations, which ae collected so that they povide the dense infomation about facial defomations. MUs captue the second-ode statistic infomation about the facial defomation and encode the coelation infomation of the movements of the facial featue points. 4. Real-Time Audio-to-MUP Mapping The nonlinea elation between audio featues and the visual featues is complicated, and thee is no existing analytic expession fo the elation. MLP, as a univesal nonlinea function appoximato, has been used to lean the nonlinea AVMs [11], [15], [17]. We also tain MLPs as an AVM. Diffeent fom othe woks using MLPs, we divide the AV taining data into 44 subsets. A MLP is tained to estimate MUPs fom audio featues using each AV taining subset. The audio featues in each goup ae modeled as a Gaussian model. Each AV data pai is classified into one of the 44 goups whose Gaussian model gives the highest scoe fo the audio component of the AV data. We set the MLPs as thee-laye peceptons. The inputs of a MLP ae the audio featue vectos of seven consecutive speech fames (3 backwad, cuent and 3 fowad time windows). The output of the MLP is the visual featue vecto of the cuent fame. We use the eo backpopagation algoithm to tain the MLPs using

9 each AU taining subset sepaately. In the estimation phase, an audio featue vecto is fist classified into one of the 44 goups. The coesponding MLP is selected to estimate the MUPs fo the audio featue vecto. By dividing the data into 44 goups, lowe computational complexity is achieved. In ou expeiments, the maximum numbe of the hidden units used in those thee-laye peceptons is only 25 and the minimum numbe of the hidden units is 15. Theefoe, both taining and estimation have vey low computational complexity. A method using tiangula aveage window is used to smooth the jeky mapping esults. 5. Expeimental Results We videotape the font view of the same subject as the one in Section 2 while he is eading a text copus. The text copus consists of one hunded sentences that ae selected fom the text copus of the DARPA TIMIT speech database. Both the audio and video ae digitized at 30 fame-pe-second. The sampling ate of the audio is 44.1k Hz. The audio featue vecto of each audio fame is its ten Mel-Fequency Cepstum Coefficients (MFCC) [19]. The facial defomations ae conveted into MUPs. Oveall, we have AV samples in the taining database. Eighty pecent of the data is used fo taining. We econstuct the displacements of the makes using MUs and the estimated MUPs. The evaluations ae based on the gound tuth of the displacements and the econstucted displacements. The displacements of each make ae nomalized to the ange of [-1.0, 1.0] by dividing them by the maximum absolute gound tuth displacement of the make. We calculate the Peason poduct-moment coelation coefficient and the elated standad deviations using the nomalized displacements. The Peason poduct-moment coelation coefficient between the gound tuth and the estimated data is

10 ' ' T t( E[( d µ )( d µ ) ]) R = T ' ' ' (3) ' T t( E[( d µ )( d µ ) ]) t( E[( d µ )( d µ ) ]) whee d is the gound tuth, (d µ = E ), ' d ' ' is the estimation esult, and ( d µ = E ). The aveage standad deviations ae also calculated as ν d γ = = 1 ( Cd [ ][ ]) γ 1/ 2 (4) ν ' d γ = = 1 ( C ' d [ ][ ]) d ' d T ' ' ' ' T whee C = E(( d µ )( d µ ) ) and C = E(( d µ )( d µ ) ). The Peason poductmoment coelation and the aveage standad deviations measue how good the global match between the shapes of two signal sequences is. The value ange of the Peason coelation coefficient is [0 1]. The lage the Peason coelation coefficient, the bette the estimated signal sequence matches with the oiginal signal sequence. The mean squae eos ae also calculated. The esults ae shown in Table 1. Table 1. Numeic evaluation of the tained eal-time AVM. γ 1/ 2 Taining data Testing data R ν d ν ' d MSE Figue 3 illustates the estimated MUPs of a andomly selected testing audio tack. The content of the audio tack is Stimulating discussions keep students attention. The fig-

11 ue shows the tajectoies of the values of fou MUPs (c 1, c 2, c 3, and c 4 ) vesus the time. The hoizontal axis epesents fame index. The vetical axis epesents the magnitudes of the MUPs coesponding to the defomations of the makes befoe nomalization. Figue 4 shows the coesponding y tajectoies of the six lip featue points (8.1, 8.2, 8.5, 8.6, 8.7, and 8.8) of the MPEG-4 FAPs. c 1 c 2 c 3 c 4 Figue 3. An example of audio-to-mup mapping. The solid blue lines ae the gound tuth. The dash ed lines epesent the estimated esults. MUPs coespond to the defomations of the makes befoe nomalization.

12 Figue 4. The tajectoies of six MPEG-4 FAPs. The speech content is the same as that of Figue 3. The solid blue lines ae the gound tuth. The dash ed lines epesent the estimated esults. The defomations of the featue points have been nomalized. 6. The iface System We developed a face modeling and animation system the iface system [8]. The system povides functionalities fo customizing a geneic face model fo an individual, textdiven face animation, and off-line speech-diven face animation. Using the method pesented in this chapte, we developed the eal-time speech-diven face animation function fo the iface system. Fist, a set of basic facial defomations is caefully and manually designed fo the face model of the iface system. The 2D pojections of the facial shapes of the basic facial defomation ae visually vey close to MUs. The eal-time AVM descibed in this chapte is used by the iface system to estimate the MUPs fom audio fea-

13 tues. Figue 5 shows some typical fames in a eal-time speech diven face animation sequence geneated by the iface system. The text of the sound tack is Effective communication is essential to collaboation. Figue 5. An example of the eal-time speech-diven face animation of the iface system. The ode is fom left to ight and fom top to bottom. 7. Conclusions This chapte pesents an appoach fo building eal-time speech-diven face animation system. We fist lean MUs to chaacteize eal facial defomations fom a set of labeled face defomation data. A facial defomation can be appoximated by combining MUs weighted by the coesponding MUPs. MUs encode the infomation of the coelation among those MPEG-4 facial featue points that ae elated to speech poducing. We show that MU-based FA is compatible with MPEG-4 FA. A set of MLPs is tained to pefom eal-time audio-to-mup mapping. The expeimental esults show the effectiveness of tained audio-to-mup mapping. We used the poposed method to develop the eal-time

14 speech-diven face animation function fo the iface system, which povides an efficient solution fo vey low bit-ate face-to-face communication. 8. Refeence [1] M. Band, Voice Puppety, SIGGRAPH 99, [2] C. Begle, M. Covell, and M. Slancy, Video ewite: diving visual speech with audio, SIGGRAPH 97, [3] T. Chen, and R. R. Rao, Audio-visual integation in multimodal communications, Poceedings of the IEEE, vol. 86, no. 5, pp , May [4] T. F. Cootes, C. J. Taylo, et al., Active shape models thei taining and application, Compute Vision and Image Undestanding, vol. 61, no. 1, pp , Jan [5] S. Cuinga, F. Lavagetto, F. Vignoli, Lip movements synthesis using Time-Delay Neual Netwoks, Poc. EUSIPCO-96, Tieste, [6] P. Ekman and W. V. Fiesen, Facial action coding system, Palo Alto, Calif.: Consulting Psychologists Pess Inc., [7] P. Hong, Facial expessions analysis and synthesis, MS thesis, Compute Science and Technology, Tsinghua Univesity, July, [8] P. Hong, Z. Wen, T. S. Huang, iface: a 3D synthetic talking face. Intenational Jounal of Image and Gaphics, vol. 1, no. 1, pp. 1-8, [9] P. Hong, An integated famewok fo face modeling, facial motion analysis and synthesis, Ph.D. Thesis, Compute Science, Univesity of Illinois at Ubana- Champaign, [10] I. T. Jolliffe, Pincipal Component Analysis, Spinge-Velag, 1986.

15 [11] S. Kshisaga and N. Magnenat-Thalmann, Lip Synchonization Using Linea Pedictive Analysis, Poceedings of IEEE Intenational Confeence on Multimedia and Expo, New Yok, August [12] F. Lavagetto, Conveting speech into lip movements: A multimedia telephone fo had of heaing people, IEEE Tansactions on Rehabilitation Engineeing, Vol. 3, No. 1, Mach [13] Y. C. Lee, D. Tezopoulos and K. Wates, Realistic modeling fo facial animation, SIGGRAPH 1995, pp [14] J. P. Lewis, Automated lip-sync: Backgound and techniques, J. Visualization and Compute Animation, vol. 2, pp , [15] D. W. Massao et al., Pictue My Voice: Audio to Visual Speech Synthesis using Atificial Neual Netwoks, in Poc. AVSP 99, Aug. 1999, Santa Cuz, USA. [16] S. Moishima, K. Aizawa and H. Haashima, An intelligent facial image coding diven by speech and phoneme, Poc. IEEE ICASSP, p Glasgow, UK, [17] S. Moishima and H. Haashima, A media convesion fom speech to facial image fo intelligent man-machine inteface, IEEE J. Selected Aeas in Communications, 4: , [18] L. R. Rabine, A tutoial on hidden Makov models and selected applications in speech ecognition, Poc. of the IEEE, vol. 77, no. 2, pp , [19] L. R. Rabine and B. H. Juang, Fundamentals of Speech Recognition, Pentice Hall, 1993.

16 [20] R. Rao, T. Chen, and R. M. Meseeau, Exploiting audio-visual coelation in coding of talking head sequences, IEEE Tans. on Industial Electonics, vol. 45, no.1, pp 15 22, [21] H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, Quantitative association of vocaltact and facial behavio, Speech Communication, vol. 26, no. 1-2, pp , 1998.