Real time 3D face and facial feature tracking

Transcription

1 J Real-Time Image Proc (2007) 2:35 44 DOI /s ORIGINAL RESEARCH PAPER Real ime 3D face and facial feaure racking Fadi Dornaika Æ Javier Orozco Received: 23 November 2006 / Acceped: 26 June 2007 / Published online: 28 July 2007 Ó Springer-Verlag 2007 Absrac Deecing and racking human faces in video sequences is useful in a number of applicaions such as gesure recogniion and human-machine ineracion. In his paper, we show ha online appearance models (holisic approaches) can be used for simulaneously racking he head, he lips, he eyebrows, and he eyelids in monocular video sequences. Unlike previous approaches o eyelid racking, we show ha he online appearance models can be used for his purpose. Neiher color informaion nor inensiy edges are used by our proposed approach. More precisely, we show how he classical appearance-based rackers can be upgraded in order o deal wih fas eyelid movemens. The proposed eyelid racking is made robus by avoiding eye feaure exracion. Experimens on real videos show he usefulness of he proposed racking schemes as well as heir enhancemen o our previous approach. Keywords Real-ime racking 3D face racking Facial feaure racking Eyelid racking Online appearance models 1 Inroducion F. Dornaika (&) Insiu Géographique Naional Laboraoire MATIS, 2 Avenue Paseur, Sain Mandé, France fadi.dornaika@ign.fr J. Orozco Compuer Vision Cener, Campus UAB, Bellaerra, Barcelona, Spain orozco@cvc.uab.es Deecing and racking human faces in video sequences is useful in a number of applicaions such as gesure recogniion and human-machine ineracion. Faces play a major role in any human compuer ineracion sysem, because hey represen a rich source of informaion. Faces are he main cue humans use for person deecion/idenificaion. Besides his, faces are he main gaeway o express our feelings and emoional saes. Being able o esimae 3D face pose in real-ime, we can ge a clue abou user s inenions. Vision-based racker sysems provide an aracive alernaive since vision sensors are no invasive. Of paricular ineres are vision-based markerless head and/or face rackers. Since hese rackers do no require any arificial markers o be placed on he face, comforable and naural moions can be achieved. On he oher hand, building robus and real-ime markerless rackers for head and facial feaures is a difficul ask due o he high variabiliy of he face and he facial feaures in videos. To overcome he problem of appearance changes recen works on faces adoped saisical facial exures. For example, he Acive Appearance Models have been proposed as a powerful ool for analyzing facial images [5]. Deerminisic and saisical appearance-based racking mehods have been proposed [2, 4, 12]. These mehods can successfully ackle he image variabiliy and drif problems by using deerminisic or saisical models for he global appearance of a special objec class: he face. A few algorihms exis which aemp o rack boh he head and he facial feaures in real ime, e.g., [2, 12]. These works have addressed he combined head and facial feaure racking using he Acive Appearance Model principles. However, [2, 12] require edious learning sages ha should be performed beforehand and should be repeaed whenever he imaging condiions change. Recenly, we have developed a head and facial feaure racking mehod based on online appearance models (OAMs) [6]. Unlike he acive

2 36 J Real-Time Image Proc (2007) 2:35 44 appearance models, he OAMs offer a lo of flexibiliy and efficiency since hey do no require any facial exure model ha should be compued beforehand. Insead he exure model is buil online from he racked sequence. This paper exends our previous work [6] in wo direcions. Firs, we show ha by adoping a non-occluded shape-free facial exure ha excludes he eye region, more accurae and sable 3D head pose parameers can be obained. Second, unlike feaure-based eyelid rackers, we show ha he OAMs can be used o rack he eyelids. Thus, we can infer he eye sae wihou deecing he eye feaures such as he irises and he eye corners. A shor version of his paper has appeared in [7]. In his paper, we addiionally provide (i) exended experimens, (ii) a descripion abou how o build an appearance-based racker able o deal wih fas eyelid movemens, and (iii) a performance evaluaion aiming a quanifying he eye blink deecion in real condiions. Tracking he eyelids and he irises can be used in many applicaions such as drowsiness deecion and inerfaces for handicapped individuals. For applicaions such as driver awareness sysems, one needs o do more han racking he locaions of he person s eyes bu obain heir deailed descripion. Deecing and racking he eye and is feaures (eye corners, irises, and eyelids) have been addressed by many researchers [11, 14, 16, 19]. However, mos of he proposed approaches rely on inensiy edges and are ime consuming. In [16], deecing he sae of he eye is based on he iris deecion in he sense ha he iris deecion resuls will direcly decide he sae of he eye. In [14], he eyelid sae is inferred from he relaive disance beween he eyelid apex and he iris cener. For each frame in he video, he eyelid conour is deeced using edge pixels and normal flow. The auhors repored ha when he eyes were fully or parially open, he eyelids were successfully locaed and racked 90% of he ime. Their proposed approach depends heavily on he exraced inensiy edges. Moreover, i assumes high-resoluion images depicing an essenially fronal face. In our sudy, we do no use any edges and here is no assumpion on he head pose. In our work, he eyelid moion is inferred a he same ime wih he 3D head pose and oher facial acions, ha is, he eyelid sae does no rely on he deecion resuls of oher feaures such as he eye corners and irises. Tracking he rapid eyelid moion is no a sraighforward ask. In our case, we like o rack he eyelid moion using he principles of OAMs. The challenges are as follows. Firs, he upper eyelid is a highly deformable facial feaure since i has a grea freedom of moion. Second, he eyelid can compleely occlude he iris and sclera, ha is, a facial exure model will have wo differen appearances a he same locaions. Third, he eyelid moion is very fas. The remainder of his paper proceeds as follows. Secion 2 inroduces our deformable 3D facial model as well as he concep of shape-free facial paches. Secion 3 describes he online adapive appearance model. Secion 4 presens a generic racking algorihm ha racks in realime he 3D head pose and some facial acions. I describes how he eyelids are racked using his generic algorihm. Secion 5 gives some comparisons obained wih differen facial paches. In Sec. 6, we presen some racking resuls and one performance evaluaion for eye-blinking deecion. 2 Modeling faces 2.1 A deformable 3D model In our sudy, we use he 3D face model Candide [3]. This 3D deformable wireframe model was firs developed for he purpose of model-based image coding and compuer animaion. The 3D shape of his wireframe model is direcly recorded in coordinae form. I is given by he coordinaes of he 3D verices P i, i =1,, n, where n is he number of verices. Thus, he shape up o a global scale can be fully described by he 3n-vecor g; he concaenaion of he 3D coordinaes of all verices P i. The vecor g is wrien as g ¼ g s þ As a ð1þ where g s is he saic shape of he model, s a he facial acion conrol vecor, and he columns of A are he animaion unis. In his sudy, we use seven modes for he facial animaion unis (AUs) marix A. We have chosen he seven following AUs: lower lip depressor, lip srecher, lip corner depressor, upper lip raiser, eyebrow lowerer, ouer eyebrow raiser and eyelid raiser. These AUs are enough o cover mos common facial animaions. Moreover, hey are essenial for conveying emoions. Thus, he lips are conrolled by four facial acions, he eyebrows are conrolled by wo facial acions, and he eyelids by one facial acion. Figure 1 illusraes hese seven facial acions. In his sudy, Fig. 1 The Candide model and he seven facial acions

3 J Real-Time Image Proc (2007) 2: Fig. 2 The configuraion of he 3D face mesh (fronal view) when each facial acion is se o is maximum value. From lef o righ: lower lip depressor, lip srecher, lip corner depressor, upper lip raiser, eyebrow lowerer, ouer eyebrow raiser he eyelids pair, he inner eyebrows pair, and he ouer eyebrows pair are each conrolled by one facial acion. In Eq. 1, he 3D shape is expressed in a local coordinae sysem. However, one should relae he 3D coordinaes o he image coordinae sysem. To his end, we adop he weak perspecive projecion model. We neglec he perspecive effecs since he deph variaion of he face can be considered as small compared o is absolue deph. Thus, he sae of he 3D wireframe model is given by he 3D head pose parameers (hree roaions and hree ranslaions) and he inernal face animaion conrol vecor s a : This is given by he 13-dimensional vecor b: b ¼½h x ; h y ; h z ; x ; y ; z ; s T a ŠT where: ð2þ h x, h y, and h z represen he hree angles associaed wih he 3D roaion beween he 3D face model coordinae sysem and he camera coordinae sysem. x, y, and z represen he hree componens of he 3D ranslaion vecor beween he 3D face model coordinae sysem and he camera coordinae sysem. Each componen of he vecor s a represens he inensiy of one facial acion. This belongs o he inerval [0, 1] where he zero value corresponds o he neural configuraion (no deformaion) and he one value corresponds o he maximum deformaion. In he sequel, he word facial acion will refer o he facial acion inensiy. Figure 2 illusraes he configuraion of he 3D wireframe model when each facial acion is se o is maximum value. Noe ha if only he aspec raio of he camera is known, hen he componen z (he in-deph ranslaion) is replaced by a scale facor s having he same mapping role beween 3D and 2D. This scale is given by s ¼ a z where a is he focal lengh of he camera in pixels. 2.2 Shape-free facial paches A facial image is represened as a shape-free exure, as shown in Fig. 3b. In his secion, we briefly describe how his shape-free exure is compued from he inpu image and he geomerical parameers b. More deails can be found in [1]. The 2D mesh associaed wih he shape-free exure is obained by projecing he saic shape g s (wireframe), using a cenered fronal 3D pose, ono an image wih a given resoluion. The exure of he shape-free facial image is obained by exure mapping from he riangular 2D mesh covering he face in he inpu image (see Fig. 3a) using a piece-wise affine ransform, W: Similarly o [1], we have aken advanage of he fac ha he barycenric coordinaes of he pixels wihin each riangle are invarian under affine ransforms. In oher words, since he geomery of he 2D mesh in he shape-free image is fixed, he barycenric coordinaes are fixed and can be compued once for all, which considerably reduces he CPU ime associaed wih he exure mapping process he warping process. Once an insance of he 3D model (encoded by he vecor b) is projeced ono he inpu image, he warping process proceeds as follows. The shape-free image bounding he fixed 2D mesh is scanned pixel by pixel. For every scanned pixel in his image, we know is riangle as well as is barycenric coordinaes wihin his riangle. Therefore, he 2D locaion of he corresponding pixel in he inpu image can be easily inferred using a linear combinaion of he coordinaes of he riangle verices where he Fig. 3 a An inpu image wih correc adapaion. b The corresponding shape-free facial pach. c The same pach wihou he eyes region

4 38 J Real-Time Image Proc (2007) 2:35 44 coefficiens are given by he barycenric coordinaes. The greylevel of he scanned pixel is hen se by blending he greylevels associaed wih he four closes pixels o he non-ineger coordinaes of he reurned locaion he bilinear inerpolaion. Mahemaically, he warping process applied o an inpu image y is denoed by xðbþ ¼Wðy; bþ ð3þ where x denoes he shape-free facial exure and b denoes he geomerical parameers. Wihou loss of generaliy, we have used wo resoluion levels for he shape-free exures, encoded by 1,310 and 5,392 facial pixels bounded by and recangular boxes, respecively. Obviously, oher levels can be used. Generally speaking as he resoluion increases, he racking accuracy increases. However, by experience we found ha he second resoluion level is a good rade-off beween he accuracy and he compuaional cos. To parially compensae for conras variaions, he original shape-free exure x is ransformed ino an image having a mean equal o 0 and a variance equal o 1. The complee image ransformaion is implemened as follows: (i) ransfer he exure y using he piece-wise affine ransform associaed wih he geomeric parameers b ¼½h x ; h y ; h z ; x ; y ; z ; s T a ŠT ; and (ii) perform zero-meanuni-variance normalizaion on he obained pach. Figure 3 illusraes wo shape-free paches associaed wih an inpu image. 3 Problem formulaion and adapive appearance models Given a video sequence depicing a moving head/face, we would like o recover, for each frame, he 3D head pose and he facial acions encoded by he conrol vecor s a : In oher words, we would like o esimae he vecor b (2) a ime given all he observed daa unil ime, denoed y 1: {y 1,, y }. In a racking conex, he model parameers associaed wih he curren frame will be handed over o he nex frame. For each inpu frame y, he observaion is he shape-free facial pach associaed wih he geomeric parameers b. We use he HAT symbol for he racked parameers and paches. For a given frame ; ^b represens he compued geomeric parameers and ^x he corresponding shape-free pach, ha is, ^x ¼ xð^b Þ¼Wðy ; ^b Þ ð4þ The esimaion of he curren parameers ^b from he previous ones ^b 1 and from he sequence of images will be presened in Sec. 4. In our work, he iniial parameers ^b 1 corresponding o he firs frame are manually provided. The auomaic iniializaion can be obained using he saisical echnique proposed in [2]. By assuming ha he pixels wihin he shape-free pach are independen, we can model he appearance of he shape-free facial pach using a mulivariae Gaussian wih a diagonal covariance marix R: Le l be he Gaussian cener and r be he vecor conaining he square roo of he diagonal elemens of he covariance marix R: l and r are d-vecors (d is he size of x) represening he appearance parameers. In summary, he observaion likelihood a ime is wrien as pðy jb Þ¼pðx jb Þ¼ Yd i¼1 Nðx i ; l i ; r i Þ where N(x i ; l i,r i ) is a normal densiy: " Nðx i ; l i ; r i Þ¼ð2pr 2 i Þ 1=2 exp 1 # x i l 2 i 2 r i ð5þ ð6þ We assume ha he appearance model summarizes he pas observaions under an exponenial envelope, ha is, he pas observaions are exponenially forgoen. When he appearance is racked for he curren inpu image, i.e. he exure ^x is available, we can updae he appearance and use i o rack in he nex frame. I can be shown ha he appearance model parameers, i.e., l and r can be updaed using he following equaions (see [9] for more deails on online appearance models): l iðþ1þ ¼ð1 aþl iðþ þ a^x iðþ ð7þ r 2 i ðþ1þ ¼ð1 aþr 2 i ðþ þ að^x iðþ l iðþ Þ 2 ð8þ In he above equaions, he subscrip i denoes a pixel in he pach ^x: This echnique, also called recursive filering, is simple, ime-efficien and herefore suiable for real-ime applicaions. The appearance parameers reflec he mos recen observaions wihin a roughly L = 1/a window wih exponenial decay. Noe ha l is iniialized wih he firs pach ^x 1 corresponding o he geomerical parameers ^b 1 : However, Eq. 8 is no used unil he number of frames reaches a given value (e.g., he firs 40 frames). For hese frames, he classical variance is used, ha is, Eq. 8 is used wih a being se o 1=: Here, we used a single Gaussian o model he appearance of each pixel in he shape-free pach. However, modeling he appearance wih Gaussian mixures can also be used on he expense of some addiional compuaional load (e.g., see [10, 18]).

5 J Real-Time Image Proc (2007) 2: Tracking using adapive appearance regisraion 4.1 A generic racking algorihm In his secion, we describe he racking algorihm ha is used for racking he head and he facial feaures in a monocular video sequence. The racked facial acions as well as he shape-free facial pach can be arbirary. We consider he sae vecor b ¼½h x ; h y ; h z ; x ; y ; z ; s T a ŠT encapsulaing he 3D head pose parameers and he facial acions. If we only rack he lips and he eyebrows hen he vecor s a will encode six facial acions, i.e., he vecor b will have 12 componens. If we rack he lips, he eyebrows, and he eyelids hen s a will encode seven facial acions, i.e., he vecor b will have 13 componens. We will show how his sae vecor can be recovered for ime using (i) he previous known sae ^b 1 ; (ii) he curren inpu image y, and (iii) he curren appearance parameers encoded by he mean l and he covariance marix R: The sough parameers b a ime are esimaed by regisering he inpu image (warped version) o he curren appearance model. For his purpose, we minimize he Mahalanobis disance beween he warped exure and he curren appearance mean, min b eðb Þ¼min Dðxðb Þ; l Þ¼ Xd b i¼1 x i l 2 i ð9þ r i The above crierion can be minimized using ieraive firs-order linear approximaion which is equivalen o a Gauss-Newon mehod. I is worhwhile noing ha minimizing he above crierion is equivalen o maximizing he likelihood measure given by (5). Moreover, he above opimizaion is made robus by using robus saisics [6]. For every frame in he video sequence, he corresponding sae vecor b is esimaed using he following. The whole racking algorihm is oulined in Fig Tracking algorihm Saring from b ¼ ^b 1 ; we compue he error vecor Wðy ; ^b 1 Þ l ¼ x ð^b 1 Þ l and he corresponding Mahalanobis disance e(b) (given by Eq. 9). We find a shif Db by muliplying he error vecor wih he negaive pseudo-inverse of he gradien marix G ¼ ox ob using (10). Db ¼ G y ðwðy ; ^b 1 Þ l Þ ð10þ where G =(G T G ) 1 G T is he pseudo-inverse of G. The vecor Db gives a displacemen in he search space for which he error, e, can be minimized. We compue a new parameer vecor and a new error: b 0 ¼ b þ qdb e 0 ¼ eðb 0 Þ ð11þ Fig. 4 The generic appearancebased racking algorihm. For every image in he monocular video sequence, he 3D head pose parameers as well as he facial acions are simulaneously esimaed by regisering he curren exure wih he curren facial exure model he curren appearance model ) ( D( ( ), ) ( ( ) y, x, y, z, x, z, T a 1 ( ) s a ^ ^ +1 = (1 - ) = (1 - ) 2 + (x - ) 2

6 40 J Real-Time Image Proc (2007) 2:35 44 where q is a posiive real. If e < e, we updae b according o (11) and he process is ieraed unil convergence. If e e, we ry smaller updae seps in he same direcion (i.e., a smaller q is used). Convergence is declared when he error canno be improved anymore or he number of ieraions reaches a maximum. In he above opimizaion, he gradien marix G ¼ owðy ;b Þ ob ¼ ox ob is approximaed by numerical differences similar o he work of Cooes [5]. Noice ha he gradien marix is compued for each ime sep. The advanage is wofold. Firs, a varying gradien marix is able o accommodae appearance changes. Second, i will be closer o he exac gradien marix since i is compued for he curren geomeric configuraion (3D head pose and facial acions) whereas a fixed gradien marix can be a source of errors. More deails abou his opimizaion echnique can be found in [6]. 4.2 Eyelids racking As we have menioned earlier, racking he eyelid moion is a challenging ask, and mos of he proposed approaches for locaing and racking he eyelids rely on he exraced inensiy edges. In our case, he generic racking algorihm (Sec. 4.1) is used for boh cases: (i) racking he lips and he eyebrows, and (ii) racking he lips, he eyebrows, and he eyelids. However, in he second case, here are wo main differences. Firs, we adop a shape-free facial pach whose eye region corresponds o closed eyes configuraion (see Fig. 5), which excludes he iris and sclera regions. Noe ha he 2D shape of he eyelids in he shape-free pach is fixed like any oher facial feaure, ha is, he eyelids appear closed in he facial pach regardless of he sae of he eyelids in he inpu image. This is illusraed in Fig. 5. The maximum value of he eyelid facial acion corresponds o wide open eyes while he zero value corresponds o closed eyes. Noe ha when he eyes are open in he inpu image, he exure of he eyelid region in he shape-free pach (associaed wih a correc eyelid facial acion) will be a disored version of a very small area in he inpu image. However, he global The inpu image The 2D shape The facial pach Fig. 5 The shape-free facial pach used o rack 13 degrees of freedom including he eyelid moion appearance of he eyelid is sill preserved since he eyelids have he skin appearance. Second, since he eyelid moion is very fas a good esimaion of is gradien (a column in he global gradien marix G ¼ ox =ob) is compued wih a large number of perurbaion seps ha cover almos all he variaion inerval. 5 Tracking comparisons In his secion, we compare he 3D head pose esimaes obained wih wo differen shape-free facial paches using he same racking algorihm described above (Sec. 4.1) and he same sae vecor b given by he six head pose parameers and he six facial acions associaed wih he lips and eyebrows, ha is, he eyelid facial acion is no used. To his end, we use he wo shape-free paches depiced in Fig. 3. The firs pach includes a region for he eye feaures namely he iris and he sclera. The second pach is obained from he firs one by only removing he eyes region. We have used a 1,000-frame sequence feauring a alking subjec 1 as a es video. Noe ha alking is a sponaneous aciviy. Figure 6 illusraes he esimaed 3D head pose parameers associaed wih a 150-frame segmen using he wo differen shape-free facial paches. The displayed parameers are (from op o boom): he pich angle h x, he yaw angle h y, he scale s, and he verical ranslaion y. The video segmen sars a frame 500 and conains hree eye blinks a frames 10, 104, and 145. The solid curves correspond o he firs facial pach (wih eye region) while he doed curves correspond o he second pach (wihou eye region). One can noice ha (i) he solid curves and he doed curves coincide for almos all frames, and (ii) he mos significan discrepancies occur a hose frames associaed wih an eye blink (e.g., see he scale plo). Alhough here is no ground-ruh daa for he head moions, we found ha hese discrepancies correspond well o some inroduced errors since acually he head has no suddenly moved a hese ime insans (see frames 104 and 145). Thus, o some exen hese discrepancies can be considered as errors associaed wih he esimaed parameers. Whenever an eye blink occurs he pach wihou he eyes region has provided more accurae and sable parameers han he pach wih he eye region. This is explained by he fac ha he pach conaining he eye region (sclera and iris) does no model he shape of he eyelids. Therefore, despie he use of robus saisics he esimaion 1 hp://

7 J Real-Time Image Proc (2007) 2: Pich angle 4 Wih Eyes Region Wihou Eyes Region 2 0 Deg Frames 6 7 Yaw angle Wih Eyes Region Wihou Eyes Region 8 Deg Frames Scale Wih Eyes Region Wihou Eyes Region Frames 0.8 Open Eyelid moion Closed Frames Fig. 7 Tracking he 3D head pose, he lips, he eyebrows and he eyelids associaed wih a 1,000-frame sequence. Only frames 280, 284, 975 are shown. The plo depics he esimaed eyelid facial acion as a funcion of he sequence frames Pixels Y Translaion Wih Eyes Region Wihou Eyes Region of he 3D head pose parameers wih a pach conaining he eye region (sclera and iris) is affeced by he eyelid moion. One can noice ha he roaional discrepancies seem o be small (abou one degree). However, he verical and indeph ranslaion errors can be large. For example, a frame 145 he obained scale discrepancy is abou 0.025, which corresponds o an in-deph error of abou 3 cm Frames Fig. 6 3D head pose parameers obained wih wo differen facial paches ha differ by he eyes region 2 The exac value depends on he camera-inrinsic parameers and he absolue deph.

8 42 J Real-Time Image Proc (2007) 2:35 44 Fig. 9 A frame belonging o he low resoluion video depiced a he op of Fig. 8 zoomed views of hose frames. Noice how he eyelids are correcly racked in he inpu images. The esimaed eyelid facial acion reflecs he degree of he eye openness. The upper lef corner of each image shows he curren appearance ðl Þ and he curren shape-free exure ð^x Þ: The boom of his figure displays he esimaed eyelid facial acion as a funcion of he sequence frames where zero value corresponds o a closed eye and one value o a wide open eye. Figure 8 displays some racking resuls obained wih four video sequences. The firs video has a low resoluion. Figure 9 displays a snapsho of he low resoluion video. 6.1 Eye blink deecion Fig. 8 Tracking resuls associaed wih four video sequences. The firs one has a low resoluion 6 Head, lips, eyebrows, and eyelid racking In he previous secion, we have shown ha he accuracy of he racked 3D head pose parameers can be affeced by he eyelid moion (eye blinking) if he sclera and iris region is included in he exure model. This is no surprising since eye blinking corresponds o a sudden occlusion of a small par of he face. Thus, if he eyelid moion is racked one can expec ha he 3D head pose parameers can be more sable. We have racked he head, lips, eyebrows, and eyelids using he 1,000-frame sequence. Figure 7 displays he racking resuls (13 degrees of freedom) associaed wih frames 280, 284, and 975. The lef column displays Eye blinking is a discree and imporan facial acion [8, 13, 15]. The rae of blinking varies, bu on average he eye blinks once every 5 s. 3 In our case, he eye-blinks can be direcly deeced and segmened by hresholding he racked eyelid facial acion. As can be seen, he dual sae of he eye can easily be inferred from he coninuous curve associaed wih he eyelid facial acion. For he racked sequences, all eye blinks are correcly deeced and segmened. Figure 10 illusraes he racking resuls associaed wih a 594-frame sequence. This sequence depics a subjec performing head moions and facial animaions. This sequence depics 18 eye blinks. The proposed algorihm was able o correcly deec 17 blinks. The non-deeced blink happened a he same ime when he subjec pu on his glasses, ha is, he eyelids are suddenly occluded by he frame of he eyeglasses, which corresponds o a sudden appearance variaion. However, once he glasses are pu on all subsequen eye blinks are correcly deeced. 3 hp://

9 J Real-Time Image Proc (2007) 2: akes 333 ms on a Penium II 400 MHz PC. This mehod only racks he iris locaion as well as he eye sae. In [17], he average running ime for racking he eyelids and eye corners in one frame is abou 100 ms. However, his mehod only racks he eye corners and he eyelids. 7 Conclusion In his paper, we have exended our appearance-based 3D head and facial acion racker o deal wih eyelid moions. The 3D head pose and he facial acions associaed wih he lips, eyebrows, and eyelids are simulaneously esimaed in real-ime using OAMs. Compared wih oher eyelid racking echniques our proposed approach has several advanages. Firs, compuing and segmening inensiy edges has been avoided. Second, he eyelid racking does no depend on he deecion of oher eye feaures. Third, he eyelid moion is racked using a coninuous facial acion. Experimens on real video sequences including low-resoluion videos indicae ha he eye sae can be deeced using he eyelid racking resuls. Acknowledgmens The auhors hank Dr. Franck Davoine from CNRS, Compiegne, France, for providing he video sequence shown in Figure 10. References Fig. 10 Tracking resuls associaed wih one video sequence depicing 18 eye blinks Processing ime On a 3.2-GHz PC, a non-opimized C code of our proposed approach compues he 3D head pose and he seven facial acions in 70 ms. The edge-based mehod presened in [16] 1. Ahlberg, J.: Real-ime facial feaure racking using an acive model wih fas image warping. In: Inernaional Workshop on Very Low Birae Video (VLBV). Ahens, Greece (2001) 2. Ahlberg, J.: An acive model for facial feaure racking. EURA- SIP J. Appl. Signal Process. 2002(6), (2002) 3. Ahlberg, J: Model-based coding: exracion, coding, and evaluaion of face model parameers. Ph.D. hesis, No. 761, Linköping Universiy, Sweden (2002) 4. Cascia, M., Sclaroff, S., Ahisos, V.: Fas, reliable head racking under varying illuminaion: an approach based on regisraion of exure-mapped 3D models. IEEE Trans. Paern Anal. Mach. Inell. 22(4), (2000) 5. Cooes, T., Edwards, G., Taylor, C.: Acive appearance models. IEEE Trans. Paern Anal. Mach. Inell. 23(6): (2001) 6. Dornaika, F., Davoine, F.: On appearance based face and facial acion racking. IEEE Trans. Circuis Sys. Video Technol. 16(9): (2006) 7. Dornaika, F., Orozco, J., Gonzalez, J.: Combined head, lips, eyebrows, and eyelids racking using adapive appearance models. In: LNCS IV Conference on Ariculaed Moion and Deformable Objecs, pp (2006) 8. Grauman, K., Beke, M., Gips, J., Bradski, G.R.: Communicaion via eye blinks: deecion and duraion analysis in real ime. In: Inernaional Conference on Compuer Vision and Paern Recogniion (2001) 9. Jepson, A., Flee, D., El-Maraghi, T. Robus online appearance models for visual racking. IEEE Trans. Paern Anal. Mach. Inell. 25(10): (2003) 10. Lee, D.: Effecive Gaussian mixure learning for video background subracion. IEEE Trans. Paern Anal. Mach. Inell. 27(5): (2005)

10 44 J Real-Time Image Proc (2007) 2: Liu, H., Wu, Y., Zha, H.: Eye saes deecion from color facial image sequence. In: SPIE inernaional conference on image and graphics, vol. 4875, pp (2002) 12. Mahews, I., Baker, S.: Acive appearance models revisied. In. J. Compu. Vis. 60(2): (2004) 13. Moriyama, T., Kanade, T., Cohn, J., Xiao, J., Ambadar, Z., Gao, J., Imamura, H.: Auomaic recogniion of eye blinking in sponaneously occuring behavior. In: Inernaional Conference on Paern Recogniion (2002) 14. Sirohey, S., Rosenfeld, A., Duric, Z.: A mehod of deecing and racking irises and eyelids in video. Paern Recogni. 35(6): (2002) 15. Tan, H., Zhang, Y.J.: Deecing eye blink saes by racking iris and eyelids. Paern Recogni. Le. 27(6): (2006) 16. Tian, Y., Kanade, T., Cohn, J.F.: Dual-sae parameric eye racking. In: Inernaional Conference on Auomaic Face and Gesure Recogniion (2000) 17. Uzunova, V.I.: An eyelids and eye corners deecion and racking mehod for rapid iris racking. Maser s hesis, Universiy of Magdeburg, Zhou, S., Chellappa, R., Mogghaddam, B.: Visual racking and recogniion using appearance-adapive models in paricle filers. IEEE Trans. Image Process. 13(11): (2004) 19. Zhu, J., Yang, J.: Subpixel eye gaze racking. In: Inernaional Conference on Auomaic Face and Gesure Recogniion (2002)