Facial Feature Tracking and Expression Recognition for Sign Language

Transcription

1 Facial Feature Tracking and Expression Recognition or Sign Language İsmail Arı Computer Engineering Bo gazic. i University İstanbul, Turkey ismailar@boun.edu.tr Asli Uyar Computer Engineering Bo gazic. i University İstanbul, Turkey asli.uyar@boun.edu.tr Lale Akarun Computer Engineering Bo gazic. i University İstanbul, Turkey akarun@boun.edu.tr Abstract Expressions carry vital inormation in sign language. In this study, we have implemented a multi-resolution Active Shape Model(MR-ASM) tracker, which tracks 116 acial landmarks on videos. Since the expressions involve signiicant amount o head rotation, we employ multiple ASM models to deal with dierent poses. The tracked landmark points are usedtoextractmotioneatureswhichareusedbyasupport Vector Machine(SVM) based classiier. We obtained above 90% classiication accuracy in a data set containing 7 expressions. I. INTRODUCTION There has been a growing interest in extracting and tracking acial eatures and understanding the perormed expression in automatically. Many approaches[1],[2] include three distinct phases: First, beore a acial expression can be analyzed, theacedetectionmustbedoneinascene.thisprocessis ollowed by the extraction o the acial expression inormation rom the video and localizing(in static images) or tracking (in image sequences) these eatures under dierent poses, illumination, ethnicity, age and expression. The outputs o thisprocessaregivenastheinputortheollowingstep whichistherecognitionotheexpression.thisinalstepisa classiication problem where the expression is classiied into one o the predeined classes o expressions. A. Face Detection Inmostotheresearch,aceisalreadycroppedandthe system starts with tracking and eature extraction. In others, vision-based automated ace detectors[3],[4] or pupil tracking with inrared(ir) cameras[5] are used to localize the ace. Alternatively, a ace detector can be used to detect the aces in a scene automatically[6]. B. Facial Feature Extraction and Tracking Numerouseaturesweretriedinordertomakeabetter recognition o the acial expression. Image-based models rely onthepixelvaluesothewholeimage(holistic)[7]orrelated parts o the image(local)[8]. On the other hand, model-based approaches create a model that best represents the ace by using training images[4],[9]. Moreover, dierence images are used to ind the eye coordinates rom the image pairs gathered by IR cameras[10]. Statistical model-based approaches have three main components: capture, normalization and statistical analysis. Inthecapturepart,onedeinesacertainnumberopoints (landmarks)onthecontourotheobjectortheshapeand uses image warping or the texture. The ollowing shape normalization is done using Procrustes Analysis and texture normalization is done by removing global illumination eects between rames. Finally, Principal Component Analysis(PCA) is perormed to analyze the correlations between object shapes or textures and this inormation is also used or synthesis. Active Shape Models(ASM) and Active Appearance Models (AAM) are two widely used statistical approaches. AAM approach is used in acial eature tracking due to its ability in detecting the desired eatures[9]. In addition, ASMswhicharethesimplerversionotheAAMsthatonlyuse shape inormation and the intensity values along the proiles perpendicular to the shape boundary- are also used[11] to extract eatures. Because o the diiculty in describing dierent ace views using a single model, view-based AAMs are also proposed[12]. C. Facial Expression Recognition It is a common approach to use variants o displacement based motion eature vectors or expression recognition. Generating a motion displacement vector or each pixel in the image[13], measuring the total amount o motion relative to eachothethreeaxes[14],andconverting3dpositiondata into a representation o motion composed o displacement vectors[15] are some eature extraction methods used in gesture recognition. The extracted eatures are input to the expression classiier. Some popular machine learning algorithms used in classiication are Hidden Markov Models(HMM), Support Vector Machines(SVM), Bayesian networks, decision-tree based classiiers and neural networks[1],[2],[16],[17]. D. Sign Language Scope Sign language expressions are composed o manual(hand gestures) and non-manual components(acial expressions, head motion, pose and body movements). Some expressions are perormed only using hand gestures whereas some change the meaning where a acial expression accompanies hand gestures. Thereore, a robust high-perormance acial eature tracker and acial-expression classiier is a must in sign language recognition. Although most o the sign language

2 recognition systems rely only on hand gestures and lack nonmanualcomponentsoasign[14],[18],therearealsosome uniied systems[19]. sampleshapespaceφ s containingshapess i where s i =(x 1,y 1,x 2,y 2,...,x L,y L ),i=1,...,n is a shape containing the coordinates o the landmarks shown infigure2. Fig. 2. Selected Feature Points Secondly, shape normalization is done using Procrustes Analysis where we choose the irst shape as the reerence shapeandalignallotherstoit.thenwerecalculatethe estimateothemeanromallandrepeattheprocessuntil convergence. Finally, PCA is applied to the normalized shape space to ind the orthogonal basis vectors satisying Fig.1. Flowchartotheproposedsystem The subject o this research is eature point based expression recognition in sign videos. The system low o our approach is illustrated in Figure 1. We employ state-o-the-art techniques such as ASMs or tracking and SVMs or classiication to deal with this problem. Since standard ASMs have diiculty dealing with extreme poses[12], we train multiple viewbased ASMs to track these diicult poses. Section II describes the details o our tracking approach. Section III gives our expression classiication methodology. Section IV gives the properties o the database we used and the experiments we perormed on this database with the results are given. We conclude in Section V with relevant discussion. II. FACIAL POINT TRACKING A. Statistical Analysis o Samples The statistical analysis o shapes can be divided into three distinct steps; capture, normalization and PCA. We start with annotating L eature points manually or each othenrandomlycapturedramesromvideosandcreatethe where C s e k =λ k e k (1) C s isthecovariancematrixconstructedusingnormalized shapes e k isthek th eigenvector λ k isthek th eigenvalue Anyshapescanalsobedescribedusingalleigenvectorsin a lossless way and the coeicients o eigenvectors orm the parameter vector b. where s= 1 N b=e T 2L(s s) (2) s= s+e 2L b (3) N s i, E 2L = [ ] e 1... e 2L i=1 The motivation behind applying PCA is to reduce the dimension and use K < 2L eigenvectors, yet preserving the mostothevariation.kischosenwhereitsatisies K λ k 0.95 k=1 2L λ k k=1 Letλ 1,...,λ k betheirstkeigenvectors.thenwith ˆb=(b 1,...,b K ),wecansynthesizeŝwhichisanestimate osthatissimilartotheshapesinφ s. ˆb=E T K(s s) (4) ŝ= s+e Kˆb (5)

3 B. Creating a Model rom Landmark Proiles Let p j,i bethej th landmarkinthei th shape,suchthat p j,i =(x j,i,y j,i ).g j,i isthegradientopixelintensitiesalong theproileop j,i asinfigure3. F. Tracking in an Image Sequence Since there is head motion in addition to acial expressions insignvideos,asingleviewmodelisnotsuicientor handling all views o the ace. So, three dierent models are trained which are or rontal, let and right views. Sample imagesareshowninfigure4.weusetwometricstocalculate the spatial and temporal errors to choose the best itting model during tracking. Fig. 3. Proiles o landmarks Thenwecalculateḡ j asthemeangradientvectorandc j as the covariance o gradients. Thus a single model is composed oparticular s,e K,ḡ j andc j (j=1,...,l). C.FittingaShapetoaTestImage The initialization is done by detecting the ace using OpenCV sacedetector[6]and sisplacedontheoundace. Then the shape is iteratively perturbed along the proile until convergence. Each iteration involves two steps as ollows 1)FindingtheBestFit:Letussaynistheproilewidth andmisthesearchwidthasinfigure3wherem>n. Foreachlandmark,weindthebestitalongtheproile wherethebestproilegradientĝ j givestheminimummahalanobis distance with the model, i.e. the term (ĝ j ḡ j ) T C 1 j (ĝ j ḡ j ) (6) is minimized. 2)ConstrainingtheBestFit:Thebestitisconstrainedby indingtheapproximateshapeparametersˆbusingeq.(4)and constrainingeachcoeicientb k satisying 3 λ k b k 3 λ k ork=1,...,k.thatis,ithevalueisoutothe allowed limits, then it is changed to the nearest allowed value. Finally, the constrained parameters are projected back to ŝ using Eq.(5).This way, the itted shape avoids deormation andwillbesimilartotheonesinφ s. D. Multi-resolution Approach InsteadousingasinglelevelASMsearch,amodelis created or each level o the pyramid where the original size images are in the lowest level and higher models involve sub-sampled images. The search is irst done at the highest levelandtheoundshapeispassedtothelowerlevelasthe initialshapeorthatlevel.soaroughestimateisoundinthe highest level and ine-tuned at each level it goes through. This procedure is called Multi-resolution ASM(MRASM). E. Data Reitting SinceΦ s isgatheredbyclickingeaturepointsmanually, the training set is error-prone to human mistakes. To reduce this bias, data reitting is perormed. So, a model is trained usingφ s.then,eachshapeisinitializedbyitsownshape and reitted using this model. Finally, a new model is trained byusingtheittedshapesasdescribedin[20]. Fig. 4. Three dierent views rms g,rontal,rms g,let andrms g,right aretherootmean square errors between the itted shape s proile gradient values and the used model s(rontal, let or right) mean gradient values.so,theygiveasimilaritymeasureorustodecideon a model view. rms, 1 istherootmeansquareerroroshapesoundin th and( 1) th rame,where=2,...,f.itinormsus aboutthechangeinshapeinthegiveninterval.weassume thattheirstrameisrontal,sowestartwithasingleview MRASMsearchor=1.Thenthealgorithmis or=2,...,fdo or each model(rontal, let, right) do apply MRASM search; end eliminatethemodelsgivingrms g,model >threshold; inomodelremainsthen mark this rame as empty(not ound); re-initialize the tracker; else choosethemodelgivingtheminimumrms, 1 ; settheoundshapeasinitialshapeornext iteration; end end Finally, the empty rames are illed by interpolation and α-trimmed mean ilter is applied to eliminate the spikes encountered during tracking. III. EXPRESSION CLASSIFICATION Classiication is the inal stage o the acial expression recognition system and consists o two sub-stages which are motion eature extraction and classiication using SVM. Since thedatahavethenatureoatimeseries,oneneedstoeither normalize it or extract eatures that would represent the whole sequence. Motion history images used by Michel and Kaliouby [16]isonesucheature.Here,weollowasimilarapproach and use maximum displacements. The tracker extracts the coordinates o acial landmarks in consecutive rames o the video sequences. Then these coordinates are used to evaluate

4 the maximum displacement values or each eature point in ourdirectionsx +,x,y + andy acrosstheentireimage sequence. A. Motion Feature Extraction We use displacement based and time independent motion eaturevectorastheinputtothesvmclassiier.themotion eature vector includes inormation about both the magnitude and the direction o motion or each landmark. A similar displacement based approach has been applied in [16] to extract the motion inormation between the initial and the peakrames inimagesequences.inourstudy,weind the maximum displacement o points where peak location oreachpointmaybeindierentrames. WedeineV i asthei th video V i = [ s i 1, s i 2, s i ] F (7) where s i = ( ) x i,1 y i,1 x i,2 y i,2 x i,l y i,l isthesetolandmarksinthe th rameoi th video. Foreachvideo,theinitialrame(s 1 )ischosenasthe reerence rame and the displacements o the points between each rame and the reerence rame have been measured. Then, the maximum displacement values o each point in our directions have been chosen as the motion eatures. dx i,l max=max x i,l xi,l 1 dx i,l min =min x i,l xi,l 1 dymax i,l =max y i,l yi,l 1 dy i,l min =min y i,l yi,l 1 Theoutputothisprocessisasinglemotionvectorz i or each video. ( z i = dx i,1 max dxi,l max dxi,1 min dxi,l min (9) ) dymax i,1 dymaxdy i,l i,1 min dyi,l min (8) order to achieve a linear solution. This mapping is done using kernel unctions. Final decision unction is in the orm: ( ) (x)= α i y i K(x i x)+b (10) i wherek(x i x)isthekerneltransormation.thetraining sampleswhoselagrangecoeicients α i arenon-zeroare called support vectors (SV) and the decision unction is deined by only these vectors. A. Used Database IV. EXPERIMENTS& RESULTS WeusedthedatabaseoAranetal.[17]whichinvolves8 dierent classes o non-manual sign videos. The classes(as showninfigure5)arebriely 1) neutral expression 2)headshakingtoletandright(usedornegation) 3) head up, eyebrows up(used or negation) 4) head orward(used when asking a question) 5) lips turned down, eyebrows down(sadness) 6)headupanddown(usedoragreement) 7) lips turned up(happiness) 8) classes 6 and 7 perormed together(happy agreement) B. Classiication Using SVM Because o the superior classiication perormance and its ability to deal with high dimensional input data, SVM is the choice o the classiier in this study or acial expression recognition. A motion eature vector is extracted in the previous stage and classiied into one o the predeined expression classesinthisstage.abriedeinitionosvmisgivenbelow. Givenasetotrainingdatapairs(x i,y i ), y i {+1, 1}, theaimothesvmclassiieristoestimateadecision unction by constructing the optimal separating hyperplane in theeaturespace[21].thekeyideaosvmistomapthe original input space into a higher dimensional eature space in Fig.5. Theclassesinthedatabase Thereare5repetitionsoreachclassrom11(6emale, 5 male) dierent subjects and each video takes about 1-2 seconds. B. Facial Point Tracking We selected 2 subjects and trained rontal, let and right models or each. So, we trained a person-speciic multiviewasmrom35rontal,15letand15rightimagesper

5 subject. These images were randomly selected rom videos and manually annotated. A ew(8-10) eigenvectors seemed to beenoughtodescribe95%othetotalvariationineachmodel (Figure6). Fig.8. Foundlandmarksinavideothatbelongsto3 rd class Fig. 6. Eigenvector contributions to the total variance ForMRASMsearch,weused3levelsandn={13,9,9} andm={15,13,13}aretheparameterswechoseorl= {0,1,2}respectively.Foreachlevel,atmost4iterationsare allowedandtheconvergenceratiorisselectedas90%othe number o landmarks. SampletrackingresultscanbeseeninFigures7and8 wheretheindexonthetoprightstandsortheramenumber in that video. Fig.7. Foundlandmarksinavideothatbelongsto2 nd class C. Expression Classiication We perormed the classiication experiments by LIBSVM [22]. Gaussian kernel has been the kernel choice and the kernel parameters cost and gamma have been searched in the ranges [ 2 5,2 15] and [ 2 15,2 3] respectively. We prepared the ollowing sets or our experiments: Φ 1 andφ 2 :Eachinvolves7classesand5repetitionsor a single subject ound with the tracker. The class numbers are2to8.(35sampleseach) Φ 1,2 :Involves7classesand5repetitionsortwosubjects oundwiththetracker.theclassnumbersare2to8.(70 samples) Φ gt :Involves4classesand3repetitionsoreacho9 subjects(excluding the subjects tracked) in the ground truthdata.theclassnumbersare2,3,4and7.(108 samples) Then we designed three tests on these sets: I-UseΦ 1 ortrainingandφ 2 ortestingandviceversa. II-Use Φ 1,2 ortrainingandtestingby5-oldcrossvalidation. III-UseΦ gt ortrainingandφ 1,2 ortesting. TestsIandIIareperormedusingboth4and7classesto compare with Test III. The accuracy results ound with the best parameters are shownintablei. Itisobservedthatthebestaccuracyisobtainedwhen training data includes instances o the test subject(test II). When we train the system with a dierent person, perormance drops(testi).however,whenthenumberosubjectsinthe training set is increased, high perormance person-independent expression classiication is possible(test III). Since we have tracking results o only two subjects, we have used handlabeled landmarks in the training set o Test III. Perormance is expected to increase urther i tracked landmarks are used instead. The conusion matrix showing the results(in percentage) or theworstcasearegivenintableii.overallaccuracyis67.1%. It is observed that the most conused expressions are classes 5-7(happiness-sadness) and 6-8(agreement-happy agreement). V. CONCLUSION In this study, we presented an automatic acial expression and head motion recognition system. First, we have tracked

6 TABLE I EXPRESSION CLASSIFICATION RESULTS Optimized Gaussian Dataset Training Options # expression # training # test Kernel Parameters Accuracy(%) classes samples samples Cost Gamma Automatically TestI Tracked Features Test II Ground Truth Test III Features TABLE II CONFUSION MATRIX FOR 7 EXPRESSION CLASSES(TEST I) Class acial eature points in prerecorded non-manual sign videos by using multi-view MRASM. Then maximum displacement oeachlandmarkhavebeenusedasacomponentomotion eature vector and these vectors are given to SVM classiier as input. Inthetrackingpart,weusedasinglesubject3-view MRASM(rontal, let and right). The results were promising andweshowedawaytotrackacialeaturepointsrobustly. The proposed tracking system is easily extendible by training themodelormorethanthreeviewsandmoresubjects.since itisbasedonasm,thetimecomplexityislessthanaam approach because it uses only the intensity values along the landmark proiles. WehavetestedtrackerandSVMbasedclassiieron7 expressions rom Turkish Sign Language. We have obtained above 90% person independent recognition accuracy. Our utureworkwillocusonmakingthesystemworkinrealtime and integrate the expression classiication in our sign language recognizer. ACKNOWLEDGMENT This study has been supported by TUBİTAK under project 107E021. REFERENCES [1] M. Pantie and L. Rothkrantz, Automatic analysis o acial expressions: the state o the art, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 22, no. 12, pp , [2] B. Fasel and J. Luettin, Automatic acial expression analysis: A survey, Pattern Recognition, vol. 36, no. 1, pp , [3] L. Jordao, M. Perrone, J. Costeira, and J. Santos-Victor, Active ace and eature tracking, Proceedings o the 10th International Conerence on Image Analysis and Processing, pp , [4] J. Ahlberg, An active model or acial eature tracking, EURASIP Journal on Applied Signal Processing, vol. 2002, no. 6, pp , [5] Y. Zhang and Q. Ji, Active and dynamic inormation usion or acial expression understanding rom image sequences, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 27, no. 5, pp , [6] I. Inc, The opencv open source computer vision library. [7]X.Wei,Z.Zhu,L.Yin,andQ.Ji, Arealtimeacetrackingand animation system, Computer Vision and Pattern Recognition Workshop, 2004 Conerence on, pp , [8] J. Lien, T. Kanade, and J. Cohn, Automated acial expression recognition based on acs action units, Automatic Face and Gesture Recognition, Proceedings. Third IEEE International Conerence on, pp , [9] T. Cootes, G. Edwards, and C. Taylor, Active appearance models, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp , [10]A.KapoorandR.Picard, Areal-timeheadnodandshakedetector, Proceedings o the 2001 workshop on Percetive user interaces, pp. 1 5, [11] G. Votsis, A. Drosopoulos, and S. Kollias, A modular approach to acial eature segmentation on real sequences, Signal Processing: Image Communication, vol. 18, no. 1, pp , [12] T. Cootes, G. Wheeler, K. Walker, and C. Taylor, View-based active appearance models, Image and Vision Computing, vol. 20, no. 9-10, pp , [13] T. Gee and R. Mersereau, Model-based ace tracking or dense motion ield estimation, Proceedings o the Applied Imagery and Pattern Recognition Workshop, [14] P. Vamplew and A. Adams, Recognition o sign language gestures using neural networks, Australian Journal o Intelligent Inormation Processing Systems, vol. 5, no. 2, pp , [15] H. Hienz, K. Grobel, and G. Oner, Real-time hand-arm motion analysis using a single video camera, Proceedings o the Second International Conerence on Automatic Face and Gesture Recognition, pp , [16] P. Michel and R. El Kaliouby, Real time acial expression recognition in video using support vector machines, Proceedings o the 5th international conerence on Multimodal interaces, pp , [17]O.Aran,I.Ari,A.Guvensan,H.Haberdar,Z.Kurt,I.Turkmen, A.Uyar,andL.Akarun, Adatabaseonon-manualsignsinturkish sign language, Signal Processing and Communications Applications, 2007.SIU2007.IEEE15th,pp.1 4,2007. [18] S. Ong and S. Ranganath, Automatic sign llanguage analysis: A survey and the uture beyond lexical meaning, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 27, no. 6, pp , [19]O.Aran,I.Ari,A.Benoit,A.Carrillo,F.Fanard,P.Campr,L.Akarun, A. Caplier, M. Rombaut, and B. Sankur, Sign language tutoring tool, Proceedings o enterface 2007, The Summer Workshop on Multimodal Interaces, vol. 21, [20] R. Gross, I. Matthews, and S. Baker, Generic vs. person speciic active appearance models, Image and Vision Computing, vol. 23, no. 12, pp , [21] C. Burges, A tutorial on support vector machines or pattern recognition, Data Mining and Knowledge Discovery, vol. 2, no. 2, pp , [22] C. Chang and C. Lin, Libsvm: a library or support vector machines, 2001.