ROBUST FACIAL EXPRESSION RECOGNITION USING SPATIALLY LOCALIZED GEOMETRIC MODEL

Transcription

1 Internatonal onference on Systemcs, ybernetcs and Informatcs, February 5, 004 ROBUST FAIAL EXRESSION REOGNITION USING SATIALLY LOALIZED GEOMETRI MODEL Ashutosh Saxena Department of Electrcal Engneerng IIT Kanpur Kanpur 0806, Inda Ankt Anand Dept. of omputer Sc. and Engg. IIT Kanpur Kanpur 0806, Inda rof. Amtabha Mukerjee Dept. of omputer Sc. and Engg. IIT Kanpur Kanpur 0806, Inda ABSTRAT An effcent, local mage-based approach for extracton of ntransent facal features and recognton of four facal expressons from D mage sequences s presented. The algorthm uses edge projecton analyss for feature extracton and creates a dynamc spato-temporal representaton of the face, followed by classfcaton through a feed-forward net wth one hdden layer. A novel transform for extractng lp regon for color face mages based on Gaussan modelng of skn and lp color s proposed. The proposed lp transform for colored mages results n better extracton of lp regon n the feature extracton stage. The algorthm acheves an accuracy of 90.0% for facal expresson recognton from grayscale mage sequences. ategores and Subject Descrptors [ybernetcs]: attern Recognton and Analyss, Artfcal Intellgence & Applcatons facal expresson recognton, color mages. General Terms Algorthms, Desgn. Keywords olor mages, face, facal expresson recognton, lp regon extracton.. INTRODUTION Identfyng human facal expressons has become an mportant feld of study n recent years because of ts nherent ntutve appeal and also due to possble applcatons such as humancomputer nteracton, face mage compresson, synthetc face anmaton and vdeo facal mage queres. opyrght 004 aper Identfcaton Number: -. Ths paper has been publshed by the entagram Research entre () Lmted. Responsblty of contents of ths paper rests upon the authors and not upon entagram Research entre () Lmted. Indvdual copes could be had by wrtng to the company for a cost. Whle approaches based on 3D deformable facal model have acheved expresson recognton rates of as hgh as 98% [], they are computatonally neffcent and requre consderable apror tranng based on 3D nformaton, whch s often unavalable. Recognton from D mages remans a dffcult yet mportant problem for areas such as mage database queryng and classfcaton. The accuracy rates acheved for D mages are around 90% [3,4,5,]. In a recent revew of expresson recognton, Fasel [] consders the problem along several dmensons: whether features such as lps or eyebrows are frst dentfed n the face (local [4] vs holstc []), or whether the mage model used s D or 3D. Methods proposed for expresson recognton from D mages nclude the Gabor-Wavelet [5] or Holstc Optcal flow [] approach. Ths paper descrbes a more robust system for facal expresson recognton from mage sequences usng D appearance-based local approach for the extracton of ntransent facal features,.e. features such as eyebrows, lps, or mouth, whch are always present n the mage, but may be deformed [] (n contrast, transent features are wrnkles or bulges that dsappear at other tmes). The man advantages of such an approach s low computatonal requrements, ablty to work wth both colored and grayscale mages and robustness n handlng partal occlusons [3]. Edge projecton analyss whch s used here for feature extracton (eyebrows and lps) s well known [6]. Unlke [6] whch descrbes a template based matchng as an essental startng pont, we use contours analyss. Our system computes a feature vector based on geometrcal model of the face and then classfes t nto four expresson classes usng a feed-forward bass functon net. The system detects open and closed state of the mouth as well. The algorthm presented here works on both color and grayscale mage sequences. An mportant aspect of our work s the use of color nformaton for robust and more accurate segmentaton of lp regon n case of color mages. The novel lp-enhancement transform s based on Gaussan modelng of skn and lp color. To place the work n a larger context of face analyss and recognton, the overall task requres that the part of the mage nvolvng the face be detected and segmented. We assume that a near-frontal vew of the face s avalable. Tests on a grayscale and two color face mage databases ([8] and [9,0]) demonstrate a superor recognton rate for four facal expressons (smle, surprse, dsgust and sad aganst neutral). 4

2 Robust Facal Expresson Recognton usng Spatally Localzed Geometrc Model. IDENTIFIATION OF FEATURE REGIONS We have used ntegral projectons [6] of the edge map of the face mage for extracton of facal features. Let I (x, y) be the nput mage. Vertcal and horzontal projecton vectors (fg ) n the rectangle [x, x ] x [y, y ] are defned as y y V ( x) H ( y) y y x x I( x, y) x x I( x, y) (a ) (b ) 5. The boundng box so obtaned s processed further to get an exact bnary mask of the feature. The followng sectons descrbe n detal the applcaton of ths generc algorthm for each partcular case of eyebrow, lp and nose... Eyebrow The approxmate boundng box s the top half of the face. The generc algorthm uses horzontal sobel edges to compute boundng box contanng eye and eyebrow. The segmentaton algorthm cannot gve boundng box for the eyebrow exclusvely because the edges due to eye also appear n the chosen boundng box. Brunell [6] suggests use of template matchng for extractng the eye, but we use another approach as descrbed below. Eyebrow s segmented from eye usng the fact that the eye occurs below eyebrow and ts edges form closed contours (fg ), obtaned by applyng Laplacan of Gaussan operator at zero threshold. These contours are flled and the resultng mage contanng masks of eyebrow and eye s morphologcally fltered by horzontally stretched ellptc structurng elements. From the two largest flled regons, the regon wth hgher centrod s chosen to be the mask of eyebrow. Fgure. Generc algorthm to get boundng box. () Edge map, () Vector H (y), (3) Vector V (x) n upper half porton, (4) Vector V (x) n lower half porton, (5) Orgnal mage. A typcal human face follows a set of anthropometrc standards, whch have been utlzed to narrow the search of a partcular facal feature to smaller regons of the face. We use the followng generc algorthm for the facal feature extracton from the localzed face mage. An approxmate boundng box for the feature s obtaned usng the anthropometrc standards.. Sobel edge map (fg.) s computed to obtan edges along the boundary of the feature. 3. The ntegral projectons V(x) and H(y) are calculated on the edge map (fg.,.3, and.4). 4. Medan flterng followed by Gaussan smoothng smooths the projecton vectors so obtaned. Hgher value of projecton vector at a partcular pont ndcates hgher probablty of occurrence of the feature. The relatve probablty E() of the th regon contanng the feature s calculated as y y E( ) + y y H ( y) w( y) () Where, y s the poston of the th regon; and w(y) s the gaussan weghng factor, calculated on the bass of anthropometrc standards. The regon wth maxmum E() gves the vertcal extent of the regon contanng the feature. Smlar approach s used for gettng the vertcal extent from the vertcal projecton V(x). Fgure. ost processng of approxmate boundng box to get exact bnary mask of eyebrow... Lp The approxmate boundng box s the lower half of the face. In case of colored mages, lp pxels sgnfcantly dffer from those of skn n Ybr color space. Therefore the color mage s preprocessed to produce pronounced demarcaton between lp and other skn regons (see secton 3 for the detals of the proposed lp enhancement transform). For the case of grayscale mages, no such preprocessng s appled. The generc algorthm calculates edge maps on the transformed mage. Edges for lps occur both n horzontal and vertcal drecton. In the boundng box computed by the generc algorthm, closed contours are obtaned by applyng Laplacan of Gaussan operator at zero thresholds. These contours are flled and morphologcally fltered usng ellptc structurng elements to get bnary mask for the lps..3. Nose The approxmate boundng box for the nose les between the eyes and the mouth. The generc algorthm uses vertcal sobel edges to compute the vertcal poston, whch s requred as a reference pont on face. 5

3 Internatonal onference on Systemcs, ybernetcs and Informatcs, February 5, LI ENHANEMENT TRANSFORM Sadegh [] proposed a lp segmentaton method based on Gaussan mxture modelng of mouth area mages, followed by Bayesan decson-makng system, whch labels the pxels as lp or non-lp. Ths approach gves a bnary mage, on whch usual feature extracton algorthms cannot be appled. Approach by Levn [3,4] usng Bayesan segmentaton also results n a bnarzed mage. To follow the usual feature extracton algorthms (for example as n [5]), the method would be to frst convert the vector (color) mage to a scalar (grayscale) mage by S + ar + ag a3b (3) Where (R,G,B) are the components n the RGB color space. Ths s followed by other operatons lke calculatng edge maps to fnd the lp regon n a color mage. Ths approach would use ntensty nformaton only. Ths compromses the accuracy of lp segmentaton, because ntensty dfference need not be sgnfcant between the lp and skn pxels, and lghtenng varatons may also be present. We propose a transform, based on Gaussan modelng of mouth area mages, to convert the vector (color) mage to a novel scalar mage, on whch further feature extracton algorthms can be appled. Ths method uses the chromatcty components n Ybr color space. 3.. Gaussan modelng of skn and lp color olor of skn and lp s a very mportant property that can be used for dentfcaton of the skn and lp regons. To model skn color, one has to look for color spaces n whch the dstrbuton of color components s concentrated n a small area. Researchers have looked at varous color spaces normalzed RGB, Ybr, HIS, etc. ha [7] has studed varous color spaces for modelng skn pxels. Skn and lp pxels are localzed n a small porton n a two-dmensonal b-r space. Note that the color space chosen should be such that t s ndependent of ntensty component. The sample dstrbutons were calculated expermentally from over randomly selected 93 mages (000 pxels for lp), and (40,000 pxels for skn). The skn and lp pxels n b-r space are confned n a small regon (fgure 3). On one extreme, dstrbuton of skn and lp each can be modeled by a smple vector ( b, r ), whch s the mean n b-r space. On the other extreme, one can model the dstrbuton by summaton of G gaussans. In former case, the dstrbuton could not be properly modeled. In latter case, the number of parameters ( 6 G for each) becomes very large, renderng the transform computatonally ntensve. We used a compromsng soluton, usng one two-dmensonal Gaussan to model each of skn and lp pxels as n [7]. Therefore, we get statstcally estmated mean of skn pxels and lp pxels as skn lp (, ) (4a) b r (, ) (4b) b r We get one covarance matrx each for skn and lp dstrbutons. The b and r components have very hgh crosscovarance br. The covarance matrces are gven by Skn Lp b br b br br r br r 3.. The proposed Transform (5a) (5b) Let p skn and p lp be apror probabltes of skn and color,.e., what s the expected fracton of skn and lp pxels n the chosen wndow. The probablty of an mage pxel to be lp ndependently s lp and the probablty of an mage pxel to be skn ndependently s skn. π exp ( I I Lp Lp ) Lp Lp Skn Lp exp ( I ) Skn Skn I π Skn Where, Lp and Skn are estmated the covarance matrces n br space. I (b,r) s the mage pxel under consderaton. Skn T T (6) (7) The lp regon s surrounded by skn area. Hence, probablty of an mage pxel to be a lp-pxel n presence of skn pxels s gven by Lp: b Lp: r Skn: b Skn: r Fgure 3. Sample dstrbuton of lp and skn pxels along b and r axs. p p Lp / Skn Lp Lp Skn Skn The above value s normalzed to gve S [ 0,] S s shown n fgure 4. (8). The plot of { ( )} max( ) (9) S lp / skn mn lp / skn lp / S 3 Y log + KS log + K skn ( ) ( ) (0) 6

4 Robust Facal Expresson Recognton usng Spatally Localzed Geometrc Model Where, Y s the Y-component of Ybr representng ntensty (smlar to grayscale), and K s a parameter that decdes the relatve mportance to be gven to color nformaton. A large value of K mples less mportance to color and more mportance to ntensty nformaton. S 3 s the scalar value of the pxel n the transformed mage. Thus, calculatng ths transform for each pxel, we get a new mage S 3 that s lp-enhanced. To localze lp regon n the face mage, K should be small (e.g. K ). However, to study features nsde lp or fnd exact curves after localzaton of lp, K should be made large (e.g. K 00). Thus, transform has the flexblty gven varable level of features wthn lp. Whle calculatng edges ether for segmentng lps or for approxmatng lps wth curves, t s desrable that the edges occur only at the boundary of lp and skn but not n other regons of the face. The equatons (8), (9) and (0) tlt the projecton plane n such a way that the gradent whle movng from a skn pxel to a lp pxel s maxmzed n ths plane. Therefore, the lp and skn regons are demarcated more accurately. The gradent whle movng from skn pxel to a lp pxel s shown by an arrow n fgure 4. Not only the gradent s maxmzed, but also the absolute value of S (eq. 9) s maxmum for the lp pxels. Ths makes the transform sutable for varous applcatons lke edge projecton, template based methods, or bnarzng by thresholdng. revously [3-6], Mahalnobs dstance (.e. the dstance from the Gaussan) served as a crteron to dentfy a lp pxel, wthout nvolvng any concept of maxmzng the gradent. The mprovement n lp regon segmentaton on applyng the lp enhancement transform s shown n fgure FEATURE VETOR AND LASSIFIATION A spato-temporal representaton of the face s created, based on geometrcal relatonshps between features usng Eucldean dstance (fgure 6). Such a representaton allows robust handlng of partal occluson. Seven parameters form the feature vector F F {He, W e, Hm, Wm, R ul, R ll, NL} () All components of the vector are normalzed aganst the frst frame to acheve scale ndependence. Rad of curvature of the upper and lower lps R ul and R ll are computed by approxmatng the bnary mask of the lps wth two parabolas. N L s the number of dstnct peaks detected for upper and lower lps durng edge projecton analyss, ndcatng whether mouth was open or closed. The change n feature vector F when the face undergoes change from neutral state to some expressonal state F { H e, W e, H m, Wm, R ul, R ll, N L} () Such dynamc characterstc of the feature vector provdes shapendependence. F serves as an nput to the classfer. The classfer s a feed-forward bass functon net wth one hdden layer (fgure 7). The actvaton functon (fgure 8) mplemented n the hdden layer s Fgure 4. ontours for Normalzed S n b-r space. Legend: Whte, Black 0. Note the arrow shows the movement from an deal skn pxel to an deal lp pxel. Gradent s maxmzed n the drecton ndcated by the arrow. Fgure 5. Frst Row: (A) olor mage converted to grayscale, (B) Edges for A, () Bnary mage obtaned by thresholdng A. Second Row: (A) Transformed color mage, (B) Edges for A, () Bnary mage obtaned by thresholdng A. Notce the mprovement n the second row, n terms of edges for lps, and bnarzed mage. H e : Heght of eyebrow W e : brows dstance H m : mouth heght W m : mouth wdth R ul : upper lp curvature R ll : lower lp curvature Fgure 6. Geometrcal parameters of the face, formng the feature vector. 7

5 Internatonal onference on Systemcs, ybernetcs and Informatcs, February 5, 004 Where σ s the varance of th component of F. As compared to the standard sgmod functon, ths actvaton functon has a small value n a larger nterval near zero. Thus, t makes the net tolerant towards small errors present n F. For output layer, O Y x ) exp x σ sgn( j 7 w O j (3) (4) Where, weghts w j are correlaton coeffcents between O and desred output Y j. The output at each node Y j gves the confdence level of the correspondng expresson. Fgure 7. The bass functon net for classfcaton. Fgure 8. The actvaton functon for the hdden layer. 5. RESULTS Experments were performed on colored as well as grayscale mage databases. ohn Kanade [8] database conssts of grayscale mage sequences depctng the four facal expressons smle, surprse, sad and dsgust. The subjects vared n ethncty, age and skn color. A set of 50 sequences (85 mages) was randomly selected as test samples. Ths database was used to test the accuracy of the facal expresson recognton algorthm. To test the effectveness of lp enhancement transform n mprovng lp regon extracton n case of colored mages, 93 mages from AR [9] and VL [0] database (contanng consderable lghtenng varatons) were used. In the followng sectons, we descrbe the accuracy obtaned n the conducted tests. 5.. Feature Extracton The accuracy of facal feature extracton for colored and grayscale mages s shown n fgure 9. In case of colored mages, the lp enhancement transform was appled to the mages. An average accuracy of 9.% was obtaned for grayscale mage database (data sze85 mages). In case of color mage database, a slghtly better accuracy of 95.4% was obtaned (data sze 93). % orrect Recognton Left Eyebrow Rght Eyebrow Lp Fgure 9. Accuracy for facal feature extracton 5.. Facal expresson recognton olored Grayscale Each grayscale mage sequence n the database depcted one of the expresson classes (smle, surprse, sad and dsgust aganst neutral). The frst mage n the sequence was a neutral mage. onfdence level of each expresson was calculated for each of the subsequent mages aganst the neutral mage. The calculated vector of confdence levels was added to gve total confdence for each of the expressons. Expresson havng the hghest total confdence level was declared as the expresson of the sequence. On a test set of 50 sequences (85 mages), an accuracy of 90.0% was acheved for grayscale mages (fgure 0). Bourel [3] has reported an accuracy of 89% over the same database. Ther system has used feature pont trackng followed by k-nearest neghbor algorthm. The holstc approach of Yacoob [] has reported an accuracy of 89% Smle (5) Surprse (5) Sad (0) Dsgust (0) All (50) Fgure 0. Accuracy for grayscale mage sequences. 6. ONLUSION An effcent, local mage-based approach for extracton of ntransent facal features and recognton of four facal expressons was presented. A lp-enhancement transform for better segmentaton of lp regon n color mages was proposed. Our system shows superor performance n comparson to the other facal expresson recognton systems. The system requres no manual nterventon (lke ntal manual assgnment of feature ponts, as n system descrbed by Bourel [3]). The system, based on a local approach, s able to detect partal occlusons also. 8

6 Robust Facal Expresson Recognton usng Spatally Localzed Geometrc Model 7. AKNOWLEDGEMENT We thank Dr. Sumana Gupta for helpng us n formalzng the statstcal approach for lp enhancement transform. 8. REFERENES [] B. Fasel, et al. Automatc Facal Expresson Analyss: A Survey. attern Recognton, 36, 59-75, 003 [] I. Essa, A. entland. odng, analyss, nterpretaton and recognton of facal expresson. IEEE Trans. attern Anal. Mach. Intel., 9(7), , 997. [3] F. Bourel, et al. Robust Facal Expresson Recognton Usng a State-Based Model of Spatally-Localsed Facal Dynamcs. roc Ffth IEEE Int l onf on Automatc Face and Gesture Recognton (FGR-0), Washngton D.., 3-8, 00. [4] M. Black, et al. Recognzng facal expressons n Image Sequences usng local parameterzed models of mage moton. Int l J. omp Vson, 5(), 3-48, 997. [5] Z Zhang, et al. omparson Between Geometry-Based and Gabor-Wavelets-Based Facal Expresson Recognton Usng Mult-Layer erceptron. roc Thrd IEEE Int l onf on Automatc Face and Gesture Recognton, Nara, Japan, , 998. [6] R. Brunell, et al. Face Recognton: Feature versus Templates. IEEE Trans attern Anal. Mach. Intel., 5(0), 04-05, Oct 993. [7] D. ha, et al. Locatng Facal Regon n Head and Shoulder colored mage. roc. Thrd IEEE Int l onf of Automatc Face and Gesture Recognton, Nara, Japan, 4-9, 998. [8] Kanade T., et al. omprehensve Database for Facal Expresson Analyss. roc. Fourth IEEE Int l onf on Automatc Face and Gesture Recognton (FG 00). Grenoble, France. March 000. [9] Martnez AM, et al. The AR Face Database. V Tech Report #4, 998. [0] VL Face Database. Avalable: [Onlne] ŠV, TERŠ, Velenje. [] Y Yacoob, et al. Recognzng Human Facal Expressons from Long Image Sequences Usng Optcal Flow. IEEE Trans on attern Anal. Mach. Int., vol 8, no. 6, 996. [] Sadegh M., et al. Segmentaton of Lp pxels for lp tracker ntalzaton. IEE roc on Vson, Image and Sgnal roc., vol 49 (3), 79-84, June 00. [3] Levn, M.; Luthon, F. Lp features automatc extracton. roceedngs Internatonal onference on Image rocessng, II 98, vol. 3, 68-7, 4-7 Oct [4] Levn, M.; Delmas,.; oulon,.y.; Luthon, F.; Frstol, V. Automatc lp trackng: Bayesan segmentaton and actve contours n a cooperatve scheme. IEEE Internatonal onference on Multmeda omputng and Systems, vol., , 999. [5] Kaucc, R.; Reynard, D.; Blake, A. Real-tme lp trackers for use n audo-vsual speech recognton. IEE olloquum on Integrated Audo-Vsual rocessng for Recognton, Synthess and ommuncaton (Dgest No: 996/3), 3/ - 3/6, 8 Nov [6] Ramos Sanchez, M.U.; Matas, J.; Kttler, J. Statstcal chromatcty-based lp trackng wth B-splnes. IEEE Internatonal onference on Acoustcs, Speech, and Sgnal rocessng IASS-97, vol. 4, , -4 Aprl