Facial Expression Recognition Based on Histogram Sequence of Local Gabor Binary Patterns

Transcription

1 Facial Expressio Recogitio Based o Histogram Sequece of Local Gabor Biary Patters Xighua Su, Hogxia Xu, Chuxia Zhao, ad Jigyu Yag School of Computer Sciece ad Techology Najig Uiversity of Sciece ad Techology Najig, Chia xighuasu@vip.163.com Abstract Durig the drivig, the good emotio ca beefit the vehicle safety. The good emotio will result i the certai facial expressios ad vice versa. The facial expressio is used as a useful cue to perform the surveillace of driver's status. Cosiderig the characteristics of drivig safet the facial expressios of ager, happiess, sadess ad fear are ivestigated. The mai cotributio of this paper is to preset a ew facial expressio recogitio method based o the histogram sequece of local Gabor biary patters. First, the Gabor Coefficiets Maps (GCM are extracted by covolvig the face image with the multi-scale ad multi-orietatio Gabor filters. The, the local biary patter operator is performed o each GCM to extract the local Gabor biary patter. Next, the face image is described usig the histogram sequece of all these local Gabor biary patters. Fiall the multi-class Support Vector Machie (SVM is used to perform the feature classificatio. The experimetal results show that the facial expressio recogitio algorithm proposed i this paper is effective ad superior to the other similar methods both i the recogitio rate ad speed. Keywords facial expressio recogitio, Gabor coefficiets maps, local Gabor biary patter, support vector machie I. INTRODUCTION Facial expressio, as a kid of body laguage, ca reflect the huma emotio effectively [1]. Based o the emotio observatio of oe perso, we ca get to kow some iformatio about his or her mood, feelig ad persoality etc. I 1971, Ekma ad Friese preseted that the huma emotio ca be divided ito six types, icludig the happiess, sadess, fear, disgust, surprise ad fear. Each of these six basic emotios correspods to a certai type of facial expressio. Besides the eutral expressio, these seve types of expressios have bee studied widely i the world. So far the facial expressio recogitio, as a importat techolog has bee applied i may domais, such as psycholog image uderstadig, vehicle drivig safet video retrieval, virtual reality ad huma computer iteractio. Our research domai is the vehicle safety ad the facial expressio is used as a useful cue to perform the surveillace of driver's status. Cosiderig the characteristics of drivig safet the facial expressios of ager, happiess, sadess ad fear are ivestigated i this paper. Our objective is to recogize these four types of facial expressios effectively ad efficietly. Accordig to [], the techiques of facial expressio recogitio ca be divided ito two types, oe is based o the static image ad aother is based o the dyamic video. To the former type, we recogize the facial expressio just from the sigle static image ad the available iformatio is very limited. The correspodig methods iclude the Pricipal Compoet Aalysis (PCA [3], Idepedet Compoet Aalysis (ICA [4] ad the oe based o wavelet trasformatio. To the latter type, the facial expressio ca be recogized based o the multiple frames of video. The model of optical flow [5], as oe of the latter correspodig methods, ca extract the dyamic features of facial expressio. Accordig to the existig practical applicatios, it ca be leared that the former techique is simple ad ca be used i the real-time situatio. The latter techique costs the high computatio ad caot satisfy the real-time requiremet. This paper performs the facial expressio recogitio based o the static image. Zhu et al. [6] proposed a method combiig the Gabor filter ad Support Vector Machie (SVM. The achieved recogitio rate is high, but the computatioal cost is also high ad oly the local texture iformatio is adopted. I [7], the local biary patters ad SVM are applied i the expressio recogitio. Due to the dimesio of local biary patters beig lower tha the oe of Gabor features, the computatioal cost of the proposed method i [7] has bee reduced quite a lot. To combie the advatages of the two methods proposed i [6] ad [7], this paper proposes a ew facial expressio recogitio method combiig the Gabor coefficiets with the histogram of Local Biary Patter (LBP [8]. First, the Gabor Coefficiets Maps (GCMs are extracted by covolvig the face image with the multi-scale ad multiorietatio Gabor filters. The usage of GCM ca reduce the dimesio of feature vectors. The, the Local Biary Patter (LBP operator is performed o each GCM to extract the Local Gabor Biary Patter (LGBP. Next, the face image is described usig the Histogram Sequece of all these Local Gabor Biary Patters (HSLGBP [9]. By combig the Gabor trasformatio, LBP ad local histogram, the proposed method is robust to the illumiatio variatio, image rotatio ad deformatio. Fiall the multi-class SVM is used to perform the feature classificatio, cosiderig that SVM ca achieve the good performace i the case of small samples [10]. Accordig to the traiig samples, the prior kowledge of /08 /$ IEEE CIS 008

2 HSLGBP i each scale ad each orietatio ca be achieved. The fial recogitio result is based o the combiatio of prior kowledge ad estimatio result of testig samples. II. IMAGE PREPROCESSING The image preprocessig icludes the image segmetatio, size ormalizatio ad histogram equalizatio. We take Fig. 1(a as a example to describe how to perform the image preprocessig. The size of Fig. 1(a is 56 x 56 i pixels. Geerally speakig, the huma face i a image is slatig to some extet, which will ifluece the precisio of expressio recogitio. So the image eeds to be rotated to make the huma face perpedicular to the horizotal lie. We use the method of template matchig to locate the two eyes of huma face i the image, as show i Fig. 1(b. Coect these two eyes ito a lie ad compute the agle betwee this lie ad the horizotal lie. Accordig to the achieved agle, rotate the image so that the lie coectig the two eyes is horizotal. The rotatio result is show i Fig. 1(c. Next, we eed to extract the pricipal face regio from the backgroud regio, ad here the geometrical feature of huma face is used. The segmetatio result is show i Fig. 1(d, whose size is 13 x 135 i pixels. Followig the image segmetatio is the size ormalizatio, i.e. the image zooms i or out to a costat size. Here all the face regio images are ormalized to the size of 11 x 96 i pixels. The size ormalizatio result is show i Fig. 1(e. To reduce the ifluece of illumiatio variatio, the histogram equalizatio is applied to perform the illumiatio compesatio, as show i Fig. 1(f. The Gabor filter is a liear filter whose impulse respose is defied by a harmoic fuctio multiplied by a Gaussia fuctio. Because of the multiplicatio-covolutio property (Covolutio theorem, the Fourier trasform of a Gabor filter's impulse respose is the covolutio of the Fourier trasform of the harmoic fuctio ad the Fourier trasform of the Gaussia fuctio. The two-dimesioal Gabor filter is composed of the real part ad imagiary part, doated as Gk ( = Gr( + igi(. (1 The real part ca be calculated as follows Gr( = Kv K exp( v ( x + y *[cos( Kv cos( Qu x + Kvsi( Qu exp( ]. ( Ad the imagiary part is calculated as follows Kv K Gi( = exp( v ( x + y *[si( Kv cos( Qu x + Kv si( Qu exp( ], (3 where = exp( ( v + / =. Kv π, Qu π *u / 6 (a (b (c (d (e (f Figure 1. Example of image preprocessig III. FEATURE EXTRACTION A. Gabor Filter It is commo to use the base poits istead of the etire image to extract the target features. The Gabor filter ca be viewed as a siusoidal plae at the certai frequecy ad orietatio, modulated by a Gaussia evelope [11]. The Gabor filter has bee applied i may domais, such as texture segmetatio, object detectio, fractal dimesio maagemet, documet aalysis, edge detectio, retia idetificatio, image codig ad represetatio. Due to the characteristics of Gabor filter, we use the multiple Gabor filters i the differet scales ad orietatios to perform the feature extractio. Based o the cosideratio of Gabor filters spatial locality ad orietatio selectivit we desig a group of Gabor filters i four scales ad six orietatios. I (3, the multiplicative factor Kv esures that the filters tued to the differet spatial frequecy bads have the approximately equal eergies. The term exp( is subtracted to make the filters isesitive to the overall level of illumiatio. The terms Qu ad Kv have defied the orietatio ad scale of the Gabor wavelets respectively. The parameter is the ratio of the Gaussia widow s width to the Gabor wavelets legth ad the parameter u is the orietatio of Gabor wavelets. I this paper is set to beπ. B. Gabor Feature Extractio With the differet scales ad orietatios, a filter bak cosistig of several Gabor filters is built. The filters are covolved with sigal, resultig i the Gabor space. This process is closely related to the activity i the primary visual cortex. The Gabor space is very useful i several domais such as iris recogitio. To a image whose size is 56 x 56 i pixels, the four scales are deoted as v =0, 1,, 3 ad the six orietatios are deoted as u =1,, 3, 4, 5, 6, i.e. from 30 to 180 degrees by the step of 30 degrees. The face image is covolved with the Gabor filters ad the covolutio result is called the Gabor Coefficiet Map (GCM. The correspodig covolutio equatio is followig

3 G( u = Gk( * I(, (4 where Gk( is the Gabor filter ad I( is the face image. After the preprocessig the face image is set to be 11 x 96 i pixels. With four scales ad six orietatios, the umber of Gabor filters is 4 (=4*6. So the dimesio of Gabor features is very high ad equals (=4*11*96. Fig. illustrates the Gabor features of the face image. where 1, fp fc S( fp fc =. (6 0, fp < fc Fig. 3 illustrates how to compute the LBP operator. I Fig. 3, the left part gives the eighborhood iformatio ad the right part gives the computatioal result of (6. The sequece of biary result is ad the computatioal result of (5 is 11 (=1*1+0*+0*4+1*8+1*16+1*3+1*64+0*18. Figure 3. A example of LBP operator I this paper, the LBP operator is applied to the output of GCM ad the result is deoted as Figure. Gabor features of face image C. Local Gabor Biary Patter The features extracted based o the Gabor filter are usually high-dimesioal, but their iformatio is rather redudat. There are two classic methods which ca be used to perform the dimesio reductio. Oe method is to use the sparse poits to extract features, e.g. the sparse poits are achieved every several lies ad rows. The example of this method is PCA [1] ad the shortage of this method is that it is possible to lose some pricipal iformatio. Aother method, e.g. the elastic buch graph matchig [13], applies the Gabor filter to the feature poits of the face image. Here, the shortage is that the method depeds too much o the selectio of feature poits. Agaist the shortage of the existig methods, the Local Biary Patter (LBP operator [14] [15] [16] is applied i the eighborhood regio to get the Gabor coefficiets ad the histogram techique is used to solve the partial variatio. By extractig the GCM we ca reach two goals, oe is to avoid losig pricipal iformatio ad ather is to reduce the dimesio simultaeously. The LBP as a ovel low-cost image descriptor for texture classificatio has bee applied i may domais [7]. To a pixel, its itesity is deote as f c, where c=4. From top to bottom ad from left to right, except the cetral pixel, each pixel i the 3 x 3 eighborhood regio is labeled to be 0 to 8 except 4. The itesity of each pixel is deoted as f p, where p= 0, 1,,3,5,6,7,8. To each pixel, the LBP operator is computed as follows p LBP S( fp fc, (5 = 8 p= 0 8 p LGBP = S( Gp( u Gc( u. (7 p= 0 D. Histogram Sequece of Local Gabor Biary Patters Cosiderig that the histogram techique has may good characteristics, the histogram is applied to the LGBP. If the histogram is computed based o the etire face image, it is possible to lose some detailed iformatio. So, before computig the histogram the face image is divided ito several o-itersectioal regios. The the histogram is applied to each regio s LGBP of the face image respectively. Meawhile, the dimesio of LGBP ca be reduced by dividig the etire face image ito several regios. To the image f ( whose gray-level is betwee 0 ad L-1, the histogram ca be computed as hi = I{ f ( == i}, i = 0,1,... L 1, (8 where i is the gray-level, whose gray-level is I, ad hi meas the umber of pixels 1, X is ture I{ X} =. (9 0, X is false We divide the face image ito m regios ( R 0, R1,... Rm 1, as show i the left part of Fig. 4 ad compute the histogram of each regio, as show i the right part of Fig. 4. Here the histogram is deoted as H u, Rj, i = I LGBP( u} I{( Rj} ( Rj {,(10 where i =0,1,...L-1, v =0,1,,3, u =1,,.6, j =0,1,..m-1. I the scale v ad orietatio u, all the histograms are cascaded as the fial feature, deoted as

4 HSLGBP ( u = ( H u,0, H u,1,... H u, m 1, (11 where v =0,1,,3, u =1,,.6. I this paper the face image is divided ito 4(=7x6 regios ad each regio is 16 x 16 i pixels. Fig. 4 illustrates the computatio of HSLGBP. Figure 4. IV. The computatio of HSLGBP FEATURE CLASSIFICATION A. Support Vector Machie As the last step the expressio classificatio must be based o a certai patter classifier. All the classifiers ca be divided ito two types, oe is the liear ad aother is the o-liear. Support Vector Machie (SVM belogs to a family of geeralized liear classifiers [17] ad has proved to be more effective tha other classifiers i the case of small samples. Moreover, SVM is a statistical learig method based o the miimum structural risk criterio. Cosiderig that the facial expressio is the high-dimesioal problem with the samples beig relatively small, we use the method of SVM to perform the expressio classificatio. SVM is a very effective biary classificatio algorithm. Give a liearly separable biary classificatio problem {( Xi, Yi} i= 1 (as show i Fig. 5, where Y i = { + 1, 1 }, Xi is a -dimesio vector ad Yi is the label of the class that the vector belogs to. SVM separates the two classes of poits by a T hyper plae, doated as ω x + b = 0, where ω is a iput vector, x is a adaptive weight vector ad b is a bias. SVM fids the parametersω ad b for the optimal hyper plae to maximize the geometric margi / ω, subject T to yi( ω xi + b + 1. The solutio ca be foud through a Wolfe dual problem with the Lagragia multiplied by α i : Q = αi i = 1 i, j = 1 toαi 0 ad α iyi = 0. i= 1 Figure 5. ( α αiαjyiyj( xi xj /, subject SVM for a liearly separable biary classificatio problem I the dual format, data poits oly appear i the ier product. To get a potetially better represetatio of the data, the data poits are mapped ito a Hilbert Ier Product space through a replacemet x i xj φ ( xi φ( xj = K( xi, xj, where K( is a kerel fuctio. We the get the kerel versio of the Wolfe dual problem i= 1 Q( α = αi αiαjyiyjk ( xi xj /. (1 i, j= 1 Thus, for a give kerel fuctio, the SVM classifier is give by F ( x = sg( f ( x, where f ( x = αiyik ( xi, x + b is i = 1 the decisio fuctio of the output hyper plae of the SVM. Some commo kerels iclude: d 1. Polyomial (homogeeous: K ( = ( x x'. Polyomial (ihomogeeous: K ( = ( x x' Gaussia Radial basis fuctio: x x' K ( = exp( 4. Sigmoid: K ( = tah( kx x' + c, for some (ot ever k>0 ad c<0 5. Radial Basis Fuctio (RBF: K( = exp( γ x x',for γ > 0 I this paper, we choose the RBF kerel fuctio to perform the SVM classificatio of two classes. It is oticeable that SVM makes the biary decisio. Here, the multi-class classificatio is accomplished by a cascade of biary classifiers together with a votig scheme. If there is ot a sigle maximum value of the votig result, we will use the algorithm of Nearest Neighborhood Classifier to make the fial judgmet. B. Classificatio Result If we cascade all the HSLGBP ( u as the fial feature to perform the sample traiig, we will fid that the feature dimesio is too high. Through the large-scale experimets, we discover that the Gabor wavelet coefficiet i the differet scale ad orietatio has the differet cotributio to the recogitio result. The reaso may be that the differet expressio results i the differet orietatio ad degree of face orga deformatio, such as eyebrow, mouth, eye ad so o. If oe expressio causes the large deformatio i oe orietatio, the recogitio rate based o the Gabor wavelet coefficiet at that orietatio will be high. Based o the above observatio, we desig the experimets to ivestigate the relatioship betwee the recogitio rate ad the Gabor wavelet coefficiet i the differet scale ad orietatio. If we determie all the relatioships, we ca assig the differet weight to the expressio i the differet scale ad d

5 orietatio properly. The fial recogitio result is achieved by fusig all the recogitio results based o the differet Gabor wavelet coefficiets. This method is superior to the oe of usig the wavelet coefficiet i a sigle scale ad orietatio, or the oe of usig all the wavelet coefficiet sequece i the differet scale ad orietatio. The relatioship betwee HSLGBP(u ad the recogitio result is give i Fig. 6. (d Figure 6. The relatioship betwee HSLGBP(u ad recogitio result (a Ager, (b Fear, (c Happiess, (d Sad The term R( u, i is used to deote the recogitio result of the i th expressio i the scale v ad orietatio u. We get R( u, i through the sample traiig. The term r ( u, i is the recogitio result based o the testig samples. The fial recogitio result ca be got by fusig R( u, i ad (a r ( u, i, i.e. R = max[ R( u, i r ( u, i]. i v (13 u V. EXPERIMENTAL RESULT I experimets the JAFFE facial expressio database is used, which icludes 13 images of te persos. Each perso has six basic expressios ad two to four eutral expressios. Fig. 7 gives two examples of differet expressios, which are the ager, disgust, fear, happiess, eutral, sadess ad surprise. (b Figure 7. Two groups of images i the JAFFE facial expressio database Cosiderig the case of small samples, the method of tefold cross validatio is applied. That is to sa the database is divided radomly ito te equal-sized parts, from which ie parts are used as traiig samples ad the other as testig sample. The above process is repeated te times so that each part has a opportuity to be used as testig sample. The experimetal results show that the facial expressio recogitio algorithm proposed i this paper is effective ad superior to the other similar methods. (c Table I compares the recogitio results amog the differet methods. The comparative study is carried out based o the same facial expressio database, i.e. JAFFE, as the oes adopted by the other methods listed o the table. The 16

6 recogitio results of Gabor+SVM, LBP+SVM ad LBP+Template Matchig ca be foud i [6] [7] [18]. From the table we ca see that the recogitio result of our method is best. The facial features extracted based o Gabor + LBP is more effective tha the oes based just o the Gabor filter or LBP operator. The dimesio of features extracted based o the Gabor filter is rather high ad equals (= Zhu et al. [6] applied the techique of AdaBoost to perform the feature selectio ad achieved the relatively good result. But it is possible for this method to lose some pricipal iformatio about the face image. Fig. 8 compares the recogitio results of four classic expressios betwee our proposed method based o Gabor+LBP+SVM ad the oe based o Gabor+AdaBoost+SVM. It ca be see that the method based o Gabor+LBP+SVM is superior to the oe based o Gabor+AdaBoost+SVM. The dimesio of features based o LBP is 1075 (=7 6 56, which is much lower tha the oe based o the Gabor filter. The average processig time per image with our proposed method is 3 secods, which outperforms the average time of 38 secods with Gabor+Adaboost+SVM. So it ca be said that our proposed method is more efficiet. TABLE I. RECOGNITION RESULT COMPARISON AMONG DIFFERENT METHODS Mehthod Recogitio Result Gabor+LBP+SVM 97.8% Gabor+SVM [18] 95.18% Gabor+Adaboost+SVM [6] 97.1% LBP+SVM [7] 87.6% LBP+Template Matchig [7] 79.1% Gabor+LBP+Template Matchig 85.14% Figure 8. Recogitio result compariso betwee two methods VI. CONCLUSION Compared with the expressio recogitio method based o the video sequece, the oe based o the static image is more difficult due to the lack of temporal iformatio. The mai cotributio of this paper is to ivestigate the facial expressio recogitio based o the static image ad to propose a ew recogitio method based o the histogram sequece of local Gabor biary patters. The experimetal results show that the facial expressio recogitio algorithm proposed i this paper is effective ad superior to the other similar methods. The good performace of our proposed method depeds o the variability of Gabor filter i represetig expressios, the effectiveess of LBP i gettig the global iformatio ad reducig dimesio, the discrimiatio of SVM i the case of small samples. ACKNOWLEDGMENT This paper is supported by the NSFC grat ad the project of sciece ad techology pla of Jiagsu Provice (o. BG REFERENCES [1] Y. Tia, L. Brow, A. Hampapur, S. Pakati, A. Seior,ad R. Bolle, Real world real-time automatic recogitio of facial expressio, IEEE PETS, Australia, March 003. [] B. Fasel, ad J. Luetti, Automatic facial expressio aalysis: a survey. Patter Recogitio, vol. 36, o. 1, pp , 003. [3] M. Turk ad A. Petlad, Eigefaces for recogitio, Joural of Cogitive Neurosciece, vol. 3, pp , [4] Buciu, C. Kotropoulos, ad I. Pitas, ICA ad Gabor represetatio for facial expressio recogitio, Image Processig, vol., pp , 003. [5] K. Mase. Recogitio of facial expressio from optical flow, IEICE Trasactios, vol. 74, pp , [6] J. Zhu, G. Su, ad Y. Li, Facial expressio recogitio based o Gabor feature ad Adaboost. Joural of Optoelectroics Laser, vol. 17, o [7] C. Sha, S. Gog ad P. W. McOwa, Robust facial expressio recogitio usig local biary patters. I: Deparmet of Computer Sciece, Quee Mar Uiversity of Lodo, Lodo, E1 4NS, UK [8] Z. Zhag, M. J. Lyos, M. Schuster, ad S. Akamatsu, Compariso betwee geometry-based ad gabor-wavelets-based facial expressio recogitio usig multi-layer perceptro, I IEEE FG, April [9] W. Zhag S. Sha, H. Zhag, J. Che, X. Che, ad W. Gao, Histogram sequece of local Gabor biary patter for face descriptio ad idetificatio. Joural of Software, vol. 17, o. 1, pp , 006. [10] M. Dumas, Emotioal expressio recogitio usig Support Vector Machies. I Proceedigs of Iteratioal Coferece o Multimodal Iterfaces, 001. [11] T. Lee, Image represetatio usig -D Gabor wavelets. IEEE Tras Patter Aalysis ad Machie Itelligece, vol. 18, o. 10, pp , [1] C. Liu, ad H. Wechsler, Gabor feature based classificatio usig the ehaced fisher liear discrimiat model for face recogitio. IEEE Tras. o Image Processig, vol. 11, o. 4, pp , 00. [13] L. Wiskott, J. Fellous, N. Kruger, ad C. Malsburg, Face recogitio by elastic buch graph matchig. IEEE Tras. o Patter Aalysis ad Machie Itelligece, vol. 19, o. 7, pp , [14] A. Timo, H. Abdeour, ad P. Matti, Face recogitio with local biary patters. I: Pajdla T, Matas J, eds., Proc. of the Europea Cof. o Computer Visio. LNCS 301, Prague: Spriger-Verlag, pp , 004. [15] T. Ojala, M. Pietikie, ad T. Mep, Multiresolutio grayscale ad rotatio ivariat texture classi_catio with local biary patters. IEEE PAMI, vol. 4, o. 7, July 00. [16] H. Ji, Q. Liu, H. Lu, ad X. Tog, Face detectio usig improved LBP uder Bayesia framework. I: Zhag D, Pa ZG, eds. Proc. of the It l Cof. o Image ad Graphics. Hog Kog: IEEE Computer Society Press, pp , 004. [17] V. Vapik, Statistical Learig Theor Wile New York,1998. [18] I. Buciu, C. Kotropoulos ad I. Pitas, ICA ad Gabor Represetatio for Facial Expressio Recogitio, Proceedigs of ICIP, 003.