A Vision-based Facial Expression Recognition and Adaptation System from Video Stream

Transcription

1 Internatonal Journal of achne Learnng and Comutng, Vol. 2, No. 5, October 202 A Vson-based Facal Exresson Recognton and Adataton System from Vdeo Stream Nurul Ahad awhd, Nasr Uddn Laskar, and Hader Al system for detecton of basc emotonal exressons from 2D facal mages for a artcular race. he system automatcally categorzes facal exresson from frontal faces to fve dmensons: Neutral, Hay, Sadness, Surrse and Dsgust. Abstract he am of ths aer s to develo a real tme vson-based facal exresson recognton and adataton system for human-comuter nteracton. ajor objectve of ths research s to detect face, to dentfy and recognze user's facal exresson usng face mage n real tme and to be able to adat wth new user's facal exresson. It also works on mxed race exresson detecton. It s based uon the egenface algorthm. Whch a small set of feature vectors are used to descrbe the varaton between exresson mages. It s also beng able to adat new exresson mage n real tme. he roosed system makes major contrbuton n mlementng facal exresson recognton and adataton n real tme. he facal exresson recognton task s dvded nto two arts: frst art conssts of automatc face detecton from vdeo stream and rerocessng, second art conssts of a classfcaton ste that emloys Prncal Comonent Analyss (PCA) to classfy the exresson nto one of fve categores. he algorthm has been tested usng both statc and dynamc mages. he average recson and recall rate acheved by the system s about 88 for erson secfc recognton. II. RELAED WORKS All facal exresson recognton methods can be classfed nto two broad-based categores: feature based aroach and robablstc aroach. he feature-based method utlzes the Facal Acton Codng System (FACS) desgned by Ekman and Frser []. In FACS, the motons of the face are dvded nto 44 acton unts (AU), and ther combnatons may descrbe any facal exresson. ore than 7,000 combnatons of AU have been observed [2]. he robablstc-based method does not gve reference to facal features such as eyes and mouth. Instead, the feature vector can be the random dstrbuton of mage ntenstes and these vectors may dffer from each emoton. he vectors are calculated er emoton and classfcaton algorthms lke H, Neural Network (NN) or a hybrd aroach (H and NN) [3] are aled. here are some other technques lke PEG-4 Facal Anmaton Parameter (FAP), measurement of facal moton through otc f. urk and Pentland [7] roosed Egenfaces emloyed PCA whch s an unsuervsed learnng method and treats samles of the dfferent classes n the same way. Nowadays, varous researchers reorted the model-based methods [4,5] for feature extracton, such as actve aearance model [6], ont dstrbuton model and labeled grahs. ut those methods requre heavy comutaton or manually detected feature nodes to construct the model, whch can hardly be mlemented n real-tme automatc facal exresson recognton (FER). Index erms Facal acton codng system, facal exresson recognton, hdden markov model (H), neural network (NN), rncal comonent analyss (PCA). I. INRODUCION Facal exresson s one of the most owerful, natural, and mmedate means of human bengs to convey ther emotons and ntentons. he extracton and recognton of facal exresson by machne can contrbute to transmsson of facal data and user-frendly communcaton between user and ther comuter system. Facal exressons are related to one's emoton state and lay an mortant role n smooth communcaton among ndvduals. In ths way, comuters n the future wll be able to offer advce n resonse to the mood of the users. Snce the facal structure s mostly same for humans of a artcular race so ther facal structure for an exresson s also close. hs s the man thng whch we tred to use n ths work. Egenface method extracts the features from the tranng mages and uses t to categorze the test mages. So f a system s traned wth the facal exresson of a artcular race, then t can dentfy the exressons of that race. In ths aer we resent results of a fully automatc III. In the egenface method, the test mage s comared aganst the tranng set mages to fnd the best match of the exresson. o ass through the egenface method the test mage s rerocessed, so that the features of the face, whch are mortant to exresson detecton, are get sharer. Face detecton from the mage and the re-rocessng stes are as fols: A. Face Detecton In ths aer we wll attemt to detect face from an mage usng a temlate matchng technology rovded by the [FaceVACS]. o locate the face, an mage yramd s formed from the orgnal mage. An mage yramd s a set of coes of the orgnal mage at dfferent scales, thus reresentng a anuscrt receved July 8, 202; revsed Setember 0, 202. hs work was suorted n art by Unversty of Informaton echnology and Scences (UIS), Dhaka, angladesh. d. Nurul Ahad awhd s wth the det. of II, Unversty of Dhaka, angladesh (e-mal: tawhd_du@yahoo.com). D. Nasr Uddn Laskar s wth the det. of CSE and I, UIS, Dahaka, on study leave. He s now ursung S n Kyung Hee Unversty (e-mal: nasr_csedu@yahoo.com). d. Hader Al s wth the de. of CSE, Unversty of Dhaka (e-mal: hader@cse.unvdhaka.edu) /IJLC.202.V2.83 EHODOLOGY 535

2 Internatonal Journal of achne Learnng and Comutng, Vol. 2, No. 5, October 202 set of dfferent resolutons. A mask s moved xel wse over each mage n the yramd, and at each oston, the mage secton under the mask s assed to a functon that assesses the smlarty of the mage secton to a face. If the smlarty value s hgh enough, the resence of a face at that oston and resoluton s assumed. From that oston and resoluton, the oston and sze of the face n the orgnal mage can be calculated. From the oston of the face, a frst estmate of the eye ostons can be derved. In a neghborhood around these estmated ostons, a search for the exact eye ostons s started. hs search s very smlar to the search for the face oston, the man dfference beng that the resoluton of the mages n the yramd s hgher than the resoluton at whch the face was found before. he ostons yeldng the hghest smlarty values are taken as fnal estmates of the eye ostons.. Image Normalzaton he face mage s normalzed before assng to the facal exresson recognton module. Sequences of mage re-rocessng technques are aled so that the mage s lght and nose nvarant and the facal feature onts become sharer. hen t needs to aly some standard exresson recognton re-requste such as grey mage converson and scalng nto a sutable szed mage. ) Converson to Grey Image and Scalng Detected face s converted to grayscale usng () and scaled to xels usng (2) and saved as a gray bm mage. Lnear nterolaton technque was emloyed to determne the scaled outut mage. where, Gr s the gray level value of th xel of the gray mage. R, G, corresonds to red, green, blue value of the th xel n the color mage. q q x q y q Q( x, y ) P( (2) 32 x, 32 y ) where, we want to re-scale mage P[(0,0) (x, y )] to mage Q[(0,0) (32 32 )] 2) Contrast Stretchng Frequently, an mage brghtness values do not make full use of the avalable dynamc range. he folng formula s used n stretchng the hstogram over the avalable dynamc range: b[m,n] R + G + Gr 3,, 2,..., ( 0 a[m,n] 2 ) hgh ( 2 ) a[m,n] < a[m,n] < a[m,n] hgh x N hgh Here, we mght choose the and 99 values for and hgh, resectvely, nstead of the 0 and 00 values reresented by the equaton. It s also ossble to aly the contrast-stretchng oeraton on a regonal bass usng the hstogram from a regon to determne the arorate lmts for the algorthm. 3) Hstogram Equalzaton o comare two or more mages on a secfc bass, such as texture, t s common to frst normalze ther hstograms to () (3) a standard hstogram. he most common hstogram normalzaton technque s hstogram equalzaton where one attemts to change the hstogram through the use of a functon b F(a) nto a hstogram that s constant for all brghtness values. For a sutable functon F(*) the relaton between the nut robablty densty functon, the outut robablty densty functon, and the functon F(*) s gven by: a (a)da b (b)db a (a)da df b (b) (4) From (4) we see that sutable means F(*) s dfferentable and df/da > 0. For hstogram equalzaton we desre b (b) constant and ths means that: f(a) ( 2 ) P(a) (5) where, P(a) s the robablty dstrbuton functon. In other words, the quantzed robablty dstrbuton functon normalzed from 0 to 2 - s the look-u table requred for hstogram equalzaton. IV. FACIAL EXPRESSION RECOGNIION A. ackground uch of the revous work on automated facal exresson recognton has gnored the ssue of just what asects of the face stmulus are mortant for exresson recognton. In the language of nformaton theory, the relevant nformaton n a face mage s extracted, encoded as effcently as ossble, and then comared wth a database of models encoded smlarly. A smle aroach to extractng the nformaton contaned n an mage of a facal exresson s to somehow cature t h e varaton n a collecton of facal exresson mages, ndeendent of any judgment of features, and use ths nformaton to encode and comare ndvdual facal exresson mages to detect exresson. In mathematcal terms, the rncal comonents of the dstrbuton of facal exresson, or the egenvectors of the covarance matrx of the set of facal exresson mages, treatng an mage as ont (or vector) n a very hgh dmensonal sace s sought. urk and Pentland [7, 8] roosed Egenfaces emloyed rncal comonent analyss (PCA). PCA s an unsuervsed learnng method, whch treats samles of the dfferent classes n the same way. he egenvectors are ordered, each one accountng for a dfferent amount of the varaton among the facal exresson mages. hese egenvectors can be thought of as a set of features that together characterze the varaton between exresson mages. Each mage locaton contrbutes more or less to each egenvector, so that t s ossble to dslay these egenvectors as a sort of ghostly face mage whch s called an egenface [7]. Each ndvdual facal exresson can be reresented exactly n terms of a lnear combnaton of the egenfaces. Each facal exresson can also be aroxmated usng only the best egenfaces, those that have the largest egenvalues, and whch therefore account for the most varance wthn the set of exresson mages. he best egenfaces san an -dmensonal subsace whch we call the facal exresson 536

3 Internatonal Journal of achne Learnng and Comutng, Vol. 2, No. 5, October 202 From whch we see that Av are the egenvectors of C AA Folng these analyss, we construct the matrx L sace of all ossble mages.. Calculatng Egenfaces Let a face mage I(x,y) be a two-dmensonal N N array of 8-bt ntensty values. An mage may also be consdered as a vector of dmenson N2, so that a tycal mage of sze becomes a vector of dmenson 65,536, or equvalently a ont n 65,536-dmensonal sace. An ensemble of mages, then, mas to a collecton of onts n ths huge sace. hese vectors defne the subsace of face mages, whch we call face sace. Each vector s of length N2, descrbes an N N mage, and s a lnear combnaton of the orgnal face mages. ecause these vectors are the egenvectors of the covarance matrx corresondng to the orgnal face mages, and because they are face-lke n aearance, we refer to them as egenfaces. Examles of egenfaces of Fg. are shown n Fg. 2. Let the tranng set of face mages be Γ, Γ 2,, Γ then the average of the set s defned by Ψ n Γ n AA where L mn Φ m Φ n and fnd the egenvectors, vl, of L. hese vectors determne lnear combnatons of the tranng set face mages to form the egenfaces ul. u l k v lk Φ k l, 2, (2) Wth ths analyss, the calculatons are greatly reduced, from the order of the number of xels n the mages (N2) to the order of the number of mages n the tranng set (). (6) Each face dffers from the average by the vector Φ Γ Ψ (7) An examle tranng set s shown n Fg.. hs set of very large vectors s then subject to rncal comonent analyss, whch seeks a set of ortho-normal vectors un whch best descrbes the dstrbuton of the data. he k th vector, uk, s chosen such that λk n ( u k Φ n )2 Fg.. Examle of tranng face mages [9]. (8) s a maxmum, subject to, u l u k δ lk 0, f l k otherwse (9) he vectors uk and scalars λk are the egenvectors and egenvalues, resectvely of the covarance matrx C Φ nφ n n AA where, the matrx A [ ΦΦ 2...Φ (0) Fg. 2. Egenfaces wth hghest egenvalues ] the covarance matrx C, however s N2 N2 real symmetrc matrx, and determnng the N2 egenvectors and egenvalues s an ntractable task for tycal mage szes. We need a comutatonally feasble method to fnd these egenvectors. If the number of data onts n the mage sace s less than the dmenson of the sace ( < N2 ), there wll be only -, rather than N2, meanngful egenvectors. he remanng egenvectors wll have assocated egenvalues of zero. We can solve for the N2 dmensonal egenvectors n ths case by frst solvng the egenvectors of an matrx such as solvng 6 6 matrx rather than a 6,384 6,384 matrx and then, takng arorate lnear combnatons of the face mages Φ. Consder the egenvectors v of a AA such that AAVl μ l vl C. Usng Egenfaces to Classfy a Facal Exresson Accurate reconstructon of the mage s not a requrement - based on ths dea, the roosed exresson recognton system lets the user secfy the number of egenfaces (') that s gong to be used n the recognton. For maxmum accuracy, the number of egenfaces should be equal to the number of mages n the tranng set. ut, t was observed that, for a tranng set of fourteen face mages, seven egenfaces were enough for a suffcent descrton of the tranng set members. In ths framework, dentfcaton becomes a attern recognton task. he egenfaces san an ' dmensonal subsace of the orgnal N2 mage sace. he ' sgnfcant egenvectors of the L matrx are chosen as those wth the largest assocated egenvalues. A new face mage ( Γ ) s transformed nto ts egenface comonents (rojected onto "face sace") by the oeraton, ω k u k () 537 (Γ Ψ ) (3)

4 Internatonal Journal of achne Learnng and Comutng, Vol. 2, No. 5, October 202 For k,...,'. hs descrbes a set of ont by ont mage multlcatons and summatons, oeratons erformed at aroxmately frame rate on current mage rocessng hardware. Classfcaton s erformed by comarng the feature vectors of the face lbrary members wth the feature vector of the nut mages (mouth and eye). hs comarson s based on the Eucldean dstance between the two members. If the comarson falls wthn the user defned threshold, then the mage s classfed as known exresson for that, otherwse t s classfed as unknown and can be added to exresson lbrary wth ts feature vector for later use. V. ARCHIECURE AND EXPERIENAL RESULS A. System Organzaton Fg. 3 gves a hgh level descrton of the roosed system. he facal exresson recognton module uses egensace-based rncal comonent analyss aroach to recognze known exressons. If an exresson cannot be recognzed as known, then the new erson s exresson adataton module wll be actvated. he adataton module n ths system catures an mage of the unknown erson s exresson and re-comutes ts erson secfc feature vectors usng the egenface method descrbed earler. he roosed facal exresson recognton system asses through three man hases durng an exresson recognton rocess. hey are: ) Face Lbrary Formaton Phase In ths hase, face mages are stored n a face lbrary n the system. Every acton such as tranng set or egenface formaton s erformed on ths face lbrary. In order to start the exresson recognton rocess, ths ntally emty face lbrary has to be flled wth facal exresson mages. After acquston and re-rocessng, face mages under consderaton s added to the face lbrary. Weght vectors of the face lbrary members are emty untl a tranng set s chosen and egenfaces formed. 2) ranng Phase After choosng the tranng set, egenfaces are formed and stored for later use. Egenfaces are calculated from the tranng set, keeng only the mages that corresond to the hghest egenvalues. hese egenfaces defne the -dmensonal "face sace". As new faces are exerenced, the egenfaces can be udated or recalculated. Accordngly the corresondng weght vector of each face lbrary member has been udated whch were ntally emty. 3) Recognton and Learnng Phase After obtanng the weght vector, t s comared wth the weght vector of every face lbrary member wthn a user defned "threshold". If there exsts at least one face lbrary member that s smlar to the acqured mage wthn that threshold then, the face mage s classfed as known. Otherwse, a mss has occurred and the face mage s classfed as unknown. After beng classfed as unknown wth certanty, ths new face mage can be added to the face lbrary wth ts corresondng weght vector for later use (learnng to recognze). Fg. 3. System archtecture for exresson recognton and adataton. Exermental Results and Performance Evaluaton able I shows the confuson matrx for the results of facal exresson recognton algorthm wth mage rerocessng and adataton wth one mage er erson s facal exresson. he dagonal elements reresent the correct recognton of corresondng exresson. ALE I: CONFUSION ARIX OF EXPRESSION RECOGNIION (PERSON SPECIFIC) Detected facal exresson otal Su Success Neu Hay Sad Dsg r Neutral Inut Exresso n Hay Sad Surrs e Dsgust We need to defne two arameters for the evaluaton of our method s erformance. able II resents the recson () and recall () rates of facal exresson recognton method. he recson () s defned by the rato of the numbers of correct recognton to total numbers of recognton for each exresson. he recall rate () s defned by the rato of the numbers of correct exresson recognton to total numbers of nut exresson mages for each erson. ALE II: PERFORANCE OF EXPRESSION RECOGNIION (PERSON SPECIFIC) Exresson () Neutral () 95.0 Hay Sad Surrsed Dsgust Next we erform a varaton above settngs where we do not adat the system wth one mage er erson s exresson. Instead we randomly selected 40 mages from 20 mages. hese 40 mages contan 0 mages er exresson ndeendent of the ersons. able III shows the confuson matrx for the results of the system for ths confguraton 538

5 Internatonal Journal of achne Learnng and Comutng, Vol. 2, No. 5, October 202 ALE III: CONFUSION ARIX OF EXPRESSION RECOGNIION (PERSON INDEPENDEN) otal Detected facal exresson Success Inut Exresso n Neu Hay Sa d Sur Ds g Neutral Hay Sad Surrs e Dsgust other hand, can run n real tme wth reasonable success rate. he system acheved on average 88.7 and 88 recson and recalls rates wth erson secfc and and 73 recson and recall rates for erson ndeendent exresson recognton. Another area of contrbuton of the roosed work s the new user s exresson adataton algorthm that facltates regstraton of unknown ersons facal mages wth the system n real tme. Real tme exresson adataton s an mressve achevement that eases nteracton between user and the system. ALE IV: PERFORANCE OF EXPRESSION RECOGNIION (PERSON INDEPENDEN) Exresson () Neutral () 85.0 Hay Sad Surrsed Dsgust If the lght condton s good and not constantly changng, f the user s face orentaton does not undergo on major varaton, the system always erforms accordng to above success rates. As we can see n case of erson secfc method, the recall rate and the recson rates are qute hgh than the exstng results, suggestng that wth mage rerocessng and one tranng mage/erson s exresson our recognton system roduces satsfactory outcome on a wde range of real tme envronment Fg. 4 comares the results of the above exerments. ercentage recson recall erson secfc erson ndeendent Fg. 4. Performance comarsons wth tranng mage varaton VI. CONCLUSION Facal exresson recognton n real tme s at the crossroads of some crtcal otmzaton ssues. hs work makes major contrbuton n areas of facal exresson recognton. here are several systems for facal exresson recognton. hese systems, although were successful n terms of recall and recson rates, but most of these systems could not be used n real tme. he roosed system, on the REFERENCES [] P. Ekman and W. Frsen, Facal acton codng system," Consultng Psychologsts Press, 977. [2] P. Ekman, ethods for measurng facal actons," Handbook of ethods n Nonverbal ehavor Research, Cambrdge: Cambrdge Unversty, 982, [3] I. Hu, L. C. D. Slva, and K. Senguta, A hybrd aroach of nn and hmm for facal emoton classfaton," ELSEVIER Pattern Recognton Letters Journal, vol. 23, no., 2002, [4] I. A. Essa and A. P. Pentland, Codng, analyss, nterretaton, and recognton of facal exressons," IEEE rans. Pattern Analyss and achne Intellgence, vol. 9, no., 997, [5]. Fasel and J. Luettn, Automatc facal exresson analyss: a survey," Pattern Recognton, vol. 36, 2003, [6]. Cootes, G. Edwards, and C. aylor, Actve aearance models," IEEE rans. Pattern Analyss and achne Intellgence, vol. 23, 200, [7]. A. urk and A. P. Pentland, Face recognton usng egenfaces," IEEE, 99, [8] K... Sabrn,. Zhang, S. Chen,. Nurul Ahad awhd,. Hasanuzzaman,. Hader Al, and H. Ueno, An Intensty and Sze Invarant Real me Face Recognton Aroach, Proc. Of ICIAR 2009, LNCS 5627, Srnger-Verlag erln Hedelburg 2009, [9] chael J. Lyons, Shgeru Akamatsu, yuk Kamach, Jro Gyoba Proceedngs, "Codng facal exressons wth gabor wavelets" hrd IEEE Internatonal Conference on Automatc Face and Gesture Recognton, Nara Jaan, IEEE Comuter Socety, 998, d. Nurul Ahad awhd was born n 985 n angladesh. He has comleted hs graduaton from Unversty of Dhaka, Dhaka, angladesh, n Comuter Scence and Engneerng n He also comleted hs ost graduaton for the same unversty n same subject n 200. Hs major feld of study was Image Processng and Artfcal Intellgence. He s currently workng as Lecturer n Insttute of Informaton echnology, unversty of Dhaka, Dhaka, angladesh. efore jonng here, he was worked as Software Engneer n &H Informatcs D Ltd (An IS Health Comany), angladesh. d. Nasr Uddn Laskar receved hs. Sc. Degree n Comuter Scence and Engneerng from Unversty of Dhaka, angladesh n Currently he s a Faculty member n the det. of Comuter Scence and Engneerng, Unversty of Informaton echnology and Scences (UIS), Dhaka, angladesh. At resent, he s on study leave and ursung the S degree n Artfcal Intellgence Lab, det. of Comuter Engneerng, Kyung Hee Unversty, Korea. Hs current research nterests nclude neural network, machne learnng and robotcs. r. Nasr s a member of IACSI. 539