Extraction of Lip Contour from Face

Transcription

1 Internatonal Journal of Current Engneerng and Technology ISSN INPRESSCO. All Rghts Reserved. Avalable at Research Artcle Extracton of Lp Contour from Face Rt Kushwaha a, Neeta Nan a* and Promla Jangra a a Department of Computer Engneerng Malavya Natonal Insttute of Technology, Japur, Rajasthan, Inda Accepted 3June 202, Avalable onlne 8 June 202 Abstract Lp segmentaton s an essental stage n many multmeda systems such as vdeo-conferencng, lp readng, or low bt rate codng communcaton systems. It s also useful n varous mage / vdeo acquston devces encouraged the development of many computer vson applcatons, such as vson-based survellance, vson based man machne nterfaces, vson-based bometrcs, and so on. Hence extracton of lp becomes a broad feld. Ths paper presents an algorthm for extracton of lp contour. We have used two algorthms for ths. Our frst algorthm s a color based algorthm derved from a quadratc polynomal, wth the assumpton that color of lps should le n the range of blue. The accuracy of the frst proposed algorthm s 9% n case of normal skn people, and 95% n case of farer skn color people. Our second algorthm s model a based algorthm whch gves accuracy up to 9 6%. Keyword: Lp contour detecton, Lp segmentaton, Lp extracton from face.. Introducton Detecton of the lp contour s a fundamental procedure of mouth feature extracton. It forms a prelmnary stage of face mage analyss, whch s essental for numerous applcaton areas ncludng man-machne nteracton based on the observed human behavor, vdeo-telephony, face and person dentcaton, bmodal speech recognton, face and vsual speech synthess, facal expressons classcaton etc. All of these applcatons requre an effcent and fully automated mouth feature extracton method that can be acheved usng an automatc lp contour detecton technque. Observng the pxels of lps, we fnd that lp color of farer skn people ranges from dark red to purple, and for normal skn color people t s n the range of blue under normal lght condtons. From the perspectve of human vsual percepton, the lps are very easy to be dferentated from the face, cause of dferent contrasts n colors. The outer labal contour of the mouth s havng very poor color dstncton when compared aganst ts skn background; t makes extracton of lp a dfcult problem. In order to mprove the contrast between lp and the other face regons we need dferent type of lp extracton technques. But we are takng the assumpton that lp color vares from dark red to purple then ths condton s satsfed wth the people who have very far complexon. For the normal skn people we are assumng that the color of lp also vares n the range of blue. We thus used the chromatc color space (Cheng Chn et al, 2003, whch s constructed from the RGB color space (Cheng Chn et al, 2003, to exclude dark pxels because * Correspondng author: Neeta Nan chromatc color space s not sutable to dstngush between brght pxels and dark pxels. For chromatc color space, each pxel s represented by two values, denoted by (r,g. Ther relatonshp s explaned n secton IV-A. Both dark pxels and brght pxels mght have the same converted r and g values due to the normalzaton effect n brghtness n chromatc color space thus we have used both the color spaces. Our second algorthm s model based algorthm (T.F.Cootes et al, 995 hence we don t need any color constran for that. Secton II descrbes the related work, secton III descrbes the dataset taken of dferent types. Secton IV and V descrbes the proposed algorthms for lp contour extracton technques. In secton VI result analyss of the proposed methods are obtaned. Secton VII contans the whole summary of the experments. 2. Related work A number of technques are used for detecton of lp contours from a gven face mage, whch can be generally classed nto categores, whch are model based approaches lke deform-able templates (L.Yulle et al, 989, actve shape models J.Luettn et al, 996 and snakes (M.Kass et al, 987. These approaches generally use a set feature pont for approxmatng the lp contours wth splne functons and gve a model for the lp contour. Hgh robustness s acheved relable constrants on the deformaton of mouth model are ncluded n the cost functon, whch s defned on the bass of heurstcs or through generally supervsed learnng from examples. However, the mnmzaton of the cost functon nvolves computatonally expensve algorthms, and the model 265

2 defnton often employs sem-automated user asssted procedures or a pror knowledge of some characterstcs of the mouth mage. The second category s Colour based approach lke (Yao Hongxun et al, Another category s level set based approach and rule based approach (Cheng Chn et al, We have used rule based approach. In ths approach frstly colour devaton s done by some technque then rules for extractng the lp contour s appled. 3. Database acquston For testng both the methods we have chosen dferent databases. If a person s havng very far complexon then we have appled dark red to purple color on hs lps, he s havng normal skn color then we choose blue color. A dataset of 50 persons for farer skn color people wth red color on lps s taken from nternet for testng results. For normal skn color people we construct the database by applyng blue color on lps, we are havng 20 person s face. Two types of pctures are taken n both the cases whch are enlsted below. Images when a part of face wth lps s avalable Images when whole face s avalable We have tested all the lsted type of mages for frst algorthm. And the second algorthm s tested on the mages when a part of face wth lps s avalable. 4. Proposed Method The overvew of the frst proposed method for extractng the lp contour from face s shown n Fgure converson from conventonal RGB color space to chromatc color space s defned as follows: r R R G B g G R G B ( (2 Where R, G, B denotes the ntenstes of pxels n red, green and blue channels respectvely. Accordng to the above transformaton, t s easy to see that the brghtness varaton n mages can be normalzed n chromatc color space. The color representaton n ths two-dmensonal chromatc color space enables us to vsualze and analyze very easly when buldng the color model for skn pxels. 2 lp ( r.776r.560r.223 ( r.560r.766 (4 As the objectve of our algorthm s to detect lps from face n real tme, we adopt a smple technque whch uses two quadratc polynomals to fnd out the upper and lower contour of lp regon usng equaton and 2. Ths extracts the darker regon of face wth a O(2 computaton cost. After applyng ths dscrmnate functon the pxels from darker part of face lke eyes, eyebrows, nose holes etc., are also ncluded n the extracted regon whch are removed by applyng threshold as explaned n the secton 4.2. b Applyng threshold As we know that the threshold vares for dferent skn color people hence we calculate two threshold condtons. One for those wth farer complexon and another for those who are havng normal complexon as gven below: For (FSC Farer Skn Color Threshold condton for farer skn color people s not havng lots of varaton n ther ranges, hence the ntensty value of red green and blue pxels le n approxmately same range. The threshold s calculated as: F( N 0 g lp( r & R 20 & G 20 & B 20 (6 Fg. flow chart for color based approach a Rules for extracton Ths algorthm adopts the chromatc color coordnate for color representaton. For chromatc color space, each pxel s represented by two values, denoted by (r, g. The 2 For (NSC Normal Skn Color As we all know there are lots of varaton n the normal skn color that mples varaton n the ntensty value of the pxels. Thus R, G and B values have varatons n ther range. The threshold s calculated as shown below. Where L (N = ndcates a lp regon pxel. And f(r and lp(r s taken from the equaton 3 and 4. We have used standard morphologcal operaton for mage enhancement and nose 266

3 removal, followed by edge detecton technque for fndng the edge detecton technque for fndng the edges of extracted component. Connected component labelng s also done for the better regon extracton. The algorthm s dscussed below: L( N 0 g lp( r & R 30 & 30 G 70 & 55 B 22 (7 Step. Morphologcal Operaton: After applyng threshold stray pxels as sde effects also called salt nose s left n the mage (I. Ths s removed by morphologcal Eroson operaton usng a square (sze varyng from 3-5 as the StrEl (structurng element as shown n equaton 8. I IStrEl (8 These small components from the face up to the sze of 5 pxels. The Eroson operaton also removes some part from the lp component whch s restored by Dlatng I (the lps as shown n equaton 9, usng Dsc as a StrEl2. I I SrtEl 2 (9 Ths restores the orgnal lps and also connects both upper and lower lp portons whch sometmes break due to Eroson. Both the StrEl s are shown n the table. Step2. Edge Detecton: To extract the upper and lower contour of lps whch are mostly dagonal edges we apply edge detecton technque usng Sobel edge detector as t smoothes the mage n the drecton perpendcular to edge detecton whch makes t a good edge detector for dagonal edges. Step3. Connected Component: As we know that the largest connected component n the face wll be lps, we have appled the 8-connected component algorthm to denty lps. Fgure 3 shows an nstance of the above steps for normal skn color people and Fgure 2 shows the result for farer skn people. Fg. 3 Result on Normal skn color people (a Input mage (b Applyng threshold condtons (c Morphologcal operaton and edge detecton (d Extracted lps. 5. Proposed method 2 Ths algorthm s desgned for specc set of frontal face mages wth open mouth, from the whole database. Ths model based algorthm uses Mean Shape Model (MSM. The database used for tranng s hand desgned. The algorthm conssts of four major steps: (a Tranng the data set, (b calculate MSM, (c normalze the MSM by algnng t wth the mn angle to prncpal axs, (d MSM deformaton. 5. Tranng Set The tranng set contans any set of 20 open mouth mages to make MSM. Open mouthed mages are selected to cater for the purpose of maxmum outer lp contour segmentaton of any nput mage n any sze database. Thus for ths experment, the tranng set contans only open mouth mages to make the large shape model whch covers up all other type of movement of lps by just nner deformaton of the model. 5.2 Mean Shape Model A SV (Shape Vector s defned manually for all mages of the tranng set. The SV conssts of 66 landmark ponts whch specy the outer lp contour as shown n Fg. 4 (b. These landmark ponts are chosen wth the constrant that these ponts exactly defne the lp contour shape. The th SV s defned as equaton 0. SV ( x, y, ( x 2, y 2,..., ( x 66, y 66 (0 After evaluatng the SV of all tranng set mages, the MSM s calculated usng equaton. Fg. 2 Result on farer skn color people (a Input mage (b Applyng threshold condtons (c Morphologcal operaton and edge detecton (d Extracted lps. SV N N SV ( Here, N s the number of landmark ponts for any tranng set mage. One nstance of MSM s shown n Fgure 4 (c. 267

4 5.3 Algnment of MSM The MSM calculated n the above step s algned wth mnmum angle wth the prncpal axs as shown n Fgure 4(d. To acheve the requred algnment we have to perform scalng, rotaton and translaton so that mage and MSM corresponds as closely as possble. 5.4 Mean Shape Model Deformaton After proper normalzaton of MSM, we start the deformaton of MSM to extract the lp contour of the nput mage. Before performng any deformaton, we threshold the RGB mage to a bnary mage (B usng equaton 2. B 0 R 20 & & G 20 & & B 70 The deformaton can be done as: (2 The MSM s dvded nto two parts- upper lp model and lower lp model. The upper lp model contans 35 landmarks ponts whereas; lower lp model contans 3 landmarks ponts of the MSM. 2 MC (model curve for these both parts s extracted usng curve fttng algorthm ( We fnd the polynomal of degree n that fts the data ponts x to. y Let suppose the mean shape ponts of upper lp contour are: X u x, x x2,..., Y u y, y y2,..., (3 (4 4 Fnd tangent normal for each pxel whch occurred n between the MSM and the lp contour of nput mage (ths s nothng but the ntensty change of the bnary mage. Normal lne s the lne perpendcular to the tangent lne to the curve. The slope of normal lne s -/(dy/dx and thus the equaton of normal lne s dy ( x x ( y y 0 dx (7 The results of model deformaton are shown n Fgure 4(e. Length of the normal vector of each tangent s bascally decded by the pont on the model curve and end pont of deformaton where ntensty change occurred between the lp and the skn. Here, we check the ntensty value at each sngle pxel ncrement of normal vector length. So, the task of pxel by pxel checkng becomes lttle bt tme consumng. To reduce ths tme, we can use the bnary search approach nstead of pxel by pxel approach. In ths approach, we can set the maxmum value of length for normal vector that s feasble under the model deformaton. For every teraton reduce the length by half and agan make a test of ntensty value at the pont. If that pont les on the lp then agan reduce the length by half ncrease the length up to half of prevous normal vector length. Repeat these tll we get a sngle pont. Ths s the end pont of the normal vector and whch defne the lp contour. A new SV s extracted that contans the landmark ponts whch defne the shape of our nput mage. An nstance of the fnal output of the deformaton of MSM s shown n Fgure 4(f. p n n ( x p. x p2. x... pn. x pn (5 Here, p, p 2 and p n+ are the polynomal coeffcent of a polynomal equaton. The degree n of the curve s chosen such that all data ponts related to * les on the polynomal equaton of the curve Where, * s ( x, y 3 Construct the tangent on the MSV curve. For example, y=p(x then slope of tangent s dy/dx. Then the tangent lne at ( x, y s defned as: y y dy.( x x dx (6 These tangents drawn on the model curve are equal to the number of landmark ponts whch defne the MSM. Fg. Lp contour extracton processng: (a Tranng set Image, (b Landmark pont represented n blue color (c MSM, red color defne the mean pont (d Algnment of MSM on nput/test mage (e Deformaton of MSM: green lne represents the normal vector (f Output after deformaton 6. Result analyss Algorthm s tested on 270 mages. 50 mages of farer skn color people wth dark red to purple color lps, and 20 mages of normal skn color people where, the test 268

5 mages contans both full face and part of face. The accuracy of the proposed algorthm s 9% for the normal skn, and 95% for the farer skn people, where the test mages contans full face, as 42 mages among 50 gves correct result for farer skn color people. And 0 among 20 gves correct result for normal skn color people. Algorthm 2 s tested on 50 mages of normal skn people database contanng frontal face mages havng mouth only. Out of 50 mages, 48 mages successfully fulflled the crtera for the acceptance of lp contour. Fgure 5 shows the results n detal. Both the algorthms work well even for low-resoluton mages, but performance degrades wth poor llumnaton. Conclusons Proposed algorthm gves good results for both skn color people, and works n dferent well-constraned condtons of database. It has an accuracy of 9% for normal skn color and 95% for far skn color. The accuracy s consstent across databases provded the lp color s red and mage qualty s good. Fg. 2 Hstrogram depcts the correctly classed mages The performance degrades the lp color s neutral and any other face feature s smlar to lp color. Ths s proposed as future enhancement of the algorthm. In algorthm 2 the lp contour segmentaton s performed by usng MSM deformaton approach whch gves good results for outer landmarks. Ths work can be further extended for nner landmark ponts of the mouth, whch defnes the poston of mouth. Ths could also denty how much lps are open whch s hghly requred n the vsual speech recognton systems. Both the algorthms work well for good qualty mages. Small degradaton s observed n poorly llumnated mages. References L.Yulle, D.S.Kohen, and P.W.Hallnan (989, Feature Extracton from Faces Usng Deformable Templates, In: Proceednds of IEEE Internatonal Conference Computer Vson and Pattern Recog., pp , (San Dego,CA. J.Luettn, N.A.Thacker, and S.W.Beet (996, Vsual Speech Recognton Usng Actve Shape Models and Hdden Markov Models, In Proc. IEEE Intl. Conf. on Acoustcs, Speech, and Sgnal. Vol. 2, pp , Atlanta, GA. M.Kass, A.Wtkn, and D.Terzopulos (987, Snakes: Actve Contour Models. In Intl. J. Computer Vson, Vol., No. 4, pp Yao Hongxun, L Yajuan, GaoWen (2002, Lp- Movement Features Extracton and Recognton Based on Chroma Analyss. In Acta Electronca Snca, pp H.Mehrotra, G.Agrawal, M.C.Srvastava (2009, Automatc Lp Contour Trackng and Vsual Character Recognton for Computerzed Lp Readng, Internatonal Journal of Computer Scence,pp.627. Cheng Chn Chang, Wen Ka Ta, Mau Tsuen Yang, Y Tng Huang, Ch Jaung Huang (2003, Novel method for detectng lps, eyes and faces n real tme (Elsever Ltd. Abu Sayeed, Md.Sohal and Prabr Bhattacharya (2007, Automated Lp Contour Detecton Usng the Level Set Segmentaton Method. In Proceedngs 4th Internatonal Conference on Image Analyss and Processng. M.Sadegh, J.Kttler and K.Messer (2002,. Modellng and segmentaton of lp area n face mages. In Proceedng of onlne confrence on Image sgnal processng, Vol. 49, No.3, June. Fahmeh Salm, Mohammad T Sadegh (2009, Decson Level Fuson of Colour Hstogram Based Classers for Clusterng of Mouth Area Images. In Proceedngs of Internatonal Conference on Dgtal Image Processng. Pham The Bao and Huynh Nguyen Duy Nhan (2009, A New Approach to Mouth Detecton Usng Neural Network In Proceedngs Internatonal Conference on Control, Automaton and Systems Engneerng. Jagdsh Lal Raheja, Radhey Shyam, Jatn Gupta, Umesh Kumar, P Bhanu Prasad (200, Facal Gesture Identcaton usng Lp Contours, In Proceedng of Second Internatonal Conference on Machne Learnng and Computng. Qang Wang, Hazhou A, Guangyou Xu (2002, A Probablstc Dynamc Contour Model for Accurate and Robust Lp Trackng, Proceedngs of the Fourth IEEE Internatonal Conference on Multmodal Interfaces (ICMI02. Y.P.Guan (2008, Automatc extracton of lps based on mult-scale wavelet edge detecton, In Proceedngs of IET Comput. Vs, Vol. 2, No., pp Lan.Matthews, T.F.Cootes, J.Andrew Bangham, Stephen Cox, Rchard Harvey (2002, Extracton of Vsual Features for Lpreadng In Proceedngs of IEEE Transactons on Pattern Analyss and Machne Intellgence, Vol. 24, No. 2 T.F.Cootes, C.J.Taylor, D.H.Cooper, J.Graham (995, Actve Shape Models- Ther Tranng and Applcaton In Proceedngs of Computer Vson And Image Understandng, Vol. 6, No., pp