Lip Segmentation with the Presence of Beards

Transcription

1 Lip Segentation with the Preence of Beard Author ang, S., Lau, W., Leung, S., Liew, A. Publihed 004 Conference Title Proceeding of the IEEE International Conference on Acoutic, Speech, and Signal Proceing, ICASSP'04 DOI Copyright Stateent 004 IEEE. Peronal ue of thi aterial i peritted. Howeve periion to reprint/ republih thi aterial for advertiing or prootional purpoe or for creating new collective work for reale or reditribution to erver or lit, or to reue any copyrighted coponent of thi work in other work ut be obtained fro the IEEE. Downloaded fro Griffith Reearch Online

2 LIP SEGMENTATION WITH THE PRESENCE OF BEARDS S.L.Wang, W.H.Lau, S.H.Leung * and A. W. C. Liew Departent of Coputer Engineering and Inforation Technology *Departent of Electronic Engineering City Univerity of Hong Kong, 83 Tat Chee Avenue, Hong Kong ABSTRACT 1 Lip iage analyi ha attracted uch interet in recent year becaue oe iportant peech inforation i contained in the hape and oveent of the lip. To extract uch inforation fro the lip iage, accurate and robut lip region egentation i of vital iportance. Howeve ot of the current lip egentation ethod fail to provide accurate reult if the peron ha beard. In thi pape we propoe a one object, ultiple background clutering ethod to olve the proble. Since the non-lip region becoe inhoogeneou in the preence of beard, ultiple background cluter can produce better fitting to a rather coplex background region than one ingle cluter. Spatial inforation in ter of the phyical ditance toward the lip center i incorporated to enhance the differentiation between the lip and background region. Experiental reult deontrate that our algorith provide accurate lip egentation reult for the iage with beard. 1. INTRODUCTION The perforance of autoatic peech recognition (ASR) yte decreae dratically due to the preence of the noie. It ha been hown that incorporating viual inforation extracted fro lip iage analyi can greatly enhance the accuracy of peech recognition yte olely baed on acoutic ignal [1,]. Lip region egentation becoe very iportant ince it provide lip pixel inforation for the ubequent lip odeling proce. Many reearcher have propoed variou algorith for robut lip region egentation. Soe ethod are baed on color pace analyi, i.e., egenting lip region directly fro the color pace [3] or tranforing the color repreentation to other color pace to enlarge the color difference between the lip and the background [4]. Thee algorith can egent the lip region in a very efficient anner. Howeve it i difficult to find the optiu value of the threhold etting for variou lip iage. Markov Rando Field (MRF) technique have been ued in oe lip egentation algorith [5,6] to iprove the perforance by exploiting the local patial inforation. The hue edge are alo conidered to provide ueful inforation for lip egentation [6]. Thee MRF baed The work decribed in thi paper i fully upported by a reearch grant (CityU 115/01E) fro the RGC of the HKSAR, China. ethod perfor well in dealing with pepper noie. Fuzzy clutering i another kind of widely ued iage egentation technique. The claical fuzzy c-ean (FCM) algorith attept to aign a probability value for each pixel to iniize a fuzzy entropy eaure [7]. A the clutering ethod i an unupervied learning ethod where neither prior auption about the underlying feature ditribution nor training i needed, it i able to handle variou lip and kin color due to different huan race or ake-up. Howeve for lip iage with variou kind of beard, ot of the ethod entioned above cannot egent the lip region accurately. In thi pape we propoed a new fuzzy clutering algorith that provide accurate lip region egentation even for the lip iage with beard. Detail decription of the objective proble i hown in Section. In Section 3, the theory and ipleentation of the propoed algorith are preented. Experiental reult are given in Section 4 and Section 5 draw the concluion.. OBJECTIVE PROBLEM DESCRIPTION Segenting lip iage with beard of different color in the kin region reain an open quetion for ot of the previou lip egentation technique [5]. In order to deontrate the difficulty of egenting thee iage, an iage with beard i hown in Fig.1 (a) along with the color ditribution of the lip pixe l (in red) and non-lip pixel (in blue) in the CIELAB color pace. To facilitate the invetigation of the uefulne of edge inforation in egentation, the edge ap of the hue [6] and luinance of the original iage are alo given in Fig. 1. Three iportant apect of the lip iage are oberved fro Fig. 1. Firtly, the pixel of lip region overlap with the background pixel in the color pace (Fig1 (b)-(e)). Thu, ethod olely depend on the color inforation will experience unolvable trouble in partitioning thee overlapping pixel. Exploiting local patial inforation uing MRF ay iprove the perforance to oe extent. Howeve if the nuber of pixel of iilar color i large enough, hole inide the lip and patche outide are often oberved. Secondly, a the color ditribution of the background region i quite coplex, the traditional one background cla i not enough to odel the background region ufficiently. Thirdly, there are any erroneou edge in the edge ap a hown in Figure 1(f) and 1(g) and they

3 can hardly be ued a an indicator of the boundary inforation between the lip and the background. (b) (d) (a) (f) (g) Fig.1 (a) original lip iage, (b) color ditribution of (a) in CIELAB color pace, (c)-(e) color ditribution projection on the L-a, b-a, L-b plane, repectively, (f) the edge ap of the hue iage, (g) the edge ap of the luinance iage. 3. THE PROPOSED FUZZY CLUSTERING ALGORITHM 3.1 General decription In order to overcoe the difficultie entioned above, we propoe a one object, ultiple background clutering ethod for lip region egentation. In our algorith, one cluter i et for repreenting the lip region and everal cluter for the background. The phyical ditance toward the lip center i included in the diiilarity eaure a the patial inforation. We can forulate the patial diiilarity eaure of the lip and non-lip repectively and thu the differentiation between the lip region and the background i enhanced. The purpoe of uing ultiple background i to ufficiently odel the non-lip region. Since the FCM clutering ethod ue Euclidean ditance a it diiilarity eaure, the cluter extracted by FCM are of pheroidal hape. Thu, one cluter i not ufficient to odel the non-lip region due to the large color variation in the background. Increaing the nuber of cluter (c) (e) iprove the fitne of the background cluter catering for the wide color ditribution. 3. Objective function forulation Let X={x 1,1,, x,, x N,M } be the et of feature vector aociated with an iage I of ize N by M, where x R q i a q-dienional color vector for a pixel located at (). Le t d i, repreent the Euclidean ditance between the feature vector x and the centroid v i of the i th cluter (i=0 for the lip cluter and i 0 for the background cluter). A patial ditance eaure i deigned in an elliptic for ince the hape of the outer lip contour i reebled to an ellipe. Let p = {x c, y c, w, h, θ} denote the paraeter et that decribe the elliptic ditance, where (x c, y c ) i the center of a of the ellipe, w and h are repectively the ei-ajor axi and ei-inor axi, and θ i the inclination angle about (x c,y c ). With the lip cluter pecified, the diiilarity eaure cobining both the color and patial inforation, denoted by D i,, i forulated a follow: ρl Di, = di, + α dit( ) i = 0 (1) ρ B D = d + α dit( ) i 0 i, i, where (( r xc) coθ + ( yc ) inθ ) (( yc) coθ ( r xc ) inθ ) dit( ) = +. w h The weighting paraeter α in (1) i for adjuting the weight of the phyical ditance againt the color feature. The exponent ρ L and ρ B deterine the hape of the patial diiilarity function in lip and background cluter. Generally they are aigned with poitive value o that the patial diiilarity provide a correct guidance for differentiating the lip and background region. The objective function of our algorith i given by, C 1 J ( U, V, p) = u i, r, Di, = J c + α ( J L + J B ) r= 1= 1 i= 0 () C ubject to 1 u = 1, ( I (3) i = 0 i, r, ) where the N M C atrix U Mfc i a fuzzy c-partition of X, V={v 0,v 1, v C-1 } R Cq with v i R q i the et of fuzzy cluter centroid, (1,8 ) define the fuzzine of the clutering and the value u i, give the eberhip of the ()-th pixel in the fuzzy cluter C i. J c in () i the ae a the objective function for fuzzy c-ean, which i reponible for the color part. J L and J B repreent the patial cot function of lip and background cluter, repectively. 3.3 Paraeter updating ethod Since the optiu olution for in{j (U,V,p)} i the tationary point of the objective function J, Picard iteration i ued to olve for the optiu point (U *,V *,p * ). The derivation for the paraeter updating forula in each

4 iteration i outlined in the following. By keeping V R cq and p R 5 unchanged and taking the partial derivative of J with repect to U ubject to the contraint in (3), we have: 1 1/( 1) + ( ),, = C u i r Di, r, D j, r, (4) j = 0 Next, we fix U Mfc and p R 5 and take the derivative of J with repect to V. Since both J L and J B reain contant when U Mfc and p R 5 are fixed, the derivative i the ae a that of FCM. Hence we have: v + = i u i, r, xr u (5), i, r, r= 1= 1 r = 1= 1 Siilarly, the partial derivative of J with repect to p i given by: ( U, V, p) c L B = + α + (6) It hould be noted that the firt ter of the right hand ide in (6) vanihe ince the patial paraeter et p i independent to the color part of the objective function. Setting the derivative equal to zero, we have: C 1 ρ 1 1 ditr (, ) u0,, ditr (, ) L ρ ρl r ρ u,, ditr (, ) B B i r = 0 r= 1 = 1 i = 1 (7) Since directly coputing p + fro (7) i quite coplicated, intead the Conjugate Gradient (CG) ethod i adopted to olve p + nuerically for it fat convergence. In each paraeter updating proce, the objective function J i a continuou function in (U,V,p) with poitive value and it i alway decreaing. Hence, the iterative proce decribed above ut converge to a local iniu of J. 3.4 Ipleentation procedure In our experient, the RGB lip iage are quantized to 8 bit per color. Since the RGB color pace i not viually unifor, the lip iage will be tranfored to the approxiately unifor CIELAB and CIELUV color pace with detail given in [8]. The color vector { L *, a *, b *, u *, v * } i ued to repreent the color inforation of pixel. Since the preence of teeth can diturb the eberhip ditribution which in turn caue bia to the color centroid, the ethod decribed in [9] i ued to ak the teeth pixel if teeth do appear in the iage. The clutering algorith run a follow: 1. Initialize the color centroid V.. With the patial diiilarity et to zero, copute the initial eberhip ditribution U given in (4). 3. Update V given in (5) and copute the patial paraeter et p uing the CG ethod. 4. Calculate the diiilarity eaure in (1) and update U given in (4). 5. Repeat tep 3 and 4 for k = 1,, 3, until 1 ( t + 1) t U U < ε or k=k T ax, where ε T i a all threhold and k ax i the axiu nuber of iteration. After clutering, the eberhip value in each cluter are oothed by a 3x3 Gauian low-pa filter before egentation take place. The final lip-background egentation i a hard claification proce by aigning the pixel to the lip cluter if the correponding eberhip value for the lip cluter i the highet aong all the cluter. 4. EXPERIMENTAL RESULTS 110 lip iage with beard and 90 lip iage without beard have been taken fro the AR Face Databae [10] for teting the perforance of our algorith. In our experient, the fuzzine i et to and the paraeter et {α,ρ L,ρ B } in (1) are et to {10,5,3} epirically. Thee choice are hown to provide good balance between the color and patial inforation. Since the etiation of an appropriate nuber of background cluter i rather coplicated and tie -conuing, the nuber of background cluter i then initially et to one, i.e. C = 1. After clutering, the tandard deviation of the color ditance to the centroid for all pixel in each cluter i calculated. The clutering proce will be repeated with C=C+1 if any of thee tandard deviation i larger than a preet value σ (σ =7 in our experient). To find the initial color centroid, FCM i applied to analyze an arbitrary iage. The color centroid of the cluter ituated in the center portion of the iage i aigned a the lip cluter. Thee initial color centroid are ued for all the 00 lip iage. An exaple iage with beard i ued to illutrate the egentation reult, a hown in Fig., of the propoed algorith. The reult obtained fro both the traditional FCM and the Zhang ethod [6] are alo given for coparion. It can clearly be een fro Fig. that the egentation reult produced by our algorith i uch better than that of the other. It ha been oberved that the traditional FCM approach i unable to differentiate the color iilarity for oe of the lip pixel and non-lip pixel. A a reult, patche cloe to or far away fro the lip are reulted a hown in Fig. (b). The two-cla etting of Zhang ethod doe not produce atifactory egentation reult a hown in Fig. (c). Even with the aid of the hue edge, it perforance i till unatifactory. To invetigate the nuber of cluter required for egenting the lip region fro the 110 iage with beard, it i found that 1 iage require cluter, 9 iage require 3 cluter and only 6 iage require 4 cluter. It i clear that ajority of the iage require 3 cluter for egenting the lip region accurately. If the color of the beard i cloe to that of the kin or it i light colo cluter are ufficient for the egentation. In order to objectively evaluate the perforance of the propoed algorith, we apply a quantitative technique

5 decribed in the following. The boundary of the lip for the original iage in Fig. i anually drawn. The Segentation Error (SE) i defined a [11]: SE = P( O) P( B O) + P( B) P( O B) (7) where P(B O) i the probability of claifying background a object, P(O B) i the probability of claifying object a background. P(O) and P(B) are the a priori probabilitie of the object and the background of the iage, repectively. The SE a well a the two iclaifying probability of the three algorith are hown in Table 1. It i oberved fro the table that the SE of our algorith i uch aller than that of the other two algorith. More egentation reult for iage with beard produced by our algorith are given in Fig. 3. The SE for the 00 lip iage elected fro the AR Face Databae i ranged fro 0.5% to 5%. Thee reult deontrate that the egented lip region obtained by our algorith fit well to the actual lip. P(B O) P(O B) SE (%) FCM Zhang Our ethod Table.1 Coparion of P{B O}, P{O B} and SE aong FCM, Zhang and our ethod for the lip iage hown in Fig.. 5. CONCLUSIONS A fuzzy clutering baed algorith for egenting the lip region in the preence of beard i preented in thi paper. Since the background region of a lip iage i inhoogeneou, we forulate the lip region egentation proce a a one object, ultiple background clutering proble. With the aid of the patial inforation which i incorporated in the diiilarity eaure, our algorith i able to differentiate pixel with iilar color feature but located in different region. Experiental reult how that our algorith provide accurate lip egentation reult even for lip iage with beard. 6. REFERENCES [1] M. T., Chan, HMM-baed audio-viual peech recognition integrating geoetric and appearance-baed viual feature, Proc of IEEE 4 th Workhop on Multiedia Signal Proceing, pp. 9-14, Canne, Oct [] E. D. Petajan, Autoatic Lipreading to enhance Speech Recognition, Proc of IEEE Conf on Coputer Viion and Pattern Recognition, pp , [3] T. Wark, S. Sridharan, V. Chandran, An approach to tatitical lip odelling for peaker identification via chroatic feature extraction, Proc of 14 th Int l Conf on Pattern Recognition, vol.1, pp.13-15, Bribane, Aug [4] N. Eveno, A. Caplie P. Y. Coulon, New color tranforation for lip egentation, Proc of IEEE 4 th Workhop on Multiedia Signal Proceing, pp. 3-8, Canne, Oct [5] M. Lievin, F. Luthon, Lip feature autoatic extraction, Proc of IEEE ICIP, vol.3, pp , Chicago, March [6] X. Zhang, R. M. Merereau, Lip feature extraction toward an autoatic peechreading yte, Proc of IEEE ICIP, vol.3, pp.6-9, Vancouve Sept [7] J. C. Bezdek, Pattern recognition with fuzzy objective function algorith, Plenu Pre, New York, [8] R. W. G.. Hunt, Meauring Colo nd Ed., Elli Horwood Serie in Applied Science and Indutrial Technology, [9] A. W. C. Liew, S. H. Leung and W. H. Lau, Segentation of color lip iage by patial fuzzy clutering, IEEE Tran on Fuzzy Syte, vol.11, pp , Augut 003. [10] A. M. Martinez, R. Benavente, The AR face databae [Online] CVC Technical Report #4, June [11] S. U. Lee, S. Y. Chung, R. H. Park, A coparative perforance tudy of everal global threholding technique for egentation, Coputer Viion, Graphic and Iage Proceing, vol. 5, pp , (a) (c) (d) Fig. (a) The original lip iage; egentation reult by: (b) FCM; (c) Zhang ; and (d) the propoed ethod. (b) Fig.3 More lip egentation reult obtained by the propoed algorith (with the lip region hown in white color).