Lip Contour Extraction based on Active Shape Model and Snakes

Transcription

1 48 IJCSNS Iteratioal Joural of Computer Sciece ad Network Security, VOL.7 No.0, October 007 Lip Cotour xtractio based o Active Shape Model ad Sakes Kyug Shik Jag Departmet of Multimedia gieerig, Dog-ui Uiversity, Busa, Korea Summary I this paper, we propose a efficiet method for extractig lip cotour. Lip shape deformatio is modeled by a statistically deformable model based o Active Shape Model(ASM). I the traditioal ASM, each ladmark poit is moved idepedetly to the best matchig poit with lip profile model, so it deforms the lip shape to implausible oe i may cases ad may cause may errors for extractig a correct lip cotour. I this paper, we have defied a eergy fuctio which is cosisted of a iteral eergy ad a exteral eergy. he ladmark poits are moved to positios at which the total eergy is miimized. he iteral eergy is used to miimize the differece of the displacemets of adacet ladmark poits ad keeps the cotour from bedig abruptly. he exteral eergy is what derives the cotour towards matchig the profile model. he experimets have bee performed for may lip images, ad showed very ecouragig result. Key words: Lip cotour, Shape deformatio, Active shape model, Sake. Itroductio Recetly, there is a icreasig requiremet for a system to track ad locate huma lip[, ]. Huma lip has much more iformatio tha ay other face features, so the lip iformatio could be used i image codig[]. o improve the performace of speech recogitio, the lip iformatio is used together with the acoustic sigal[3, 4]. he iformatio is also be applied to the graphic aimatio systems, which eed it for geeratig the lip shape of the speaker[, 4]. Accurately ad robustly trackig lip motio i image sequeces is especially difficult because lips are highly deformable ad they vary i shape ad color. Gradiet based techiques[5, 6] for edge detectio of lip ofte fail due to the poor cotrast betwee lip ad surroudig ski regio. For methods usig color iformatio to build a parametric deformable model for the lip cotour, these require optimizatio techique to refie estimates of cotour model to the huma lip[7, 8]. May papers have described the applicatios of active cotour model(sake) for lip boudary detectio[9, 0]. he sake methods are able to resolve fie cotour details but shape costraits are difficult to icorporate. Furthermore, the sake methods ofte coverge to the wrog result whe the lip edges are ot distict or whe the lip color is very close to the face ski. May methods have localized oly the outer lip cotour whe the mouth is closed because the presece of togue ad teeth could obscure the ier cotour whe the mouth is ope. Delmas[9] ad Lievi[0] have proposed a statistical approach based o markov radom field to segmet mouth usig the color iformatio ad the motio i a spatiotemporal eighborhood. his method has show good result at the outer lip cotour, but show bad result at the ier lip cotour. Methods usig Active Shape Model(ASM) have show good results i extractig ot oly the outer lip cotour but also the ier oe[4, ]. However they have show poor results i some cases. I the traditioal ASM[, 3], each ladmark poit is moved to the best matchig poit with its local profile model idepedetly, so it deforms the lip shape to much implausible oe. hat is, the resultat lip shape has a deformed cotour with abruptly varyig displacemets of adacet poits or a deformed cotour with bedig abruptly. As a result, it may cause may errors for extractig a correct lip cotour. hough the updated shape is resolved by proectig the shape ito the shape parameter space, most of errors occurred because each ladmark poits are moved idepedetly. I this paper, we propose a effective method for extractig lip cotour. he lip shape is represeted as a set of ladmark poits ad the lip deformatio is modeled by a statistically deformable model based ASM. I the traditioal ASM, each ladmark poit is moved idepedetly to the best matchig poit with its local profile model, so it deforms the lip shape to implausible oe ad may cause may errors for locatig a correct lip cotour. I this paper, we have defied a eergy fuctio which is cosisted of a iteral eergy ad a exteral eergy. he iteral eergy is used to miimize the differece of the displacemets of adacet ladmark poits ad keeps the cotour from bedig abruptly. he exteral eergy is what derives the cotour towards matchig the profile model. he ladmark poits are moved to positios at which the total eergy is miimized. he experimets have bee performed for may lip images of ulip database, ad showed that our method extract lip cotour tha a traditioal ASM more exactly. Mauscript received October 5, 007 Mauscript revised October 0, 007

2 IJCSNS Iteratioal Joural of Computer Sciece ad Network Security, VOL.7 No.0, October Lip Model. Lip Shape Model Lip shape is described by a set of 4 ladmark poits as show i Fig.. he lip shape of ith traiig image is described by a shape vector cotaiig the coordiates of the ladmark poits as show i equatio (). he lip shape model is represeted usig equatio (), where h is a mea shape, P is a matrix of the first t colum eigevectors correspodig to the largest eigevalues ad shape parameter b is a vector cotaiig the weights for each eigevector[4]. h ) = ( x, y, x, y, L, x, y () i i i i i 4i 4i Durig searchig lip cotour, we sample a profile F of legth M p (M p >N p ) either side of a poit of the lip cotour produced by the shape model. he ormalized profile F is derived by usig equatio (4). We make f by selectig a sample of legth N p at each of the M p -N p + possible positios alog F ad compare it with g which is the profile model at th ladmark poit. he updated locatio for curret poit is selected by choosig a poit that miimizes the fuctio i equatio (5). ' g i g = where, i ' g ik (4) ' g = { g g g, k =, L, N } i ik i ( k + ) ik p h = h + Pb where, P = λ i λ i + [ P P P LP ],, 3 t b = ( b, b, L, b ) t () profile 3. Sake = ( f g ) S ( f g ) (5) A sake is a eergy-miimizig splie guided by exteral costrait forces ad iflueced by image forces that pull it toward features such as edges[4, 5, 6]. he sake is represeted by a vector, v ( s) = ( x( s), y( havig arc legth, s, as parameter. he total eergy for the sake is the itegral of the eergy at each poit as equatio (6).. Lip Profile Model Fig. Lip shape model ad Itesity profile Lip profile model is costructed by statistical aalysis of the appearace of the image itesity i the eighborhood of each ladmark poit. As show i Fig., for every ladmark poit i the image i of the traiig set, we choose to sample oe dimesioal profile g of legth i N p perpedicular to the cotour ad cetered at the poit as show i equatio (3). It is sesitive to the light coditio to use the absolute gray level value, so we ormalize the profile g to obtai g usig equatio (4). A mea profile i i g ad a covariace matrix i S are derived. his is repeated for all ladmark poits at outer ad ier cotour, givig a profile model for each ladmark poit. g = ( g, g, g, L, g ) i i i i 3 inp where, g is a gray value. ik (3) ))) 0 sake + 0 it image = ds = ( s ds (6) he iteral eergy is represeted as equatio (7). he iteral eergy term tries to keep the sake smooth. I the origial sakes implemetatio, the first derivative term makes the sake act like a membrae(elasticity) ad is used to keep the sake from stretchig or cotractig alog its legth. he secod derivative term makes it act like a thi plate (stiffess) ad is used to keep the sake from bedig. Parameters α (s) ad β (s) are weightig parameters, ad cotrol the sesitivity with respect to the first ad secod derivative respectively. he derivatives may be approximated by fiite differeces. If v = ( x, y ) is a poit o the cotour, the approximatios i i i i equatio (8) is used. α ( s) v ( s) + β ( s) v ( s) it s ss = (7) v s) v v v ( s) v v v (8) ( + s i i ss i i i+ he image eergy term is what derives the sake towards matchig the image. It is usually iversely based o image itesity, gradiet magitude(edges), or similar image

3 50 IJCSNS Iteratioal Joural of Computer Sciece ad Network Security, VOL.7 No.0, October 007 features. I equatio (9), γ (s) is a approximate weightig fuctio. I equatio (0), the image eergy term is represeted as gradiet magitude, where I ( x, y) is a pixel value ad is a gradiet operator. image = γ ( s) (9) edge multiple adacet poits are moved towards a icorrect boudary so that the cotour locally fail to coverge towards the correct boudary. I this paper, we have defied a eergy fuctio similar to a eergy fuctio used i sake. ach ladmark poit is moved to positio at which the total eergy is miimized usig the eergy fuctio ad lip profile model. edge y = I( x, ) (0) 4. Lip Cotour xtractio o reduce the depedecy of iitial positio, we fid out a ceter lie of lip which coects two corer poits of lip. he y-coordiate of the lie is used as y-coordiate for the iitial search. For each colum of the image, y-coordiate of the smallest pixel value is foud by usig equatio (), where I(x, y) represets a pixel value at coordiate (x, y). H ad W represet image height ad image width, respectively. K ad K are costats. he highest peak of L(y) i equatio () gives y-coordiate of the ceter lie. L(y) for a sample image is show i Fig. ad the result is show i Fig.. I Fig., y-axis represets a image height, ad x-axis does L(y). M ( x) = arg mi I( x, y) () y H H M ( x) K cosh H = K L( y) () x W K Fig. Lip Ceter Lie ASM is a method that fids cotour based o the stadard model ad each ladmark poit is moved idepedetly. O the other had, sake has o stadard model ad each poit is moved based o the eergy of the cotour. I ASM, image regio i a eighborhood of each ladmark is examied ad each ladmark poit is moved to the best matchig poit with its profile model. Because each poit is moved idepedetly without ay costrait for the shape, the search result is a implausible shape as show i Fig. 3. I Fig 3, is a search result from, i which Fig. 3 Lip Shape after Search A lip cotour is geerated usig lip shape model i equatio () ad a iitial search poit is examied. A regio of a image i a eighborhood of each ladmark poit is examied, ad image feature profile for the ladmark poit is geerated. ach ladmark poit is moved to a poit that miimizes a eergy fuctio usig its profile model ad the image feature profile. he resultat shape is resolved by proectig the shape ito the shape parameter space ad by rescalig the shape parameter. he iterative search process is described below, where θ, t ad s are rotatio, traslatio (t x, t y ) ad scale factor respectively. Step. Geerate a lip cotour h from the lip shape model i equatio () usig shape parameter b, ad perform liear trasform usig equatio (3). Step. xamie the image regio i a eighborhood of each ladmark to fid poits that miimize a eergy fuctio as equatio (6), ad fid out cotour H. Step 3. Obtai θ, t, s ad h usig equatio (4) ad (5). Step 4. Fid out shape parameter b usig equatio (6). Step 5. Check the coditio for b usig equatio (7). If ot satisfied, it is rescaled usig equatio (8). Step 6. Iterate from step to step 5 util b ad ( θ, t, s) coverge. x cosθ siθ x t x ( s, θ, t) = s + (3) y siθ cosθ y t y ( s, θ, t ) = arg mi [( ( s, θ, t)( h) H ) ( ( s, θ, t)( h) H )] (4)

4 IJCSNS Iteratioal Joural of Computer Sciece ad Network Security, VOL.7 No.0, October h = ( s, θ, t )( H ) (5) b = P ( h h) (6) D = t i = k bi ( ) D (7) λ i max D max b = b (8) D he iteral eergy i equatio (6) is represeted as equatio (9). he first term is approximated as equatio (0), where v is the displacemet of ith ladmark i poit. he term is used to miimize the differece of the displacemets of adacet poits. he term pealizes a cotour for which the displacemets of adacet poits vary abruptly alog the cotour ad favors a cotour with smoothly varyig displacemet of adacet poits. he secod term is used to keep the cotour from bedig abruptly. As a result, the iteral eergy term tries to keep the sake smooth. he image eergy term is usually iversely based o gradiet magitude i sake. However, i this paper, it is what derives the ladmark poits towards matchig the profile model by usig equatio (5) ad (). I equatio (9) ad (), α( s), β ( s) ad γ (s) are approximate weightig fuctios. xamples of lip cotour extractio for all subects are show i Fig. 6. able shows the lip cotour extractio results by our method alog with oes by the traditioal ASM for all test images. All the test images were labeled by had ad were used as the groud truth. rue-positive meas that lip pixels are classified to lip pixels, ad false-egative meas that lip pixels are classified to o-lip pixels. rue-egative meas that o-lip pixels are classified to o-lip pixels, ad false-positive meas that o-lip pixels are classified to lip pixels. wo methods are compared i Fig. 7. Fig. 7 is a test image ad Fig. 7 is a iitial state. I the traditioal ASM, each poit is moved idepedetly without costrait. herefore it would geerate a cotour alog which the displacemets of adacet poits vary abruptly. As a result, it would geerate a implausible shape as show Fig. 7(c), ad the cotour may fail to coverge towards the correct boudary. However, i our method, the iteral eergy is used to miimize the differece of the displacemets of adacet poits ad keeps the cotour from bedig abruptly. As a result, we ca get a plausible shape as show Fig 7(d), ad the cotour may coverge towards the correct boudary. he result of our method ad the traditioal ASM are show i Fig. 7(e) ad Fig. 7(f) respectively. α ( s) Δv( s) +β ( s) v ( s) it ss = (9) Δv (0) ( s) Δv i Δvi image 5. xperimetal Result = γ ( s) () profile he experimets have bee performed for the images of ulip database of isolated digits[7]. It cosists of 96 gray image sequece of speakers each sayig the first four digits i glish twice. We referred to the set of words spoke the first times as Set ad the set of words spoke the secod times as Set. I this paper, the lip shape model was built usig 468 images from Set. We used shape modes t for the lip shape model. For experimet, 4 images i Set were used. A iitial value for parameter ( θ, t, s ) is (0, (Y Lip, W/), 70). Y Lip represets y-coordiate of a lie coectig two corer poits of lip, ad W is a image width. o evaluate the proposed method, we implemeted our method ad a traditioal ASM usig Matlab. Fig. 4 shows a result by our method, ad Fig. 5 shows oe by traditioal ASM. he result usig our method is more similar to real lip shape tha a result usig the traditioal ASM. Fig. 4 Result by our method Fig. 5 Result by the traditioal ASM he errors i the ier lip cotour occurred maily due to the gradiet origiatig from the teeth ad togue. For the

5 5 IJCSNS Iteratioal Joural of Computer Sciece ad Network Security, VOL.7 No.0, October 007 outer lower lip, some errors occurred because there was o obvious gray level differece betwee the lip ad their eighborhood ski. Fig. 6 Lip localizatio examples able : Lip cotour extractio performace Proposed method ASM ruth Positive % False Negative % rue Negative % False Positive % rror % Coclusio I this paper, we propose a efficiet method for extractig lip cotour. We defie a eergy fuctio which is cosisted of a iteral eergy ad a exteral eergy, ad the ladmark poits are moved to positios at which the total eergy is miimized. he proposed method was tested to may samples of various shapes ad the result showed that it extracted correctly lip shapes that were ot extracted by a traditioal ASM. he better performace may be obtaied by defiig the iteral eergy with more global iformatio o lip shape ad this is left to the ext topic. Refereces (c) (e) Fig. 7. Result compariso (d) (f) [] Mlrhosseil A. R., H. Ya ad K. M. Lam, "Adaptive Deformable Model for Mouth Boudary Detectio", Optical gieerig, Vol. 37 No. 3(998), pp [] Oliver N., A. Petlad, "LAFR: Lips ad Face Real ime racker", Proceedigs of the 997 Cof. o Computer Visio ad Patter Recogitio, (997), pp [3] Kaucic R.. A. Blake, "Accurate, Real-ime, Uadored Lip rackig", Proceedigs of the 6th Iteratioal Cof. o Computer Visio, pp , 998. [4] Iai Matths, imothy F. Cootes, J. Adrew Bagha, Stephe Cox ad Richard Marvey, "xtractoi of Visual Features for Lipreadig", I as. o Patter Recogitio ad Machie Aalysis, Vol 4, No.,, pp. 98-3, Feb. 00. [5] L. Zhag, "stimatio of the mouth features usig deformable templates", I Iteratioal Coferece o Image Processig, Vol. III, pp , 997.

6 IJCSNS Iteratioal Joural of Computer Sciece ad Network Security, VOL.7 No.0, October [6] M. Lievi, F. Lutho, "A Hierarchical Segmetatio Algorithm for Face Aalysis : Applicatio to Lipreadig", I Cof. o Multimedia & xpositio'000, August, 000. [7] Wark., Sridhara ad V. Chadra, "A Approach to Statistical Lip Modellig for Speaker Idetificatio via Chromatic Feature xtractio", Proceedigs of the 4th Iteratioal Cof. o Patter Recogitio, Vol., pp. 3-5, 998. [8]. Wark, S. Sridhara, "A Sytatic Approach to Automatic Lip Feature xtractio for Speaker Idetificatio", I Iteratioal Coferece o Acoustics, Speech ad Sigal Processig, p 7 -, 998 [9] Delmas P., Y. Coulo ad V. Fristot, "Automatic Sakes for Robust Lip Boudaries xtractio", I Iteratioal Cof. o Acoustics, Speech ad Sigal Processig, Vol. 6, pp , 999. [0] Lievi M., F. Lutho, "Usupervised Lip Segmetatio uder Natural Coditios", I Iteratioal Cof. o Acoustics, Speech ad Sigal Processig, Vol. 6, pp , 999. [] Luetti, J, ad hacker, NA, "Speechreadig usig probabilistic models," Computer Visio ad Image Uderstadig, vol. 65, pp , 997. [] M. B. Stegma, R. Fisker, "O Properties of Active Shape Models", Iformatics ad Mathematical Modellig, echical Uiversity of Demark, 000. [3] Cootes, aylor, Cooper ad Graham, "Active Shape Models-heir raiig ad Applicatio," Computer Visio ad Image Uderstadig, Vol. 6, No., pp , 995. [4] Michael Kass, Adrew Witki ad Demetri erzopoulos, Sake : Active Cotour Model, Iteratioal Joural of Computer Visio. v.,. 4, pp. 3-33, 987. Kyug Shik Jag received the M.S. ad Ph.D. degrees i lectroics gieerig from Yosei Uiversity, Seoul, Korea, i 99 ad 996, respectively. Durig , he stayed i Daewoo lectroics Co. Ltd., where he developed Video o Demad system. Sice 998, he oied the Multimedia gieerig, Dogeui Uiversity, Busa, Korea. Curretly he is a associate professor. His mai research iterests are face detectio, face recogitio, image processig ad multimedia system.