Enhance the Alignment Accuracy of Active Shape Models Using Elastic Graph Matching

Transcription

1 Enhance the Algnment Accuacy of Actve Shape Models Usng Elastc Gaph Matchng Sanqang Zhao 1,, Wen Gao, Shguang Shan, Baoca Yn 1 1) Multmeda and Intellgent Softwae Technology Lab, Beng Unvesty of Technology, Beng, Chna, 1000 ) ICT-ISVISION DL fo Face Recognton, Insttute of Computng Technology, CAS, Beng, Chna, {sqzhao, wgao, sgshan}@dl.ac.cn, ybc@but.edu.cn Abstact Actve Shape Model (ASM) s one of the most popula methods fo mage algnment. To mpove ts matchng accuacy, n ths pape, ASM seachng method s combned wth a smplfed Elastc Bunch Gaph Matchng (EBGM) algothm. Consdeng that EBGM s too tmeconsumng, landmaks ae gouped nto contou ponts and nne ponts, and nne ponts ae futhe sepaated nto seveal goups accodng to the dstbuton aound salent featues. Fo contou ponts, the ognal local devatve pofle matchng s exploted. Whle fo evey goup of nne ponts, two pe-defned contol ponts ae seached by EBGM, and then used to adust othe ponts n the same goup by usng an affne tansfomaton. Expemental esults have shown that the poposed method geatly mpoves the algnment accuacy of ASM wth only a lttle ncease of tme equement snce EBGM s only appled to a few contol ponts. Keywods Face Algnment, Actve Shape Models (ASMs), Actve Appeaance Models (AAMs), Elastc Bunch Gaph Matchng (EBGM), Pont Dstbuton Model (PDM) 1 Intoducton The accuate localzaton and algnment of facal featue ponts s of geat mpotance fo face ecognton, anmaton and tackng, etc. Ove the last two decades, vaous methods have been poposed to deal wth ths task. Afte Kass et al poposed Actve Contou Models (ACMs) [1], o snakes n 1987, Cootes and Taylo s Actve Shape Models (ASMs) [] and late Actve Appeaance Models (AAMs) [3] have poven to be among the most poweful tools n ths feld. ASM and AAM ae both based on statstcal models. In ASM, the devatve pofle, o local textue along the nomal of the shape bounday, s exploted to model the local featue of each landmak pont, and used to seach each landmak poston. Then the global shape models ae appled to adust the local seach esult based on the statstcal shape model. Yet, as futhe eseach shows [4, 5, 8], due to ts ambguous local-textue seachng stategy, ASM pefoms most successfully only when the obect o stuctue class s faly consstent n shape and ntensty appeaance. On the othe hand, AAM [3] combnes global statstcal shape models wth textue constants to buld appeaance models and uses a lnea pedcton to obtan the appeaance paametes fo optmzaton; thus the shape can be calculated by mnmzng the textue econstucton eo. To some extent, AAM may gve a qute good match to mage textue, but when the taget mage and backgound vay sgnfcantly, t s stll unable to locate featue landmaks accuately. Meanwhle, both the tanng pocess and the seachng pocedue of AAM ae qute complex and slow. Theefoe, much mpovement was put fowad to advance ASM fo face algnment. Gnneken et al [4] pesented a non-lnea gay-level appeaance nstead of the ognal devatve pofle to model the local textue, and got a bette matchng esult. Roges et al [5] suggested a obust paamete estmaton method usng M-estmato and andom samplng appoaches to evaluate the shape paametes moe elably. Howeve, ASM stll depends heavly on the ntalzaton and may easly be stuck n local mnma. Among othe methods, Wskott et al [6] constucted a stack-lke stuctue, called Face Bunch Gaph, and used t to seach the whole mage to fnd the pe-defned featue ponts. Though the teatng dstoton of the gaph, ths Elastc Bunch Gaph Matchng (EBGM) algothm can toleate a cetan degee of pose and expesson changes

2 and demonstate a successful esult. Howeve, snce ths method s Gabo featue based, ts tme-consumng nodes seachng pocess n the ente mage egon to lage extent confnes ts futhe pogess. On account of all above, n ou wok, the landmaks ae soted nto contou ponts and nne ponts, and the nne ponts ae futhe sepaated nto seveal goups accodng to the dstbuton aound salent featues n a face egon. Fo the contou ponts, the ognal nomal-based seachng stategy s opeated; fo evey goup of nne ponts, two contol ponts ae chosen, and a smplfed EBGM algothm s utlzed to adust them. Fully undestandng that the EBGM algothm s qute a tmeconsumng pocess, we select the contol ponts vey caefully and epesentatvely. When the new postons of the contol ponts ae dentfed, an affne tansfomaton s appled to adust othe ponts n evey goup. Ths pocess s seamlessly combned wth the ASM teatng seachng algothm and thus the accuacy can be geatly mpoved. The expemental esults show that ths method pefoms sgnfcantly bette, wth a speed not much slowe than standad ASM algothm. The emanng pat of ths pape s oganzed as follows. In Secton, the fundamentals of standad ASM ae descbed. The smplfed EBGM algothm s pesented n Secton 3. Afte the detaled demonstaton of ou algothm n Secton 4, the expemental esults ae lsted n Secton 5, and the last secton concludes the pape. Standad ASM ASM s defntely one of the best-suted appoaches n the shape detectng task. The advantage of ASM s that t allows fo consdeable vaablty but stll specfc to the class of obects o stuctue they ntend to epesent. The followng s a bef descpton of the standad ASM technque. Gven a set of annotated mages, the manually labeled n landmak ponts n each mage can be epesented as a vecto T x = x, y, x, y, L, x, ]. (1) [ y n 1 n 1 Afte algnng these vectos nto a common coodnate, pncpal component analyss s appled to get a set of othogonal bass P. Evey algned shape can then be appoxmately epesented as x x + Pb, () whee b s the shape paametes vecto. Meanwhle, n the standad ASM, Cootes et al [] used a nomalzed devatve pofle to buld the local textue models fo each landmak. The smlaty between a nomalzed devatve seach pofle g and the model pofle g can be s measued by ts Mahalanobs dstance, whch s a weghted squae eo functon T 1 f ( gs) = ( gs g) Cg ( gs g), (3) whee C g s the covaance matx of the nomalzed devatve pofle. Based on the Pont Dstbuton Model (PDM) and the gay level model, the seach pogess can then be opeated. Afte ntalzaton, each landmak n the model s optmzed by selectng the pont wth a mnmum dstance mentoned above n the decton pependcula to the contou wthn a cetan ange. As the esult of new shape s possbly mplausble, PDM s used to adust the shape paametes. Such pocedue s epeated untl no consdeable change s obseved. 3 A Smplfed EBGM Algothm 3.1 Gabo ets Repesentaton Gabo wavelet conssts of a plane snusod multpled by a Gaussan envelope, and t s fundamental to the EBGM algothm. Daugman [7] poneeed the -D Gabo wavelet epesentaton n compute vson n 1980 s. A famly of complex-valued -D Gabo kenel functons can be expessed as k k x σ ϕ x = exp exp k x exp, (4) σ σ wth k = k x kv cos u ( ) = ( ) k y k v sn u v + π, k v = π, = u u. (5) 8 Fgue 1. Gabo Wavelet. Real pat of 40 Gabo kenels. The shape of a wavelet.

3 And we employ 5 fequences v = 0, L, 4 and 8 oentatons u = 0, L, 7 wth ndex = u + 8v. Fgue 1 shows the eal pat of these 40 Gabo kenels. Thus, we can use a et to descbe the local fequency nfomaton aound each featue pont. If gven a pxel x n an mage wth gay level I ( x ), a convoluton pocess ( x ) = I ( x ) ϕ ( x x ) d x (6) wll poduce 40 complex coeffcents. Ths collecton of 40 Gabo coeffcents obtaned fo evey sngle mage pont s efeed to as a et, whch s used to epesent the local featues. Snce complex, a et can be wtten as = a exp( ), (7) whee magntudes a (x) vay slowly wth poston and phases (x) otate at a ate appoxmately detemned by the spatal fequency k of the kenels. When we obtan many ets efeng to the same featue pont n dffeent mages, ths set of ets s called a bunch. 3. Dsplacement Estmaton To measue the smlaty fo two ets, two smlaty functons ae appled. The fst one s magntude smlaty measue, whch does not use the phase nfomaton: a a. (8) S ( a, ) = a a It only computes the smlaty of the enegy of the fequences and thus vaes smoothly wth the change of the poston. As t s toleant of small dsplacements, ths method can be easly confused and may esult n an ncoect spatal featue. The second functon s phase smlaty measue: a a cos( dk ) S (, ) =, (9) a a whch s based on the smlaty of both the magntudes and the phase angles. Ths method uses a dsplacement d to compensate fo small shfts fom the ognal locaton, consdeng the phase nfomaton s apd change wth the dsplacement. The advantage of ths dsplacement estmated smlaty measue s that t can geatly educe the numbe of Gabo wavelet convolutons that need to be computed by extactng one et to epesent the egon nea the landmak. To detemne the dsplacement d between ets and, we can use a two-tem Taylo expanson to appoxmate the cosne tem n the phase smlaty functon (9) by S (, ) a a [1 0.5( a dk ) ]. (10) a So t s possble to solve the dsplacement by maxmzng (10). The fnal equaton fo calculatng d s: dx 1 Γyy Γyx Φx d(, ) = = (11) dy Γ Γ Γ Γ Γxy Γxx Φ xx yy xy yx y f Γxx Γyy ΓxyΓyx 0, wth Φx = a a k x ( ), Γ = xy a a k x k. (1) y Equaton (11) yelds a staghtfowad method fo estmatng the dsplacement between two ets taken fom obect locatons close enough that the Gabo kenels ae hghly ovelappng. When all fve fequency levels ae eventually used, the dsplacement wll be estmated accuately and quckly [6]. 3.3 A Smplfed EBGM Algothm As [6] pesented, the standad EBGM algothm uses a coase-to-fne pocedue to fnd the landmaks and thus to extact fom the mage a gaph that can maxmze the smlaty wth the Face Bunch Gaph (FBG), whch s a stack-lke stuctue that seves as a epesentaton of both ets and edges nfomaton. Howeve, ths pocedue s qute a slow pocess, because the fnal optmal mage gaph s obtaned by dstotng FBG n vaous szes and postons ove the ente mage egon, and that actually needs evey pe-defned featue pont to seach n a lage ange. Fotunately, n most pactcal applcatons, ASM can povde a qute close appoxmaton to the ntal landmak postons, whch makes the ough seachng stage n the EBGM unnecessay. Besdes, the PDM n ASM can successfully confne the landmak locatons to a class of acceptable geometcal stuctues, so edge constants n EBGM s not needed to patcpate n the smlaty measue fo ou algothm. In ths sense, based on [6], a smplfed EBGM algothm s pesented as follows: 1) Intalze the ough poston of the landmaks by ASM. ) Fo a landmak x that needs to be accuately postoned, calculate ts et usng equaton (6). 3) Fom the bunch coespondng to that landmak, th select the et as a model et, and use equaton (11) to calculate an estmated dsplacement and m m d, wth all fve fequency levels used. between ets 4) Use equaton (9) to compute the smlaty wth the dsplacement d S between and m

4 obtaned fom the fome step. Then fnd anothe et n the bunch and go to step 3) to epeat the above pocedues untl all the ets n the bunch have been opeated. 5) Fnd the hghest S value as the best one: * = ag max { S }. And the coespondng d s * used as the optmal dsplacement fo the landmak x. 6) Change to the next landmak to be adusted and go to step ) fo anothe seachng pocess. One pont to be notced s that, n step 4), we do not take ; nstead, we a et fom poston x = x + d appoxmate the smlaty between and computng the smlaty between and dsplacement d m by m wth the. Ths s because equaton (9) can estmate the smlaty as f was extacted fom a dsplacement d fom ts cuent locaton [9]. As a whole, et computaton fo a landmak seachng pocess s only once n ths smplfed EBGM algothm, whch wll educe computatonal effot geatly. 4 ASM Combned wth EBGM Seachng Stategy 4.1 Futhe Adustment fo Is Locatons The pope ntalzaton of the mean shape s ctcal to ASM seachng pogess n the sense that a good ntalzaton would be less possble to lead to ncoect local mnma. The standad ASM patally solves ths poblem by mult-esoluton stategy at the cost of moe expendtue spent dung tanng n evey esoluton level. In most cuent face detecton systems, two ses, thanks to the natual dstnct featue, can be located easly and quckly, though sometmes not vey accuately (Fgue a). And t has been poved that two ses can povde enough nfomaton to ntalze the tanslaton, the scale and the otaton angle fo the mean shape model [8]. Expements show ses slght naccuacy wll nduce ASM s sgnfcant ntalzaton eo (Fgue b). Theefoe, we use the smplfed EBGM algothm n Secton 3.3 to efne the face detecto s esult. Fgue (c) gves an example. Ths efnement esult also eveals the accuate seachng pefomance of ths algothm. 4. Featue-Based Landmak Goupng and Algnment The pupose of ths eseach s to use EBGM seachng method to efne the ASM seachng pefomance so that we can get a moe desable localzaton. So t s natual to thnk about changng all the nomal-based seachng stategy nto EBGM algothm, ust as [10] poposed. Unfotunately, even when EBGM s smplfed, t s stll qute slowe than ASM s nomal-based seachng method; moeove, expements show that EBGM algothm does not always pefom bette than the nomal-based seachng method n some outlyng ponts along the mage bode [11]. (c) (d) Fgue. Ises' slght naccuacy wll nduce ASM's sgnfcant ntalzaton eo. An example of ses naccuacy. Sgnfcant AS M ntalzaton eo fom. (c) EBGM efnement to. (d) ASM ntalzaton fom (c). Fgue 3. Face landmaks goupng. A manually labeled face mage wth 103 landmaks. 78 nne ponts sepaated nto sx goups; ed cosses epesentng contol ponts. Unde ths condton, we sot the face landmaks (Fgue 3a) nto contou ponts and nne ponts; the nne ponts

5 ae futhe sepaated nto seveal goups based on the dstbuton aound salent featues n a face egon. Fgue 3 shows an example of sx goups of nne ponts coespondng to two bow aeas, two eye aeas, one nose aea and one mouth aea. Fo the contou ponts, the ognal nomal-based seachng stategy s opeated; as to evey goup of nne ponts, we select two contol ponts, whch ae used to contol all the othe nne ponts n the same goup. Snce these two contol ponts ae vey mpotant, the smplfed EBGM algothm s utlzed to efne them. To speed the seachng pefomance, we defne only 11 epesentatve contol ponts (Fgue 3b) totally n all sx goups. As an excepton, one of the contol ponts n the nose aea s omtted, fo t can be estmated fom othe contol ponts n the two eye aeas. Afte the new postons of contol ponts ae dentfed by the smplfed EBGM algothm, we apply an affne tansfomaton to adust othe ponts n evey coespondng nne goup. Actually ths s a poblem of algnment between two sets of featue ponts. We defne ths algnment as the otaton, scalng and tanslaton whch mnmzes the sum of squaed dstances between pas of coespondng ponts, then such a coodnate tansfomaton n -D can be wtten n the followng fom: x x scosθ ssnθ x tx a b tx = + = y. (13) y ssnθ scosθ y t y b a ty 1 Snce we have two coespondng sets of ponts (two contol ponts befoe and afte efnement), the equaton (13) can be futhe ewtten as follows: x1 y a x1 y 1 x1 0 1 b y 1 = (14) x y 1 0 t x x y y 0 1 t y y By equaton (14), we can compute the coespondng affne tansfomaton paametes, and then the emanng nne ponts postons n the same goup wll be easly calculated by equaton (13). Fgue 4 gves an nstance of efnement to landmaks n the mouth aea usng ou algothm. 4.3 ASM Combned wth EBGM Seachng Stategy In standad ASM matchng pocess, evey pont along the nomal of the shape contou s seached to fnd a best one, and the accuacy of evey seachng step s vey mpotant to the fnal esult. Based on the above landmak goupng and algnment, ou algothm eaches a compomse between computatonal effot and seachng accuacy. The ente seachng pocedue of ou algothm s lsted as follows: 1) Intalze the statng shape by the model shape and two s locatons, whch ae dentfed by a face detecto and efned by the smplfed EBGM algothm. See Fgue 5. ) Fo the contou ponts, use the standad nomalbased seachng method to fnd the equed movements. 3) Fo evey goup of nne ponts, the smplfed EBGM algothm s opeated fo seachng the two contol ponts. Afte that, the method n Secton 4. s used to adust othe ponts n the same goup. We then get a new shape (Fgue 5c). 4) Calculate addtonal pose and shape paamete changes equed to move the model shape as close as possble to the new shape. 5) Update the shape and pose paametes by the above changes, and act on the model shape. Tll now, we fnsh an teaton step. Go to step ) fo anothe matchng ccle untl no consdeable change s obseved. (c) Fgue 5. Compason between ASM and ou algothm seachng pocess. Intalzaton. Standad ASM seachng esult. (c) ASM combned wth EBGM seachng stategy. 5 Expemental Results Fgue 4. Landmak algnment. Intal landmaks n the mouth aea. Result of EBGM adustment to. Ou expement s based on a 500 manually labeled faces database, most of whch ae nea fontal. We have nomalzed 350 of them fo tanng the PDM and othes fo testng; we also selected 70 epesentatve face mages fo tanng the EBGM bunches. To evaluate the pefomance of ou algothm, the aveage Eucldean dstance eo s calculated by the followng equaton:

6 N n 1 1 E = ( x x) + ( y y), (15) N = 1 n = 1 whee N s the total numbe of test mages, and n s the numbe of the landmaks n one face mage. x, y ) s the ( th manually labeled landmak locaton of the th test mage; and x, y ) s the coespondng ( th landmak locaton we calculated. We also calculate the oveall mpovement pecentage of ou algothm to the standad ASM by: EASM EASM EBGM I = 100%. (16) E ASM Table 1 lsts ou fnal expemental esults based on 150 test mages. The aveage eo fo standad ASM s about 3.08 pxels. Afte we apply EBGM to efne two s locatons, the eo s educed to.54 pxels. The oveall mpovement of ou algothm s about 39.9% compaed wth the standad ASM. Table 1. Pefomance of ou algothm Method Aveage Impovement Eo ASM 3.08 ASM wth s -efnement % ASM combned wth % EBGM (a1) (b1) Fgue 6. Matchng compason between standad ASM and ou algothm. (a1), (b1) Result of ASM., Result of ou algothm. Fgue 6 lsts some matchng esults of ou algothm as well as standad ASM. Though compason between them, we can see ou algothm mpoves the seachng accuacy sgnfcantly, especally that of the nne ponts. 6 Conclusons and Futhe Wok In ths eseach, we combne the standad ASM algothm wth a smplfed EBGM to mpove the algnment accuacy. Because EBGM has the advantage of Gabo wavelet epesentaton, the landmaks can be moe accuately localzed compaed wth usng gay level epesentaton. In ode to each a compomse between the seachng effcency and the algnment accuacy, we classfy all the landmaks nto contou ponts and nne ponts, and futhe sepaate the nne ponts nto seveal goups accodng to the dstbuton aound salent featues n a face egon. We apply the smplfed EBGM algothm to accuately localze the two contol ponts of evey goup, and then, an affne tansfomaton s used to adust othe ponts n the coespondng nne goup. Expemental esults show that ou algothm pefoms much bette than the standad ASM. Futue wok wll be focused on pefectng a obust EBGM seachng stategy to deal wth face mages wth otaton n depth and poo qualty. Meanwhle, applyng the poposed method to face ecognton s also a pactcal eseach effot. Acknowledgements Ths eseach s patally suppoted by Natonal H-Tech Pogam of Chna (No. 001AA114160, No. 001AA and No. 00AA118010), Natual Scence Foundaton of Chna (No and No ), Natonal 973 Pogam of Chna (No. 001CCA03300), Natual Scence Foundaton of Beng of Chna (No. D ) and ISVISION Technologes Co., Ltd. Refeences [1] M. Kass, A. Wtkn and D. Tezopoulos, "Actve Contou Models", 1st Intenatonal Confeence on Compute Vson, London, une, 1987, pp [] T.F. Cootes, C.. Taylo, D.H. Coope, and. Gaham, "Actve Shape Models - The Tanng and Applcaton", Compute Vson and Image Undestandng, 61(1), 1995, pp [3] T.F. Cootes, G.. Edwads, and C.. Taylo, "Actve Appeaance Models," Poceedng of the 5th Euopean Confeence on Compute Vson, vol., 1998, pp [4] B.V. Gnneken, A.F. Fang et al, "A Non-lnea Gaylevel Appeaance Model Impoves Actve Shape

7 Model Segmentaton", IEEE Wokshop on Mathematcal Models n Bomedcal Image Analyss, MMBIA 001, pp [5] M. Roges and. Gaham, "Robust Actve Shape Model Seach", Poceedngs of the Euopean Confeence on Compute Vson. May, 00. [6] L. Wskott,.M. Fellous, N. Kuge et al, "Face Recognton by Elastc Gaph Matchng", IEEE Tansactons on Patten Analyss and Machne Intellgence, Vol. 19, No. 7, uly, [7].G., Daugman, "Complete Dscete -D Gabo Tansfom by Neual Netwok fo Image Analyss and Compesson", IEEE Tansactons on Acoustcs, Speech and Sgnal Pocessng, 36(7), 1988, pp [8] W. Wang, S. Shan, W. Gao and B. Cao. "An Impoved Actve Shape Model Fo Face Algnment", The 4th Intenatonal Confeence on Mult-modal Inteface, IEEE ICMI 00, Pttsbugh, USA, pp , Oct., 00. [9] D.S. Bolme, "Elastc Bunch Gaph Matchng", Mastes Thess, CSU Compute Scence Depatment, une, 003. [10] F. ao, S. L, H. Y. Shum and D. Schuumans, "Face Algnment Usng Statstcal Models and Wavelet Featues", Intenatonal Confeence on Compute Vson and Patten Recognton, CVPR 003. [11] B. Zhang, W. Gao, S. Shan and W. Wang. "Constant Shape Model Usng Edge Constant And Gabo Wavelet Based Seach", 4th Intenatonal Confeence on Audo and Vdeo Based Bometc Peson Authentcaton, AVBPA 003.