Mult-Vew Face Algnment Usng 3D Shape Model for Vew Estmaton Yanchao Su 1, Hazhou A 1, Shhong Lao 1 Computer Scence and Technology Department, Tsnghua Unversty Core Technology Center, Omron Corporaton ahz@mal.tsnghua.edu.cn Abstract. For mult-vew face algnment (MVFA), the non-lnear varaton of shape and texture, and the self-occluson of facal feature ponts caused by vew change are the two major dffcultes. The state-of-the-art MVFA methods are essentally vew-based approaches n whch vews are dvded nto several categores such as frontal, half profle, full profle etc. and each of them has ts own model n MVFA. Therefore the vew estmaton problem becomes a crtcal step n MVFA. In ths paper, a MVFA method usng 3D face shape model for vew estmaton s presented n whch the 3D shape model s used to estmate the pose of the face thereby selectng ts model and ndcatng ts selfoccluded ponts. Experments on dfferent datasets are reported to show the mprovement over prevous works. Keywords: Actve Shape Model, face algnment, 3D face model 1 Introducton Automatcally locatng facal feature ponts on face mages,.e. face algnment (FA) s a crtcal task n many face related computer vson areas such as 3D face labelng, expresson analyss and face recognton. For face algnment, there are two fundamental approaches, Actve Shape Model (ASM) [1] and Actve Appearance Model (AAM) []. Many varatons of these two methods have been developed to mprove ther robustness and accuracy [3-5]. Whle for frontal FA there s already some robust algorthm can be used n practce [4], for MVFA where face s wth large vew change, t remans a challengng problem snce both the shape and the texture of face n mages change dramatcally when the vew changes. In the lterature of MVFA, non-lnear modelng method such as Gaussan Mxture Model [6], kernel PCA [7], Bayesan Mxture Model wth learnng vsblty of label ponts [8] and vew-based methods such as vew based DAM [9] and vew-based ASM [10] are developed whch are manly D approaches wth no appealng to 3D face nformaton. Due to the ntrnsc dffcultes caused by face appearance changes n D face mages of a 3D face, MVFA s stll not a solved problem. The state-of-the-art MVFA methods are essentally vew-based approaches n whch vews are dvded nto several categores such as frontal, half profle, full profle etc. and each of them has ts own shape and texture models. Snce the texture model used n local search of each label pont of a partcular shape model depends on
ts vew category, these methods are very senstve to the estmaton of the vew category. When the ntal vew s not predcted correctly, the results of local search become unrelable. And f the estmaton of shape parameter does not deal wth the potental outlers, ASM approach wll fal. In the orgnal vew-based ASM method [10] a non-lnear optmzaton method for model selecton s used n whch each feature pont s weghed dynamcally so that only the feature ponts that are consstent wth the shape model wll have large weghts, whle the effect of outlers wll be elmnated. Snce ths method does not completely depend on the local search of each label ponts, t s more robust aganst the ntal vew and cluttered background. Vew based methods swtch between dfferent models of dfferent vews to cover the non-lnear space of mult-vew faces, thus, the selecton of models, n other words, the estmaton of vew s a crtcal step n the algorthm. Although the overlapped defnton of vew ranges can mtgate the error caused by mproper ntalzaton of vew, the automatc vew estmaton n the algnment procedure s stll an mportant problem to be solved. There are other MVFA approaches usng 3D face model. In [11], vew-based local texture models and a sparse 3D shape model whch are traned usng syntheszed faces are ntegrated n an ASM-lke framework to algn faces wth vew changes. In [1], a parameterzed 3D deformable face model s used to help wth vew based ASM but buldng ts 3D face model s a very tough work. In ths paper, we combne a vew-based ASM and a smple 3D face shape model bult on 500 3D-scanned faces [13] to buld a fully automatc MVFA system. Intalzed by a mult-vew face detector [14], we frst use vew based local texture model to local search the feature ponts around the ntal shape [10], then a 3D face shape s reconstructed from those ponts usng the 3D face shape model. Accordng to the reconstructed 3D shape, we can get ts vew nformaton from whch self-occluded ponts can be ndcated, and then the D shape model of ths vew s adopted to refne the observed non-occluded shape by non-lnear parameter estmaton. Vew-Based Actve Shape Model In the case of MVFA, the shape and texture of faces n mages change dramatcally when the vew changes. A sngle PCA model can only represent face shapes wth lmted vew change due to the non-lnear change of face shape. And further textures around the label ponts wth large vew changes are also hard to be characterzed n a sngle model. So as n [10] we dvde vews nto 7 categores and for each vew we tran a set of local texture models and a shape model. Therefore n MVFA, a face shape s represented by a PCA model of the vewv t belongs to: ( ) S = T U p+ S q v v So the objectve of MVFA s to fnd the best PCA parameter p and pose q wth some vew v : (1)
Fg. 1. Mean shapes of 7 vew categores ( ) ( ) ( ) p, q = argmax P S v P( I x, y, v) () where PSv ( ) NS ( v, dagλ ( 1, v,..., λm, v)) s the shape pror of specfed vew v (λ,v s the -th egenvalue of the covarance matrx of shape samples wth vew v). And P( I ( x, y), v) s the probablty of the pont (, ) x y to be the -th label pont, whch s determned by local texture model. The local texture model of each vew s traned usng Haar-lke feature based boosted classfer [4] whch gven a texture patch can output the lkelhood of ths patch to be around the -th label pont. The whole algnment procedure s as follow: 1. Gven an mage I, the boundng box and the estmated vew v 0 s provded by the face detecton module. And the ntal shape S 0 s estmated by fttng the mean shape of v 0 nto the boundng box of face. See Fgure 1 for mean shape llustraton.. For each label pont, search locally around ts current poston for the best dsplacement (x *, y * ) wth the largest lkelhood usng the local texture models of current vew. 3. Parameter estmaton: for each vew, estmate the parameter p v and q v usng non-lnear parameter estmaton and then fnd the best vew v and ts correspondng parameter p and q wth the mnmum reconstructon error of the shape. 4. Set the new shape S = Tq' ( Uv' p' + Sv' ), and current vew v = v 5. Iterates from step untl the shape S converged. In the optmzaton of MVFA, the proper selecton of the hdden vew v, s a crtcal step whch wll severely affect ts accuracy and robustness. So we have to develop a robust pose estmaton method to select a proper vew when an naccurate ntal vew s gven by the face detector. 3. 3D face shape model Whle a D face model suffers when vew changes, a 3D face model can easly overcome ths obstacle. The D ASM deals both the ntrnsc change (caused by the change of expresson and dfferent person) and the extrnsc change (caused by mage projecton) wth a sngle lnear model; whle the 3D shape model reflects only the ntrnsc change. Smlar as the D face shape model, a 3D face shape can be denoted by a lst of 3D coordnates S 3d =[x 1,y 1, z 1,,x n,y n,z n ] and here we use n=88 ponts n 3D-scanned faces
whch correspond to the 88 face algnment feature ponts n D face mages. And our 3D face shape model s a PCA pont dstrbuton model of the 88 feature ponts bult on a 3D-scanned BJUT-3D Face Database [13]. The frst step to construct the 3D face shape model s to acheve the 3D locatons of the 88 feature ponts n each 3D-scanned face as follows: Each 3D-scanned face s rendered at varous vews through orthogonal projecton, and then the D ASM s employed to obtan D feature pont locatons n the rendered mages. And the correspondng 3D locaton of each feature pont can be acheved by the followng method: T Gven a projecton matrx (n fact, t s a coordnate transformaton matrx) P =(U V T W T ), the projected coordnate of the -th feature pont X=S 3d, s (x,y,z ) T =P S 3d,. The D ASM gves an observaton of x as x. The depth channel of the rendered face gves an observaton of Z as z = φ ( x, y ) D ASM s Gaussan: x N x (, σ ) (, σ ) y N y. We assume the error p.d.f. of the For smplcty we assume the errors on dfferent axes dstrbute ndependently. Thus the error of can be derved as z ( x, y ) ( x x, y y) ( x, y) x x( x, y) y y( x, y) δ φ (, ) δ φ (, ) = φ = φ + δ + δ φ + δ φ + δ φ = z + x x y + y x y z N z x y (, θ ) ( ) where = ( x, y ) + ( x, y ) θ φx φy σ Rotatng the 3D face and dong D ASM on the rendered faces gve several sets of projecton matrx and observed coordnates( P, x, y, z ). The log jont lkelhood s: ( ) ( ) = log ((,, ), = 1 ) = ( log Px ( X) + log Py ( X) + log Pz ( X) ) L X P x y z n X ( x U X) ( y VX) ( z WX) = λ + + σ σ θ where λ s constant wth respect to X. The maxmum lkelhood estmaton can be obtaned analytcally by lettng the dervatve to be zero. (3) (4) (5)
X ML X = 0 ( ) = arg max L X X In practce the D ASM s not always accurate. So we use RANSAC algorthm for robust estmaton of X. After achevng the 3D locatons of 88 feature ponts for each of the 500 3Dscanned faces n BJUT-3D Face Database [13], we select 389 3D face shapes among them whch are good estmatons to buld the 3D pont dstrbuton model usng PCA as the 3D face shape model. 3d 3d (6) S = U p+ S (7) Fg.. Achevng 3D locatons of feature ponts. Frst row shows D ASM results on dfferent vews. Second row shows reconstructed 3D poston of feature ponts on those vews. 4. Vew estmaton usng 3D shape model Gven an nput mage, suppose the orthogonal projecton holds, the D shape of the face denoted as Sd n the mage should be: S = M P S + t (8) d, 3 d, Each label pont S 3d,, whch s determned by the PCA parameter p n equaton (7), s frst transformed (scaled and rotated) by the transform matrx P, then projected nto the mage plane orthogonally wth translaton t. Where P s an orthogonal projecton matrx and 1 0 0 M = s the projecton matrx. 0 1 0 Accordng to equaton (8), gven a canddate D face shape S d, we could get the PCA parameter p and the pose nformaton (s, R, t) usng mnmum square error estmaton: ( ) ( ) Pt,, p = argmn M P U p+ S d + t S d 3,, (9)
We frst reform the object functon by denotng A=M P, so that: where A has the constrant: 3,, (10) ( ) ( ) At,, p = arg mn A U p+ S d + t S d a + a + a = a + a + a, a a + a a + a a = 0 (11) 1,1 1, 1,3,1,,3 1,1,1 1,, 1,3,3 whch makes A an orthogonal projecton. We can solve the above optmzaton problem by optmzng pose parameter (A,t) and shape parameter p alternatvely n an teraton procedure as follows: 1. Gven p, solve (A,t) ( ) ( ) At, = argmn AU p+ S3 d, + t S d, An affne projecton (A,t) can be estmated analytcally and then we can get an orthogonal projecton A by optmzng the object functon usng gradent descend ntated wth the affne projecton.. Gven (A,t), solve p ( ) p= arg mn A U p+ S3 d, + t S d, It s a lnear MSE and the soluton goes straght forward. Step 1 and Step are terated untl the convergence of the reconstructon error. (1) (13) 5. Automatc MVFA Gven a face mage, we ntalze our algorthm by applyng mult-vew face detecton [14] whch provdes a boundng box and a roll angle of the face. The roll angle corresponds to 5 vew categores n {-90,-45,0,45,90 }. We select the ntal vew accordng to ths angle and then compute the ntal D shape by rotatng and scalng the mean shape of ntal vew to ft the boundng box of face. Then the algorthm goes teratvely as follow: 1. Local Search: For the -th label pont, compute the lkelhood P(I (x,y ),v) usng the local texture model of current vew on every pont around the current locaton of label pont, then select the best canddates {(x *,y * )}wth the largest lkelhood as the new locaton. The observaton shape s S d * ={(x *,y * )}.. Pose Estmaton usng the 3D face shape model: use the observaton D shape and the 3D shape model to estmate the pose parameter (A,t) and the shape parameter p of the 3D face model. Then compute the roll angle accordng to the projecton matrx A and select current vew. At the same tme we can ndcate the self-occluded label ponts by the reconstructed 3D shape. 3. D parameter estmaton: estmate the shape and pose parameter usng D shape model of current vew. Gven the observed shape S d * and the vsbltes of each label pont, the new shape s reconstructed by mnmzng the weghted
reconstructon error of vsble ponts. The dynamc weghtng method used n [10] s stll adopted n our algorthm to mprove robustness. Here s the flow chart of our automatc MVFA system. Intalaton Intal vew& shape Local Search Use Current vew Observed shape D parameter estmaton New shape N Converge? Y Algnment result Observed shape Vew & vsbltes 3D reconstructon Fg. 3. Flowchart of MVFA The whole algnment procedure s shown n Fgure 4. (a) (b) (c) (d) (e) Fg. 4. Illustraton of algnment procedure ((a) The algorthm s ntalzed by the mean shape of current vew. (b) The observed shape s got by local search usng local texture model. (c) Wth the observed shape, a 3D shape s reconstructed usng 3D shape model and the pose s estmated. (d) The D shape s reconstructed from the observed shape. (e) The fnal shape when the teraton converged.) 6. Experments 6.1 Tranng A mult-vew face database ncludng totally 1800 mages whch are taken by a camera array wth poses n the set {-90,-75,-60,-45,-30,0 } s set up. The face mages are about 50 by 50 n sze each of whch are manually labeled wth 88 label ponts. 1500 mages are used n tranng and the other 300 are used n testng. 4 ASMs of correspondng vews shown n table 1 are traned usng Haar-lke feature based boosted classfers whch dstngush the textures around a feature pont from the textures far away from the feature pont [10] (here the vew 5-7 are omtted snce they are the mrrors of the vew -4 whch can use ther mrrored models). Notce that the angle ranges of dfferent vews have overlaps n order to make each model more robust to vew selecton.
Table. 1 Tranng models and roll angles Vew 1 3 4 Angle Frontal 0 to 45 30 to 60 60 to 90 6. Vew estmaton Vew estmaton results are tested on the 1500 tranng mages. Table gves the comparson results between the 3D approach and the D vew-based approach. It can be seen that the 3D method can apparently mprove the vew estmaton accuracy especally for those vews wth large off-mage-plane (roll) angles whch are very crtcal n MVFA snce face algnment for faces of non-frontal vews are much more senstve to vew selecton. Table. Comparson between the 3D approach and the D vew-based approach Vew 0 1 3 3D method 95% 93% 9% 95% D method 93% 90% 87% 85% 6.3 MVFA On the 300 testng mages, the performance of MVFA s measured by the average pont-to-pont errors between algnment result and the ground truth and s shown n Fgure 5. It can be seen that the proposed approach outperforms the tradtonal vewbased ASM algorthm [10]. In average, MVFA takes about 304ms to algn a face. Fg. 5. Error dstrbuton of algnment results. We also tested our method on the CMU-PIE database. Some results are shown n Fgure 6. Snce there are no ground truth data, we can only subjectvely judge the correctness of algnment on a subset of CMU-PIE database. Among all the 1145 face mages from the c0, c37, c7, c11, c14 vew categores, our algorthm acheved 86.7% n correct rate, whle the orgnal method [10] can only acheve 74.5%.
Addtonal tests have also been taken on the Labeled faces n the wld database [16], whch contans mult-vew faces n unconstraned envronments. Our method can deal wth these faces rather well even though our tranng mages are taken n constraned envronments and do not cover such large varatons n pose, llumnaton, background, focus and expresson. See fgure 7 for some results. Fg. 6. Addtonal results on CMU-PIE database Fg. 7. Addtonal results on Labeled faces n the wld database 7 Concluson In ths paper, we presented an automatc MVFA framework by ntegrate D vew based ASM wth a 3D face shape model. Algnment s done n vew-based ASM manner whle durng the teratons, the selecton of models, n other words, the vew estmaton, s done usng 3D face shape model. In addton, the 3D reconstructed shape s used to ndcate nvsble label ponts that can further mprove the accuracy and robustness of the D vew-based ASM method. Experments show that vew estmaton usng 3D model can help the vew-based ASM method n both accuracy and robustness. Our future work wll focus on extendng the proposed method to more challengng datasets such as the Labeled faces n the wld database and consumer mages over the nternet.
Acknowledgement Ths work s supported by Natonal Scence Foundaton of Chna under grant No.60673107, and t s also supported by a grant from Omron Corporaton. References 1. A. Hll, T.F. Cootes, and C.J. Taylor, Actve shape models and the shape approxmaton problem, BMVC 1995.. T.F Cootes, G.J. Edwards, and C.J. Taylor, Actve appearance models, IEEE Transactons on pattern analyss and machne ntellgence, vol. 3, NO. 6, June 001. 3. F. Jao, S.Z. L, et.al, Face algnment usng statstcal models and wavelet features, CVPR 003. 4. L. Zhang, H. A, et.al, Robust Face Algnment Based on Local Texture Classfers, ICIP 005. 5. A.U. Batur, M.H. Hayes. A Novel Convergence for Actve Appearance Models. CVPR 003. 6. T.F. Cootes and C.J. Taylor. A mxture model for representng shape varaton. BMVC 1997. 7. S. Romdhan, S. Gong, and A. Psarrou. A mult-vew non-lnear actve shape model usng kernel PCA. BMVC 1999. 8. Y. Zhou, W. Zhang, et.al, A Bayesan Mxture Model for Mult-vew Face Algnment, CVPR 005 9. S.Z. L, S.C. Yan, et.al, Mult-vew face algnment usng drect appearance models, AFG 00 10. L. Zhang, H. A, Mult-Vew Actve Shape Model wth Robust Parameter Estmaton, ICPR 006. 11. L. Gu and T. Kanade, 3D Algnment of Face n a Sngle Image, CVPR 006. 1. C. Vogler, Z.G. L, A. Kanauja, The Best of Both Worlds: Combnng 3D Deformable Models wth Actve Shape Models, ICCV 007 13. The BJUT-3D Large-Scale Chnese Face Database. Techncal Report No ISKL-TR-05- FMFR-001. Multmeda and Intellgent Software Technology Bejng Muncpal Key Laboratory, Bejng Unversty of Technology, 005. 14. C. Huang, H. A, et.al, Hgh Performance Rotaton Invarant Multvew Face Detecton, IEEE Transactons on Pattern Analyss and Machne Intellgence, Vol.9, No.4, pp. 671-686, APRIL 007. 15. T. Sm, S. Baker, and M. Bsat. The CMU Pose, Illumnaton,and Expresson (PIE) database of human faces. the robotcsnsttute, Carnege Mellon Unversty. Techncal report, 001. 16. Huang, G.B., Ramesh, M., Berg, T., Mller, E.L.: Labeled faces n the wld: A database for studyng face recognton n unconstraned envronments. Techncal Report, 5(1):07 49 (October 007).