Texture-Constrained Active Shape Models

Transcription

1 107 Texture-Contrained Active Shape Model Shuicheng Yan, Ce Liu Stan Z. Li Hongjiang Zhang Heung-Yeung Shum Qianheng Cheng Microoft Reearch Aia, Beijing Sigma Center, Beijing , China Dept. of Info. Science, School of Math. Science, Peking Univerity, Beijing , China Contact: zli Abtract In thi paper, we propoe a texture-contrained active hape model (TC-ASM) to localize a face in an image. TC-ASM effectively incorporate not only the hape prior and local appearance around each landmark, but alo the global texture contraint over the hape. Therefore, it perform table to initialization, accurate in hape localization and robut to illumination variation, with low computational cot. Extenive experiment are provided to demontrate our algorithm. I. Introduction Accurate extraction and alignment of target object from image are required in many computer viion and pattern recognition application. Active Shape Model (ASM) and Active Appearance Model (AAM), propoed by Coote et. al.[1], are two popular hape and appearance model for object localization. They have been developed and improved for year [2], [3], [4], [5]. In ASM, the local appearance model, which repreent the local tatitic around each landmark, efficiently find the bet candidate point for each landmark in earching the image. The olution pace i contrained by the properly trained global hape model. Baed on the accurate modeling of the local feature, ASM obtain nice reult in hape localization. AAM [6], [7], [8], [9] combine contraint on both hape and texture in it characterization of face appearance. In the context of thi paper, texture mean the intenity patch contained in the hape after warping to the mean hape [1]. There are two linear mapping aumed for optimization: from appearance variation to texture variation, and from texture variation to poition variation. The hape i extracted by minimizing the texture recontruction error. According to the different optimization criteria, ASM perform more accurately in hape localization while AAM give a better match to image texture. On the other hand, ASM tend to be tuck in local minima, dependent on the initialization. AAM i enitive to the illumination, in particular if the lighting in the tet i ignificantly different from the training. Meanwhile, training an AAM model i time conuming. In thi paper, a novel hape model, called Texture- Contrained Active Shape Model (TC-ASM), i propoed to addre the above problem of ASM and AAM. TC- ASM inherit the local appearance model in ASM for the robutne of varying lighting. We borrow the global texture in AAM to TC-ASM, acting a a contraint over hape and providing an optimization criterion for determining the hape parameter. In TC-ASM, the conditional ditribution of a hape given it aociated texture i modeled a a Gauian ditribution. Thu, the texture correponding Fig. 1. Left and middle: two face intance labeled with 83 landmark. Right: the meh of the mean hape. to the hape obtained from the local appearance model, could linearly predict a texture-contrained hape. It converge to a local optimum when the hape from the local appearance model i very cloe to the texture contrained hape. Extenive experiment how that TC-ASM outperform ASM and AAM in facial hape localization. It i alo demontrated that TC-ASM perform no wore than AAM in texture recontruction and ynthei. Thi paper i organized a follow. In Section 2, we briefly review the hape and appearance model. The detail of TC-ASM are dicued in Section 3. Experiment are preented in Section 4. We conclude thi paper in Section 5. II. Claical Shape and Appearance Model Aume a training et of hape-texture pair to be Ω = {(S i,ti )}N i=1. The hape S i = {(x i j,yi j )}K j=1 R2K i a equence of K point in the image lattice. The texture Ti i the image patch encloed by S i. Let S be the mean hape of all the training hape, a illutrated in Fig.1. S i calculated from an iterative procedure [10] uch that all the hape are aligned to the tangent pace of the mean hape S. After hape warping [1], the texture Ti i warped correpondingly to T i R L, where L i the number of pixel encloed by the mean hape S. The warped texture are alo aligned to the tangent pace of the mean texture T in the ame approach a computing S. A. ASM In ASM, a hape i repreented a a vector in the low dimenional hape eigenpace R k, panned by k (< 2K) principal mode (major eigenvector) learned from the training hape. A hape S could be linearly obtained from hape eigenpace: S = S + U, (1)

2 108 where U i the matrix coniting of k principal mode of the covariance of {S i }. The local appearance model, which decribe local image feature around each landmark, are modeled a the firt derivative of the ampled profile perpendicular to the landmark contour [1]. For the jth landmark (j = 1,,K), we can derive the mean profile g j and the covariance matrix Σ g j from the jth profile example directly. At the current poition (x (n 1) j,y (n 1) j )ofthejth landmark, the local appearance model find the bet candidate (x n j,yn j ) in the neighborhood N(x(n 1) j,y (n 1) j ) urrounding (x (n 1) j,y (n 1) j ), by minimizing the energy: (x n j,y n j ) = arg min (x,y) N(x (n 1) j,y (n 1) j ) g j (x, y) g j 2 Σ g j where g j (x, y) i the profile of the jth landmark at (x, y) and X 2 A =XT A 1 X i the Mahalanobi ditance meaure with repect to a real ymmetric matrix A. After relocating all the landmark uing the local appearance model, we obtain a new candidate hape S n lm. The olution in hape eigenpace i derived by maximizing the likelihood: (2) n =arg max p(slm )=arg n min Eng(S n lm; ), (3) where 1 Eng(Slm; n )=λ Slm n S lm n 2 + n lm 2 Λ. (4) In above equation, n lm =U T (Slm n S) i the projection of Slm n to the hape eigenpace, Sn lm = S + U n lm i the recontructed hape, Λ i the diagonal matrix of the larget eigenvalue of the training data {S i }. The firt term i the quared Euclidean ditance from Slm n to the hape eigenpace, and the econd i the quared Mahalanobi ditance between n lm and. λ balance the two term. Uing the local appearance model lead to fat converge to the local image evidence. However, ince they are modeled baed on the local feature, and the bet candidate point i only evaluated in local neighborhood, the olution of ASM i often uboptimal, dependent on the initialization. B. AAM In AAM, the texture eigenpace i panned by the l principle mode of {T i }. The texture model i imilar to the hape model: T = T + Vt, (5) where V i the matrix coniting of l principal orthogonal mode of the covariance in {T i }, and t i the vector of texture parameter. Let appearance a =(γ T,t T ) T [1] be a weighted vector of hape parameter and texture t with the weighting parameter γ. AAM aume that both the appearance 1 It i a deviation of the motly ued energy function with a quared Euclidean ditance between Slm n and hape S R2K derived from parameter. It i more reaonable to take into account the prior ditribution in the hape pace. diplacement δa and the poition (including the centroid (x, y), cale and orientation θ) diplacement δp are linearly dependent on the texture recontruction error δt: δa = A a δt δp = A p δt δt i continuouly minimized in AAM by hifting the hape and poition parameter a in Eqn.6. However, due to the high dimenion of the pace T and a, the training of A a and A p i time and memory conuming. Meanwhile, ince illumination do not compoe the image intenity linearly, δt could not accurately predict δa or δp under irregular lighting via linear mapping. Therefore, AAM olution are often affected by varying illumination. III. Texture Contrained Active Shape Model From above analyi, it i natural to develop a novel model to inherit the merit and reject the demerit of ASM and AAM. We propoe a texture-contrained active hape model (TC-ASM) to borrow local appearance model from ASM for landmark localization, and incorporate the global texture contraint over the hape from AAM for more accurate hape parameter etimation. It conit of everal type of model: a hape model, a texture model, K local appearance model, and a texture-contrained hape model. The former three type are exactly the ame a ASM and AAM. The texture-contrained hape model, or the mapping from texture to hape, i imply aumed linear and could be eaily learnt. In each tep of the optimization, a better hape i found under Bayeian framework. The detail of the model will be introduced in the following. A. Texture-Contrained Shape Model In the hape model, there are ome landmark defined on the edge or contour. Since they have no explicit definition for their poition, there exit uncertainty of the hape given the texture, whilt there are correlation between the hape and the texture. The conditional ditribution of hape parameter given texture parameter t i imply aumed Gauian, i.e., (6) p( t) N( t, Σ t ), (7) where Σ t tand for the covariance matrix of the ditribution, and t i linearly determined by the texture t. The linear mapping from t to t i: t = Rt, (8) where R i a projection matrix that can be pre-computed from the training pair {( i,t i )} by ingular-value decompoition. For implicity, Σ t i aumed to be a known contant matrix. Fig.2 demontrate the accuracy of the prediction in the tet data via the matrix R. Wemayeethat the predicted hape i cloe to the labeled hape even under varying illumination. Thu, the contraint over the hape from the texture can be ued a an evaluation criterion in the hape localization tak. The prediction of matrix R

3 109 Fig. 2. The comparion of the manually labeled hape (middle row) and the hape (bottom row) derived from the encloed texture uing the learned projection matrix: t = Rt. In the top row are the original image. All the image are tet data. i alo affected by illumination variation, yet ince Eqn.(8) i formulated baed on the eigenpace, the influence of the unfamiliar illumination can be alleviated when the texture i projected to the eigenpace. The ditribution (7) can alo be repreented a the prior ditribution of given the hape t : p( t ) exp{ Eng(; t )}, (9) where the energy function i: Eng(; t )= t 2 Σ t. (10) B. TC-ASM in Bayeian Framework TC-ASM earch tart with the mean hape, namely the hape parameter 0 = 0. The whole earch proce i outlined a below: 1. Set the iteration number n =1; 2. Uing the local appearance model in ASM, we may obtain the candidate hape Slm n with the hape parameter n lm baed on the hape S(n 1) of the previou iteration; 3. The texture encloed by Slm n i warped to the mean hape, denoted by t n. The texture-contrained hape n t i predicted from t n by Eqn.(8); 4. The poterior (MAP) etimation of S n or n given Slm n and n t i derived baed on the Bayeian framework; 5. If the topping condition i atified, exit; otherwie, n=n+1, goto tep 2. In the following, we illutrate the tep 4 and the topping condition in detail. To implify the notation, we hall omit the upercript n in following deduction ince the iteration number i contant. In tep 4, the poterior (MAP) etimation of given S lm and t i: p( S lm, t )= p(s lm, t )p(, t ). (11) p(s lm, t ) Aume that S lm i conditionally independent to t, given, i.e., p(s lm, t )=p(s lm ). (12) Then p( S lm, t ) p(s lm )p( t ). (13) The correponding energy function i: Eng(; S lm, t )=Eng(S lm ; )+Eng(; t ) (14)

4 110 From the Eqn.(4) and Eqn.(10), the bet hape obtained in each tep i = arg min[eng(; S lm )+Eng(; t )] = arg min lm 2 Λ + t 2 Σ t = arg min[ T (Λ 1 + Σ 1 t ) 2 T (Λ 1 lm + Σ 1 t t )] =(Λ 1 + Σ 1 t ) 1 (Λ 1 lm + Σ 1 t t ). After retoring the upercript of iteration number, the bet hape obtained in tep n i n =(Λ 1 + Σ 1 t ) 1 (Λ 1 n lm + Σ 1 t n t ). (15) Thi indicate that the bet hape derived in each tep i an interpolation between the hape from the local appearance model and the texture-contrained hape. In thi ene, TC-ASM could be regarded a a trade-off between ASM and AAM method. The topping condition of the optimization i: if the hape from the local appearance model and the texturecontrained hape are the ame, i.e., the olution generated by ASM i verified in AAM, the optimal olution mut have been touched. In practice, however, thee two hape would hardly turn to be the ame. A threhold i introduced to evaluate the imilarity and ometime the convergence criterion in ASM i ued (if the above criterion ha not been atified for a long time). For higher efficiency and accuracy, a multi-reolution pyramid method i adopted in optimization proce. IV. Experiment A data et containing 700 face image with different illumination condition and expreion are elected from the AR databae[11] in our experiment, each of which i , 256 gray image containing the frontal view face about landmark point are manually labeled on the face. We randomly elect 600 for training and the other 100 for teting. For comparion, ASM and AAM are trained on the ame data et, in a three-level image pyramid (Reolution i reduced 1/2 level by level) a TC-ASM. By mean of PCA with 98% total variation retained, the dimenion of the hape parameter in ASM hape pace i reduced to 88, and the texture parameter vector in AAM texture pace i reduced to 393. The concatenated vector of the hape and texture parameter vector with the weighting parameter γ =13.77 i reduced to 277. Two type of experiment are preented: (1) the comparion of the point-poition accuracy and (2) the comparion of the texture recontruction error. The experiment are all performed in the 3-level reolution image pyramid. A. Point Poition Accuracy The average point-point ditance between the earched hape and the manually labeled hape of the three model are compared in Fig.3. The vertical axi repreent the percentage of the olution for which the average point-point ditance to the manually labeled one are maller than the correponding horizonal axi value. The tatitic are calculated from 100 tet image with different initialization, with random diplacement to the ground truth of 10, 20, 30 and 40 pixel. The reult how that TC-ASM outperform both ASM and AAM in mot cae ince the curve of TC-ASM lie above the curve for ASM and AAM. It alo ugget that AAM outperform ASM when the initial diplacement i mall, while ASM i more robut to the increaing of the initial diplacement. We compare the tability of TC-ASM with ASM in Fig.4. The value of horizonal axi i the index number of the elected example, wherea the value of the vertical axi i the average tandard deviation of the reult obtained from 10 different initialization which deviate from the ground truth by approximately 20 pixel. The reult convince that TC-ASM i more table to initialization. An example i given in Fig.5. B. Texture Recontruction Error The texture recontruction error comparion of the three model in Fig.6 illutrate that TC-ASM improve the accuracy of the texture matching. The texture accuracy of TC-ASM i cloe to that of AAM while it poition accuracy i better than AAM (ee Fig.3). Although AAM ha more cae with mall texture recontruction error, TC- ASM ha more cae with the texture recontruction error maller than 0.2. An example in which AAM fail for a different illumination condition from the training data, yet TC-ASM perform well i preented in Fig.8. Fig.7 how a cenario of AAM and TC-ASM alignment. From the experiment, TC-ASM i more computationally expenive than ASM, but it i much fater than AAM. In our experiment (600 training image, 83 landmark and a P-III 667 computer with 256M memory), it take averagely 32 m per iteration, which i twice of ASM (16 m) but one fifth of AAM (172 m). The training time of AAM i more than two hour, while TC-ASM i only about 12 minute. V. Concluion In thi paper we propoed a novel hape model, TC- ASM, for object localization. TC-ASM efficiently incorporate the local information around each landmark and the global texture information for alignment. It i more robut to initialization, more accurate in hape localization and le enitive to illumination, when compared with conventional method. For future work, the tatitical correlation between the hape and the texture can be derived from the joint ditribution of the hape and texture. In addition, the generalization of the hape prediction from the texture can be evaluated on a larger data et. Reference [1] T. F. Coote and C. J. Taylor, Statitical model of appearance for computer viion, [2] Y. Li, S. Gong, and H. Liddell, Modelling face dynamically acro view and over time, in Proceeding of IEEE Inter-

5 111 Fig. 3. Accuracy of ASM, AAM, TC-ASM. From upper to lower, left to right are the reult obtained with the initial diplacement of 10, 20, 30 and 40 pixel. Fig. 4. Standard deviation in the reult of each example for ASM (dotted) and TC-ASM (olid) with training et (left) and tet et(right). national Conference on Computer Viion, Vancouver, Canada, July 2001, pp [3] X. W. Hou, S. Z. Li, and H. J. Zhang, Direct appearance model, in Proceeding of IEEE Computer Society Conference on Computer Viion and Pattern Recognition, Hawaii, December , pp [4] B. van Ginneken, A. F. Frangi, J. J. Staal, B. M. ter Haar Romeny, and M. A. Viergever, A non-linear gray-level appearance model improve active hape model egmentation, In Proceeding of Mathematical Method in Biomedical Image Analyi, [5] S. Mitchell, B.Lelieveldt, R. van der Geet, J. Schaap, J. Reiber, and M. Sonka, Segmentation of cardiac mr image: An active appearance model approach, SPIE Medical Imaging, Feb [6] T. F. Coote, G. J. Edward, and C. J. Taylor, Active appearance model, IEEE Tranaction on Pattern Analyi and Machine Intelligence, vol. 23, no. 6, pp , [7] M. B. Stegmann, Active appearance model: Theory, extenion & cae, Mater thei, Department of Mathematical Modelling, Technical Univerity of Denmark, Lyngby, Denmark, [8] T. F. Coote and C. J. Taylor, Contrained active appearance model, Proceeding of IEEE International Conference on Computer Viion, vol. 1, pp , [9] T.F.Coote and C.J.Taylor, On repreenting edge tructure for model matching, Proc. CVPR 2001, vol. 1, pp [10] T. F. Coote, G. J. Edward, and C. J. Taylor, Active appearance model, in ECCV98, 1998, vol. 2, pp [11] A.M. Martinez and R. Benavente, The AR face databae, Technical Report 24, CVC, June 1998.

6 112 Fig. 5. Stability of ASM (Middle column) and TC-ASM (Right column) in hape localization. The different initialization condition are howed in the left column. Fig. 6. Ditribution of the texture recontruction error in ASM (dotted), AAM (quare) and TC-ASM (aterik) with training data (left) and tet data (right).

7 113 Fig. 7. Scenario of AAM (upper) and TC-ASM (lower) alignment with texture recontruct error and repectively. From left to right are reult obtained at the 0-th, 5-th, 10-th, 15-th iteration and the original image. (Reult in different level of image pyramid i caled back to the original cale) Fig. 8. Senitivitie of AAM (upper) and TC-ASM (lower) to illumination condition not een in the training data. From left to right are the reult obtained at the 0-th, 2-th, and 10-th iteration.( Reult in different level of image pyramid i caled back to the original cale)