3D Face Modeling Using the Multi-Deformable Method

Transcription

1 Sensors 2012, 12, ; do: /s Artcle OPEN ACCESS sensors ISSN D Face Modelng Usng the Mult-Deformable Method Jnkyu Hwang, Sunjn Yu, Joongrock Km and Sangyoun Lee * Department of Electrcal and Electroncs Engneerng, Yonse Unversty, 134 Shnchon-dong, Seodaemun-gu, Seoul , Korea; E-Mals: wnsprt@yonse.ac.kr (J.H.); bometrcs@yonse.ac.kr (S.Y.); jurock@yonse.ac.kr (J.K.) * Author to whom correspondence should be addressed; E-Mal: syleee@yonse.ac.kr; el.: Receved: 24 July 2012; n revsed form: 14 September 2012 / Accepted: 19 September 2012 / Publshed: 25 September 2012 Abstract: In ths paper, we focus on the problem of the accuracy performance of 3D face modelng technques usng correspondng features n multple vews, whch s qute senstve to feature extracton errors. o solve the problem, we adopt a statstcal model-based 3D face modelng approach n a mrror system consstng of two mrrors and a camera. he overall procedure of our 3D facal modelng method has two prmary steps: 3D facal shape estmaton usng a multple 3D face deformable model and texture mappng usng seamless clonng that s a type of gradent-doman blendng. o evaluate our method s performance, we generate 3D faces of 30 ndvduals and then carry out two tests: accuracy test and robustness test. Our method shows not only hghly accurate 3D face shape results when compared wth the ground truth, but also robustness to feature extracton errors. Moreover, 3D face renderng results ntutvely show that our method s more robust to feature extracton errors than other 3D face modelng methods. An addtonal contrbuton of our method s that a wde range of face textures can be acqured by the mrror system. By usng ths texture map, we generate realstc 3D face for ndvduals at the end of the paper. Keywords: 3D face modelng; face deformable model; face shape estmaton; statstcal face model

2 Sensors 2012, Introducton hree-dmensonal (3D) face modelng s a challengng topc n computer graphcs and computer vson. Unlke 2D face models, 3D face models can realstcally express face deformaton and pose varaton wth depth nformaton. Wth these advantages, 3D face models have been appled to varous applcatons, ncludng moves, 3D anmaton and telecommuncatons [1,2]. hree dmensonal modelng systems can be categorzed nto actve and passve vson systems [3]. An actve vson system calculates 3D nformaton by measurng a beam of lght radated from an external devce such as a beam projector or laser. he typcal 3D face modelng method usng an actve vson system constructs a 3D face mesh usng a captured 3D pont cloud [4 9]. In such systems, a 3D laser scanner or calbrated stereo camera wth structured lght can be used to capture 3D coordnates and texture nformaton. Whle these methods are hghly accurate, they are also tme consumng, and the necessary equpment s expensve. Nowadays, the passve vson-based 3D modelng system s preferred for human faces because the glare from lght-emttng devces can be unpleasant for the users. Passve vson-based system means a system that needs no lght-emttng devces and estmates 3D nformaton from 2D mages. In passve vson-based 3D face modelng, 3D nformaton can be calculated by analyzng camera geometry from correspondng features n multple vews [10] or adjustng the statstcal 3D face model to captured facal mages [2,11]. For convenence, we call the former the correspondng feature-based 3D face modelng method and the latter the statstcal model-based 3D face modelng method. Among the 3D face modelng methods usng the passve vson system, the most commonly used one s the correspondng feature-based 3D face modelng method. hs method s less computatonally expensve because t uses only a few feature ponts to generate a 3D facal shape. Addtonally, ths method can generate hghly accurate 3D facal shapes by usng real 3D nformaton calculated from the camera geometry. However, the accuracy of the 3D facal shapes declnes rapdly f the extracted locatons of the correspondng ponts are not exact. hs problem should be solved to apply to automatc 3D modelng system because even excellent feature extracton technques such as the actve appearance model (AAM) [12] and the actve shape model (ASM) [13,14] can produce erroneous feature extracton results for ndstnct parts of the face. In ths paper, we am to develop a realstc 3D face modelng method that s robust to feature extracton errors and generates accurate 3D face modelng results. o acheve ths, we propose a novel 3D face modelng method whch has two prmary steps: 3D facal shape estmaton and texture mappng wth a texture blendng method. In the 3D facal shape estmaton procedure, we take a statstcal model-based 3D face modelng approach as a fundamental concept. Among the statstcal model-based methods, we use n partcular the deformable face model that utlzes locaton nformaton of facal features n the nput mage. hs method s robust to feature extracton errors because t uses pre-traned 3D face data to estmate 3D facal shapes from nput face mages but t s a lttle less accurate than the correspondng feature-based 3D face modelng methods. o mprove accuracy of the 3D facal shapes, we propose a 3D face shape estmaton method usng multple 3D face deformable models. In the texture mappng procedure, we apply a cylndrcal mappng and a sttchng technque to generate a texture map. When sttchng each face part, we apply a modfed gradent-doman blendng

3 Sensors 2012, technque [15] to remove the seam that appears at the boundares of each face part because of photometrc nconsstency. hs paper s organzed as follows: n Secton 2, we ntroduce prevous 3D face modelng technques and the 3D face deformable model that s bass of proposed method. In Secton 3, we address our 3D facal shape estmaton method wth mrror system. In Secton 4, we descrbe our texture mappng and texture map generaton method usng a modfed gradent-doman blendng technque. hen, we dscuss the 3D face modelng results and evaluate our method s performances compared wth those of other 3D face modelng methods and ground truth n Secton 5. Fnally, we conclude our paper and address future work n Secton Prelmnary Study In ths secton, we address prevous works and the fundamental concept of the proposed 3D face modelng method. In Sectons 2.1 and 2.2, we ntroduce prevous 3D face modelng methods usng passve vson systems. We categorze them nto two groups: correspondng feature-based 3D face modelng and statstcal model-based 3D face modelng. hen, we study strengths and weaknesses of these methods. In Secton 2.3, we concretely descrbe the 3D face modelng method usng a deformable model whch s a type of statstcal model-based 3D face modelng because t s a fundamental concept of the proposed method that wll be descrbed n Secton Correspondng Feature-Based 3D Face Modelng he smplest and fastest way to generate a 3D face model usng the correspondng features s to use orthogonal vews [16 18]. In ths method, the 3D coordnates of the features can be easly calculated from manually selected feature ponts n two orthogonal vews of the face. hs method s qute easy to mplement, but orthogonalty between the two vews s necessary. Some researchers construct 3D faces from several facal mages. Fua et al. [19] proposed a regularzed bundle-adjustment on trplet mages to reconstruct 3D face models from mage sequences. her method takes advantage of a rough knowledge of the head s shape (from a generc face model), but t s computatonally expensve because t requres dense stereo matchng. In a smlar approach, Lee et al. [20] constructed a 3D head model usng two-pass bundle adjustments. In the frst pass, the method computes several feature ponts of a target 3D head and then uses these features to obtan a roughly matched head model by modfyng a generc head. Next, the second pass bundle adjustment s carred out to obtan a detaled 3D head model. Pghn et al. [21] developed a method that generates 3D faces by fttng a generc 3D face model on pre-defned 3D landmark ponts whch are reconstructed from sequentally captured facal mages. he pre-defned 3D landmark ponts can be calculated by the structured from moton (SfM) method after extractng the correspondng feature ponts n mages captured from dfferent vews. he texture map s generated by combnng the mult-vew photographs nto a cylndrcal map. In ther method, face mages from dfferent vews provde a wde range of textures and features (e.g., the ears) whch can be used to generate a realstc 3D face. However, the method s somewhat nconvenent because the user must hold a statonary pose durng mage capture. In addton, the 3D reconstructon s qute senstve to the accuracy of the feature extracton so that a manually ntensve procedure s requred.

4 Sensors 2012, As another example of a correspondng feature-based method, Ln et al. [22] proposed a 3D face modelng system usng two mrrors postoned next to the face to smultaneously capture three dfferent vews of the face. hen, they reconstruct 3D ponts annotated wth markers by analyzng the relaton between drectly captured markers and the markers reflected n the mrrors. In ther method, a wde range of face data can be acqured from the captured face mages. In addton, ths approach avods the problem of synchronzaton of multple cameras, although t s stll senstve to feature extracton error Statstcal Model-Based 3D Face Modelng In 3D face modelng usng statstcal model, the 3D morphable face model suggested by Blanz and Vetter [23] s the most well-known. o generate a 3D morphable face model, they construct a database ncludng the 3D coordnates and skn texture from a real human face captured by a 3D laser scanner. hen, statstcal analyss s carred out to determne control parameters for the 3D face shape and skn texture deformaton. Durng the modelng procedure, the model parameters are teratvely adjusted n order to ft the model to the nput mage. hs gves remarkably realstc results, but the computatonal cost s very hgh. o mprove the speed, researchers have proposed 3D face modelng methods usng a sngle-vew mage. Kuo et al. [24,25] proposed a method to synthesze a lateral face from a sngle frontal-vew mage. hey construct a facal mage database contanng both frontal and lateral vews and then defne anthropometrc parameters that represent the dstance between two features manually extracted by anthropometrc defnton. In the modelng stage, they estmate the lateral facal parameters from the nput frontal mage usng the relatonshp between the frontal and lateral facal parameters. Baek et al. [26] suggested an anthropometry-based 3D face modelng technque. hey bult a database after measurng anthropometrc nformaton from anatomcally meanngful 3D ponts among a 3D pont cloud captured by a 3D laser scanner. hen, they created a statstcal model to control the overall 3D face shape after statstcal analyss of the database. hs method s much faster than the 3D morphable face model, but the depth estmaton results are senstve to head poses, whch may result n naccurate dstance measurement between landmarks. Importantly, ths approach s also lmted to the reconstructon of frontal vews D Face Shape Model Generaton he 3D face deformable model s a type of parametrc model that can deform shapes and textures by changng related parameters. he morphable face model [27], whch generates a textured 3D face by controllng parameters that can be acqured from statstcal analyss of 3D face scans contanng geometrc and textural data, s the most representatve model n ths class. Generally, the computatonal costs of morphable face models are very hgh because they use entre face data (vertces and texture) and requre many parameters to ft on nput face mages. On the other hand, a 3D face deformable model s less computatonally expensve because t uses only geometrc nformaton (.e., the 3D coordnates of 3D face scans) and s composed of sparsely dstrbuted vertces rather than full vertces of 3D scans. For clarty, the 3D deformable model that s composed of sparsely dstrbuted vertces s called the 3D face shape model (FSM) n ths paper. he 3D FSM s composed

5 Sensors 2012, of anatomcally meanngful 3D vertces whch can represent the shape of the entre face. Each vertex s called a 3D FSM landmark, and a set of landmarks s called a 3D FSM face shape. he deformaton of 3D FSM can be carred out by global and local deformatons. In a global deformaton, the poston and shape of the 3D FSM can be determned by a 3D affne transformaton. he 3D affne transformaton can be expressed as a 3 1 translaton matrx () and a 3 3 matrx (A) of rotaton and skew transformatons. Under a global deformaton, the coordnates of a vertex ([X,Y,Z]) can be transformed nto ([X t,y t,z t ]) by: X t A1 A2 A3 X 1 Y A A A Y t Z t A7 A8 A 9 Z 3 (1) In a local deformaton, a detaled deformaton of 3D FSM can be parameterzed by statstcal analyss. Frst, 3D face shape data sets are constructed from 3D vertces correspondng to the poston of 3D FSM landmarks n each 3D face scan. hese sets (S) can be represented as: S x, y, z, x, y, z,, x, y, z S S, S,, S 1 2 M N N N where M s the total number of face shape sets and N s the number of landmarks of 3D FSM. After constructng the 3D face shape data sets, a shape algnment procedure s carred out usng 3D procrustes analyss [28], and the mean face shape s calculated as: 1 M M 1 S S (3) hen, feature vectors of the face data sets are calculated by prncpal component analyss (PCA). In PCA, the feature vectors of the shape data sets are the egenvectors (Φ) of the covarance matrx (Σ) of the normalzed shape data set (D): Γ S S Σ D D, D Γ, Γ,, Γ Σ Φ Φ Λ where denotes a dagonal matrx of egenvalues: 1 2 M 1 Λ (5) 3M hen, the local deformaton can be parameterzed wth model parameter (b) and the sample mean of 3D face shape data ( S ). Fnally, new 3D coordnates can be generated by the followng equaton: m 1 where m s the number of the modes of feature vectors. Sˆ S Φ b (6) (2) (4)

6 Sensors 2012, In Equaton (6), the local deformaton of 3D FSM depends on only the model parameter (b) because the other varables are fxed. herefore, determnng the model parameter s a key to local deformatons. Fgure 1 shows deformaton result of 3D FSM when the model parameters of frst and second prncpal mode are changed. Fgure 1. 3D FSM deformaton usng model parameters of frst and second prncpal mode. Deformaton results applyng a frst prncpal mode parameter (up). Deformaton results applyng a second prncpal mode parameter (bottom). 3. 3D Facal Shape Estmaton In ths secton, we mprove our proposed 3D face shape estmaton method usng a mrror system that was ntroduced at our prevous work [29]. In the 3D face shape calculaton usng the 3D FSM, to use only the frontal face mage may produce an naccurate 3D face shape result that s dfferent from the objectve face shape because of uncertanty n the depth drecton. o solve the problem, we calculate the 3D face shape by fttng multple 3D FSMs to face mages n three vews. he multple 3D FSMs nclude two vrtual 3D FSMs whch are appled to lateral face mages and an orgnal 3D FSM whch s appled to a frontal face mage. he vrtual 3D FSMs can be generated by transformng the orgnal 3D FSM symmetrcally onto the mrror plane. After calculatng the 3D face shape, detaled 3D face shapes can be nterpolated by a generc 3D face model D Face Shape Estmaton Usng Multple 3D FSMs n the Mrror System Our proposed face modelng system conssts of two mrrors placed on ether sde of the face and a camera n front of the face. Frontal and lateral face mages are captured smultaneously, and the pre-defned feature ponts are extracted from the captured mage as descrbed n Fgure 2. After feature extracton, the 3D FSM fttng procedure s carred out to calculate 3D coordnates from the extracted 2D feature ponts. Durng the fttng procedure, the 3D FSM parameters are adjusted to match the landmarks of the 3D FSM wth the extracted feature ponts. hs can be thought as least square optmzaton problem, and then the sum of the dstances between the projected landmarks and objectve feature ponts can be the cost functon to be mnmzed. hs cost functon can be represented as:

7 Sensors 2012, F S S S Obj S x, y, x, y,, x, y Obj N N ProjFSM ProjFSM N N 2 x, y, x, y,, x, y (7) where N s the number of landmarks of 3D FSM, S Obj s the set of objectve feature ponts n the mage, and S ProjFSM s the set of 3D FSM landmarks projected to the mage plane. Fgure 2. A mrror system for capturng facal mages and a conceptual dagram of the system. he extracted feature ponts are categorzed nto three groups, S FObj, S LObj and S RObj dependng on the drecton of the face. hen, three cost functons can be generated by the followng equatons: F F F S Front FObj FProjFSM 2 Left SLObj SLProjFSM 2 S S S Rght RObj RProjFSM 2 (8) he total cost functon s then the sum of these three varables: Fotal FFront FLeft FRght (9) In practce, 30 left sde face features are extracted, whle features lke the rght ear, rght eye, etc., reman occluded. hen, a cost functon for left sde face s determned as descrbed n Equaton (8). Next, 30 rght sde face features are extracted whle the left sde features lke left ear, left eye, etc. are occluded, and a cost functon can be determned lke n the left sde face case. In the frontal face case, only features of the ears are occluded, so we extract 40 features as descrbed n Fgure 3. Meanwhle, S FProjFSM can be drectly calculated from the perspectve projecton of 3D FSM n Equaton (8). However, addtonal calculatons wth respect to the mrror reflecton are requred to acqure S LProjFSM and S RProjFSM. In mrror geometry, the mrror mage of an object can be explaned by the projecton of a vrtual 3D object reflected by the mrror plane onto the mage plane, as descrbed n Fgure 2. A vrtual 3D face can be generated by transformng the objectve face symmetrcally onto the mrror plane. herefore, the locaton and orentaton of the mrror planes should be calculated frst.

8 Sensors 2012, Fgure 3. Feature extracton n smultaneously captured mages and cost functon generaton. An deal, perfectly flat mrror plane can be represented by: ax by cz d 0 (10) where (a,b,c) s the normal vector of the mrror plane, and (X,Y,Z) are the 3D coordnates of an arbtrary pont on the mrror plane. hen, the unknown coeffcents a,b,c and d can be easly calculated by solvng a lnear equaton usng sngular value decomposton (SVD). he 3D ponts on the mrror plane can be obtaned by Zhang s camera calbraton method [30] after attachng checkerboards to the mrrors. Once the plane equaton s calculated, a vrtual 3D face can be generated usng a Householder reflecton. Gven the 3D FSM landmarks (P real ) and the Householder matrx (H u ), the vrtual 3D face can be calculated by: P H P H P P mg u real u plane plane u H I uu u 33 2 A, B, C where u s the normal vector of the mrror plane, I 3 3 s the 3 3 dentty matrx, and P plane s an arbtrary pont on the mrror plane. Fgure 4 descrbes the landmarks of 3D FSM and vrtual 3D FSMs generated by the Householder reflecton, where the red dots n the center represent orgnal 3D FSM landmarks, and the green and blue dots represent the vrtual 3D FSM landmarks derved from both mrrors. he red and green grds represent the mrror planes. Fgure 4. 3D FSM and two vrtual 3D FSMs generated by a Householder reflecton. (11)

9 Sensors 2012, After generatng the vrtual 3D FSMs, we can calculate S LProjFSM and S RProjFSM by perspectve projecton under the camera coordnate system shown n Equaton (12): x x x 0 cx y y 0 y c y z (12) In Equaton (12), α (α x and α y ) s the scale parameter, and c (c x and c y ) s the prncpal pont wth respect to the x and y axes. hese parameters can be easly determned from the camera calbraton result when we calculate the 3D ponts on the mrror plane usng Zhang s method [30]. he second term on the left-hand sde s the normalzed perspectve projecton matrx. After the total cost functon (F total ) s determned, we calculate optmal soluton to mnmze t to ft the 3D FSM on the nput face mage. hs can be expressed as: x * x argmn x{ Ftotal } x ˆ,, A b where  s the column vector after stackng the elements of A. In global deformaton, the unknown parameters are elements of the matrces A and, and the vector b s an unknown parameter of local deformaton. o calculate the mnmzer (x * ), the teratve Levenberg-Marquardt optmzaton method s appled. In ths method, n case of orgnal 3D FSM, the partal dervatves of the Jacoban matrx can be easly calculated usng the chan rule: (13) n 1 2, Obj ProjFSM 1 n F f x f S S f f f1 f1 x1 x m Jx, x ˆ,, A b fn fn x1 x m f j f j P X, Y, Z A X t, Yt, Zt A P X, Y, Z f j f j P X, Y, Z X t, Yt, Zt P X, YZ, m f j f j Pt X, Y, Z Sb 1 b X t, Yt, Zt X, Y, Z m b Pt X, Y, Z S 1 b (14) where m s the number of 3D FSM parameters, n s the dmenson of resdual f.

10 Sensors 2012, However, for the vrtual 3D FSM, the partal dervatves are changed due to the Householder transformaton terms. After applyng Equaton (11) to Equaton (1), the partal dervatves of the resdual f can be calculated as: f j f j Pmg X t, Yt, Zt P X, Y, Z k k k A X,, X t, Yt, Z mg Ymg Z mg t A P P mg f j f j Pmg X t, Yt, Zt P X, Y, Z X,, X t, Yt, Z mg Ymg Z mg t P P mg f j f m j Pmg X t, Yt, Z t Pt X, Y, Z Sb 1 b X mg, Ymg, Zmg X t, Yt, Zt X, Y, Z b P P mg t P (15) where A and are the parameters of the 3D affne transformaton, and b s a model parameter. After calculatng the Jacoban matrx about the three 3D FSM, the entre Jacoban matrx (J All ) can be reformulated by stackng each ndvdual matrx. hen, the gradent of the cost functon ( F ) can be calculated by: J J, J, J All F LS RS F JAllf where J F s the Jacoban matrx from the front 3D FSM, J LS s the Jacoban matrx of the vrtual 3D FSM from the left mrror, and J RS s the Jacoban matrx of the vrtual 3D FSM from the rght mrror. Fgure 5 shows estmaton results of the 3D FSM, where the blue dots represent the landmarks of the ntal 3D FSM, and the red dots represent the landmarks of the 3D face shape estmated by our method. Fgure 5. he 3D face shape estmaton results. (a) Frontal vew, (b) brd eye s vew, (c) sde vew. he blue dots represent the landmarks of the ntal 3D FSM, and the red dots represent the landmarks of the 3D face shape estmated by our method. (16) 3.2. Generc Model Fttng (a) (b) (c) After 3D face shape estmaton, the 3D postons of other vertces can be determned by a generc 3D face model. he generc face model has been used n varous applcatons because t has a unform

11 Sensors 2012, pont dstrbuton and can provde detaled face shape wth a small number of ponts [3,4,26,31]. Among the prevous approaches, the most common s a deformaton technque based on the radal bass functon (RBF) whch can deform the vertces of the generc model by establshng the deformaton functon between the estmated 3D face feature ponts and the correspondng ponts of the generc model. Generally, the deformaton functon P = g(p) takes the form of low order polynomal terms M and t added to a weghted sum of the radal bass functon r wth constrants w 0 and P 0 : w g( P) w P P MP t In Equaton (17), P s a 3D feature pont of the face, and P s a correspondng vertex of the generc model. o determne the weghts of the radal bass functon w and the affne matrces M and t, Equaton (18) s reformulated as lnear equaton Ψ X = Y, where Ψ, X and Y can be expressed as follows: Ψ p1 p1 p1 pn p1 x p1 y p1 1 z pnx pny p 1 nz p1x p2x p1y p2y p1z p2z pn p1 pn pn n n w w ( n4) ( n4) X M t Y q q n 1 n Once all of the parameters of the deformaton functon are determned, the vertces of the generc model can be deformed by multplyng Ψ by X. Smlar to Ψ n Equaton (18), Ψ can be establshed from other landmarks of the 3D FSM. Fgure 6 shows the ntal generc face model and deformed generc model after RBF nterpolaton. We edted the generc model created by Pghn [5] for use n our system. In Fgure 6, the red crcles n the generc model represent the 3D shape landmarks estmated n Secton 2.2. Fgure 6. (a) Intal generc model and (b) deformed generc model after RBF nterpolaton. (17) (18) (a) (b)

12 Sensors 2012, exture Map Generaton and Mappng In ths secton, we ntroduce cylndrcal mappng and address the sttchng method usng a seamless clonng method. A seam appears at the boundares of each face part because of photometrc nconsstency after sttchng. o solve ths problem, a seamless clonng method [15] wth a gradent-doman blendng technque s appled. hs method successfully removed the seams at the boundares, and thus a texture map of the whole face could be created exture Extracton Usng Cylndrcal exture Mappng o map textures on the 3D face model, a texture map s created by extractng the texture drectly from the captured face mage. For the sake of smplcty, cylndrcal mappng s appled. In common cylndrcal mappng methods, mesh vertces that are ntersected wth the ray passng through the center of a cylnder are projected onto the mage plane after a vrtual cylnder s placed around the 3D face model. hen, the colors of the correspondng pxels n the mage are extracted and mapped to the texture map. However, ths s tme consumng because the postons of the vertces on the face mesh must be calculated. hus, n our texture mappng procedure, vertces of a trangle mesh are projected onto the mage plane, and then textures n the projected mesh are warped on the texture map, as shown Fgure 7. Our system sttches together texture maps from the three dfferent vews to create a texture map of the entre face. Fgure 7. Cylndrcal texture mappng for facal texture extracton exture Map Generaton Usng Modfed Image Sttchng Method As addressed n Secton 4.1, a texture map of the entre face can be created by sttchng each face texture parts. However, a seam appears at the boundares of each face parts because of photometrc nconsstency, as descrbed n Fgure 8(a). o solve ths problem, a seamless mage sttchng method wth gradent-doman blendng technques [15] s appled. Generally, gradent-doman blendng technques are more effcent at reducng photometrc nconsstences than s general mage-doman blendng. o mplement the seamless clonng method, the entre texture mage s frst dvded nto three parts along the boundares. hen, overlappng regons are created by expandng both parts by

13 Sensors 2012, pxel at the encounterng regon. Next, a h(x,y) map s created that s the same sze as the orgnal texture map of the whole face. Fgure 8. exture map refnement usng seamless clonng. (a) Face texture map after mergng each face texture part, (b) ntal map for seamless clonng, (c) syntheszed face texture map after seamless clonng (d) h(x,y) map after 100 teratons of Equaton (12). (a) (b) (c) (d) For ntalzaton of the h(x,y) map, the sde facal regon havng 0 pxel values s changed to have 1 pxel values. he pxel values n the overlappng regons are set to the rato of the pxel value of the front vew (g) to the pxel value of the sde vew (f): where R s the sde regon. h x, y 1 f ( x, y) g( x, y) g( x, y) 0 or( x, y) R otherwse he h(x,y) value s teratvely updated by calculatng the soluton of Laplace s equaton at the correspondng pxel: 1, 1 1, 1,, 1, 1 n n n n n h x y h x y h x y h x y h x y (20) 4 Fgure 8(b,d) shows an ntal h(x,y) map and the resultng h(x,y) map after 100 teratons. In Fgure 8(b), the h(x,y) values at the boundary are propagated nto the vald area. After suffcent teratons, the fnal pxel values on the sde of the face are calculated as the product of the updated h(x,y) and orgnal pxel value, whch allows for creaton of a contnuous texture map at the boundary: n (19) f ˆ x, y h ( x, y ) g x, y (21) After usng morphologcal operatons to fll n holes, a fnal face texture map can be created, as shown n Fgure 8(c). Fgure 9 shows the fnal 3D face after refnng the texture map and the addton of artfcal eyes.

14 Sensors 2012, Fgure 9. 3D face modelng result after texture mappng. Generally, mult-resoluton splnng [32] s well known as a blendng method that operates well when the overlapped regons between mages to be splned are broad enough. In our system, the overlapped regon between each face part s only one pxel wde, therefore we couldn t get a satsfactory result when usng mult-resoluton splnng method as descrbed n Fgure 10(b). On the other hand, the gradent doman mage sttchng method that s appled to our system shows excellent performance n spte of the narrow overlapped regon as descrbed n Fgure 10(c). Fgure 10 shows that the seam between boundares of texture parts as descrbed n Fgure 10(a) s completely removed, whle the seam remans after applyng mult-resoluton splnng. Fgure 10. Image sttchng results usng mult-resoluton splnng and our method. (a) Image sttchng wthout any blendng methods. (b) Image sttchng usng mult-resoluton splnng. (c) Image sttchng usng our method. 5. Experments and Results 5.1. Expermental Settngs (a) (b) (c) Before constructng the proposed face modelng system, we completed statstcal analyses wth respect to the 3D scan landmarks n order to calculate the feature vector elements of the local deformaton parameters n the FSM, as descrbed n Secton 2.1. For the statstcal analyss, we used prncpal component analyss (PCA). We recorded 3D face vews of 100 ndvduals wth a M Cyberware laser scanner and then selected 50 landmarks n each face. Durng the PCA procedure, we retaned 90% of the egenvalue energy spectrum to reduce computatonal complexty.

15 Sensors 2012, o defne the ground truth, we captured 3D faces of 30 ndvdual wth a 3D laser scanner at the same tme that we captured the mage wth our proposed system. We attached color markers on each user s face to dentfy the feature ponts as shown n Fgure 11 and then manually measured ther 3D coordnates. Fgure 11. he user s face captured wth our mrror system. he red color markers are attached on user s face for performance test. For a relatve comparson, we used Ln s method [9]. hat work s a good reference to evaluate the performance of our method because ther mrror-based face modelng system s smlar to ours. Addtonally, we can ndrectly compare our method and ordnary 3D reconstructon methods usng eppolar geometry because they already compared ther work wth ordnary 3D reconstructon methods usng eppolar geometry Accuracy ests We mplemented Ln s method [22] and calculated the 3D coordnates of the marked feature ponts. In Ln s method, only the 3D coordnates of vsble features can be reconstructed, so we compared the 3D reconstructon results of the 40 features on the frontal face wth the actual faces. o compare the results wth the actual faces, we algned the 3D ponts from both methods wth the 3D ponts of the actual faces. 3D procrustes analyss [28] was used to algn the 3D coordnates of the reconstructed ponts. After algnng the 3D ponts, we calculated the average sum of the Eucldean dstances between each pont on the reconstructon and the actual faces, as descrbed n Equaton (22): M 1 E SRe const S M S S 1 Ground x, y, z, x, y, z, x, y, z Reconst N N N x, y, z, x, y, z, x, y, z Ground N N N hen, we compared the accuracy of ther method wth that of our method, as shown n able 1, where our proposed method exhbts slghtly hgher absolute errors than that of Ln, but the standard devaton (Std) of our method was about two tmes lower. 2 (22)

16 Sensors 2012, able 1. Mean, standard devaton and medan of absolute dstance errors of the proposed and Ln. I-C s method compared to the actual faces. Estmaton Method Absolute Error (mm) Mean Std Medan Ln. s method [22] Proposed method est on Robustness to Feature Extracton Error o test on the robustness of our method wth respect to feature extracton errors, we artfcally generated erroneous feature ponts wth normally dstrbuted random dstances and drectons. Frstly, we calculated a two-dmensonal matrx contanng normally dstrbuted random numbers usng the Box-Muller method. hen, we generate the nosy feature ponts by addng each column vector of the matrx to the 2D coordnates of the feature ponts n the nput face mage. he feature ponts on the face for measurement were annotated by color markers. We assumed that the 2D postons of the marked feature ponts were the reference poston. We frst tested the results of our proposed method and Ln s method accordng to error strength, whch can be adjusted by changng the standard devaton of the random numbers. able 2 shows the maxmum error dstances accordng to the standard devaton. able 2. he maxmum error dstances accordng to the standard devaton. Standard devaton of the error Maxmum error dstances (pxel) We carred out the 3D face shape reconstructon by applyng the proposed method and Ln s method wth nosy feature ponts. he standard devaton of the error was vared from 0 to 5 n ntervals of hen, we calculated the average sum of the Eucldean dstances (average absolute error) between each 3D reconstructon pont and the truth. As shown n Fgure 12(a), the average absolute error of the proposed method ncreases monotoncally, whle the average absolute error of Ln s method ncreases and fluctuates much more rapdly. Next, we fxed the standard devaton and measured the average absolute error as the number of nosy feature ponts was ncreased from 0 to 100 n ntervals of 1. As shown n Fgure 12(b), the results are smlar to those of the frst robustness test. he average absolute error of the proposed method ncreases monotoncally, but the average absolute error of Ln s method ncreases and fluctuates much more rapdly. Fgure 13 shows the 3D face modelng results of proposed method and that of Ln wth erroneous feature ponts. he standard devaton of the error s fxed at 3. As shown n Fgure 13, the result of Ln s method shows a sgnfcantly dstorted shape near the erroneous feature ponts. On the other hand, the proposed method mantans the overall face shape, even wth the nosy feature ponts.

17 Sensors 2012, Fgure 12. Robustness test results. (a) Error measurement results wth ncreasng error strength by changng standard devaton of normally dstrbuted random numbers. (b) Error measurement results wth ncreasng number of nosy feature ponts. (a) (b) Fgure 13. (a) 3D facal shape estmaton results of Ln s method. (b) Proposed method wth erroneous feature ponts. (a) (b)

18 Sensors 2012, extured 3D Face Model Generaton Results We generated a textured 3D face model of users usng the proposed face modelng method. After applyng the generc model fttng as descrbed n Subsecton 3.2, we appled our texture mappng method descrbed n Secton 4. Eyeballs are not ncluded n the generc model, and so we nserted artfcal eyeballs wth the 3D Max program. After producng the eyeballs, we algn the center of the eyeball to the center of the eye regon. Fgure 14 shows the nput face mage, the texture map and the textured 3D face. Fgure 14. he results of the textured 3D face model for ndvduals. 6. Conclusons In ths paper, we propose a realstc 3D face modelng method that s robust to feature extracton errors and can generate accurate 3D face models. In the facal shape estmaton procedure, we propose a 3D face shape estmaton method usng multple 3D face deformable models n a mrror system. he proposed method shows hgh robustness to feature extracton errors and hghly accurate 3D face modelng results, as descrbed n Sectons 5.2 and 5.3. In the texture mappng procedure, we apply cylndrcal mappng and sttchng technque to generate a texture map. We apply the seamless clonng method, whch s a type of gradent-doman blendng technque, to remove the seam that caused by photometrc nconsstency and fnally can thus acqure a natural texture map. o evaluate our method s performance, we carry out accuracy and robustness tests wth respect to 30 ndvduals 3D facal shape estmaton results. Our method shows not only hghly accurate 3D face shape results when compared wth the ground truth, but also robustness to feature extracton errors. Moreover, the 3D face renderng results ntutvely show that our method s more robust to feature extracton errors than other 3D face shape estmaton methods. An addtonal contrbuton of our method s that wde range of face textures can be acqured by the mrror system. Lastly, we generate textured 3D faces usng our proposed method. he results show that our method can generate very realstc 3D faces, as shown n Fgure 14. Our ultmate goal s to create an automatc 3D face modelng system, and so we plan to apply automatc feature extracton processes to our method. hs may be a problem for sde vew mages and wll requre the development of new technques that wll be descrbed n future works. Acknowledgments hs research was supported by Basc Scence Research Program through the Natonal Research Foundaton of Korea (NRF) funded by the Mnstry of Educaton, Scence and echnology ( ). hs work was supported by the Natonal Research Foundaton of Korea (NRF) grant funded by the Korea government (MES) (No ).

19 Sensors 2012, References 1. Ersotelos, N.; Dong, F. Buldng hghly realstc facal modelng and anmaton: A survey. Vs. Comput. 2008, 24, Sanson, G.; rebesch, M.; Doccho, F. State-of-the-art and applcatons of 3D magng sensors n ndustry, cultural hertage, medcne, and crmnal nvestgaton. Sensors 2009, 9, Remondno, F.; El-Hakm, S. Image-based 3D modelng: A revew. Photogram. Rec. 2006, 21, Lee, Y.; erzopoulos, D.; Waters, K. Realstc Modelng for Facal Anmaton. In Proceedngs of the 22nd Annual Conference on Computer Graphcs and Interactve echnques, Los Angeles, CA, USA, 6 11 August 1995; pp Yu, Z.; Edmond, C.P.; Erc, S. Constructng a realstc face model of an ndvdual for expresson anmaton. Int. J. Inform. echnol. 2002, 8, Crocombe, A.D.; Lnney, A.D.; Campos, J.; Rchards, R. Non-contact anthropometry usng projected laser lne dstorton: hree-dmensonal graphc vsualzaton and applcatons. Opt. Lasers Eng. 1997, 28, Fua, P. From multple stereo vews to multple 3D surfaces. Int. J. Comput. Vs. 1997, 24, Ku, A. Implementaton of 3D optcal scannng technology for automotve applcatons. Sensors 2009, 9, Gonzlez-Agulera, D.; Gmez-Lahoz, J.; Snchez, J. A new approach for structural montorng of large dams wth a three-dmensonal laser scanner. Sensors 2008, 8, Hartley, R.I.; Zsserman, A. Multple Vew Geometry n Computer Vson, 2nd ed.; Cambrdge Unversty Press: Cambrdge, UK, Blanz, V.; Vetter,. A Morphable Model for the Synthess of 3D Faces. In Proceedngs of the 26th Annual Conference on Computer Graphcs and Interactve echnques, Los Angeles, CA, USA, 8 13 August 1999; pp Cootes,.; Edwards, G.; aylor, C. Actve Appearance Models. In Proceedngs of the 5th European Conference on Computer Vson, Freburg, Germany, 2 6 June 1998; pp Cootes,.; aylor, C. Constraned Actve Appearance Models. In Proceedngs of the 8th IEEE Internatonal Conference on Computer Vson, Vancouver, BC, Canada, 7 14 July 2001; pp Cootes,.; aylor, C.; Cooper, D.; Graham, J. Actve shape models her tranng and applcaton. Comput. Vs. Image Understand. 1995, 61, Georgev,. Covarant Dervatves and Vson. In Proceedngs of the 9th European Conference on Computer Vson, Graz, Austra, 7 13 May 2006; pp Lee, W.S.; Magnenat-halmann, N. Generatng a Populaton of Anmated Faces from Pctures. In Proceedngs of IEEE Internatonal Workshop on Modellng People, Kerkyra, Greece, 29 September 1999; pp Akmoto,.; Suenaga, Y.; Wallace, R. Automatc creaton of 3D facal models. IEEE Comput. Graph. Appl. 1993, 13, Ip, H.H.S.; Yn, L. Constructng a 3D ndvdualzed head model from two orthogonal vews. Vs. Comput. 1996, 12,

20 Sensors 2012, Fua, P. Regularzed Bundle-adjustment to model heads from mage sequences wth calbraton data. Int. J. Comput. Vs. 2000, 38, Lee,.Y.; Ln, P.H.; Yang,.H. Photo-Realstc 3D Head Modelng Usng Mult-Vew Images. In Computatonal Scence and Its Applcatons ICCSA 2004; Sprnger: Berln/Hedelberg, Germany, 2004; Volume 3044, pp Pghn, F.; Hecker, J.; Lschnsk, D.; Szelsk, R.; Salesn, D. Syntheszng realstc facal expressons from photographs. In Proceedngs of the 25th Annual Conference on Computer Graphcs and Interactve echnques, Orlando, FL, USA, July 1998; pp Ln, I.; Yeh, J.; Ouhyoung, M. Extractng 3D facal anmaton parameters from mult-vew vdeo clps. IEEE Comput. Graph. Appl. 2002, 22, Blanz, V.; Scherbaum, K.; Sedel, H.P. Fttng a Morphable Model to 3D Scans of Faces. In Proceedngs of the 11th IEEE Internatonal Conference on Computer Vson, Ro de Janero, Brazl, October 2007; pp Kuo, C.; Huang, R.; Ln,. Syntheszng Lateral Face from Frontal Facal Image Usng Anthropometrc Estmaton. In Proceedngs of Internatonal Conference on Image Processng, Washngton, DC, USA, October 1997; pp Kuo, C.; Huang, R.S.; Ln,.G. 3-D facal model estmaton from sngle front-vew facal mage. IEEE rans. Crc. Syst. Vdeo echnol. 2002, 12, Baek, S.; Km, B.; Lee, K. 3D Face Model Reconstructon from Sngle 2D Frontal Image. In Proceedngs of the 8th Internatonal Conference on Vrtual Realty Contnuum and Its Applcatons n Industry, Yokohama, Japan, December 2009; pp Blanz, V.; Vetter,. A Morphable Model for the Synthess of 3D Faces. In Proceedngs of the 26th Annual Conference on Computer Graphcs and Interactve echnques, Los Angeles, CA, USA, 8 13 August 1999; pp Ansar, A.; Abdel-Mottaleb, M. 3D Face Modelng Usng wo Vews and A Generc Face Model wth Applcaton to 3D Face Recognton. In Proceedngs of the IEEE Conference on Advanced Vdeo and Sgnal Based Survellance, Mam, FL, USA, July 2003; pp Hwang, J.; Km, W.; Ban, Y.; Lee, S. Robust 3D Face Shape Estmaton Usng Multple Deformable Models. In Proceedngs of the 6th IEEE Conference on Industral Electroncs and Applcatons, Mam, FL, USA, June 2011, pp Zhang, Z. Flexble Camera Calbraton by Vewng a Plane from Unknown Orentatons. In Proceedngs of the 7th IEEE Internatonal Conference on Computer Vson, Kerkyra, Greece, September 1999; pp Wdanagamaachch, W.; Dharmaratne, A. 3D Face Reconstructon from 2D mages. In Proceedngs of 2010 Internatonal Conference on Dgtal Image Computng: echnques and Applcatonsn, Canberra, AC, Australa, 1 3 December 2008, pp Lee, W.-S.; halmann, N.M. Fast head modelng for anmaton. J. Image Vs. Comput. 2000, 18, by the authors; lcensee MDPI, Basel, Swtzerland. hs artcle s an open access artcle dstrbuted under the terms and condtons of the Creatve Commons Attrbuton lcense (