3D Face Reconstruction With Local Feature Refinement. Abstract

Size: px

Start display at page:

Download "3D Face Reconstruction With Local Feature Refinement. Abstract"

Winfred Cox
6 years ago
Views:

1 , pp D Face Reconstructon Wth Local Feature Refnement Rudy Adpranata, Kartka Gunad and Wendy Gunawan 3, formatcs Department, Petra Chrstan Unversty, Surabaya, donesa 3 wendygunawan87@gmal.com Abstract ths paper, we propose 3D face reconstructon from two or three mages. The reconstructon process begns wth a face localzaton and facal feature extracton. Global transformaton process s done for facal features whch have been obtaned. Local feature refnement s used to refne the result of global transformaton. Last, texture mappng process s done to provde texture of 3D model. The method used for face reconstructon can generate accurate and fast result for varous poses and facal expressons, and by usng local feature refnement can mprove the qualty of reconstructon output more smlar to the orgnal face mage. Keywords: face reconstructon, local feature refnement, texture mappng. troducton 3D reconstructon of the human face s a topc that s stll actvely developed today. The human face s one of the models that are dffcult to reconstruct n three dmensons due to varatons of the human face shape, skn color varaton, the estmated depth of facal components. The need for 3D facal reconstructon has been growng n applcatons such as face recognton, three-dmensonal games, vrtual realty, anmaton, etc. general, the method to reconstruct can be dvded nto several categores, namely pure modelng technques from mages, 3D scannng technques, mage-based hybrd technques [6, 9]. pure modelng technques from mages, three-dmensonal models made only based on the mage [9]. These technques perform reconstructon usng D mages only wthout estmatng the 3D structure of the real, but only by dong a comparson of the rendered mage. On 3D scannng technques, there are knds of ways, namely actve and passve. 3D scannng technques are actvely carred out by means of a three-dmensonal scanner, 3D scannng technques whle passvely carred by fndng correspondences between multple forms. Passve 3D scannng technque known as Shape from X where X can be means of movement, texture, shadow or stereo [7, 0,, 3]. the hybrd technque, mages and estmaton of the data are used to perform reconstructon process. Pure modelng methods often produce structures that are less approprate. Actve 3D scannng methods requre a 3D scanner tool that s qute expensve whle the method of shape from X requres specal preparaton to obtan 3D nformaton that s not easy to do. To address the structural weaknesses that were not approprate, a generc face model wll be used to deform accordng to the feature ponts on the human face [3, 3]. To obtan 3D nformaton n the feature pont, one, two or more orthogonal dsplay faces wll be used [,, 3, 9]. ths paper we propose methods to reconstruct 3D face model from or 3 orthogonal photos usng facal feature extracton preprocessng and addton of local refnement process n order to acheve better result. ISSN: IJMUE Copyrght c 04 SERSC

. Generc Face Model Generc face model s generally defned n terms of a trangular mesh consstng of vertces (coordnate ponts), edges (relatons between two vertces) and face (set of edge).

2 . Generc Face Model Generc face model s generally defned n terms of a trangular mesh consstng of vertces (coordnate ponts), edges (relatons between two vertces) and face (set of edge). Generc face models can also be nterpreted as a set of trangular mesh wth vertces that can accommodate the mportant features of the face n performng face recognton. These features nclude the eyebrows, eyes, nose, mouth and face shape. Generc face model s an mportant part of a 3D face modelng. The level of accuracy n the modelng s also nfluenced by how detaled generc face model used [4]. ths research we used generc face model taken from a 3D human face n FaceGen Modeller applcaton [7] whch can be seen at Fgure. Fgure. Generc Face Model 3. Face Localzaton and Facal Feature Extracton Face localzaton s one of the stages of face detecton, whch ams to obtan a canddate regon of human face through the process of skn color detecton. the process of skn color detecton, the process of analyss wll be carred out n two color models, normalzaton of RGB (red, green, blue) and HSV (hue, saturaton, value). From the analyss by usng a combnaton of the two color models, t can be determned whch parts of an mage that ncludes skn-color and what s not. Accordng to Wang and Yuan [6], the boundares of whch can be accepted as the lmt of human skn color s as follows n equaton () r 0.465, 0 H 50, 0.8 g 0.363, 0.0 S 0.68, 0.35 V.0 After the dentfcaton of the parts of a dgtal mage that s ncluded n the category of skn color, next process s combnng parts to form a face canddate regon and dong facal feature canddate extracton. Facal feature canddate extracton s a step done n order to obtan the parts of an object's face. ths case, the normal human face regons generally consst of two eyes, nose and mouth. However, n the process of canddate facal feature extracton, the nasal passages are not consdered due to the consderaton that the nose has a color factor whch s almost the same as human skn color n general. Therefore, the facal feature extracton, the eyes and mouth factor s more domnant [4, 5]. Facal feature canddate extracton process begns wth the converson of RGB color component of face canddate areas to the YCbCr color models. Next, a maskng process wth the convex-hull maskng method s done. addton to convex hull algorthm, whch s used to obtan a convex area of the face canddates, there s an optonal process to solve some problem that cannot be solved by ordnary convex hull algorthms. Ths process s () 6 Copyrght c 04 SERSC

3 called recheckng convex hull process [5]. Recheckng convex hull s a process to recheck every pxel value on the mask regon area whch s resulted from the ordnary convex hull, whether the pxel s ncluded n the human skn color crtera or not. As long as the recheckng convex hull process does not fnd any pxels that are ncluded n human skn color crtera, the process wll replace those pxels wth a certan value of color whch has been determned before. If the pxel has exactly the same value wth the background pxel, the process wll replace t wth nothng. Ths recheckng convex hull process wll stop f t has found a pxel whch s ncluded n human skn color crtera. Recheckng convex hull process has two types of drecton, vertcal drecton and horzontal drecton. The face canddate area n YCbCr color models and the result of maskng wll be processed to determne the locaton of the eyes and mouth. For the mappng process of the eye, s dvded nto two stages: EyeMapL and EyeMapC. EyeMapL done based on the lumnance component factors whle EyeMapC performed by chromnance component factors. The equatons to obtan EyeMapL and EyeMapC are as follows n equaton () (6) [4]: ~ EyeMapC ( Cb ) ( Cr ) ( Cb / Cr) () 3 ~ r ~ Cb, Cr, Cb / Cr are normalzed to the value range between 0 and 55. C s the negatve value of Cr (55-Cr). EyeMapL g Y( x, g Y( x, g ( x, (3).( ( R( x, / ) R( x, y ( x, / ) R R x (5) W. H (6) (. Fe) g s hemspherc structurng element whch wll be used as a structurng element n the gray-scale dlaton and gray-scale eroson. R(x, s euchldean dstance from the center pont of structurng element to any other pont of the structurng element. W and H are the wdth and heght of the face canddate area. Fe s a constant defned as the maxmum rato between the sze of the eyes and the sze of a human face. Mappng of both equatons EyeMapL and EyeMapC wll be operated by AND operator to get EyeMap regon. To clarfy the eye regon, dlaton operaton can be performed. For the mappng of the mouth, can be obtaned through the equaton (7) (8) [4]: MouthMap n Cr (. Cr. Cr / Cb) (7) n ( x, G ( x, G Cr( x, ( Cr( x, / Cb( x, ) (4) (8) Copyrght c 04 SERSC 63

4 Cr, Cr / Cb are normalzed to the value range between 0 and 55. s the rato between the average value of Cr and average value of Cr / Cb. n s the number of pxels n face canddate area. After obtanng the canddate eye and mouth area through the facal area canddate feature extracton, selecton process wll then be carred to the facal area canddate feature tself. Ths selecton process can generally be dvded nto two man parts: the teratve thresholdng and the selecton of center of mass of the face canddate [4]. The results of facal feature extracton are 9 feature ponts consstng of 4 feature ponts of the rght eye, 4 feature ponts of left eye, and one feature pont of mouth. 4. Global Transformaton the D model, has been set 35 ponts as the startng facal features pont whch s forward-lookng facal features and the face looks the left sde and rght sde [9]. Then, usng 9 features ponts that have been obtaned from the facal feature extracton, global transformaton process s done to fnd the value of the transformaton of other 6 features pont. After calculaton, the new 35 feature ponts wll be llustrated n the D model and the user can change t back f t s consdered less approprate. Ths global transformaton ncludes scalng, rotaton and translaton to the orgnal pont. Suppose P Mo s the feature ponts of the model and P s the feature ponts from facal feature extracton. To be able to fnd the value of the transformaton from P Mo to P, t s necessary to elmnate the translaton and scale values of PMo and P [9]. Elmnatng the translatonal component of the feature ponts can done by translatng the ponts so that the mean of all the ponts s at the startng pont as shown n equaton (9)-(0). ( x', y') ( x x, y x x xk y y yk x, y k k,,,k, k s the number of ponts Scale can also be elmnated by dvdng the ponts that have been translated n equaton (9)-(0) wth the norm usng equaton ()-(). ( x', y' ) (( x x) / s,( y / s) s ( x y () x) ( y ( xk x) ( yk ),,,k, k s the number of ponts Next step s dong transpose of P and multplyng the result by usng equaton (3). T A P P Mo (9) (0) () PMo to produce A To get the value of a scale and rotaton transformaton, sngular value decomposton (SVD) process of the matrx A s done that generates U, S and V matrx usng equaton (4) [9]. USV T SVD(A) (3) (4) 64 Copyrght c 04 SERSC

5 Rotaton, scale and translaton matrx can be calculated usng equaton (5)-(6) sp Sc Tr( S) * sp Mo (5) ' Rot VU (6) sp s norm of P. spmo s norm of P Mo. Tr P P Mod P Mod ScP Rot Fnally, to transform PMo P ScPMoRot Tr to P, equaton (7) s used as follows: To perform calculatons on 3D space, smply add the z coordnate factor n the equaton (9), (0), () and () [9]. 5. Local Feature Refnement Transformng the global 3D model s not enough to hghlght mportant features on the human face. Ths s because each feature has dfferent transformatons value. It s dffcult to obtan a transformaton matrx that can precsely place all of feature ponts correspond to the nput mage. Therefore we need the local transformaton at each mportant facal feature such as eyes, nose and mouth n order to get more accurate 3D model result. Local transformaton conssts of local algnment and local feature refnement process [4]. Local algnment s the process to transform the model features such as eyes, nose, mouth, and chn to match the facal features. Local feature refnement s a process to mprove the model so as to blend wth the local features that have been transformed. The process of local algnment s done by fndng the value of the transformaton of a generc face model that has been transformed globally nto 3D ponts. Searchng on the value transformaton can be done usng two methods. Frst method s the same as the global transformaton process. The dfference s only on the number of feature ponts used. Snce global transformaton s done throughout all of feature ponts, the local transformaton s done only to feature ponts that are part of the local features. The second method s to calculate the transformaton of the square by comparng the wdth and heght of the model wth three-dmensonal coordnates from calculaton. The second method s better than the frst method that usng sngular value decomposton. Ths s because the SVD process only produces a sngle scale value. If there s a dfference n scale between x coordnate wth y coordnate, the SVD wll look for the nearest value. Square transformaton can be calculated usng equaton (8)-() [4]. l Scx l Mo (7) (8) Sc y h h Mo (9) Tr C Sc * C x x x x Mo (0) Copyrght c 04 SERSC 65

6 Tr C Sc * C y y y y Mo () Tr s translaton of x coordnate, x Tr s translaton of y coordnate y Sc s scale of x coordnate, x Sc s scale of y coordnate y l s wdth of local feature of pont calculaton result, l Mo s wdth of local feature of 3D model h s heght of local feature of pont calculaton result, 3D model C s centrod of x x from local feature of pont calculaton result, from local feature of 3D model C s centrod of y from local feature of pont calculaton result, y centrod of x from local feature of 3D model Centrod of polygon can be calculated usng equaton ()-(4). A C C N 0 (x y x y ) N x (x x )(x y x y ) 6A 0 N y (y y )(x y x y ) 6A 0 x h Mo s heght of local feature of C x Mo s centrod of x C s centrod of x, C s centrod of y, N s number of vertces of polygon y C y Mo C x Mo s The local features refnement s done by usng the dsplacement propagaton method. Dsplacement propagaton on a trangular mesh s smlar to message delvery method n computer networks. Suppose N s the number of vertex whch s connected to vertex V, J s a collecton of vertex connected to V, w s the sum of eucldean dstance of each vertex whch s connected to V, and d j s the eucldean dstance between vertex V and vertex V j. Suppose V j s value of vertex V j changes and s decay factor, a value that wll change when the dstance between two ponts changed. The calculaton of the contrbuton to the dsplacement of vertex V j from dsplacement of vertex V can be seen n the equaton (5)-(6) [4]. w dj dj V j. e, N, w d jj j Vj w.( N ) dj V j. e, N, j J (5) V V j jj 6. Texture Mappng Texture mappng s the process of attachng an mage or texture to a polygon. Texture mappng allows the depcton of the entre mage s used as a texture n a polygon. Texture mappng also ensure the proper texture delneaton when a polygon s () (3) (4) (6) 66 Copyrght c 04 SERSC

transformed. Texture tself s a collecton of data such as data color, lghtng and transparency. dvdual value at texture array s often referred as a texel [].

Due to the correlaton between the vertex and mappng area already contaned n the 3DS template we used, then the problem s how to move the rght mage nput to the exstng mappng area.

Target mage wll be checked pxel by pxel to fnd pxel match to the source mage. The advantage of the reverse mappng s each pxel n the target mage defntely gets a par n the source mage [8].

7 transformed. Texture tself s a collecton of data such as data color, lghtng and transparency. dvdual value at texture array s often referred as a texel []. Texture mappng process begns wth a movng mage nput from user to an area that s used for mappng. Next process s to determne correlaton of each vertex of 3D models n the mappng area. Due to the correlaton between the vertex and mappng area already contaned n the 3DS template we used, then the problem s how to move the rght mage nput to the exstng mappng area. To resolve ths problem, we use the mage warpng approach wth reverse mappng method. Reverse mappng s one method n mage warpng whch begn from the target mage. Target mage wll be checked pxel by pxel to fnd pxel match to the source mage. The advantage of the reverse mappng s each pxel n the target mage defntely gets a par n the source mage [8]. We mplement reverse mappng on 3D face modelng by creatng 70 regons n the target mage whch n those regons, 35 feature ponts have been determned. Then a transformaton process for each regon wll be done. After the transformaton matrx s obtaned, t can be determned the correlaton of each pxel n the source mage for each pxel n the target mage. The dvson of the facal regon by usng 35 pont features can be seen n Fgure. Fgure. Dvson of Facal Regon by Usng 35 Feature Ponts 7. Expermental Results We developed the applcaton usng C++ and the nterface can be seen n Fgure 3 as follows. Fgure 3. Applcaton terface Copyrght c 04 SERSC 67

8 The experments were conducted by usng multple mages of dfferent faces n terms of poses, gender, skn color, glance of the eyes, eyeglasses, llumnaton, face expresson and the overall result of 3D face modellng can be seen at Fgure Copyrght c 04 SERSC

9 Copyrght c 04 SERSC 69

10 Fgure 4. Expermental Result We also conducted experment on local feature refnement. We compare the result of local feature refnement usng both methods, square transformaton and SVD. The experment result on ths can be seen n Table. 70 Copyrght c 04 SERSC

11 Table. Experment Result on Local Feature Refnement Image Source () 3D Modellng Result Wreframe Square Tranformaton SVD Processng tme seconds seconds Image Source () 3D Modellng Result Wreframe Square Tranformaton SVD Processng tme 0.35 seconds seconds Copyrght c 04 SERSC 7

Image Source (3) 3D Modellng Result Wreframe Square Tranformaton SVD Processng tme 0.968 seconds 7.

However, the speed of the square transformaton process s slower than the speed of SVD process. 8.

Local feature refnement process gves better results, and along wth the ncreasng of teratons number, the better the results wll be obtaned, but f the teraton number s too large, t wll only ncrease

From experment result, the method used for face reconstructon can generate accurate and fast result and allow to synthess varous poses and facal expressons correspondng to the orgnal face lke lp mmc,

12 Image Source (3) 3D Modellng Result Wreframe Square Tranformaton SVD Processng tme seconds seconds From the results of experments conducted, t s seen that the results of 3D modelng by usng square transformaton better than usng SVD because t s more smlar to the orgnal face mages. However, the speed of the square transformaton process s slower than the speed of SVD process. 8. Concluson ths paper, we have proposed 3D face reconstructon from two or three mages usng local feature refnement. Local feature refnement process gves better results, and along wth the ncreasng of teratons number, the better the results wll be obtaned, but f the teraton number s too large, t wll only ncrease processng tme wthout sgnfcant dfference result. From experment result, the method used for face reconstructon can generate accurate and fast result and allow to synthess varous poses and facal expressons correspondng to the orgnal face lke lp mmc, glance of the eyes, etc. The local feature refnement usng square transformaton s able to produce a better result than usng sngular value decomposton but takes a lttle longer. References [] M. Pamplona Segundo, Improvng 3D face reconstructon from a sngle mage usng half-frontal face poses, Proceedngs of 9th IEEE ternatonal Conference on Image Processng (0) September 30- October 3. [] I. Kemelmacher-Shlzerman and R. Basr, 3D Face Reconstructon from a Sngle Image usng a Sngle Reference Face Shape, IEEE Transactons on Pattern Analyss and Machne tellgence, vol. 33, no., (0). [3] X. Fan, Q. Peng and M. Zhong, 3D Face Reconstructon from Sngle D Image Based on Robust Facal Feature Ponts Extracton and Generc Wre Frame Model, Proceedngs of ternatonal Conference on Communcatons and Moble Computng, (00) Aprl -4. [4] N. Patel and M. Zaver, 3D Facal Model Constructon and Expressons Synthess usng a Sngle Frontal Face Image, ternatonal Journal of Graphcs, vol., no., (00). 7 Copyrght c 04 SERSC

[5] E. R. Adpranata, C. G. Ballangan and R. P. Ongkodjojo, Fast Method for Multple Human Face Segmentaton n Color Image, ternatonal Journal of Advanced Scence and Technology, vol. 3, (009) [6] W. N.

Glles, Analyss of 3D face reconstructon, Proceedngs of the 4th IEEE ternatonal Conference on Image Analyss and Processng, (007) September 0-4. [8] T.

13 [5] E. R. Adpranata, C. G. Ballangan and R. P. Ongkodjojo, Fast Method for Multple Human Face Segmentaton n Color Image, ternatonal Journal of Advanced Scence and Technology, vol. 3, (009) [6] W. N. Wdanagamaachch and A. T. Dharmaratne, 3D Face Reconstructon from D Images, a Survey, Proceedngs of Dgtal Image Computng: Technques and Applcatons (008), December -3. [7] S. H. Amn and D. Glles, Analyss of 3D face reconstructon, Proceedngs of the 4th IEEE ternatonal Conference on Image Analyss and Processng, (007) September 0-4. [8] T. Funkhouser, Image warpng, compostng and warpng, Prnceton Unversty, Department of Computer Scence, (006). [9] N. Raswasa, The avatar: 3-d face reconstructon from two orthogonal pctures wth applcaton to facal makeover, (005). [0] D. Samaras, S. Wang and L. Zhang, Face reconstructon across dfferent poses and arbtrary llumnaton condtons, Proceedngs of Ffth ternatonal Conference on Audo and Vdeo based Bometrc Person Authentcaton, (005) July 0-. [] T. Davs, J. Neder, D. Shrener and M. Woo, OpenGL programmng gude (4th ed.). Addson-Wesley, Boston (004). [] J. Lee, R. Machraju, M. Baback and H. Pfster, Slhouette-based 3d face shape recovery, Graphcs terface, (003). [3] C. H. Esteban, and F. Schmtt, Slhouette and Stereo Fuson for 3D Object Modelng, Proceedngs of Fourth ternatonal Conference on 3-D Dgtal Imagng and Modelng, (003) October [4] R.-L. Hsu, M. Abdel-mottaleb and A. K. Jan, Face Detecton Color Images, IEEE Transactons on Pattern Analyss and Machne tellgence, vol. 4, no. 5, (00). [5] K. Sandeep and A. N. Rajagopalan, Human Face Detecton n Cluttered Color Images Usng Skn Color and Edge formaton, Proceedngs of dan Conference of Computer Vson, Graphcs and Image Processng, (00) December 6-8. [6] Y. Wang and B. Yuan, A novel approach for human face detecton from color mages under complex background. Pattern Recognton, vol. 34, no. 0, (00). [7] FaceGen Modeller Applcaton. Authors Rudy Adpranata s currently a senor lecturer n formatcs Department, Petra Chrstan Unversty, Surabaya, donesa. He receved hs bachelor degree n Electrcal Engneerng from Petra Chrstan Unversty, Surabaya, donesa, and master degree n Software Engneerng from Graduate School of Software, Dongseo Unversty, Busan, South Korea. Hs research nterests are mage processng and computer vson. Rudy Adpranata s currently a senor lecturer n formatcs Department, Petra Chrstan Unversty, Surabaya, donesa. He receved hs bachelor degree n Electrcal Engneerng from Petra Chrstan Unversty, Surabaya, donesa, and master degree n Software Engneerng from Graduate School of Software, Dongseo Unversty, Busan, South Korea. Hs research nterests are mage processng and computer vson. Wendy Gunawan receved hs bachelor degree n formatcs from Petra Chrstan Unversty Surabaya, donesa. Currently he works as a teacher n Surabaya, donesa Copyrght c 04 SERSC 73

14 74 Copyrght c 04 SERSC

3D Face Reconstruction With Local Feature Refinement

3D Face Reconstruction With Local Feature Refinement ternatonal Journal of Multmeda and Ubqutous Engneerng Vol.9, No.8 (014), pp.59-7 http://dx.do.org/10.1457/jmue.014.9.8.06 3D Face Reconstructon Wth Local Feature Refnement Rudy Adpranata 1, Kartka Gunad