Hallucinated 3D Face Model from A Single 2D Low-Resolution

Transcription

1 Scences Hallucnated 3D Face Model from A Sngle 2D Lo-esoluton Face Usng Machne Learnng H.M.M Naleer 1 and Yao LU 2 1 Facult of Appled Scences, South Eastern Unverst of Sr Lanka, Oluvl, Sr Lanka. 2 Laborator of ntellgent nformaton echnolog, Bejng nsttute of echnolog, Bejng , hna. -mal: hmmnaleer@gmal.com,vs_l@bt.edu.cn Abstract n ths paper, the 3D face hallucnaton sstem s proposed on both 2D tranng face mages as ell as respectve 3D tranng face models th gre-level. he proposed method hallucnates the 3D hgh-resoluton model patch usng same poston of each mage patches of nterpolated 2D tranng mage and 3D hgh-resoluton tranng face model for lo-resoluton nput mage. Frstl, the optmal eghts of the 2D nput lo-resoluton mage poston-patches are estmated th the correspondng 2D lo-resoluton tranng mage patches. he canoncal correlaton analss (A) s used to learn the mappng beteen the 2D nterpolated face tranng mage and the 3D face model th respect to ther eghts. Secondl, the correspondng 3D face model patch th eght b matchng hgh score among the 2D nterpolated tranng mage patches and 3D tranng face model s selected. Fnall, the 3D hgh-resoluton facal model s formed b ntegratng the hallucnated 3D patches hch are obtaned through mappng patches th respectve eghts. n order to evaluate the performances of the above approaches, e used eample based learnng methods to obtan the hgh-resoluton output for a lo-resoluton nput. n ths approach, e used the avalable frontal data sets such as FEE, AS-EL and MU to analss the performance and some parameters are also consdered, hch ma affect the results from the above proposed method. Keords: face hallucnaton, 3D face model, 2D lo-resoluton, A ntroducton he modelng face s used n several computer vson applcatons, hch are presented n (Blanz, et al., 2003, Edards et al., 1998, Blanz & Vetter, 2003, amanathan, et. al., 2006, Baker, et al., 2004). he 3D models become useful unversall b face enhancement of 3D models. n several aspects, the 3D face model super-resoluton s emploed. he capture of 3D data s lmted accordng to the dstance of the stuaton and objects. he avalablt of 3D scanners and capturng nstruments are acqured tpcall lo-resoluton 3D data. herefore, the hallucnaton of 3D face models s ver helpful hle the 3D data models are lo-resoluton. Hoever, the tme-consumng 3D scanner (Esert & Grod 1998) s not accepted for such a face hallucnaton sstem. he method of usng to orthogonal photos (Goto, et al., 2002) cannot provde real-tme processng ether. he sngle mage s used to obtan the super-resoluton of face (Feng & Yuen 2000), here the head rotaton parameters requre measurng va other mages. n addton, t gves the naccurate results for 3D face models. On the other hand, the multple mages are used to obtan 3D super-resoluton, s presented (Morrs, et al., 1999). hs process s nvolved th 2D mage and 3D model. he shape from stereo and shadng are studed n (Morrs, et al., 1999, Zhang, et al.,1995, Hom & Brooks, 1989) for 3D super-resoluton. hough, fe methods are presented for 3D super-resoluton th an nput 2D mage. resentl, 3D data has become more advance hle rapd process n 3D technolog. hs knd of 3D super-resoluton s great needs n practce, snce occasonall the 3D acqurement sstem does not provde good enough data due to 3D face recognton and length dstance (Gang & Zhaohu 2005). roceedngs, 04 th nternatonal Smposum, SEUSL age 501

2 Scences he 3D poston values of head shape are obtaned b scanner technolog hch s the man approach to create 3D face model currentl. t needs addtonal cost due to need of specfc nstrument. Meanhle, the 3D model creaton can be obtaned from 2D mages. onversel, n ths paper, a ne method based on learnng-method s focused to provde the 3D hallucnated face model for 2D lo-resoluton nput mage. ecentl, ver fe orks studed on 3D face hallucnaton. Hoever, the superresoluton of 3D face s presented (an et al., 2006), hch s pursued b the ork (Baker & Kanade 2000). Frstl, 3D shape s acqured b progressve resoluton chan (). Secondl, MA approach s used to obtan the 3D face super-resoluton (Yang et al., 2007). Furthermore, human ma be used 2D mage and t can be vsualze nto 3D mage for bettervsualzaton that s brng ne change for ee th bran role. hus 3D mage s ma not be precse. f the 2D mage s lo-resoluton, the human bran has ver mportant role n acqurng the 3D mage and t ma not be precse. o overcome ths problem, e proposed the hallucnated 3D face model from a 2D lo-resoluton nput mage. We can dvde our hole sstem nto to phases. We consder as matchng phase beteen 2D nterpolated tranng mage and 3D tranng face model and hallucnaton phase. Output of proposed method s shon n Fg he rest of ork s organzed as follos: Secton 3 s the fundamental dea of A. Secton 4 descrbes our proposed hallucnated 3D face model va mage patches and secton 5 presents epermental results on constructon of 3D hallucnated face model from a 2D lo-resoluton mage. Fnall, the concluson s gven n secton Basc dea of A he domnant multvarate analss s, anoncal orrelaton Analss (A) (Hardoon, et al., 2004), hch has been used n several applcatons such as post estmaton and face matchng (Melzer, et al., 2003 & Naleer, et al., 2013). o mamze the correlaton beteen to sets, the A s constructng the subspace. Gven N pars of sample, ) of ( X, Y), 1,..., N, here X ( j zero. he goal of A s to learn a par of drecton and and, here denotes the transpose, hch s to mamze: n andy n. he mean of both X and Y s to mamze the correlaton beteen the to projectons E E XX XY E YY, (3.1) here E f, - the epermental antcpaton of the functon f,. he covarance matr of X, Y s X X XX YX X, Y E E, Y Y XY YY here XX and YY are thn sets covarance matrces. Hence, can be rtten as (3.2) roceedngs, 04 th nternatonal Smposum, SEUSL age 502

3 Scences 0 XY Let A, YX 0 B XX 0 0 YY, e can sho that the soluton W, amounts to the etreme ponts of the alegh quotent: r W W AW BW (3.3) he soluton and can be obtaned as solutons of the generalzed egen problem: AW BW (3.4) n case of the sample sze s comparatvel small; the A has a tendenc to over-ft to the tranng mage [17]. f 0 and 0 are regularzaton parameters, addton of and to and respectvel, gves encouragng avoded outputs. Let, the object functon of regularzed A s to mamze, here. 2. Hallucnated 3D Face Model va mage atches 2.1 A egresson for 2D-3D Matchng hase he follong secton used A to fnd out the results of mappng beteen 3D and 2D face models. n the learnng phase, the 2D (nterpolated 2D tranng dataset ( 24 32) )-3D (3D tranng face model ( ) ) mappng s learnt from the tranng set hch conssts of K pars of 2D-3D face mage face model respectvel. n the mappng phase, the most correlatve 3D face shape to the 2D (2D lo-resoluton nput mage ( 24 32) ) s found b 2D-3D mappng. n the learnng stage, N pars of 2D-3D face are gven as ( X, Y) ( k, k ),( k 1,2... N), here ( k, k ) s a correspondng par of 2D and 3D. Furthermore, to reduce the computatonal comple, A s appled frstl to transform spaces (knon as dmenson reducton). A transform matrces and k and k nto the loer dmensonal are learnt from 2D and 3D tranng sets respectvel. he A projectons are computed as X ( X X ) and Y ( Y Y), here X andy are the mean faces of 2D face mage and 3D face model respectvel. On the other hand, to further projecton drectons learnt for lnear A b performng A on X and Y respectvel (knon as A regresson). Also, let are best correlated. n the mappng phase, also need to project ne par of mages X ts ( Xts X ) and Yts ( Yts X ) respectvel. For lnear A, that s Fnall, the matchng score (So) can be calculated as 2.2 3D Face Model Hallucnaton va Learnng Method out ts Xts and and X and X ts and Yts nto A frstl, that s are Y Yts are projected nto A sup-space, X X, Y Y (4.1) out ts X out Yout So ( X out, Yout) (4.2) ( X Y ) out out roceedngs, 04 th nternatonal Smposum, SEUSL age 503

4 Scences n ths secton, the to phases as mappng phase and hallucnaton phase are measured. As mentoned n secton 4.1, the 3D face model patch correspondng to 2D nput lo-resoluton mage patch can be found. he descrpton of eghts calculaton on 2D lo-resoluton tranng mage patches th nput 2D lo-resoluton mage s descrbed n ths secton frstl. Secondl, hallucnaton phase s consdered alculaton of Mappng Scores We adopted the A to get the eact model of 2D-3D face mappng b patch based A, the follong steps are consdered n mappng stage. A large number of patches are created from 2D face mage and 3D face models. A s used to learn the performance beteen 2D mage and 3D face model for ever patch. n the mappng phase, the testng mages are also departed nto several parts n the same a. hen e can get a score for each patch. he fnal score s obtaned b combnng the ndvdual scores of all partcpatng patches. Fnall, selected the 3D patches th eghts on 3D tranng face model set usng mappng score to the correspondng 2D patches on nterpolated 2D tranng set. he eghts are calculated as follos and the stle of the patch s shon n Fg. 4.1n the patch based eghts n the partcular locaton (, j) s calculated b the follong Eq. (4.3), hch s descrbed n (Naleer, et al., 2013) Algorthm-, here S - 2D q n column vector of lo-resoluton tranng mages L q partcular locaton, the eght s calculated b Eq. (4.3) 1.f j, s ndcated as coordnate of the mage patch n the. q n L q 1 (, j) [( L (, {[( L (, 1 1 ( L (, 1] ( (, ] L 1 1 [( L (, [( L (, 1 1 ( L(, j) ( (, L 1 1 ] ]} 1 (4.3) he selecton of patch sze s an mportant ssue. t s too large or t s too small, n both cases the local mappng of hole face nformaton ll be dffcult. Hence the scores of each patch are combned to such a decsve level so that t s smplfed. Varous combnng schemes nclude Medan ule and, Sum ule, Majort Votng, Ma ule, roduct ule and Mn ule [6]. th Eq. (4.4) gves the results of fnal matchng score here, So and are the output score and correspondng eght for patch respectvel. F 1 So (4.4) roceedngs, 04 th nternatonal Smposum, SEUSL age 504

5 Scences D Hallucnaton hase Once e found the 2D face tranng mage patch poston th respectve eghts to correspondng 2D lo-resoluton mage patch n the same poston, accordng to the fnal matchng score usng A regresson mappng beteen 3D tranng face model and 2D nterpolated tranng mages usng eghts, the follong process are used to construct the fnal output as hallucnated 3D face model.accordng to the orgnal poston, the fnal 3D hallucnated face model s obtaned b combnng the 3D face model patches. Average of the pels values n the overlappng areas among to adjacent hallucnated patches s used to acqure the fnal result for pel of the overlappng areas. he proposed 3D face hallucnaton model has been summarzed belo and the entre frameork s gven n Fg Step1:Fnd the eghts beteen 2D lo-resoluton nput and 2D lo-resoluton tranng mages n the same postons usng Eq. (4.3). Step2:Fnd the fuson score beteen 2D tranng mage and 3D face mage model usng Eq. (4.4). Step3:Once selected the 3D face mode patch th respectve eghts accordng to the fuson score, the hallucnated 3D face model H can be obtaned b follong Eq. (4.5). H n (, j) ( (, (, j) (4.5) q 1 q H roceedngs, 04 th nternatonal Smposum, SEUSL age 505

6 Scences L q n L q 1 A egresson 3 D ran n g F ace M o d el D ataset A egresson A egresson q n H q A egresson A egresson 3 D S u b sp aces A egresson F u so n S core o f E ach atch th W eg h t H F n al D ecso n th 3 D atch and W eg h t o n stru cted 3 D F ace M o d el roceedngs, 04 th nternatonal Smposum, SEUSL age 506

7 Scences Eperments and esults Our eperments are conducted th the AS-EAL and FEE face datasets consstng of 200 ndvduals. he tranng face data sets are formed as 2D face mages and 3D face models pars for respectve each ndvdual. he laser scanner s used to obtan 3D tranng face model, hch s offer range mages and the faces are essentall 2.7D data. n the 3D face model tranng set, have a lttle facal epresson and head pose varng for correspondng 2D tranng data set of each ndvdual. All 3D tranng face models ( ) and 2D lo-resoluton face mages ( ), and all tranng mages are manuall algned based on eeballs, mouths postons th centers of left and rght eeballs and center of the mouth, and the nput loresoluton mage s also manuall algned. Our proposed approach s evaluated on 80 test lo-resoluton nputs. Some output of the test mages on ASE-EL are presented n Fg (96 128) he comparatve results for nput lo-resoluton face mage for each patch ( 3 3) are gven n Fg. 5.2 n terms of cumulatve match curves. Meanhle the performance th tranng samples s also consdered the results b cubc nterpolaton method and t s shon n Fg 5.3. o ehbt the performance of our proposed method, the amount of tranng mages th MS values has been gven n Fg n fact, the cubc nterpolaton s not dependng on the number of tranng mages. Meanhle, the performance of the proposed method s ver poor hle less than about 73 tranng mages. t means, our methods s enclosed most of the varance of the 3D face model. roceedngs, 04 th nternatonal Smposum, SEUSL age 507

8 roceedngs, 04 th nternatonal Smposum, SEUSL age 508 Scences

9 Scences ubc nterpolaton roposed Method Number of ranng mage Samples ars oncluson hs ork has presented a learnng-based 3D face hallucnaton method for sngle 2D lo-resoluton face mage th 2D loresoluton face mage and 3D face model tranng mage pars based on matchng scores. he eperments sho that, t could be applcable to generate a 3D hallucnated face mage verson of the 2D lo-resoluton nput face mage n absence of 3D hgh-resoluton face model n the tranng sets. eferences vol. 22(3), pp omputer Vson, vol. (EV) [], pp and Machne ntellgence, vol. 25(9), pp amanathan, S, Kassm, A, Venkatesh nter. onf. on mage rocessng () [], pp n omputer Graphcs Forum, European onference on EEE rans. on attern Analss EEE rans. on attern Analss and Machne ntellgence, vol. 26 (10), pp EEE omputer Graphcs and Applcaton, vol. 18(5), pp Goto,, Le Sgnal rocess and mage ommuncaton, vol. 17(3), pp &-Shoulder Face mage Usng Vrtual Frontal- EEE rans. Sstems Man and bernetcs art A, vol. 30(6), pp EEE Sgnal rocessng Workshop [], pp Artfcal ntellgent, vol. 78(1), pp roceedngs, 04 th nternatonal Smposum, SEUSL age 509

10 Scences 21. (EV) [], vol. 3952, pp [], pp M ress, pp nter. Journal of mage and Graphcs, vol. 5(3) pp n roc. European onf. on omputer Vson n roc. nter. onf. on Automatc Face and Gesture ecognton. (FG) -Depth Super omputer Vson attern ecognton (V) [], pp Hardoon, D, Szedmak S, & Shae- Neural omput, vol. 16(12), pp n roc. EEE onf. ecognton, vol. 36(9), pp attern -Step Method for Face Hallucnaton Based on Sparse ompensaton E mage processng, vol. 7(2), pp , roceedngs, 04 th nternatonal Smposum, SEUSL age 510