Model-Based Bundle Adjustment to Face Modeling

Size: px

Start display at page:

Download "Model-Based Bundle Adjustment to Face Modeling"

Arline Carr
6 years ago
Views:

1 Model-Based Bundle Adjustment to Face Modelng Oscar K. Au Ivor W. sang Shrley Y. Wong he Hong Kong Unversty of Scence and echnology Realstc facal synthess s one of the most fundamental problems n computer graphcs and computer vson. It s dffcult for generatng the transton 3D model for facal expresson, especally lacks the nformaton of camera pose. Startng from several un-calbrated vews of a human face, we attempt to recover the camera poses and the facal geometry. hen we deform a generc face mesh to ft the partcular geometry of the subject's face by offlne calbraton and modelbased Bundle adjustment and scattered data modelng respectvely. Face modelng s wdely used n varous felds, such as computer game, flmmakng, vdeo teleconferencng, etc. Now, actually, there are a number of ways for dong such model development. Indeed, there have been qute a number of research papers talkng about the technques to model realstc human faces. However, there s no perfect realstc facal anmaton to be generated by computer. Nowadays the expensve and complex technology s used, lke laserbased cylndrcal scanner. It s, however, not so practcal for people to generate a 3D model wthout those equpment. o overcome ths, we ntroduce another technque, whch only photographs of a human subject are used and the generated facal model s farly well. In ths case, hgh and expensve technology s no longer needed and we can play around the 3D face modelng very easly. he rest of ths proposal s organzed as follows. Challenges of face modelng and Background nformaton about Levenberg-Marquardt mnmzaton for parameters estmaton s covered n Secton. Secton 3 descrbes our methodology and mplementaton for fttng a generc face model from a collecton of photographs and feature ponts. he results of the facal models generated by our system are shown n the Secton 4. Lastly, Secton 5 s the concluson of ths project.

2 he man challenge of our project s to estmate the camera parameters θ ~ precsely wthout pror knowledge about nput sequence, sze and orentaton of mages. In the past, we needed to do a lot of preprocessng on the mages to fnd out values of these parameters or to use some expensve equpment to obtan mages wth some fxed camera postons. After, we recovered these parameters; we can use them such as 3D feature ponts to deform the face model and camera poses for textual mappng. hat means now the man problem s how to estmate these parameters. In order to estmate both ntrnsc parameters and external parameters of camera pose, we use offlne calbraton and Levenberg-Marquardt mnmzaton for parameters estmaton. Here, we have a bref ntroducton to Levenberg-Marquardt mnmzaton. Gven a vector relaton y = f (x) where x and y can have dfferent dmensons and an observaton ŷ, we want to fnd the vector x whch best satsfes the gven relaton. More precsely, we are lookng for the vector xˆ satsfyng yˆ = f ( xˆ) + eˆ for whch ê s mnmal. A typcal way to solve xˆ s Newton teraton. Newton's teraton approach starts from an ntal value x and refnes ths value usng the assumpton that f s locally lnear. A frst order approxmaton of f ( x + ) yelds: f ( x + ) = f ( x ) + J wth J the Jacoban matrx and a small dsplacement. Under these assumptons mnmzng e ˆ = eˆ can be solved through lnear least squares. A smple dervaton yelds J J = J eˆ. hs equaton s called the normal equaton. he soluton to ths problem s found by startng from an ntal soluton and then refne t based on successve teratons x+1 = x + wth the soluton of the above normal equaton evaluated at x. One hopes that ths algorthm wll converge to the desred soluton, but t could also end up n a local mnmum or not converge at all. hs depends a lot on the ntal value x. Another way to solve xˆ and avodng the local mnmum s Levenberg-Marquardt teraton. he levenberg-marquardt teraton s a varaton on the Newton teraton. he normal equatons N = J J = J eˆ are augmented to N ' = J eˆ where N j ' = (1 + δ jλ) N j wth δ j the Kronecker delta. he value λ s ntalzed to a 3 small value, e.g. 1. If the value obtaned for reduces the error, the ncrement s accepted and λ s dvded by 1 before the next teraton. On the other hand, f the error ncreases, then λ s multpled by 1 and the augmented

3 normal equatons are solved agan, untl an ncrement s obtaned that reduces the error. hs s bound to happen, snce for a large λ the method approaches a steepest descent.

4 ! Facal Modelng s to adapt a generc 3D face model to ft an ndvdual's face and facal expressons. We wll use several mages of the face from dfferent vewponts and a generc face model. Offlne calbraton s used to obtan ntrnsc parameters. A set of features ponts s chosen from each mage to recover the external parameters of the camera pose. hese ponts are also used to refne the generc face model to ft the face n the nput mages, along wth a precse estmate of the camera pose correspondng to each nput mage. he whole process conssts of two stages. In the pose recovery stage, we apply model-based bundle adjustment to estmate the vewng parameters (poston and orentaton) for each nput mage and smultaneously recover the 3D coordnates of a set of feature ponts on the face. hese feature ponts are selected from the face mesh vertces, and ther postons n each mage are specfed by hand. he scattered data nterpolaton stage uses the estmated 3D coordnates of the feature ponts to compute the postons of the remanng face mesh vertces. In our project, we appled some lbrares such as OpenGL for 3D renderng n both Unx and Wndow platform and Numercal Recpes n C for numercal optmzaton. he ntrnsc parameters of the camera wll be recovered before we solve the structure and moton problem. A few reference ponts wll be chosen by the users (both D and 3D coordnates) and we can wrte the relatonshp between the 3D and D ponts M = ( x, y, z,1) and m = ( u, v,1) : ~ m ~ = PM, where P s projecton matrx, m ~ and m and M. For each reference pont t q1 q t P = q q t q 3 q M ~ are the homogeneous coordnates of M, we obtan two lnear equatons n the unknowns q m and m4 q (m = 1,, 3): q M u q M 1 3 q M v q M 3 + q + q 14 4 u q v q For N ponts, we obtan a system of N homogeneous lnear equaton Aq = t t t where q s 1 x 1 vector [ q 1 q14 q q4 q3 q34 ]. hs system can be solved f N >= 6 by usng least-squares method. After P s found we can decompose P nto ntrnsc and extrnsc parameters: = =

5 α s u P = AK, A = v β, 1 K = [ R t ] he elements of matrx A are the ntrnsc parameters of the camera, and elements n K are the extrnsc parameters. We assume the mage frame s n same drecton to camera frame.(.e. s =, α, β > ). After normalzng P wth q = 1 3, we can compute the ntrnsc parameters: u v = q = q α = q 1 1 β = q q q 3 3 q 3 q 3 Fgure 1 Ponts are pcked for Calbraton " # In face modelng, we are nterested n recoverng the structure a 3D scene/object from multple mages. Once the 3D structure and moton have been obtaned for these multple mages, t s easy to deform the 3D generc face model to a specfc face model. It s recommended to recover the 3D structure through a global mnmzaton step. he maxmum lkelhood estmaton can be obtaned through bundle adjustment. he goal s to fnd the projecton matrces Pˆ k and the 3D ponts Mˆ for whch the mean squared dstances between the observed mage ponts m k and the re-projected mage ponts mˆ k s mnmzed. For m vews and n ponts the followng crteron should be mnmzed:

6 mn ˆ, ˆ n m Pk M = 1 k = 1 D( m k, Pˆ Mˆ ) where D ( m, mˆ ) s the Eucldean mage dstance. If the mage error s zero-mean Gaussan, then bundle adjustment s the Maxmum Lkelhood Estmator. Although t can be expressed very smply, ths mnmzaton problem s huge. For a typcal sequence of vews and ponts, a mnmzaton problem n more than 6 varables has to be solved. A straghtforward computaton s obvously not feasble. However, the specal structure of the problem can be exploted to solve the problem much more effcently. he detal on effcently solvng the bundle adjustment, the Levenberg-Marquardt mnmzaton was presented n secton.1. $%& he observed ponts m k beng fxed, a specfc resdual r (, ˆ ˆ k = D mk Pk M ) s only dependent on the th pont and the k th projecton matrx. hese results n a very sparse matrx for the Jacoban. hs s llustrated n the followng fgure for 3 vews and 4 ponts. k Fgure : Sparse structure of Jacoban for bundle adjustment. Because of the block structure of the Jacoban, solvng the normal equatons J J = b have a structure as seen n fgure

7 Fgure : Block structure of normal equatons. It s possble to wrte down explct formulas for each block. Let us frst ntroduce the followng notaton: wth mˆ Pˆ k coordnates of k and mˆ k 1 mˆ k mˆ k U k = ( ) σ Pˆ k Pˆ k 1 mˆ k mˆ k V = ( ) k σ Mˆ Mˆ 1 mˆ k mˆ k Wk = ( ) σ Pˆ k Mˆ 1 mˆ k ε ( Pk ) = ( ) ε k σ Pˆ k 1 mˆ k ε ( M ) = ( ) ε k k σ Mˆ mˆ k Mˆ matrces contanng the partal dervatves from the to the parameters of Pˆ k and Mˆ respectvely. Here, we assume

8 there s a zero mean Gaussan nose model wth σ varance for each mage pont mˆ k. In ths case the normal equatons can be rewrtten as U W ( P) ε ( P) = W V ( M ) ε ( M ) where the matrces U, W, V, ( P), ( M ), ε ( P) and ε( M ) are composed of the blocks defned prevously. Assumng V s nvertble both sdes of above equaton multpled on the left wth I WV I 1 to obtan U WV W 1 W 1 ( P) ε ( P) WV ε( M ) = V ( M ) ε( M ) hs can be separated n two groups of equatons. he frst one s 1 1 ( U WV W ) ( P) = ε( P) WV ε( M ) and can be used to solve for (P). he soluton can be substtuted n the other group of equatons: 1 ( M ) = V ( ε ( M ) W ( P)) Note that due to the sparse block structure of V ts nverse can be computed very effcently. he only computatonally expensve step consst of solvng equaton. hs s however much smaller than the orgnal problem. For vews and ponts the problem s reduced from solvng 6 unknowns concurrently to more or less unknowns. o smplfy the notatons the normal equatons were used n ths presentaton. It s however smple to extend ths to the augmented normal equatons. $ Instead of representng the 3D scene/object by a collecton of solated 3D features, we consder the stuatons where the scene/object s a surface defned by a set of M parameters C = { cm m = 1,..., M}, called model parameters. Let us denote the surface by S (C). Furthermore, we assume there are Q semantcally meanngful ponts { Q j j = 1,..., Q} on the scene/object. If there are no semantc ponts avalable, then Q =. In our case, Q = 5 because we use fve semantc ponts: two nner eye corners, two mouth corners and the nose tp. he

9 relatonshp between the j th semantc pont s descrbed by Q j = ϑ( C, j). Obvously, pont Q j and the scene/object parameters C Q j S(C). Fgure Fve semantc ponts: two nner eye corners, two mouth corners and the nose tp Fgure 3 Feature ponts for MBA We are gven a set of N mages/frames, and assume we have matched a number of ponts of nterest across mages. Image k s taken by camera wth unknown pose parameters Pˆ k whch descrbes the poston and orentaton of the camera wth respect to the coordnate system. We assume that the j th semantc pont Q j s observed and detected n zero or more mages. Let Ω j be the set of frame numbers n whch Q j are detected, and qlj ( l Ω j ) be the semantc pont n the l th frame. Our objectve functon s now, mn n m Q ˆ ˆ ( (, ) (, ˆ ˆ ˆ D mk Pk M + D qlj Pl Q j ) ) Pk, M, C = 1 k = 1 j= 1 l Ω j subject to Mˆ S( C), whch we fnd smultaneously the scene/object parameters and camera parameters

10 by mnmzng statstcally Gaussan nose for each ndependent pont from the mages. Note that although the part for the general feature ponts (the frst term) and the part for the semantc ponts (the second term) have the same form, we should treat them dfferently. Indeed, the latter provdes stronger constrant n the bundle adjustment than the former. Also, we can mpose some bound constrants to parameters c m C : l m c m u m, and we can defne the penalty term f cm lm pm = w( lm cm) otherwse. In our case, we defne a, b, c, d and e as our semantc parameters C n the above fgure. And use the followng nequalty constrants e 3a. ""'( Solvng for the rotaton matrx s a lttle trcker than for the other parameters, snce R must be a vald rotaton matrx. Instead of drectly estmaton of the elements of R, Rodrguez s formula of general rotaton matrx ~ R ( nˆ, θ ) = I + X ( nˆ)sn θ + X ( nˆ)(1 cosθ ) s used to replace the rotaton matrx R ~ wth R ( nˆ, θ ) R durng updatng the estmaton, where θ s an ncremental rotaton angle, nˆ s a rotaton axs. For smplcty, the formulaton of equaton of R ~ s n ~ form of ts frst order expanson R( v) = I + X ( v), where v = θnˆ = ( vx, v y, v z ), X ( v ) = vz v y v v x z v y v We can estmate v nstead of the entres of R, then substtute back nto to update rotaton matrx. )! x. ~ R ( nˆ, θ ) R After we computed the coordnates for the feature ponts p, we use these values as samplng ponts to deform the remanng vertces on the face mesh wth a latent functon of a 3D face model. herefore, we need to construct a smoothng functon that nterpolates the latent functon of 3D model. Snce ths latent functon s non-lnear, we use a non-lnear kernel functon to transform the feature ponts as a set of non-lnear bass. hen the smoothng functon s a lnear combnaton of ths bass. In general, ths smooth nterpolaton functon n that gves the 3D dsplacements between the orgnal pont postons and the new adapted postons for every vertex n the orgnal generc face mesh. hat s, gven () () a set of known dsplacement u = p p away from the orgnal postons p

11 at every constraned vertex, construct a functon that gves the dsplacement u j for every unconstraned vertex j. We choose radal bass functons as the kernel to construct the nterpolatng functon, that s, functons of the form f ( p) =cφ( p p ), where φ (r) are radcally symmetrc bass functons. Many dfferent functons for r / 64 φ (r) have been proposed, we have chosen to use φ ( r) = e to regulate the nterpolaton error. Also we add some low-order polynomal terms to model global, e.g., affne, deformatons to reducng the basng effect of samplng feature ponts from the latent functon. So that out nterpolatng functon has the form: f ( p) c φ( p p + Mp + t = ) Our algorthm determnes the coeffcents c and the affne components M and t by solvng a set of lnear equatons to do curve fttng for other ponts n face mesh. hese equatons nclude the nterpolaton constrants u = f ( p ), also the = c c p = constrants and, whch remove affne contrbutons from the radal bass functons. For computaton effcency, we further smplfy ths set of lnear equatons nto three smaller subsets of lnear equatons. * +'!! We generate a vew ndependent texture map by constructng the texture map on a vrtual cylnder enclosng the face model. But nstead of castng a ray from each pxel to the face mesh and computng the texture blendng weghts on a pxel-bypxel bass, we use a more effcent approach. For each vertex on the face mesh, we compute the blendng weght for each mage based on the angle between surface normal and the camera drecton. If the vertex s nvsble the weght s set to.. he weghts are then normalzed so that the sum of the weghts over all the mages s equal to 1.. We then set the colors of the vertces to be ther weghts, and use the rendered mage of the cylndrcal mapped mesh as the weght map. For each mage, we also generate a cylndrcal texture map by renderng the cylndrcal mapped mesh wth the current mage as texture map. Let C and W ( = 1,,, k ) be the cylndrcal texture maps and the weght maps. Let C be the fnal blended texture map. For each pxel (u, v), ts color on the fnal blendng texture map s C( u, v) = k = 1 W ( u, v) C ( u, v)

Each of them s photographed n fve dfferent postons by a dgtal camera.

12 Because the renderng operatons can be done usng graphcs hardware, the approach s very fast. * " We have used the photographs of some classmates to test our system. Each of them s photographed n fve dfferent postons by a dgtal camera. he photos were uploaded from the camera to the PC wthout any preprocessng; the feature ponts were marked n each of the photos by smple graphc nterface. Followng the steps of our mplementaton, then the facal model of dfferent human faces n the photos s generated. Here are the results: Fgure 4 Model 1

13 Fgure 5 Output of Model 1 Fgure 6 Model

14 Fgure 7 Output of Model

15 Fgure 8 Model 3 Fgure 9 Output of Model 3

16 , he powerful of our work s to model a human face from the photographs wth absence of vewng parameters nformaton, such as focal length, poston, orentaton, for each of the nput cameras. o estmate all these parameters, the pose recovery technque s used to obtan precse estmatons of the camera poston. In addton, the pose recovery also provdes the way for us to refne the 3D coordnates of the set of user nputted feature ponts n a generc 3D model. hen the scattered data nterpolaton, whch uses the estmated 3D coordnates of the feature ponts, s appled to defne the coordnates of other ponts of the face mesh vertces. he mplemented technque s just the elementary step n facal modelng. Also, we can add facal expressons to enhance the face model by usng dfferent style models. herefore, we can mplement the texture extracton technque and facal expressons to enhance our system to produce a more realstc facal model of a human face. "- Rchard Szelsk, Sng Bng Kang. Recoverng 3D Shape and Moton from Image Streams usng Non-Lnear Least Squares. Journal of Vsual Communcaton and Image Representaton, 5(1):1-8, March 1994 Frederc Pghn, Jame Hecker, Dan, Lschnsk, Rchard Szelsk Davd H.Salesn. Syntheszng Realstc Facal Expressons from Photographs. In Computer Graphcs, Annual Conference Seres, pages Sggraph, July 1998 W.H.Press, B.P. Flannery, S.A. eukolsky, and W..Vetterlng. Numercal Recpes n C: he Art of Scentfc Computaton. Cambrdge Unversty Press, Cambrdge, England, second edton, 199. Z.Zhang, R.Derche, O.Faugeras, and Q.-. Luong. A robust technque for matchng two uncalbrated mages through the recovery of the unknown eppolar geometry. Artfcal Intellgence Journal, 78:87-119, Oct Y.Shan, Z.Lu and Z.Zhang. Model-Based Bundle Adjustment wth Applcaton to Face Modelng. o appear ICCV 1 Z.Lu, Z.Zhang, C.Jacobs and M.Cohen. Rapd Modelng of Anmated Faces From Vdeo. In Proc. 3 rd Internatonal Conference on Vsual Computng, pages 58-67, Mexco Cty, Sept.

Computer Animation and Visualisation. Lecture 4. Rigging / Skinning

Computer Animation and Visualisation. Lecture 4. Rigging / Skinning Computer Anmaton and Vsualsaton Lecture 4. Rggng / Sknnng Taku Komura Overvew Sknnng / Rggng Background knowledge Lnear Blendng How to decde weghts? Example-based Method Anatomcal models Sknnng Assume