3D Face Modeling Lacey Best- Rowden, Joseph Roth Feb. 18, MSU
Outline ApplicaDon / Benefits 3D ReconstrucDon Techniques o Range Scanners o Single Image! 3DMM o MulDple Image! Shape from stereo! Photometric stereo! Shape from modon Conclusion
ApplicaDons/Benefits to FR Invariant to o IlluminaDon o Pose o Background Allows o Warping to frontal o ParDal matching
Challenges Non- rigid object Dense landmark detecdon Require user cooperadon for range scanners o Minolta Vivid 900: 2.5sec
RepresentaDon Depth map Point cloud Mesh
ReconstrucDon Direct Acquisi,on Single Image MulDple Images
Range Scanners Bounce waves off subject and measure reflecdon Requires user to be stadonary Post processing to fill holes / remove outliers
Structured Light Shine pa]ern on object Measure warping of pa]ern
ReconstrucDon Direct AcquisiDon Single Image MulDple Images
3D Morphable Model A generadve model Linear 3D shape and appearance model + imaging model Maps 3D surface onto an image Classic works: V. Blanz and T. Ve]er. Face RecogniDon Based on Fi`ng a 3D Morphable Model. PAMI, 2003. V. Blanz and T. Ve]er. A Morphable Model for the Synthesis of 3D Faces. SIGGRAPH, 1999.
3D Morphable Model Vector representadons of geometry and texture of 3D face ( ) T R 3n ( ) T R 3n S = X 1,Y 1,Z 1,X 2,,Y n,z n T = R 1,G 1,B 1,R 2,,G n,b n New shapes and new textures from m exemplar/prototypical faces in full correspondence S mod = m a i S i T mod = b i T i i=1 The morphable model is a set of faces parameterized by coefficients S mod ( a! ),T mod ( b! ) ( ) m i=1 m m a i = b i =1 i=1 i=1!! a = ( a 1,a 2,,a m ) T b = ( b 1,b 2,,b m ) T Arbitrary new faces can be generated by varying the shape!! and texture coefficients, a and b
3D Morphable Model: PCA Linear combinadons can contain non- faces Coefficient vectors need assigned probabilides of describing a face PCA on shape and texture vectors of the set of m exemplar 3D faces Instead of describing a novel shape and texture as a linear combinadon of example faces, linear combinadons of N S shape and N T texture principal components N S S = S + α i S i T = T + β i T i i=1 N T i=1
3D Morphable Model: PCA
3D Morphable Model: Segments 200 example faces Dimensionality of shape and texture spaces are limited Need more 3D scans Larger variety of faces if linear combinadons of shape and texture are formed separately for different regions MulDplies dimensionality of MM by four Image blending technique used to smooth the transidons between segments
Building a Morphable Model Database of 3D laser head scans 100 males, 100 females Mostly Caucasian Key problem: need a dense point- to- point correspondence between verdces of 3D faces Mesh- based algorithms Non- rigid IteraDve Closest Point (ICP)
Correspondence via Non- Rigid ICP Progressively deforms a template/reference face towards the measured surface Starts with a strong regularizadon First recovers global deformadons RegularizaDon is then lowered Allows progressively more local deformadons
Model- Based Image Analysis Represent a novel face in an image by model coefficients and provide a reconstrucdon of 3D shape AutomaDcally esdmates all rendering/scene parameters Analysis: model inversion problem? StochasDc Newton OpDmizaDon updates/iteradons based on 1 st and 2 nd derivadves of a MAP energy funcdon Synthesis: generadon of accurate face images Viewed under any possible rendering condidons (pose, illuminadon, expression, etc.)
Model- Based Image Analysis: FiEng
Model- Based Image Analysis: FiEng Minimize the sum of squared errors over all color channels and pixels between the image and the reconstrucdon E I = I input (x, y) I model (x, y) x,y q x, j E F = I input I q model j y, j p x,k j p y,k j MinimizaDon may cause overfi`ng 2 2 First iteradons exploit manually defined feature points ( 8) Maximum a posteriori (MAP) esdmadon of the parameters
Model- Based Image Analysis: MAP Find model parameters with maximum posterior probability. Bayes Rule, p α,β,ρ I input,f Assume independence, p α,β,ρ I input,f ( )P α,β,ρ ( ) ~ p I input,f α,β,ρ ( ) p F α,β,ρ ( ) ~ p I input α,β,ρ Priors of shape and texture esdmated with PCA Assume normal distribudon of rendering parameters ( ) ~ exp 1 p I input α,β,ρ 2 2σ E I I Posterior maximized by minimizing cost funcdon StochasDc Newton OpDmizaDon (SNO) algorithm ( ) ( )P( α)p( β)p ρ ( ) ~ exp 1 p F α,β,ρ E = 2 log p( α,β,ρ I input,f) ( ) 2 2σ E F F
Fi`ng Results (SNO)
Many FiEng Algorithms AcDve Shape Model (ASM) AcDve Appearance Model (AAM) Inverse ComposiDonal Image Alignment (ICIA) ICIA applied to 3DMM MulD- Features Fi`ng (MFF) 2D+3D AAM Linear Shape and Texture fi`ng
Face Recogni,on Paradigms RecogniDon based on nearest- neighbor of model coefficients Intrinsic shape and texture of faces Independent of imaging condidons SyntheDc views of gallery or probe images Pose correcdon to frontal Generate views at various poses
Recogni,on Results FERET database CMU- PIE database
Recogni,on Results
Recogni,on Results
NIST FRVT 2002: Morphable Models Results Gallery: 87 subjects (frontal, controlled) Probe: 87 images of 87 subjects (9 sets)
FRVT 2002: Morphable Models Results
ReconstrucDon Direct AcquisiDon Single Image Mul,ple Images
Shape from Stereo 2 cameras with known posidon Measure disparity between corresponding points
Photometric Stereo Single camera MulDple images with different light sources Why does this work?
Photometric Stereo Unknown lighdng Decompose into light and shape images pixels M nxp M = LS - L nx4, S 4xp - S i = [p, p*n x, p*n y, p*n z ] T
ReconstrucDng Faces 1. EsDmate pose and warp to near frontal 2. Solve inidal lighdng and shape 3. Refine shape via local patches 4. Integrate shape 5. Repeat with updated template
Pose EsDmaDon q = srq+t q: 2D landmarks in photo Q: 3D landmarks in template s: scale R: rotadon t: transladon
Expression NormalizaDon Low rank approximadon to get shape also removes expression from images Preserves lighdng
Results Can you guess who these people are?
Shape from MoDon MoDon of object MoDon of camera EsDmate geometry and modon simultaneously
The End Any QuesDons?
References [1] I. Kemelmacher- Shlizerman and S. Seitz. Face ReconstrucDon in the Wild. ICCV 2011. [2] V. Blanz and T. Ve]er. Face RecogniDon Based on Fi`ng a 3D Morphable Model. PAMI, 2003. [3] V. Blanz and T. Ve]er. A Morphable Model for the Synthesis of 3D Faces. SIGGRAPH, 1999.