3D Face Modeling. Lacey Best- Rowden, Joseph Roth Feb. 18, MSU

Transcription

1 3D Face Modeling Lacey Best- Rowden, Joseph Roth Feb. 18, MSU

2 Outline ApplicaDon / Benefits 3D ReconstrucDon Techniques o Range Scanners o Single Image! 3DMM o MulDple Image! Shape from stereo! Photometric stereo! Shape from modon Conclusion

3 ApplicaDons/Benefits to FR Invariant to o IlluminaDon o Pose o Background Allows o Warping to frontal o ParDal matching

4 Challenges Non- rigid object Dense landmark detecdon Require user cooperadon for range scanners o Minolta Vivid 900: 2.5sec

5 RepresentaDon Depth map Point cloud Mesh

6 ReconstrucDon Direct Acquisi,on Single Image MulDple Images

7 Range Scanners Bounce waves off subject and measure reflecdon Requires user to be stadonary Post processing to fill holes / remove outliers

8 Structured Light Shine pa]ern on object Measure warping of pa]ern

9 ReconstrucDon Direct AcquisiDon Single Image MulDple Images

10 3D Morphable Model A generadve model Linear 3D shape and appearance model + imaging model Maps 3D surface onto an image Classic works: V. Blanz and T. Ve]er. Face RecogniDon Based on Fi`ng a 3D Morphable Model. PAMI, V. Blanz and T. Ve]er. A Morphable Model for the Synthesis of 3D Faces. SIGGRAPH, 1999.

11 3D Morphable Model Vector representadons of geometry and texture of 3D face ( ) T R 3n ( ) T R 3n S = X 1,Y 1,Z 1,X 2,,Y n,z n T = R 1,G 1,B 1,R 2,,G n,b n New shapes and new textures from m exemplar/prototypical faces in full correspondence S mod = m a i S i T mod = b i T i i=1 The morphable model is a set of faces parameterized by coefficients S mod ( a! ),T mod ( b! ) ( ) m i=1 m m a i = b i =1 i=1 i=1!! a = ( a 1,a 2,,a m ) T b = ( b 1,b 2,,b m ) T Arbitrary new faces can be generated by varying the shape!! and texture coefficients, a and b

12 3D Morphable Model: PCA Linear combinadons can contain non- faces Coefficient vectors need assigned probabilides of describing a face PCA on shape and texture vectors of the set of m exemplar 3D faces Instead of describing a novel shape and texture as a linear combinadon of example faces, linear combinadons of N S shape and N T texture principal components N S S = S + α i S i T = T + β i T i i=1 N T i=1

13 3D Morphable Model: PCA

14 3D Morphable Model: Segments 200 example faces Dimensionality of shape and texture spaces are limited Need more 3D scans Larger variety of faces if linear combinadons of shape and texture are formed separately for different regions MulDplies dimensionality of MM by four Image blending technique used to smooth the transidons between segments

15 Building a Morphable Model Database of 3D laser head scans 100 males, 100 females Mostly Caucasian Key problem: need a dense point- to- point correspondence between verdces of 3D faces Mesh- based algorithms Non- rigid IteraDve Closest Point (ICP)

16 Correspondence via Non- Rigid ICP Progressively deforms a template/reference face towards the measured surface Starts with a strong regularizadon First recovers global deformadons RegularizaDon is then lowered Allows progressively more local deformadons

17 Model- Based Image Analysis Represent a novel face in an image by model coefficients and provide a reconstrucdon of 3D shape AutomaDcally esdmates all rendering/scene parameters Analysis: model inversion problem? StochasDc Newton OpDmizaDon updates/iteradons based on 1 st and 2 nd derivadves of a MAP energy funcdon Synthesis: generadon of accurate face images Viewed under any possible rendering condidons (pose, illuminadon, expression, etc.)

18 Model- Based Image Analysis: FiEng

19 Model- Based Image Analysis: FiEng Minimize the sum of squared errors over all color channels and pixels between the image and the reconstrucdon E I = I input (x, y) I model (x, y) x,y q x, j E F = I input I q model j y, j p x,k j p y,k j MinimizaDon may cause overfi`ng 2 2 First iteradons exploit manually defined feature points ( 8) Maximum a posteriori (MAP) esdmadon of the parameters

20 Model- Based Image Analysis: MAP Find model parameters with maximum posterior probability. Bayes Rule, p α,β,ρ I input,f Assume independence, p α,β,ρ I input,f ( )P α,β,ρ ( ) ~ p I input,f α,β,ρ ( ) p F α,β,ρ ( ) ~ p I input α,β,ρ Priors of shape and texture esdmated with PCA Assume normal distribudon of rendering parameters ( ) ~ exp 1 p I input α,β,ρ 2 2σ E I I Posterior maximized by minimizing cost funcdon StochasDc Newton OpDmizaDon (SNO) algorithm ( ) ( )P( α)p( β)p ρ ( ) ~ exp 1 p F α,β,ρ E = 2 log p( α,β,ρ I input,f) ( ) 2 2σ E F F

21 Fi`ng Results (SNO)

22 Many FiEng Algorithms AcDve Shape Model (ASM) AcDve Appearance Model (AAM) Inverse ComposiDonal Image Alignment (ICIA) ICIA applied to 3DMM MulD- Features Fi`ng (MFF) 2D+3D AAM Linear Shape and Texture fi`ng

23 Face Recogni,on Paradigms RecogniDon based on nearest- neighbor of model coefficients Intrinsic shape and texture of faces Independent of imaging condidons SyntheDc views of gallery or probe images Pose correcdon to frontal Generate views at various poses

24 Recogni,on Results FERET database CMU- PIE database

25 Recogni,on Results

26 Recogni,on Results

27 NIST FRVT 2002: Morphable Models Results Gallery: 87 subjects (frontal, controlled) Probe: 87 images of 87 subjects (9 sets)

28 FRVT 2002: Morphable Models Results

29 ReconstrucDon Direct AcquisiDon Single Image Mul,ple Images

30 Shape from Stereo 2 cameras with known posidon Measure disparity between corresponding points

31 Photometric Stereo Single camera MulDple images with different light sources Why does this work?

32 Photometric Stereo Unknown lighdng Decompose into light and shape images pixels M nxp M = LS - L nx4, S 4xp - S i = [p, p*n x, p*n y, p*n z ] T

33 ReconstrucDng Faces 1. EsDmate pose and warp to near frontal 2. Solve inidal lighdng and shape 3. Refine shape via local patches 4. Integrate shape 5. Repeat with updated template

34 Pose EsDmaDon q = srq+t q: 2D landmarks in photo Q: 3D landmarks in template s: scale R: rotadon t: transladon

35 Expression NormalizaDon Low rank approximadon to get shape also removes expression from images Preserves lighdng

36 Results Can you guess who these people are?

37 Shape from MoDon MoDon of object MoDon of camera EsDmate geometry and modon simultaneously

38 The End Any QuesDons?

39 References [1] I. Kemelmacher- Shlizerman and S. Seitz. Face ReconstrucDon in the Wild. ICCV [2] V. Blanz and T. Ve]er. Face RecogniDon Based on Fi`ng a 3D Morphable Model. PAMI, [3] V. Blanz and T. Ve]er. A Morphable Model for the Synthesis of 3D Faces. SIGGRAPH, 1999.