3D model-based human face modeling

Transcription

1 3D model-based human face modeling André Gagalowicz Projet MIRAGES INRIA - Rocquencourt - Domaine de Voluceau Le Chesnay Cedex Andre.Gagalowicz@inria.fr

2 II-I - INTRODUCTION II-II FIRST STEP : DEFORMATION OF THE GENERIC MODEL WITH THE AID OF FACE FEATURE POINTS II-III III SECOND STEP: USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION II-IV IV RESULTS

3 Context II-I - INTRODUCTION Application: Post-production Quality (model must be precise) No Real-time Goal: Face Tracking in a video sequence Rotoscopy Face pose estimation for virtual reality Lips and emotion tracking Must be quick, regular and reusable

4 Previous approaches How to obtain a good 3D model of face? 3D DEVICES (control on the 3D model production REQUIRED) 1. Laser capture devices 2. Structured light

5 PrevioUs Approaches How to obtain a good 3D model of face? From IMAGES 1. Graphics designers 2. Shape from shading

6 Previous Approaches How to obtain a good 3D model of face? From IMAGES 1. Graphics designers 2. Shape from shading 3. Generic face modification

7 Modeling is performed in two steps: FIRST STEP: DEFORMATION OF THE GENERIC MODEL WITH THE AID OF FACE FEATURE POINTS SECOND STEP: USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION

8 GLOBAL SCHEME Data Cooperative Calibration / Reconstruction Radial Basis Function Second step Finalize the deformation Results Calibration of the cameras using feature points Deformation of the whole 3D mesh using RBF Model Refinment using limbs Generic face 3D reconstruction of face feature points Specific face

9 II-II FIRST STEP : DEFORMATION OF THE GENERIC MODEL WITH THE AID OF CHARACTERISTIC POINTS II-II -1- Choice of characteristic points II-II -2- Interactive calibration of the cameras II-II -3- Assigning interactively the characteristic points of the generic model to the face images II-II 4 Realisation of the model deformation

10 II-II 1 Choice of characteristic points a priori Knowledge Generic model 34 Feature points

11 II FIRST STEP : DEFORMATION OF THE GENERIC MODEL WITH THE AID OF CHARACTERISTIC POINTS II-II 1 Choice of characteristic points II-II 2 - Interactive Calibration of the cameras II-II - 3- Assigning interactively the characteristic points of the generic model to the face images II-II - 4- Realisation of the model deformation

12 II-II 2 - Interactive Calibration of the cameras

13 II-II - 1 Choice of characteristic points II-II Interactive Calibration of the cameras II-II - 3- Assigning interactively the characteristic points of the generic model to the face images II-II - 4- Realisation of the model deformation

14 II-II -3- Assigning interactively the characteristic points of the generic model to the face images

15 II-II - 1 Choice of characteristic points II-II Interactive Calibration of the cameras II-II - 3- Assigning interactively the characteristic II-II points of the generic model to the face images II - 4- Realisation of the model deformation a) Cooperative Calibration/3D Reconstruction of feature points b) Extrapolation of deformations to the complete face mesh

16 a) Cooperative Calibration/3D Reconstruction of feature points Calibration of the cameras using feature points 3D reconstruction of feature points

17 a) Cooperative Calibration / 3D Reconstruction of feature points Calibration of each camera method:posit needs: the intrinsic Iterative: é R ê ë 0 Calibration intrinsic parameters (focal at least 5 3D points the associated projections T 1 ú û ù (focal length,, pixel aspect ratio ) Minimizes the distance between the estimated 3 3 projection of each 3D point and each associated real 3 projection in image feature points

18 a) Cooperative Calibration / 3D Reconstruction of feature points 3D Reconstruction 3D Reconstruction by stereo Calibration is not perfect Camera 1 Camera 2 Reconstructed point: Middle point on the segment giving the minimum distance between the two lines of sight

19 Calibration / Reconstruction Average error for all cameras during calibration / reconstruction Error value 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0, Iteration number

20 a) Cooperative Calibration/3D Reconstruction of feature points 3D Reconstruction Case where a point is seen only on one image: Use Adaptive symmetry Here point 4 is seen only on the left image

21 a) Cooperative Calibration/3D Reconstruction of feature points Adaptive symmetry (cont.) Proj 1 (P 1 ) Image 1 Camera 2 P 1 V 1 R 1 Proj 1 (P 2 ) P 2 Camera 1 Proj 2 (P 1 ) Image 2

22 a) Cooperative Calibration/3D Reconstruction of feature points Adaptive symmetry (cont.) Image 1 Camera 2 P 1 R 1 Proj 1 (P 2 ) P 2 R 2 R 2 Camera 1 Image 2

23 b) Extrapolation of deformations to the complete face mesh Given a set of deformation vectors Basic idea: 1 Compute the deformation at feature points (done( in a ))) 2 Interpolate the deformations on the whole mesh USE Radial Basis Functions (RBF)

24 b) Extrapolation of deformations to the complete face mesh For n deformation vectors D 1..n of points P 1..n choose a deformation function f(p) as : 1..n, å f( P) ( Ai. s( P-Di)) = n i= 1 Where s is the Radial Basis Function Typically: We have: s( r ) = r, for smooth result (-r / ) ( ) 2 C s r = e a a set of n unknows A 1..n n constraints:, for local effect depending on C f( P ) = For i = 1.. n i D i

25 Calibration / Reconstruction - Problems Anti-parallel positions of the cameras C 2 R I 2 I 2 C 1 I 1 C 2 C 1 I 1 R

26 Calibration / Reconstruction - Problems Solution: Angle between C 1, C 2 Symmetry plane 3D reconstruction without using symmetry plane 3D reconstruction with using symmetry plane

27 II-III III SECOND STEP USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION Model Resulting from the deformation based on feature points (in GREEN) is poor! Idea: deform the current model using face silhouette (limbs( limbs) ) in RED

28 II-III III SECOND STEP USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION For each camera Compute the model limb and project it in image plane

29 Automatic 3D model silhouette extraction Initial vertex Projections of visible facets on the image plane Silhouette construction Special processing of intersections Enhancements: Processing of only those edges which lie between the visible and invisible facets Computation of the intersections only for the currently chosen edge Result: 60 sec 1.2 sec of computation time per view

30 III SECOND STEP USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION Compute the limb and project it in image plane For each camera Construct by hand the real contour in image plane Captions: Control Point of the BézierB curve Next tangent, evolving along the curve Preceding tangent, evolving along the curve

31 II-III III SECOND STEP USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION Compute the limb and project it in image plane For each camera Construct the real contour in image plane Compute the 2D deformations transforming the projected model limb to the real contour

32 2 1 How to obtain the 2D Deformations? Use local maxima (improves( computation) and propagate the deformations from the maxima along the curve using 1D RBF R 8 P 8 R 9 P D Deformation vectors

33 III SECOND STEP USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION Compute the limb and project it in image plane For each camera Construct the real contour in image plane Compute the 2D deformations transforming the projected limb into the real contour Infer the 3D deformations from the 2D ones

34 2D à 3D Deformations

35 II-III III SECOND STEP USE OF LIMBS TO FINALIZE THE 3D FACE RECONSTRUCTION Calculate the limb and project it in image plane For each camera Define the real contour in image plane Calculate the 2D deformations transforming the projected limb to the real contour Infer the 3D deformations from the 2D ones Add this set of 3D deformations incrementally to the previous set of feature points

36 II-IV IV RESULTS

37 II-IV IV RESULTS

38 II-IV IV RESULTS Lambertian synthesis Original

39 VALIDATION The precision of the 3D reconstruction of faces from 4 images when the geometry of faces was known beforehand was 2% (similar to the precision of a conventional laser system) TIME The time required to construct a 3D model from scratch lies between 20 min to ½ hour!