Facial Feature Detection

Transcription

1 Facial Feature Detection Rainer Stiefelhagen Interactive Systems Laboratories, Universität Karlsruhe (TH)

2 Overview Resear rch Group, Universität Karlsruhe (TH H) Introduction Review of already presented approaches Statistical ti ti appearance models Active Shape Models (Cootes et al., 1995) Active Appearance Models (Cootes et al., 1998) Deformable Models (Yuille et al., 1989) Gabor Wavelets to match facial landmarks Elastic Bunch Graph Matching (Wiskott et al., 1997) 2

3 What are facial features? Facial features are referred to as salient parts of a face region which carry meaningful information. eye, eyebrow, nose, mouth, Also called facial landmarks What is facial feature detection? Facial feature detection is defined as methods of locating the specific areas of a face. 3

4 Applications of facial feature detection Face recognition Use geometric features for identification Facial lfeature points also used for normalization i in appearance based recognition Needed for some local / modular identification approaches Model-based dhead pose estimation i Compute head pose from 2D to 3D correspondences Eye Gaze tracking Need to localize eyes first Facial expression recognition Movements of eyes, eyebrows and a mouth are important cues to analyze human s emotion. 4

5 Problems in facial feature detection Identity variations Each person has unique facial parts Expression variations Some facial features change their state (e.g. eye blinks). Head rotations If a head orientation changes, the visual appearance also changes. Scale variations Changes in resolution and distance to the camera affect appearance. Lighting gconditions Light has non-linear effects on the pixel values of a image. Occlusions Hair or glasses might hide facial features. 5

6 Already presented approaches (from face detection) ti Integral projections + geometric constraints Haar-Filter Cascades (Viola & Jones) can also be applied to facial feature detection PCA-based methods (Modular Eigenspace) Search for patches which fit to the individual Eigenspaces Morphable 3D Model delivers facial landmarks implicitly but very time consuming! 6

7 Statistical Appearance Models 7 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

8 Statistical appearance models Idea: make use of prior-knowledge, i.e. models, to reduce the complexity of the task Needs to be able to deal with variability df deformable models dl Use statistical models of shape and texture to find facial landmark points Good models should: 1. Capture the various characteristics iti of fthe object tto be detected dt td 2. Be a compact representation in order to avoid heavy calculation 3. Be robust against noise 8

9 Statistical appearance models: Basic idea 1. Training stage (Construction of models) 2. Test stage (Search the region of interest (ROI)) Input image Training images Generate statistical models of shape and texture Test stage: Fit model parameters to new target image (includes translation and scaling of the model) 9

10 Appearance models Represent both texture and shape Statistical model learned from training data Modeling shape variability: Landmark points x = Model x x+ P s b s Modeling intensity variability Gray values = [ x y, x, y,...,, ] T, Gray values [ g, g,..., ] T h 1 2 P x n y n x : Mean vector b S S g k Eigenvectors of covariance matrix = P T S (x x) P b h : Mean vector i i P i Eigenvectors of T b = P (h h) h h+ i i covariance matrix 10

11 Training of appearance models 1. Construct a shape model with Principal component analysis (PCA). A shape is represented with manually labeled points The shape model approximates the shape of an object 2. Construct a texture model which represents grey-scale (or color) values at each point 3. Model the correlation between shapes and grey-level models 11

12 Principal Component Analysis (PCA) Samples are projected into a sub-space that maximizes the variances of the samples along the 1st principal axis. An example in 2-D space Original axis Principal axis x Principal axis x x ' = b x + bp x p x' : mean x Original axis We can remove redundant elements of the vectors cut off at n-th dimension 12

13 Construction of a shape model The positions of labeled points are x = x + x The mean shape Ps P b s s Orthogonal modes of variation obtained by PCA Mode 1 Mode 2 mean b Shape parameters in the projected space s The shapes are represented with fewer parameters. Dim( x ) >Dim( b ) s 13

14 Generating gplausible shapes Effects of varying each of the first shape model parameters 14 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

15 Training of an appearance model 1. Construction of a shape model with Principle component analysis (PCA). A shape is represented with manually labeled points The shape model approximates the shape of an object 2. Construction of a texture model which represents grey-scale values at each point 3. Modeling the correlation between shapes and grey-level models 15

16 Construction of a texture model Warp the image so that the labeled points fit on the mean shape warped A image Mean shape shape-free patch Normalize the intensity on the shape-free patch 16 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

17 Texture warping Triangle in mesh s Triangle in reference mesh Piece wise affine warping cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

18 Texture model The pixel values on the shape-free patch g = g + P g b g g The mean of normalized pixel values P g Orthogonal modes of variation obtained by PCA b Texture parameters in the projected space g The pixel values (appearance) are presented with fewer parameters. Dim( g ) >Dim( ) b g 18

19 Training of an appearance model 1. Construction of a shape model with Principle component analysis (PCA). A shape is represented with manually labeled points The shape model approximates the shape of an object 2. Construction of a texture model which represents grey-scale values at each point 3. Modeling the correlation between shapes and texture models 19

20 The correlation between two models The concatenated vector b = Apply PCA b = P c c W s bs b g Pcs = c P cg The parameter c can control lboth shape and grey-level lmodels The shape model The grey-level model x = x + P W s 1 s P cs c g = g + P g P cg c 20

21 Examples of synthesized faces Various objects can be synthesized by controlling the parameter c First two modes of shape First two modes of grey-level el variation (+- 3sd) variation (+- 3sd) 21

22 s 0 A 0 (x) x 23 Instances of Shape and Texture s 0 + p 1 s 1 0 p 2 s 2 s + s 0 + p 3 s 3 ( ) + A ( x ) ( x ) + A ( x ) ( x ) + A ( x ) A 0 x λ 1 1 A 0 λ 2 2 A 0 λ 3 3 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

23 Dataset for Building Model (from ongoing g diploma thesis, Hua Gao) IMM data set from Danish Technical University 240 images with 640*480 size; 40 individuals, with 36 males and 4 females. Each Subject 6 shots, with different pose, expressions and illuminations. Each image is labeled with 58 landmarks; 3 closed and 4 opened point-paths. 24

24 Image Interpretation with Models Must find the set of parameters which best match the model to the image optimize some cost function Difficult optimization problem Set of parameters defines Shape, position, appearance Can be used for further processing Position of landmarks Face recognition Facial expression recognition Pose estimation Problem: Optimizing the Model Fit Active Shape Models Active Appearance Models 25

25 Active Shape Models Given a rough starting position, create an instance of X of the model using shape parameters b, Translation T=(X t,y t ), scale s and rotation θ Iterative approach 1. Examine region of the image around X i to find the best nearby match for the point X i 2. Update parameters (b,t,s, θ) to best fit the new points X (constrain the model parameters to be within three standard deviations) 3. Repeat until convergence 26

26 Search along gprofiles In practice: search along profile normals either for edges Or for learned texture profiles Can also be done in a multi-resolution approach Coarse to fine 27 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

27 Multi-Resolution approach The amount of calculation for search is too much. The optimal parameters are searched from multi-resolution images hierarchically. 1. Search for the object in a coarse image 2. Refine the location in a series of higher resolution images. Level 2 Level 1 Level 0 This leads to a faster algorithm. 28

28 Examples of Search Search using Active Shape Model of a face 29 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

29 Examples Search using ASM of cartilage on an MR image of the knee 30 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

30 Active Appearance Models (AAM) Disadvantage of ASM: uses mainly shape constraints for search Does not take advantage of texture across the target AAM approach Optimize parameters, so as to minimize the difference of a synthesized image and the target image Solved using a gradient-descent approach 31

31 Fitting AAMs Input image Initial model Image under model c : model parameter δg : error image R : relationship matrix between error and parameter adjustments k : weight for parameter update Appearance c c + kδc Difference δg Warp to reference Convergence Predict update δc = Rδg

32 Fitting AAMs(cont.) Learning linear relation matrix R using multiple multivariate linear regression Generate training set by perturbing model parameters for training images Include small displacements in position, scale, and orientation Record perturbation and image difference Experimentally, optimal perturbation around 0.5 standard deviations for each parameter

33 34 Alignment with AAMs cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

34 Fitting Examples Fitted mesh Mismatched mesh cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

35 Pose normalization Fitted modal can be used to warp image to frontal pose E.g. using piecewise affine transformation of mesh triangles Faces with different poses from FERET data base and their pose-aligned images ci cv:hc 36

36 Results (2) Much better results under pose variations compared to simple affine transform: With pose correction Simple Affine Transf. Different warping functions can be used Piecewise affine transformation worked best Approach works well with local-dct-based approach but not so well with holistic approaches, such as Eigenfaces (PCA) (Gao, Ekenel, Stiefelhagen, ICB 09) 37

37 Pose-normalization with AAMs 3D pose correction for pose invariant face recognition, DFFS assesses the quality of model fitting 38 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

38 ASM vs. AAM ASM Seeks to match a set of model points to an image, constrained by a statistical model of shape Matches model points using an iterative technique (variant of EM- algorithm) A search is made around the current position of each point to find a nearby point which best matches texture for the landmark Parameters of fthe shape model dlare then updated dtdto move the model dl points closer to the new points in the image AAM matches both position i of model points and representation of texture of the object to an image Uses the difference between current synthesized image and target image to update parameters Typically, less landmark points are needed 39

39 Summary of ASM / AAM Statistical appearance models provide a compact represenation Can model variations such as different identities, facial expression, appearances, etc. Labeled training images are needed (very time-consuming) Original formulation of ASM and AAM is computationally expensive (i.e. slow) But, efficient extensions and speed-ups exist Multi-resolution l search, Constrained AAM search, Inverse compositional AAMs (CMU) Usage Face recognition, pose estimation Facial expression analysis Audio-visual speech recognition 40

40 Deformable Templates 41 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

41 Deformable Templates (Yuille et al.) Hand-crafted model to find eyes, using a combination of parametrised circles and arcs Energy function is defined Links edges, peaks, valleys in the image intensity to the properties of the template Model parameters are altered in order to minimize energy function Template with 11 parameters A.L Yuille, D.S. Cohen, P.W. Hallinan, Feature extraction from faces using deformable templates, Computer Vision Pattern Recognition

42 The Eye Template Circle of radius r, centered at x c Boundary attracted to edges Interior attracted to valleys (low image intensities) Bounding contour Two paraboloid sections, center x e Two points, centers of the whites X e lies at center, θ: eye orientation i Attracted to peaks Regions between bounding contour and iris Attracted t to large intensity it values Linked together through forces: Encourage x c and x e to be close together th Make width of the eye ca. 4 times the iris radius want the centers of the whites roughly at eye center 43

43 Energy Function Complete function E C = Valley potential E v : Integral over internal of the circle Edge potential E e Integral over boundaries of circle and parabolae Image potentials E i Minimize total brightness inside circle Maximize it between circle and parabolae Peak potential E p Points should lie within the white of the eye Internal potential E internal Enforce reasonable model parameters Gradient descent used for minimization Energy give a goodness of fit measure 44

44 Minimization Algorithm has several epochs Different values at each epoch for the individual coefficients (c i and k i ) E.g. high h coefficients i for valley forces initially i i will pull the template to the eye Increase intensity and edges forces later, etc. fine tuning Good initial coefficients c i and k i needed Good initial starting point needed Template has to lie over the eye 45

45 Examples Several iterations of eye fitting Same approach applied to lips 46 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

46 Other usage Fitting lip-models Model-parameters can be used for audio-visual speech recognition Fitting an ellipsoid head model Maximizes edges along ellipse contour and skin colored pixel inside ellipse See face detection lecture I 47

47 Summary: Deformable Templates Can be used to fit a detailed model E.g. eyes or lips thus recovering the exact position of these facial features Uses a-priori knowledge about the shape and appearance of the object Hand-tuned parametric model and energy function Needs good initialization Initial template has to lie above the eye already Might be combined with other more imprecise localization approaches (Viola&Jones, Integral Projection, etc.) 48

48 Approaches using Gabor Wavelets 49 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

49 Gabor wavelet transformation Input image Training stage (construction of templates) Preparation of the image patches The image patches are transformed by Gabor wavelet transformation. Scan and crop the image Gabor wavelet transformation Calculate the difference between the cropped image and templates Find the position where the difference is minimal 50

50 Gabor wavelet transformation (GWT) Gabor kernels are controlled by the scale v and orientation u u orientation Ou, v ( x, y) = I( x, y) ψ u, v ( x, y) v scale Output of GWT O u, v ( x, y ) is defined as the convolution of an input image I ( x, y) and the Gabor kernel ψ u, v( x, y) Where x and y mean the position GWT can be considered to be a filter that emphasizes the scale and direction of an image 51

51 Facial feature vector with Gabor wavelet transformation (GWT) O u, v ( x, y) = I( x, y) ψ u, v ( x, y) Typically, multiple scales u and orientations v are used. Therefore O ( x, y) ) becomes a high dimensional vector. O u, v y Techniques to reduce the dimension are applied (such as PCA). 52

52 Gabor wavelet representation of a face image At 5 different scales and 8 different orientations. orientation ti scale 53 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

53 Elastic Bunch Graphs (EBG) A Jet : is a set of 40 complex Gabor wavelet coefficients obtained for one image point A graph consists of N facial landmark points (nodes) nodes are labelled with jets edges are labelled with distance vectors A bunch graph includes different jets for different poses and appearances (e.g. closed eye, open eye,...) L.Wiskott et al., Face recognition by elastic bunch graph matching, IEEE Trans. on PAMI, 19(7): ,

54 Gabor filters 2D Gabor filter Spatial domain Orientation Scale 55 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

55 Face Modeling with EBGs Matching a Face Bunch Graph (FBG) is done by maximizing a graph similarity. It depends on the jet similarities and topography. 56 cv:hci Computer Vision for Human-Computer Interaction Research Group, Universität Karlsruhe (TH)

56 Summary Approaches Viola & Jones to find landmarks Eigenspace-matching to find landmarks Integral projection plus geometric constraints Gabor Wavelets to find landmarks Elastic network to imply geometric constraints ( Elastic Bunch Graph Matching, EBGM) Statistical appearance models Model shape and texture (using PCA) Model fitted to image delivers positions of landmarks Active Shape Model (ASM) / Active Appearance Model (AAM) 3D Morphable Model (Blanz et al.) Deformabel Models (Yuille et al.) Define a parametrized model of an eye (often also used for lips) Specific forces / energy function to fit the model to an image Most of these approaches involve the minimization of some energy function EBGM, ASM, AAM, 3D Morph. Model Gradient descent search 57

57 References L.Wiskott et al., Face recognition by elastic bunch graph matching, IEEE Trans. on PAMI, 19(7): , 1997 A.L Yuille, D.S. Cohen, P.W. Hallinan, Feature extraction from faces using deformable templates, Computer Vision Pattern Recognition,1989 T.F. Cootes and C.J. Taylor, Statistical Models of Appearance for Computer Vision, Technical Report (draft), Univ. of Manchester T.F. Cootes, G.J. Edwards, C.J. Taylor, Active Appearance Models, European Conf. on Computer Vision, Vol. 2, pp , Springer 1998 ci cv:hc 58