Facial Expression Recognition

Transcription

1 Facial Expression Recognition YING SHEN SSE, TONGJI UNIVERSITY Facial expression recognition Page 1

2 Outline Introduction Facial expression recognition Appearance-based vs. model based Active appearance model (AAM) Pre-requisite Principle component analysis (PCA) Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 2

3 Outline Introduction Facial expression recognition Appearance-based vs. model based Active appearance model (AAM) Pre-requisite Principle component analysis (PCA) Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 3

4 Introduction Smile detection Facial expression recognition Page 4

5 Introduction Facial expression recognition Happy Surprise Angry Disgust Sad Fear Facial expression recognition Page 5

6 Introduction Facial expression recognition Happy: 83% Disgusted: 9% Fearful: 6% Angry: 2% Facial expression recognition Page 6

7 Introduction Factors affect facial expression recognition accuracy Pose Illumination Facial expression recognition Page 7

8 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 8

9 Facial expression recognition Framework of automatic facial expression recognition Face Acquisition Facial Feature Extraction Facial Expression Classification Whole Face Deformation Extraction Appearance-based Model-based Facial expression recognition Page 9

10 Facial expression recognition Appearance-based methods E.g. Features are extracted using Garbor filter Facial expression recognition Page 10

11 Facial expression recognition CMU facial expression database (CK, CK+ database) images Six basic expressions Training preparation create positive examples prepare negative examples Training Train Haar-like features using OpenCV Or train a NN classifier using extracted features Facial expression recognition Page 11

12 Facial expression recognition Angry Positive Examples for Training Facial expression recognition Page 12

13 Facial expression recognition Model-based methods Facial expression recognition Page 13

14 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis (PCA) Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 14

15 Pre-requisite: Lagrange multiplier Single-variable function f( x) is differentiable in (a, b). At extremum df dx Two-variables function 0 x 0 x0 ( a, b) f ( x, y) is differentiable in its domain. At an extremum f x f ( x0, y0 ) 0, ( x0, y0 ) 0 y ( x, y ), f(x) achieves an 0 0, f(x, y) achieves Facial expression recognition Page 15

16 Pre-requisite: Lagrange multiplier In general case If x 0 is a stationary point of f ( x), x n1 f f f 0, 0,..., x x x x x x 1 2 n Facial expression recognition Page 16

17 Pre-requisite: Lagrange multiplier Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints Problem: find stationary points for under m constraints g ( x) 0, k 1,2,..., m Solution: 1 k F( x;,..., ) f ( x) g ( x) ( x,,..., m ) m k k k1 If is a stationary point of F, then, x is a stationary point of f ( x) with constraints 0 m Joseph-Louis Lagrange Jan. 25, 1736~Apr.10, 1813 Facial expression recognition Page 17

18 Pre-requisite: Lagrange multiplier Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints Problem: find stationary points for under m constraints g ( x) 0, k 1,2,..., m Solution: 1 k F( x;,..., ) f ( x) g ( x) m m k k k1 ( x0, 10,..., m 0) is a stationary point of F F F F F F F 0, 0,..., 0, 0, 0,..., 0 x x x 1 2 n 1 2 at that point n + m equations! Facial expression recognition Page 18 m

19 Pre-requisite: Lagrange multiplier Example Problem: for a given point p 0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p 0. The distance is 2 2 f ( x, y) ( x 1) ( y 0) y=x? p 0 Now we want to find the stationary point of f(x, y) under the constraint g( x, y) y x 0 According to Lagrange multiplier method, construct another function F x y f x g x x y y x Find the stationary point for F( x, y, ) 2 2 (,, ) ( ) ( ) ( 1) ( ) Facial expression recognition Page 19

20 Pre-requisite: Lagrange multiplier Example Problem: for a given point p 0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p 0. F 0 x 2( x 1) 0 x 0.5 F 0? 2y 0 y y 0.5 x y 0 F 1 p 0 0 y=x (0.5,0.5,1) is a stationary point of F( x, y, ) (0.5,0.5)is a stationary point of f(x,y) under constraints Facial expression recognition Page 20

21 Pre-requisite: matrix differentiation Function is a vector and the variable is a scalar f ( t) f1( t), f2( t),..., f ( ) T n t Definition df df ( t) df ( t) dfn t,,..., dt dt dt dt 1 2 () T Facial expression recognition Page 21

22 Pre-requisite: matrix differentiation Function is a matrix and the variable is a scalar Definition f11( t) f12 ( t),..., f1 m( t) f21( t) f22( t),..., f2m( t) f ( t) fij ( t) fn1( t) fn2( t),..., fnm( t) df dt nm df11( t) df12( t) df1 (),..., m t dt dt dt df21( t) df22( t) df2m() t,..., dfij () t dt dt dt dt dfn 1( t) dfn2( t) dfnm( t),..., dt dt dt Facial expression recognition Page 22 nm

23 Pre-requisite: matrix differentiation Function is a scalar and the variable is a vector Definition In a similar way, f x x x1 x2 x n ( ), (,,..., ) T df f f f,,..., dx x1 x2 x f x x x1 x2 x n ( ), (,,..., ) df f f f,,..., dx x1 x2 x n n T Facial expression recognition Page 23

24 Pre-requisite: matrix differentiation Function is a vector and the variable is a vector T x x, x,..., x, y y ( x), y ( x),..., y ( x) Definition dy dx 1 2 n 1 2 T y1( x) y1( x) y1( x),,..., x1 x2 x n y2( x) y2( x) y2( x),,..., x1 x2 x n y ( x) y ( x) y ( x),,..., m m m x1 x2 xn m n Facial expression recognition Page 24 m T

25 Pre-requisite: matrix differentiation Function is a vector and the variable is a vector T x x, x,..., x, y y ( x), y ( x),..., y ( x) 1 2 n 1 2 In a similar way, dy dx T y1( x) y2( x) ym( x),,..., x1 x1 x 1 y1( x) y2( x) ym( x),,..., x2 x2 x 2 y1( x) y2( x) ym( x),,..., x x x n n n n m Facial expression recognition Page 25 m T

26 Pre-requisite: matrix differentiation Function is a vector and the variable is a vector Example: x y ( x) y y x x x , x2, y1( ) x1 x2, y2( ) x3 3x2 2( x) x 3 y1( x) y2( x) x x 1 1 2x1 0 T dy y1( x) y2( x) 1 3 dx x x x3 y1( x) y2( x) x x 3 3 Facial expression recognition Page 26

27 Pre-requisite: matrix differentiation Function is a scalar and the variable is a matrix Definition df ( X) dx f ( X), X mn f f f x11 x12 x1 n f f f x x x m1 m2 mn Facial expression recognition Page 27

28 Pre-requisite: matrix differentiation Useful results (1) Then, d dx T a x xa, n1 d a, dx T x a a How to prove? Facial expression recognition Page 28

29 Pre-requisite: matrix differentiation Useful results dax (2) Then, A T dx (3) Then, T T dx A T A dx (4) T dx Ax T Then, ( A A ) x dx (5) T da Xb Then, ab dx T T da X b (6) Then, ba dx T dxx (7) Then, 2x dx Facial expression recognition Page 29 T T

30 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 30

31 Principal Component Analysis (PCA) PCA: converts a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components The number of principal components is less than or equal to the number of original variables This transformation is defined in such a way that the first principal component has the largest possible variance, and each succeeding component in turn has the highest variance possible under the constraint that it be orthogonal to (i.e., uncorrelated with) the preceding components Facial expression recognition Page 31

32 Principal Component Analysis (PCA) Illustration How to find? x, y (2.5, 2.4) (0.5, 0.7) (2.2, 2.9) (1.9, 2.2) (3.1, 3.0) (2.3, 2.7) (2.0, 1.6) (1.0, 1.1) (1.5, 1.6) (1.1, 0.9) De-correlation! Along which orientation the data points scatter most? Facial expression recognition Page 32

33 Principal Component Analysis (PCA) Identify the orientation with largest variance Suppose X contains n data points, and each data point is p- dimensional, that is Now, we want to find such a unit vector 1, Facial expression recognition Page 33

34 Principal Component Analysis (PCA) Identify the orientation with largest variance 1 1 var ( ) ( )( ) n n T T T T T X xi xi xi n1 i1 n1 i1 2 T C T T (Note that: ( x ) ( x ) ) i i 1 n i where x n i1 n 1 and C ( i )( i ) n 1 x x i1 T is the covariance matrix Facial expression recognition Page 34

35 Principal Component Analysis (PCA) Identify the orientation with largest variance T Since is unit, 1 Based on Lagrange multiplier method, we need to, T T arg max C 1 T T d C 1 0 2C 2 C d is C s eigen-vector Since, max var max max max Thus, T T T X C Facial expression recognition Page 35

36 Principal Component Analysis (PCA) Identify the orientation with largest variance 1 Thus, should be the eigen-vector of C corresponding to the largest eigen-value of C 1 What is another orientation, orthogonal to, and 2 along which the data can have the second largest variation? Answer: it is the eigen-vector associated to the second largest eigen-value of C and such a variance is 2 2 Facial expression recognition Page 36

37 Principal Component Analysis (PCA) Identify the orientation with largest variance Results: the eigen-vectors of C forms a set of orthogonal basis and they are referred as Principal Components of the original data X You can consider PCs as a set of orthogonal coordinates. Under such a coordinate system, variables are not correlated. Facial expression recognition Page 37

38 Principal Component Analysis (PCA) Express data in PCs Suppose {,,..., } are PCs derived from X, X 1 2 p p1 Then, a data point xi can be linearly represented by {,,..., }, and the representation coefficients are 1 2 p T 1 T 2 ci xi T p Actually, c i is the coordinates of x i in the new coordinate system spanned by { 1, 2,..., p } pn Facial expression recognition Page 38

39 Principal Component Analysis (PCA) Illustration x, y (2.5, 2.4) (0.5, 0.7) (2.2, 2.9) (1.9, 2.2) (3.1, 3.0) (2.3, 2.7) (2.0, 1.6) (1.0, 1.1) (1.5, 1.6) (1.1, 0.9) X cov( X) Eigen-values = , Corresponding eigen-vectors: Facial expression recognition Page 39

40 Principal Component Analysis (PCA) Illustration Facial expression recognition Page 40

41 Principal Component Analysis (PCA) Illustration Coordinates of the data points in the new coordinate system T 1 newc X T X Facial expression recognition Page 41

42 Principal Component Analysis (PCA) Illustration Coordinates of the data points in the new coordinate system Draw newc on the plot In such a new system, two variables are linearly independent! Facial expression recognition Page 42

43 Principal Component Analysis (PCA) Data dimension reduction with PCA n p1 p p1 Suppose X { x }, x, { }, are the PCs i i1 i i i1 i T 1 T If all of { } p 2 i i 1 are used, ci xi is still p-dimensional T p m If only { i} i 1, m p are used, ci will be m-dimensional That is, the dimension of the data is reduced! Facial expression recognition Page 43

44 Principal Component Analysis (PCA) Illustration Coordinates of the data points in the new coordinate system newc X If only the first PC (corresponds to the largest eigen-value) is remained newc X Facial expression recognition Page 44

45 Principal Component Analysis (PCA) Illustration All PCs are used Only 1 PC is used Dimension reduction! Facial expression recognition Page 45

46 Principal Component Analysis (PCA) Illustration If only the first PC (corresponds to the largest eigen-value) is remained newc X How to recover newc to the original space? Easy T newc Facial expression recognition Page 46

47 Principal Component Analysis (PCA) Illustration Data recovered if only 1 PC used Original data Facial expression recognition Page 47

48 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 48

49 Procrustes analysis Who is Procrustes? Facial expression recognition Page 49

50 Procrustes analysis Problem: Suppose we have two sets of point correspondence pairs (m 1, m 2,, m N ) and (n 1, n 2,, n N ). We want to find a scale factor s, an orthogonal matrix R and a vector t so that: N 2 i i i1 ( ) 2 m sr n T (1) Facial expression recognition Page 50

51 Procrustes analysis In 2D space: Facial expression recognition Page 51

52 Procrustes analysis How to compute R and T? We assume that there is a similarity transform N N between point sets and ii 1 Find s, R and T to minimize m nii 1 N N 2 2 i i i i1 i1 e m sr n T ( ) 2 (1) Let 1 1 m m n n m m m n n n N N N ' ' i, i, i i, i i i1 N i1 Note that N N ' mi, i1 i1 0 n 0 ' i Facial expression recognition Page 52

53 Then: e m sr n T m m sr n n T m m sr n sr n T ' ' ' ' i i ( i) i ( i ) i ( i) ( ) m sr( n ) T m sr( n) m sr( n ) e e0 T m sr( n) (1) can be rewritten as: N N N N 2 2 N N N 2 i ( i ) 2 i 2 ( i ) e m sr( n ) e m sr( n ) 2 e m sr( n ) Ne 2 2 ' ' ' ' ' ' 2 i i i 0 i i 0 i i 0 i1 i1 i1 i1 m sr n e m e sr n Ne ' ' ' ' i1 i1 i1 N i1 Procrustes analysis ' ' ' ' i i i i ' ' is independent from { mi, ni} ' ' 2 2 i ( i) 0 m sr n Ne Variables are separated and can be minimized separately. 2 e T m sr n 0 0 ( ) If we have s and R, T can be determined. 0

54 Procrustes analysis Then the problem simplifies to: how to minimize N 2 ' ' 2 mi sr( ni) i1 We revise the error item as a symmetrical one: Consider its geometric meaning here. N 2 N N N 2 ' ' ' 2 ' ' ' 2 mi sr( n ) i m 2 i mi R( ni ) s R( ni ) i1 s s i1 i1 i s 1 s N N N ' 2 ' ' ' 2 mi 2 mi R( ni ) s ni i1 i1 i1 P D Q Variables are separated P 2D sq s Q P s s s 2( PQ D) Thus,

55 2 1 P s Q P 0 s s Q Then the problem simplifies to: how to maximize D N i1 N i1 N ' ' mir( ni) i1 N N ' ' ' T N ' ' ' i i i i i i i1 i1 i1 m n ' i ' i 2 2 Note that: D is a real number. Determined!. D m Rn m Rn trace Rn m trace RH H Procrustes analysis T N i1 ' ni m ' i T Now we are looking for an orthogonal matrix R to maximize the trace of RH.

56 Lemma For any positive semi-definite matrix C and any orthogonal matrix B: Proof: Let Procrustes analysis a i trace C From the positive definite property of C, where A is a non-singular matrix. be the ith column of A. Then trace BC T T T i i trace BAA trace A BA a Ba According to Schwarz inequality:, Hence, x y x y a Ba a Ba a a a B Ba a a T T T T T T i i i i i i i i i i T T T i i trace BAA a a trace AA i A, C AA i T that is, trace BC trace C

57 Procrustes analysis Consider the SVD of H N H UV According to the property of SVD, U and V are orthogonal matrices, and Λ is a diagonal matrix with nonnegative elements. ' ni m i1 ' i T T Now let X VU T Note that: X is orthogonal. We have T T T XH VU UV VV which is positive semi-definite. Thus, from the lemma, we know: for any orthogonal matrix B trace( XH ) trace( BXH ) for any orthogonal matrix Ψ trace( XH ) trace( H) It s time to go back to our objective now R should be X

58 Now, s, R and T are all determined. ' ' 1 N T T i i i H n m U V T R VU 2 ' 1 2 ' 1 N i i N i i m s n ( ) T m sr n Procrustes analysis

59 Procrustes analysis Problem: Given two configuration matrices X and Y with the same dimensions, find rotation and translation that would bring Y as close as possible to X. 1) Find the centroids (mean values of columns) of X and Y. Call them x and y. 2) Remove from each column corresponding mean. Call the new matrices X new and Y new 3) Find Y newt X new. Find the SVD of this matrix Y newt X new = UDV T 4) Find the rotation matrix R = UV T. Find scale factor 5) Find the translation vector T = x- sr y. s N i1 N i1 X Y new new 2 2 Facial expression recognition Page 59

60 Some modifications Sometimes it is necessary to weight some variables down and others up. In these cases Procrustes analysis can be performed using weights. We want to minimize the function: M 2 T tr( W( X AY)( X AY) ) This modification can be taken into account if we find SVD of Y T WX instead of Y T X Facial expression recognition Page 60

61 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 61

62 Active shape model (ASM) Collect training samples 30 neutral faces 30 smile faces Facial expression recognition Page 62

63 ASM Add landmarks 26 points each face X = [x 1 x 2 x N ]; N = 60. Facial expression recognition Page 63

64 ASM Align training faces together using Procrustes analysis Transform x i to x j : Rotation (R i ), Scale (s i ), Transformation (T i ) Consider a weight matrix W: D kl represents the distance between the point k and l in one image and V Dkl represents the variance of D kl in different images w k N ( VD ) l1 m is the dimension of x i min Z * T i Zi Z x ( s * R *( x ) T ) kl i j i 1 W w 1 w m Facial expression recognition Page 64

65 ASM We want to minimize E min Z * W * Z T i i i 1) Find the centroids (mean values of columns) of x i and x j. 2) Remove from each column corresponding mean. 3) Find the SVD of this matrix x i_new *W*x j_newt = UDV T 4) Find the rotation matrix A = UV T. Find the scale factor and translation vector as shown in page 59. Facial expression recognition Page 65

66 ASM Align training faces using Procrustes analysis Steps: 1. Align the other faces with the first face 2. Compute the mean face x xi i1 3. Align training faces with the mean face 4. Repeat step 2 and 3 until the discrepancies between training faces and the mean face won t change N Facial expression recognition Page 66

67 ASM Faces after alignment Facial expression recognition Page 67

68 ASM Construct models of faces Compute mean face Calculate covariance matrix S N i 1 Find its eigenvalues (λ 1, λ 2,, λ m ) and eigenvectors P = (p 1, p 2,, p m ) Choose the first t largest eigenvalues Usually f v = 0.98 t i1 x Facial expression recognition Page 68 N x i1 N 1 ( xi x)( xi x) i, v T f V V T i i t T Dimension reduction: 26*227

69 ASM We can approximate a training face x x x Pb b P T ( x x) b i is called the i th mode of the model constraint: b 3 i Facial expression recognition Page 69 i

70 ASM Effect of varying the first three shape parameters in turn between ±3 s.d. from the mean value -3 s.d. origin +3 s.d. -3 s.d. origin +3 s.d. Facial expression recognition Page 70

71 ASM Active Shape Model Algorithm 1. Examine a region of the image around each point x i to find the best nearby match for the point x i ' 2. Update the parameters (T, s, R, b) to best fit the new found points x' 3. Apply constraints to the parameters, b, to ensure plausible shapes (e.g. limit so b i < 3 λ i ). 4. Repeat until convergence. Q1: How to find corresponding points in a new image? Facial expression recognition Page 71

72 ASM Find the initial corresponding points Detect face using Viola-Jones face detector Estimate positions of eyes centers, nose center, and mouse center Align the corresponding positions on the mean face to the estimated positions of eyes centers, nose center, and mouse center (s init, R init, T init ) on the new face x s * R * x T init init init init Facial expression recognition Page 72

73 ASM Construct local features for each point in the training samples For a given point, we sample along a profile k pixels either side of the model point in the i th training image. Instead of sampling absolute grey-level values, we sample derivatives and put them in a vector g i Normalize the sample: point i-1 g i gi g j ij point i 2k+1 pixels point i+1 Facial expression recognition Page 73

74 ASM For each training image, we can get a set of normalized samples {g 1, g 2,, g N } for point i We assume that these g i are distributed as a multivariate Gaussian, and estimate their mean g i and covariance S gi. This gives a statistical model for the greylevel profile about the point i Given a new sample g s, the distance of g s to g i can be computed using the Mahalanobis distance d T 1 ( gs, gi ) ( gs gi S g ( gs gi ) ) Facial expression recognition Page 74 i

75 ASM During search we sample a profile m pixels from either side of each initial point (m > k ) on the new face. point i-1 point i dx i 2m+1 pixels point i+1 T 1 min d( gs, gi ) ( gs gi S g ( g i s gi g ) ) Facial expression recognition Page 75 s

76 ASM Some constraints on dx i dx i = 0 dx i = 0.5d best dx i = 0.5d max if d best <= δ if δ<= d best <= d max if d best > d max Facial expression recognition Page 76

77 ASM Apply one iteration of ASM algorithm 1. Examine a region of the image around each point x i to find the best nearby match for the point x i ': x i ' = x i + dx i 2. Update the parameters (T, s, R, b) to best fit the new found points x' 3. Apply constraints to the parameters, b, to ensure plausible shapes (e.g. limit so b i < 3 λ i ). 4. Repeat until convergence. Q2: How to find T, s, R, and b for x'? Facial expression recognition Page 77

78 ASM Suppose we want to match a model x to a new set of image points y We wish to find (T, s, R, b) that can minimize 2 min ys * R( x) T min y s * R( x Pb) T 2 Facial expression recognition Page 78

79 ASM A simple iterative approach to achieving the minimum 1. Initialize the shape parameters, b, to zero (the mean shape). 2. Generate the model point positions using x = x + Pb 3. Find the pose parameters (s, R, T) which best align the model points x to the current found points y min ys * R( x) T 4. Project y into the model co-ordinate frame by inverting the transformation : 1 ' ( y ) / R T s 5. Project y into the tangent plane to x by scaling: y'' = y'/(y' x). 6. Update the model parameters to match to y'' 7. If not converged, return to step 2. y T b P ( y '' x) 2 Facial expression recognition Page 79

80 ASM Apply one iteration of ASM algorithm 1. Examine a region of the image around each point x i to find the best nearby match for the point x i ': x i ' = x i + dx i 2. Update the parameters (T, s, R, b) to best fit the new found points x' using the algorithm in page Apply constraints to the parameters, b, to ensure plausible shapes (e.g. limit so b i < 3 λ i ). 4. Repeat until convergence. Facial expression recognition Page 80

81 ASM So now we have a vector of b for the new face Classify facial expression using b Facial expression recognition Page 81

82 Classification Training Compute b for each training faces Training a classifier (e.g. SVM, Neural Network) using {b 1, b 2,, b N } and the corresponding labels Test Using the trained classifier to classify a new mode vector b new of a new face Facial expression recognition Page 82

83 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 83

84 Delaunay triangulation Terrains by interpolation To build a model of the terrain surface, we can start with a number of sample points where we know the height. Facial expression recognition Page 84

85 Delaunay triangulation How do we interpolate the height at other points? Height ƒ(p) defined at each point p in P How can we most naturally approximate height of points not in P? Let ƒ(p) = height of nearest point for points not in A Does not look natural Facial expression recognition Page 85

86 Delaunay triangulation Better option: triangulation Determine a triangulation of P in R 2, then raise points to desired height Triangulation: planar subdivision whose bounded faces are triangles with vertices from P Facial expression recognition Page 86

87 Delaunay triangulation Formal definition Let P = {p 1,,p n } be a point set. Maximal planar subdivision: a subdivision S such that no edge connecting two vertices can be added to S without destroying its planarity A triangulation of P is a maximal planar subdivision with vertex set P. Facial expression recognition Page 87

88 Delaunay triangulation Triangulation is made of triangles Outer polygon must be convex hull Internal faces must be triangles, otherwise they could be triangulated further Facial expression recognition Page 88

89 Delaunay triangulation For P consisting of n points, all triangulations contain 2n-2-k triangles 3n-3-k edges n = number of points in P k = number of points on convex hull of P Facial expression recognition Page 89

90 Delaunay triangulation But which triangulation? Facial expression recognition Page 90

91 Delaunay triangulation Some triangulations are better than others Avoid skinny triangles, i.e. maximize minimum angle of triangulation Facial expression recognition Page 91

92 Delaunay triangulation Let T be a triangulation of P with m triangles. Its angle vector is A(T) = (α 1,, α 3m ) where α 1,, α 3m are the angles of T sorted by increasing value. Let T be another triangulation of P. A(T) is larger then A(T ) iff there exists an i such that j = ' j for all j < i and i > ' i T is angle optimal if A(T) > A(T ) for all triangulations T of P Facial expression recognition Page 92

93 Delaunay triangulation If the two triangles form a convex quadrilateral, we could have an alternative triangulation by performing an edge flip on their shared edge. The edge e = P i P j is illegal if min α i min α i 1 i 6 1 i 6 Flipping an illegal edge increases the angle vector Facial expression recognition Page 93

94 Delaunay triangulation If triangulation T contains an illegal edge e, we can make A(T) larger by flipping e. In this case, T is an illegal triangulation. Facial expression recognition Page 94

95 Delaunay triangulation We can use Thale s Theorem to test if an edge is legal without calculating angles Theorem: Let C be a circle, l a line intersecting C in points a and b, and p, q, r, s points lying on the same side of l. Suppose that p, q lie on C, r lies inside C, and s lies outside C. Then arb > apb = aqb > asb Facial expression recognition Page 95

96 Delaunay triangulation If p i, p j, p k, p l form a convex quadrilateral and do not lie on a common circle, exactly one of p i p j and p k p l is an illegal edge. Lemma: The edge P i P j is illegal iff p l lies in the interior of the circle C. Facial expression recognition Page 96

97 Delaunay triangulation A legal triangulation is a triangulation that does not contain any illegal edge. Compute Legal Triangulations 1. Compute a triangulation of input points P. 2. Flip illegal edges of this triangulation until all edges are legal. Algorithm terminates because there is a finite number of triangulations. Too slow to be interesting Facial expression recognition Page 97

98 Delaunay triangulation Before we can understand an interesting solution to the terrain problem, we need to understand Delaunay Graphs. Delaunay Graph of a set of points P is the dual graph of the Voronoi diagram of P Facial expression recognition Page 98

99 Delaunay triangulation Voronoi Diagram and Delaunay Graph Let P be a set of n points in the plane The Voronoi diagram Vor(P) is the subdivision of the plane into Voronoi cells V(p) for all p P Let G be the dual graph of Vor(P) The Delaunay graph DG(P) is the straight line embedding of G Facial expression recognition Page 99

100 Delaunay triangulation Voronoi Diagram and Delaunay Graph Calculate Vor(P) Place one vertex in each site of the Vor(P) Facial expression recognition Page 100

101 Delaunay triangulation Voronoi Diagram and Delaunay Graph If two sites s i and s j share an edge (s i and s j are adjacent), create an arc between v i and v j, the vertices located in sites s i and s j Facial expression recognition Page 101

102 Delaunay triangulation Voronoi Diagram and Delaunay Graph Finally, straighten the arcs into line segments. The resultant graph is DG(P). Facial expression recognition Page 102

103 Delaunay triangulation Properties of Delaunay Graphs No two edges cross; DG(P) is a planar graph. Facial expression recognition Page 103

104 Delaunay triangulation Some sets of more than 3 points of Delaunay graph may lie on the same circle. These points form empty convex polygons, which can be triangulated. Delaunay Triangulation is a triangulation obtained by adding 0 or more edges to the Delaunay Graph. Facial expression recognition Page 104

105 Delaunay triangulation From the properties of Voronoi Diagrams Three points p i, p j, p k P are vertices of the same face of the DG(P) iff the circle through p i, p j, p k contains no point of P on its interior. Facial expression recognition Page 105

106 Delaunay triangulation From the properties of Voronoi Diagrams Two points p i, p j P form an edge of DG(P) iff there is a closed disc C that contains p i and p j on its boundary and does not contain any other point of P. Facial expression recognition Page 106

107 Delaunay triangulation From the previous two properties A triangulation T of P is a DT(P) iff the circumcircle of any triangle of T does not contain a point of P in its interior. Facial expression recognition Page 107

108 Delaunay triangulation A triangulation T of P is legal iff T is a DT(P). DT Legal: Empty circle property and Thale s Theorem implies that all DT are legal Legal DT Let P be a set of points in the plane. Any angle-optimal triangulation of P is a Delaunay triangulation of P. Furthermore, any Delaunay triangulation of P maximizes the minimum angle over all triangulations of P. Facial expression recognition Page 108

109 Delaunay triangulation Therefore, the problem of finding a triangulation that maximizes the minimum angle is reduced to the problem of finding a Delaunay Triangulation. So how do we find the Delaunay Triangulation? Facial expression recognition Page 109

110 Delaunay triangulation How do we compute DT(P)? We could compute Vor(P) then dualize into DT(P). Instead, we will compute DT(P) using a randomized incremental method. Facial expression recognition Page 110

111 Delaunay triangulation Incremental method 1. Initialize triangulation T with a big enough helper bounding triangle that contains all points P. 2. Randomly choose a point p r from P. 3. Find the triangle that p r lies in. 4. Subdivide into smaller triangles that have p r as a vertex. 5. Flip edges until all edges are legal. 6. Repeat steps 2-5 until all points have been added to T. Facial expression recognition Page 111

112 Delaunay triangulation In matlab, try delaunay(x,y) function Facial expression recognition Page 112

113 Outline Introduction Facial expression recognition Appearance-based vs. model-based Active appearance model (AAM) Pre-requisite Principle component analysis Procrustes analysis ASM Delaunay triangulation AAM Facial expression recognition Page 113

114 AAM Steps of constructing AAM 1. Apply triangulation algorithm on the training faces; Facial expression recognition Page 114

115 AAM Steps of constructing AAM 2. Warp the training face to the mean face by matching the corresponding triangles Facial expression recognition Page 115

116 AAM Steps of constructing AAM 3. Sample the grey level information g im from the shape-normalized image over the region covered by the mean face To minimize the effect of global lighting variation, normalize the faces g = (g im β1)/α α = g im g, β = (g im 1)/n Facial expression recognition Page 116

117 AAM Steps of constructing AAM 4. Apply PCA to the normalized data g = g + P g b g Facial expression recognition Page 117

118 AAM Appearance vector: b T Wb s s WsPs ( x x) T g g ( ) b P g g Apply a further PCA b Qc c is a vector of appearance parameters We can express the face as functions of c x x PW Q c, g g P Q s s s g g c Q Q Q s g Facial expression recognition Page 118