Face Detection and Alignment. Prof. Xin Yang HUST

Transcription

1 Face Detection and Alignment Prof. Xin Yang HUST

2 Applications Virtual Makeup

3 Applications Virtual Makeup

4 Applications Video Editing

5 Active Shape Models Suppose we have a statistical shape model Trained from sets of examples How do we use it to interpret new images? Use an Active Shape Model Iterative method of matching model to image

6 Building Models Require labelled training images landmarks represent correspondences

7 Building Shape Models Given aligned shapes, { } x i Apply PCA x x Pb P First t eigenvectors of covar. matrix b Shape model parameters

8 PCA: General From k original variables: x 1,x 2,...,x k : Produce k new variables: y 1,y 2,...,y k : y 1 = a 11 x 1 + a 12 x a 1k x k y 2 = a 21 x 1 + a 22 x a 2k x k... y k = a k1 x 1 + a k2 x a kk x k such that: y k 's are uncorrelated (orthogonal) y 1 explains as much as possible of original variance in data set y 2 explains as much as possible of remaining variance etc.

9 5 4 2nd Principal Component, y 2 1st Principal Component, y

10 PCA Scores 5 x i2 4 y i,1 y i, x i1

11 PCA Eigenvalues 5 λ 1 λ

12 PCA: Another Explanation From k original variables: x 1,x 2,...,x k : Produce k new variables: y 1,y 2,...,y k : y 1 = a 11 x 1 + a 12 x a 1k x k y 2 = a 21 x 1 + a 22 x a 2k x k... y k = a k1 x 1 + a k2 x a kk x k y k 's are Principal Components such that: y k 's are uncorrelated (orthogonal) y 1 explains as much as possible of original variance in data set y 2 explains as much as possible of remaining variance etc.

13 Principal Components Analysis on: Covariance Matrix: Variables must be in same units Emphasizes variables with most variance

14 PCA: General {a 11,a 12,...,a 1k } is 1st Eigenvector of correlation/covariance matrix, and coefficients of first principal component {a 21,a 22,...,a 2k } is 2nd Eigenvector of correlation/covariance matrix, and coefficients of 2nd principal component {a k1,a k2,...,a kk } is kth Eigenvector of correlation/covariance matrix, and coefficients of kth principal component

15 Building Shape Models Given aligned shapes, { } x i Apply PCA x x Pb P First t eigenvectors of covar. matrix b Shape model parameters

16 Hand Shape Model Varying b Varying b Varying b 1 2 3

17 BUILDING THE MODELS All Faces Modes of Variation First mode Second mode Third mode

18 BUILDING THE MODELS Neutral - Appearance Variation Changing the first three modes of variation simultaneously

19 Active Shape Models Match shape model to new image Require: Statistical shape model Model of image structure at each point Model Point Model of Profile

20 Placing model in image The model points are defined in a model coordinate frame Must apply global transformation,t, to place in image x x Pb T ( x; X, Y, s, ) c c X T ( x Pb ) Model Frame Image

21 ASM Search Overview Local optimisation Initialise near target Search along profiles for best match,x Update parameters to match to X. ( X i, Yi )

22 Local Structure Models Need to search for local match for each point Model Strongest edge Statistical model of profile

23 Computing Normal to Boundary Tangent Normal ), ( ), ( x y y x t t n n ), ( y t x t ), ( 1 1 i X i Y ), ( 1 1 i X i Y 2 2 ), ( ), ( y x y x y x d d d d t t i i y i i x Y Y d X X d (Unit vector)

24 Sampling along profiles Model boundary Profile normal to boundary Model point ( X, Y ) Interpolate at these points ( X, Y ) i( s n, s n i... 2, 1,0,1,2,... n x n y ) s n Tak e step s of length s n alo ng ( n x, n y ) s n n y s n n x

25 Noise reduction In noisy images, average orthogonal to profile Improves signal-to-noise along profile g i1 g i2 g i3 U se g i 0.25g i1 0.5 g i g i 3 Sampled profile is g (..., g -2, g -1, g 0, g 1, g 2,...)

26 Searching for strong edges g ( x) dg ( x) dx x dg( x) dx 0.5( g ( x 1) g ( x 1)) Select point along profile at strongest edge

27 Profile Models Sometimes true point not on strongest edge Strongest edge True position Model local structure to help locate the point

28 Statistical Profile Models Estimate p.d.f. for sample on profile Normalise to allow for global lighting variations From training set learn g p(g)

29 Profile Models For each point in model For each training image Sample values along profile Normalise Build statistical model eg Gaussian PDF using eigen-model approach

30 Searching Along Profiles During search we look along a normal for the best match for each profile g ( x) p ( g ( x )) g( x) x Form vector from samples about x

31 Search algorithm Search along profile Update global transformation, T, and parameters, b, to minimise X T ( x Pb ) 2 ( X i, Yi )

32 Updating parameters Find pose and model parameters to minimise Either Put into general optimiser Use two stage iterative approach 2 ),,, ; ( ),,,, ( s Y X T s Y X f c c c c Pb x X b

33 Updating Parameters 2 ),,, ; ( ),,,, ( s Y X T s Y X f c c c c Pb x X b 2 ) ( minimise w hich ),,, ( find and F ix Pb x X b T s Y X c c 2 ) ( minimises w hich find and ),,, ( F ix Pb x X b T s Y X c c ) ) ( ( 1 x X P b T T Repeat until convergence: Analytic solution exists (see notes)

34 Update step Hard constraints M inimise X T ( x Pb ) 2 sub ject to p( b ) p t e.g. b 3 i λ i

35 Multi-Resolution Search Train models at each level of pyramid Gaussian pyramid with step size 2 Use same points but different local models Start search at coarse resolution Refine at finer resolution

36 Gaussian Pyramids To generate image at level L Smooth image at level L-1 with gaussian filter (eg ( )/20) Sub-sample every other pixel Each level half the size of the one below

37 Multi-Resolution Search Start at coarse resolution For each resolution Search along profiles for best matches Update parameters to fit matches (Apply constraints to parameters) Until converge at this resolution

38 ASM Fitting Process

39 ASM Fitting Process

40 Face Tracking Framework: Active Face Detection Shape Model Start Shape on the ROI Feature Extraction & Search Shape Realignment on the ROI Output the Shape Landmarks of Active Shape Model

41 Algorithm Face Detection Start Shape on the ROI Feature Extraction & Search Shape Realignment on the ROI Output the Shape

42 Algorithm Face Detection Start Shape on the ROI Feature Extraction & Search Shape Realignment on the ROI Output the Shape

43 Algorithm Face Detection Start Shape on the ROI Feature Extraction & Search Shape Realignment on the ROI Output the Shape

44 Algorithm Face Detection Start Shape on the ROI Feature Extraction & Search Shape Realignment on the ROI Output the Shape

45 Algorithm Face Detection Start Shape on the ROI Feature Extraction & Search Shape Realignment on the ROI Output the Shape

46 Algorithm Face Detection Start Shape on the ROI replace Feature Extraction & Search Shape Realignment on the ROI Output the Shape

47 Experiment Data: 1000 frames, 800x600 (hy1.avi) Intel(R) Core(TM) i GHz LDB-ASM: s (64fps) HAT-ASM: s (15fps) Mobile Platform (Qualcomm 8974) LDB-ASM: 00:01:22m (12fps) HAT-ASM: 00:08:13m (2fps)

48 LDB-ASM HAT-ASM

49 LDB-ASM HAT-ASM STD-X STD-X STD-Y STD-Y

50 Active Shape Models Advantages Fast, simple, accurate Efficient to extend to 3D Disadvantages Only sparse use of image information Treat local models as independent

51 Cascade Shape Regression Framework Stage t = 0 t = 3 t = 5 Cascaded pose regression, Dollar et. al., CVPR 2010

52 Face Alignment based on LBF-Regression Tree Induced Local Binary Features learned from data global optimization much stronger than previous regression trees efficient training / testing Best accuracy on challenging benchmarks 3,000 FPS on desktop, or 300 FPS on mobile first face tracking method on mobile

53 Face Alignment based on LBF-Regression Shape-index Feature

54 Face Alignment based on LBF-Regression A simple form sum of a large number of regression trees Novel two step learning 1. Local learning of tree structure learn an easier task and better features 2. Global optimization of tree output enforce dependence between points and reduce local estimation errors

55 Local Learning of Tree Structure Target: one point Random forest learn standard random forests for each local point standard regression tree using pixel difference features only use pixels in the local patch around the point regularization of feature selection

56 From Local to Global Target: one point Random forest Fix tree structures and optimize tree leave s output

57 Global Optimization of Tree Output Regression Target Feature Mapping Function

58 Global Optimization of Tree Output

59 Tree Induced Binary Features Each leave is a binary indicator function 1 if the image sample arrives at the leaf 0 otherwise Trees -> high dimension sparse binary features Learning global linear regression

60 Experiments Benchmark #landmarks #training images #testing images LFPW Helen W Two variants of our method Accurate: LBF 1200 trees with depth 7 Fast: LBF fast 300 trees with depth 5

61 LFPW (29 landmarks) Method Error FPS [1] 3.99 ESR [2] RCPR [3] SDM [4] EGM [5] 3.98 LBF LBF fast Helen (194 landmarks) Method Error FPS STASM [6] CompASM [7] ESR [2] PCPR [3] SDM [4] LBF LBF fast W (68 landmarks) Method Fullset Common Subset Challenging Subset FPS ESR [2] SDM [4] LBF LBF fast LBF is much more accurate and a few times faster LBF fast is slightly more accurate and dozens of times faster

62 Face Alignment at 3000 FPS

63 Summary State-of-the-art face alignment Best accuracy on challenging benchmarks Dozens of times faster than previous methods faster than real time face tracking on mobile