Face2Face Comparing faces with applications Patrick Pérez. Inria, Rennes 2 Oct. 2014

Transcription

1 Face2Face Comparing faces with applications Patrick Pérez Inria, Rennes 2 Oct. 2014

2 Outline Metric learning for face comparison Expandable parts model and occlusions Face sets comparison Identity-based face search F. Jurie G. Sharma B. Bhattarai Example-guided face manipulation Automatic re-enactment I. Steiner C. Theobalt P. Garrido L. Valgaerts K. Vanarasi O. Rehmsen T. Thormaehlen 2

3 Metric learning for face comparison? [Sharma et al. ACCV 14]? [Sharma et al. 2014] [Bhattarai et al. ECCV-LBP 14] 3

4 Face analysis Central in a variety of contexts Security, surveillance, biometrics Smart environments Annotation, search and navigation Tasks (given detection) Recognition and authentication Grouping and clustering Verification and retrieval [Everingham et al.bmvc 06] [Ren CVPR 08] Fundamental problem: represent faces and compute similarities 4

5 Face representation Face-specific representation Layout of facial landmarks SIFT of facial landmarks Issue: robustness and accuracy of landmark detection (Almost) generic representation Fixed grid of overlapping blocks SIFT/HOG/LBP block description Fisher and DeepNet variants Landmarks still useful [Everingham et al. BMVC 06] [Everingham et al. IVC 2009] [Cinbis et al. ICCV 11] [Sivic et al. ICCV 09] [Simonyan et al. BMVC 13] [Chen et al. CVPR 13] 5

6 Face verification (identification) Given 2 face images: Same person? Persons unseen before Various types of supervision for learning Named faces (provide +/- pairs) Tracked faces (provide + pairs) Simultaneous faces (provide pairs) Labelled Faces in the Wild (LFW) +13,000 faces, +4,000 persons 10-fold testing with 300 +/- pairs per fold Restricted setting: only pair information for training Unrestricted setting: name information for training 6

7 Linear metric learning Powerful approach to face verification Learning Mahalanobis metric in input space Typical training data: +/- pairs should become close/distant Verification of new faces: Several approaches Large margin nearest neighbor (LMNN) [Weinberger et al. NIPS 06] Information theoretic metric learning (ITML) [Davis et al. ICML 07] Logistic Discriminant Metric Learning (LDML) [Guillaumin et al. ICCV 09] Pairwise Constrained Component Analysis (PCCA) [Mignon & Jurie, CVPR 12] 7

8 Learning in high dimension Very high dimension (in range 1, ,000) Prohibitive size of Mahalanobis matrix Scarcity of training data Low-rank Mahalanobis metric learning: Learn a linear projection (dim. reduction) Learn the metric Minimize loss over training set Rank fixed by cross-validation 8

9 Losses Probabilistic logistic loss Generalized logistic loss Hinge loss Extension to latent variables and multiple metrics 9

10 Expanded parts model [Sharma et al. CVPR 13]: Expanded parts model for human attributes and action recognition Objectives Avoid fixed layout Learn collection of discriminative parts and associated metrics Leverage the model to handle occlusions 10

11 Expanded parts model Mine n discriminative parts and learn associated metrics Dissimilarity based on comparing k < n best parts Learning Minimize hinge loss: greedy on parts + gradient descent on metrics Prune down to n a large set of N random parts Projections initialized by whitened PCA Stochastic gradient (SGD): given annotated pair 11

12 Experiments with occlusions LFW, unrestricted setting N = 500, n 50, k = 20, d = 20, 10 6 SGD iterations Random occlusions (20 80%) at test time, on one image only Focussed occlusions 12

13 Experiments with occlusions 13

14 Comparing face sets Given groups of single-person faces e.g., labelled clusters, face tracks Comparing sets Based on face pair comparison, i.e. For face tracks: a single descriptor per track [Parkhi et al. CVPR 14] [Everingham et al.bmvc 06] 14

15 Learning multiple metrics Metrics associated to k mined types of cross-pair variations Learning from annotated set pairs 15

16 Learning multiple metrics Stochastic gradient (SGD): given annotated pair Subsample the sets (to ensure variety of cross-pair variations) Dissimilarity: Sub-gradient of pair s hinge loss: if Projections initialized by whitened PCA computed on random subsets 16

17 New dataset From 8 different series (inc. Buffy, Dexter, MadMen, etc.) 400 high quality labelled face tracks, 23M faces, 94 actors Wide variety of poses, attributes, settings Ready for metric learning and test (700 pos., 7000 neg.) 17

18 Comparing face tracks Parameters: D 14000, k = 3, 10 6 SGD iterations Method Subspace dim. d Aver. Precision known persons Aver. Precision unknown persons PCA+ cosine sim + min-min PCA + cosine sim + min-min Metric Learning + min-min Latent ML (proposed) (3X)

19 Identity-based face search Use metric learning to organize face database hierarchically speed-up search Approach Dataset with sparse pair annotations Project on sub-space learned with metric learning on annotated set Cluster database in subspace Redo in each cluster Search Traverse the partition tree with query knn search with learned metrics in leaf node cluster 19

20 Identity-based face search 20

21 Clusters 16 clusters (dyadic tree of depth 4) 21

22 Search performance within LFW+1M distractors 4-depth dyadic tree d 0 = 256, d 3 = 32 3 faster 3-depth dyadic tree d 0 = 128, d 2 = 32 6 faster Baseline 0-depth, d = 32 22

23 Example-guided face manipulation [Garrido et al. CVPR 14] 23

24 Automatic reenactment Given source and target performances Transfer source face to target While preserving target performance source target Challenges Expressions and poses differ Timing and speech don t match Re-enactement should look plausible re-enactment 24

25 Method overview 25

26 Face tracking [Saragih et al. FG 11] Active Appearance Model (AAM) Fit to neutral face of interest Track 66 2D face landmarks 26

27 Face matching Match source/target faces under different head poses Break target sequence in stable segments For each segment, retrieve a source face LBP descriptions of 4 regions and motion of facial landmarks segment k segment k+1 27

28 Face transfer Landmarks of retrieved source faces sequence of new landmarks in target video (rigid + non-rigid smooth warps) Warping source faces and temporal morphing along landmark sequence 28

29 Final composite Poisson blending 29

30 Results YouTube videos 30

31 Results HD footages 31

32 Results HD footages 32