Face Recognition by Deep Learning - The Imbalance Problem

Transcription

1 Face Recognition by Deep Learning - The Imbalance Problem Chen-Change LOY MMLAB The Chinese University of Hong Kong Homepage: Twitter: CVPR Workshop on Biometrics 2018, Salt Lake City, Utah, USA, June 2018

2 WIDER Challenge 2018 WIDER Face Detection WIDER Pedestrian Detection WIDER Person Search Event Deadline: July 18!!

3 Outline Pose-Robust Face Recognition via Deep Residual Equivariant Mapping K. Cao, Y. Rong, C. Li, X. Tang, C. C. Loy CVPR 2018 Deep Imbalanced Learning for Face Recognition and Attribute Prediction H. Chen, Y. Li, C. C. Loy, X. Tang An extension of our paper in CVPR 2016, submitted to TPAMI

4 iphone X opens an era of unlocking mobile phone by face replacing fingerprint ID l 1/1,000,000 error rate l 1 billion photos for data training l 3D liveness detection

5 Vivo X20 Face Wake: unlock your mobile phone in 0.1 seconds

6

7 Captured 25 criminal suspects during the 2017 Cultural Fair in Shenzhen during 5 days

8 2015 Yang et al., From Facial Part Responses to Face Detection: A Deep Learning Approach, ICCV 2015

9 2017 Zhang et al., S 3FD: Single Shot Scale-invariant Face Detector, ICCV 2017

10 Is there anything else I can solve?

11 Is there anything else I can solve? Scaling to large N in 1:N scenario Learning in small data regime The use of unannotated data Challenging scenarios Generalization and transferability Imbalance problem

12 Pose-Robust Face Recognition via Deep Residual Equivariant Mapping K. Cao, Y. Rong, C. Li, X. Tang, C. C. Loy CVPR 2018 Wednesday 12:30-2:50

13 Profile and Frontal Face Recognition Large pose discrepancy between two face images is one of the key challenges in face recognition The number of frontal and profile training faces are highly imbalanced Profile faces of different persons are easily to be mismatched (false positives), and profile and frontal faces of the same identity may not trigger a match leading to false negatives

14 Why does not face recognition work well on profile faces? The generalization power of deep models is usually proportional to the training data size Given an uneven distribution of profile and frontal faces in the dataset, deeply learned features tend to bias on distinguishing frontal faces rather than profile faces.

15 Existing solutions I. Masi, S. Rawls, G. Medioni, and P. Natarajan. Pose-aware face recognition in the wild. In CVPR, 2016

16 Existing solutions Y. Taigman et al. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014

17 Existing solutions Zhu et al. High-Fidelity Pose and Expression Normalization for Face Recognition in the Wild, CVPR 2015

18 Existing solutions Model Input Generated Real L. Tran, X. Yin, and X. Liu. Disentangled representation learning GAN for pose-invariant face recognition. In CVPR, 2017

19 Motivation We can map profile face feature to the frontal space through a mapping function that adds residual.

20 Feature equivariance The representation of many deep layers depends upon transformations of the input image Such transformations can be learned by a mapping function from data The function can be subsequently applied to manipulate the representation of an input image to achieve the desired transformation K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In CVPR, 2015

21 Feature equivariance A convolutional neural network (CNN) can be regarded as a function! that maps an image " $ to a vector!(") ' ( The representation! is said equivariant with a transformation ) of the input image if the transformation can be transferred to the representation output " $:!()") -.!(") K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In CVPR, 2015

22 Problem formulation For simplicity, let s assume we have: frontal face image! " and profile face image! # We wish to obtain a transformed representation of a profile image! # through a mapping function $ %, so that $ % &(! # ) &(! " ) $ % &(! # ) = &(! # ) +,(! # )R(! # ) &(! " ) yaw coefficient, [0 1], a soft gate of the residuals residual function

23 Problem formulation Yaw coefficient provide a higher magnitude of residuals (thus a heavier fix) to a face that deviates more from the frontal pose! " = 0 for frontal face and gradually changes from 0 to 1 when the face pose shifts from frontal to a complete profile The soft gate can be viewed as a correction mechanism that adopts top-down information (the yaw in our case) to influence the feed-forward process

24 Network structure the DREAM block The Deep Residual EquivAriant Mapping (DREAM) block

25 Usage of DREAM Stitching Stitch the DREAM block to an existing stem CNN End-to-end + Stitching First end-to-end training Followed by DREAM block fine-tuning DREAM block training

26 Visualization

27 Visualization

28 Results on Celebrities in Frontal-Profile (CFP) Equal error rate (EER). Baselines CDFE - Two transforms are simultaneously learned to map the samples in two modalities respectively to the common feature space. JB Joint Bayesian approach for face verification FF - Face Frontalization morphs faces from profile to frontal with a generative adversarial network S. Sengupta et al. Frontal to profile face verification in the wild. In WACV, 2016

29 Results on IJB-A

30 Further analysis With DREAM, similarity decreases for unmatched pairs With DREAM, similarity increases for matched pairs

31 Summary Equivariant mapping in the deep feature space Performing frontalization in the feature space is more fruitful than the image space Easy to use, light-weight, and can be implemented with a negligible computational overhead Code is available:

32 Deep Imbalanced Learning for Face Recognition and Attribute Prediction H. Chen, Y. Li, C. C. Loy, X. Tang An extension of CVPR 2016, submitted to TPAMI

33 Other types of imbalance Face recognition Frontal faces are more than profile faces Some celebrities have more faces Face attribute recognition More negative than positive, e.g., no beard vs. beard

34 CelebA face attributes dataset 200K celebrity images, each with 40 attribute Liu et al. Deep Learning Face Attributes in the Wild, ICCV hk/projects/celeba.html

35 CelebA face attributes dataset

36 The fundamental problem Without handling imbalanced class issue Prediction biases toward the majority class Poor accuracy for the minority class CelebA positive/negative distribution

37 Existing solutions Class re-sampling [Drummond & Holte, ICML 03] Random under-sampling of majority class Remove valuable information Random over-sampling of minority class Introduce artificial noise Cost-sensitive learning [Zadrozny et al., ICDM 03] Assigns higher misclassification costs to the minority class How to design costs?

38 Motivation Is there a better way apart from sampling and cost learning? Can we introduce tighter constrains to ameliorate such invasion? Minority class: very few instances with high degree of visual variability The genuine neighborhood of these instances is easy to be invaded by other imposter nearest neighbors

39 Triplet loss helps to a certain extent Class-level constraint! " an anchor! " # a positive instance (of the same class)! " $ a negative instance (different class) Class 1 minority Class 2 majority x p i x i x n i Wearing hat Not wearing hat

40 D feature embedding of one imbalanced binary face attribute Triplet loss helps to a certain extent Class 1: cluster 1 Class 1: cluster 2 Class 2: cluster 1 Class 2: cluster 2 Class 2: cluster 3 Class 2: cluster 4 Class 2: cluster 5 PC2-10 PC1-5 NC5 0 NC4 NC3 5 NC2 10 NC Features extracted from DeepID2 model Triplet embedding NC1 10 NC2 NC3 5 NC4 0 NC5 PC PC

41 Large margin local embedding (CVPR 2016) Learning deep feature embedding for imbalanced data classification A new method that preserves locality across clusters and discrimination between classes C. Huang, Y. Li, C. C. Loy, and X. Tang, Learning deep representation for imbalanced classification, in CVPR, 2016

42 6-3 Our solution compared to triplet loss PC2 Class 1: cluster 1: cluster 2: cluster Class 2: cluster NC3 5Class 2: cluster NC2 Class 2: cluster 10 NC1 Class 2: cluster PC1-5 Class NC5 0Class NC D 10feature embedding of one imbalanced binary face attribute NC1 NC1 NC NC2 NC2 NC2 3 5NC3 5 NC3 NC NC4 NC4 NC NC5 NC5 0 NC PC1 PC1 PC1-2 PC2 PC2 PC Features extracted from DeepID2 model Triplet embedding Our solution

43 Quintuplet sampling Cluster- and class-level! " an anchor! #$ " the anchor s most distant withincluster neighbor! #% " the nearest within-class neighbor of the anchor, but from a different cluster! #%% " the most distant within-class neighbor of the anchor! & " the nearest between-class neighbor of the anchor Class 1 minority x p i x p i Cluster 1 Cluster 2 p+ xi x i x n i Class 2 majority Cluster j Cluster 1

44 Quintuplet sampling Ensure the following relationship <D(f(x i ),f(x n i )) > Class 1 minority Class 2 majority Cluster j D(f(x i ),f(x p i )) > x p i Cluster 2 D(f(x i ),f(x p i )) D(f(x i ),f(x p+ i )) > x p i Cluster 1 p+ xi x i x n i Cluster 1 D(f(x i ),f(x j )) = kf(x i ) f(x j )k 2 2 is the Euclidean distance

45 Large Margin Local Embedding Training samples Minibatches x i x p+ i x p i x p i x n i Shared parameters CNN CNN CNN CNN CNN f(x i ) f(x p+ i ) f(x p i ) f(x p i ) f(x n i ) Triple-header hinge loss Quintuplet Embedding

46 Triple-header hinge loss To constrain three margins between the four distances min X i (" i + i + i )+ kwk 2 2 s.t.: max 0,g 1 + D(f(x i ),f(x p+ i )) D(f(x i ),f(x p i )) apple " i max 0,g 2 + D(f(x i ),f(x p i )) D(f(x i ),f(x p i )) apple i max 0,g 3 + D(f(x i ),f(x p i )) D(f(x i ),f(x n i )) apple i 8i, " i 0, i 0, i 0

47 Advantages Richer information and a stronger constraint than the conventional class-level image similarity No information loss unlike under-sampling No artificial noise unlike over-sampling

48 CelebA dataset (100k train,10k test) Relative accuracy gain (%) Over Anet [28] Over PANDA [46] Over Triplet-kNN [33] Code available ojects/lmle.html Class imbalance level (%) More imbalanced Face attribute

49 Disadvantages Exponential growth of quintuplet number Potential penalty inconsistency between sampled quintuplets Enforcing Euclidean distance on a hypersphere manifold is not natural

50 Cluster-based large margin local embedding FaceNet (CVPR 2015) SphereFace (CVPR 2017) LMLE (CVPR 2016) CLMLE (Our Method)

51 Cluster-based large margin local embedding Maximize the intra-cluster similarity between!(# $ ) and its containing cluster centroid & ' Minimizing the inter-cluster similarity between!(# $ ) and other cluster centroids {& ) : + -} Larger angular margin for those between-class clusters than for same class clusters Consider class label c(.)

52 Cluster-based Large Margin Local Embedding & = number of clusters! " = cluster centroid # $ = inter-class angular margin # % = intra-class angular margin

53 Steps of CLMLE 1. Cluster for each class by spherical k-means using the latest features. Initially, we use the features extracted by the pre-trained CNN. 2. For CNN training, repeatedly construct mini- batches with one query cluster (with highest loss) from a random class, and nearest clusters retrieved from both the same and different classes. 3. Randomly sample some samples for each of the clusters in batch, and compute their cluster centroids. 4. Compute the loss with cost-sensitivities (by inverse class frequency). Backpropagate the gradients to update the CNN parameters and feature embeddings. 5. Alternate between step 1 and 2-4 periodically until convergence (often within 5 alternation rounds).

54 Cluster-based Large Margin Local Embedding

55 Results on LFW and YTF Use large-scale outside data that are not publicly available Recent imbalanced learning methods The state-of-the-art loss functions Use the small training data (CASIA-WebFace: 0.45M) and the same 64-layer CNN model as SphereFace and CosFace for fair comparison.

56 Results on MegaFace Challenge 1

57 Results on MegaFace Challenge 2

58 Facial attribute recognition

59 Face attribute recognition

60 Is there anything else I can solve? Scaling to large N in 1:N scenario Learning in small data regime The use of unannotated data Challenging scenarios Generalization and transferability Imbalance problem

61 Thanks