Image Set Classification Based on Synthetic Examples and Reverse Training

Transcription

1 Image Set Classification Based on Synthetic Examples and Reverse Training Qingjun Liang 1, Lin Zhang 1(&), Hongyu Li 1, and Jianwei Lu 1,2 1 School of Software Engineering, Tongji University, Shanghai, China {13_qingjunliang,cslinzhang,hyli, jwlu33}@tongji.edu.cn 2 The Advanced Institute of Translational Medicine, Tongji University, Shanghai, China Abstract. This paper explores a synthetic method to create the unseen face features in the database, thus achieving better performance of image set based face recognition. Image set based classification highly depend on the consistency and coverage of the poses and view point variations of a subject in gallery and probe sets. By considering the high symmetry of human faces, multiple synthetic instances are virtually generated to make up the missing parts, so as to enrich the variety of the database. With respect to the classification framework, we resort to reverse training due to its high efficiency and accuracy. Experiments are performed on benchmark datasets containing facial image sequences. Comparisons with state-of-the-art methods have corroborated the superiority of our Synthetic Examples based Reverse Training (SERT) approach. Keywords: Face recognition Image set classification 1 Introduction Face recognition conducted on multiple images can be formulated as an image set classification problem. The existing image set classification methods can be divided into two categories, parametric model based methods and nonparametric model based methods [1]. Parametric methods utilize a statistical distribution to represent an image set and measure the similarity between two sets by KL-divergence. The main drawback of such methods is that they need to tune the parameters of a distribution function and rely on strong statistical correlation between training and test image sets [2, 3]. Unlike parametric methods seeking for global characteristics of the sets, non-parametric methods put more emphasis on local samples matching. They attempt to find the overlap views between two sets and measure the similarity upon those parts of data. Nearest Neighbor (NN) matching is used to find the common parts. They model the whole image set as local exemplars [4], an affine hull or a convex hull [5], a regularized affine hull [6], or use the sparsity constraint as a means to find the nearest pair of points between two image sets [7]. Then the similarity of two sets can be reflected by the Euclidean distance between their closest points. Since the NN based methods use only a small part of the Springer International Publishing Switzerland 2015 D.-S. Huang and K. Han (Eds.): ICIC 2015, Part III, LNAI 9227, pp , DOI: / _30

2 Image Set Classification Based on Synthetic Examples 283 data information, they are more vulnerable to outliers. Later, people find that in some cases the structure of the whole image set might be a nonlinear complex manifold and a linear subspace is not sufficient for representation. Thus, researchers start to model an image set as a point on a certain manifold, e.g., a Grassmannian manifold [8, 9] ora Riemannian manifold [10]. The corresponding distance metrics can be geodesic distance [11], projection kernel metric [12] and Log-Euclidean distance (LED) [13]. In [14], Hayat et al. tried to keep each example independent and to remain the image set in its original form rather than seeking a whole representation. They argued that whatever form you use, once you model a set as a single entity, there must be loss of information. Besides, they adopted the reverse training strategy. The abovementioned methods mainly focus on devising an efficient classifier. They tacitly make an assumption that the distribution of a person s poses and view points in a probe image set are similar to those in the gallery image set. However, it is sometimes the case that there is pose or view point mismatch between the gallery and probe image sets of the same subject. In such case, the probe image set is more easily classified as the class whose gallery set contains the same head pose as the probe set but is indeed from a different subject. In this paper, to solve such a problem, we propose a simple yet effective approach by synthesizing more samples for each image set. In this way, the variety of poses and viewpoints within an image set can be apparently enriched. In terms of the classification framework, we resort to Reverse Training [14]. The proposed method is named as Synthetic Examples based Reverse Training, SERT for short. 2 Image Set Feature Extraction We propose a face sample synthesizing method in which the symmetry property of the human face is fully exploited. This approach is inspired by [14]. In [14], Hayat et al. pointed out that based on the manual inspection of the most challenging YouTube Celebrities dataset, a great amount of misclassified query image sets have a common characteristic that their head poses are not covered in the training sets. To address this issue, here we present our solution. 2.1 Horizontal Symmetry Synthetic Examples and LBP We create synthetic examples to enrich the set variations, by operating directly in data space. For each example in an image set, we flip the image horizontally and get another symmetry version of the original face. To determine the necessity of this flipping step, we use the Euclidean distance metric to measure the similarity between the original face and the flipped one. A threshold is empirically set. If the distance is less than the threshold, we neglect the flipped face since the original face itself has a good symmetry. Otherwise, we add the new flipped face column to the image set and therefore augment the number of instances in all the sets. Next, we use Local Binary Patterns (LBP) [15] for face feature extraction. It has three classical mapping table: (1) uniform LBP ( u2 ), (2) rotation-invariant LBP ( ri ),

3 284 Q. Liang et al. and (3) uniform rotation-invariant LBP ( riu2 ). Here we adopt the uniform LBP ( u2 ), whose binary pattern contains at most two bitwise transitions from 0 to 1 (or 1 to 0). There are totally 2 conditions for 0 transition and 56 conditions for 2 transitions (1 transition is impossible) in the case of (8, R) neighborhood. All the non-uniform LBP that contains more than two transitions are labeled as the 59 th bin. The details of feature extraction are listed in Table 1. Table 1. Feature extraction based on grid division LBP u2 8;1 Input: A face image 1. Divide the face image into k k non-overlapping uniformly spaced grid cells 2. For each pixel in one cell, sample its 8 neighbors with radius 1 and map its pattern into one of the 59 conditions 3. Build the histogram over each cell, which counts the frequency of each number (1 59) 4. Normalize the histograms and concatenate them one after another (either column-wise or row-wise) Output: A feature vector whose dimension is 59 k 2 Original face Flipped face Fig. 1. A synthetic feature and its original feature. Since we use grid division LBP u2 8;1 which is not rotation invariant, the flipped image must have a different LBP value from its original one. An intuitive illustration can be seen in Fig. 1. Imagine the case that a training set only composes of left profile faces, while its corresponding upcoming test set only consists of right profile faces. It is obviously that the original gallery and probe set is hard to match to each other. However, after we create synthetic examples, both gallery and probe set contains left and right profile features. It is much easier for the later classification. 2.2 SMOTE and PCA Whitening The number of instances varies a lot from set to set. Such an uneven distribution will lead to the bias in the classification stage especially for those methods who do not represent the image set as a whole entity. To solve this problem, here we use the

4 Image Set Classification Based on Synthetic Examples 285 Synthetic Minority Over-sampling Technique (SMOTE) proposed by Chawla et al. [16]. For the image sets whose sizes are smaller than 100, we generate synthetic examples by taking each minority instance and introducing synthetic ones along the line segments between itself and its k nearest neighbors in the same set. At the training phase, the LBP features are redundant since adjacent pixel intensities are highly correlated. Therefore, we use PCA whitening to make our input features uncorrelated with each other and have unit variance. 3 SERT: Synthetic Examples Based Reverse Training 3.1 Problem Formulation Denote X ={x 1, x 2,, x n } as an image set containing n face examples from a person, where x i is a feature vector of the i th single image, and is in the form of LBP. A subject can have multiple image sets. Given k training image sets X 1, X 2,, X k that belong to c classes (k >= c) and their corresponding labels y = {1, 2,, k}, when there comes in a query image set X q, our task is to find out which class it belongs to. 3.2 Reverse Training and the Proposed Framework After the preparation for features, we use the Reverse Training algorithm proposed in [14] to do the classification work. Suppose a coming query set X q has 200 images. The 20 training sets that belong to 20 classes (multiple sets per subject are combined as a whole) are marked as D ={X 1, X 2,, X 20 }. 10 images per set in D are randomly selected to form a set D 1 containing 200 images and the rest of images in D form the set D 2. As the name Reverse training suggests, we treat the 200 images in X q as training data while the images in D 2 as test data. Specifically, 200 features in X q are labeled as +1 and 200 features in D 1 are labeled as 1. A binary classifier Liblinear [17] is trained on these 400 instances and D 2 is tested on the linear decision boundary. Those who are classified as +1 (same side as X q ) are denoted as D þ 2. A normalized histogram h is computed on the y D þ (labels of 2 D þ 2 ) over the 20 class bins. Intuitively, h i (i =1,2, 20) indicates the percentage of the number of label i in y D þ over the number of label i in y D2. 2 h i ¼ X f y2y i ðþ= y X ( f D þ y2y i ðþ; y where f i ðyþ ¼ 1; y ¼ i ð1þ D2 0; y 6¼ i 2 Finally, the label of the query set X q is assigned according to h i that has the largest occurrence, y q ¼ arg max h i i ð2þ The flowchart of the proposed SERT approach is presented in Fig. 2.

5 286 Q. Liang et al. Offline Preparation Online Testing Phase gallery image sets X 1, X 2,, X c create synthetic examples for each set a query image set X q create synthetic examples gallery image sets X 1, X 2,, X c, SMOTE and PCA whitening a query image set X q, SMOTE and PCA whitening training feat. D ={X 1, X 2,, X c } divide into D 1 and D 2 a query image set X q train a linear classifier C on D 1 and X q test D 2 on C data sets D 1 and D 2 normalized histogram h return the bin index of maximum ratio the class label of query set X q Fig. 2. Illustration for the computation process of SERT. 4 Experimental Results 4.1 Datasets and Settings Honda/UCSD Dataset. The Honda/UCSD dataset [18] contains 59 video sequences involving 20 different persons. The face in each frame is first automatically extracted using Viola and Jones face detection algorithm [19] and then resized to the size of For our experiment, one video is considered as an image set. Specifically, each person has one image set as the gallery and the remaining sets as the probes. We repeat our experiment for 10 times with randomly selected training and testing combinations. CMU Mobo Dataset. The CMU Mobo dataset [20] consists of 96 video sequences of 24 different subjects. The number of frames for each video is about 300. Similar to the Honda, the faces are detected using [19] and resized to As a convention, we

6 Image Set Classification Based on Synthetic Examples 287 consider one video as an image set and select one set per person for training and the rest sets for testing (24 sets for training and 72 for testing). k = 5 for LBP grid division. YouTube Celebrities Dataset. The YouTube Celebrities [21] has 1910 video clips of 47 celebrities. We utilized the method in [22] to track the face region across the entire video, in which the face bounding boxes in initial frame is manually marked and provided along with the dataset. The cropped face region is then resized to Specifically, we divided the whole dataset into five equal folds with minimal overlapping. From the aspect of fold division, for subjects who have more than 45 videos, we randomly select 45 from them. As for subjects who don t have 45 videos, some videos are selected more than once. Then we divide 45 videos per person into 5 fold. 4.2 Comparisons with Existing Methods We compare our proposed framework with several recently proposed state-of-the-art methods which include DCC [1], MMD [4], MDA [8], AHISD [5], CHISD [5], SANP [7], CDL [10] and RT [14]. Table 2 tabulates the recognition results for our approach and all the other methods listed above on the three datasets. Table 2. Average recognition rates (%) with standard deviation of different methods on the three benchmark datasets Method Honda/UCSD CMU Mobo YouTube DCC [1] 92.6 ± ± ± 2.1 MMD [4] 92.1 ± ± ± 1.8 MDA [8] 94.4 ± ± ± 1.1 AHISD [5] 91.3 ± ± ± 2.4 CHISD [5] 93.6 ± ± ± 2.9 SANP [7] 95.1 ± ± ± 2.4 CDL [10] 98.9 ± ± ± 3.3 RT [14] 100 ± ± ± 2.0 SERT 100 ± ± ± Conclusions In this paper, we examined the value of using synthetic examples combined with reverse training, namely SERT, to increase the recognition rate of set based face recognition. SERT is simple in concept and can be implemented easily. Experimental results indicate that SERT could yield better performance than the other competitors. References 1. Kim, T.K., Kittler, J., Cipolla, R.: Discriminative learning and recognition of image set classes using canonical correlations. IEEE PAMI 29, (2007)

7 288 Q. Liang et al. 2. Arandjelovic, O., Shakhnarovich, G., Fisher, J., Cipolla, R., Darrell, T.: Face recognition with image sets using manifold density divergence. In: CVPR, pp (2005) 3. Shakhnarovich, G., Fisher III, J.W., Darrell, T.: Face recognition from long-term observations. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part III. LNCS, vol. 2352, pp Springer, Heidelberg (2002) 4. Wang, R., Shan, S., Chen, X., Gao, W.: Manifold-manifold distance with application to face recognition based on image set. In: CVPR, pp. 1 8 (2008) 5. Cevikalp, H., Triggs, B.: Face recognition based on image sets. In: CVPR, pp (2010) 6. Yang, M., Zhu, P., Gool, L., Zhang, L.: Face recognition based on regularized nearest points between image sets. In: IEEE FG, pp. 1 7 (2013) 7. Hu, Y., Mian, A.S., Owens, R.: Sparse approximated nearest points for image set classification. In: CVPR, pp (2011) 8. Wang, R., Chen, X.: Manifold discriminant analysis. In: CVPR, pp (2009) 9. Harandi, M.T., Sanderson, C., Shirazi, S., Lovell, B.C.: Graph embedding discriminant analysis on grassmannian manifolds for improved image set matching. In: CVPR, pp (2011) 10. Wang, R., Guo, H., Davis, Larry S., Dai, Q.: Covariance discriminative learning: a natural and efficient approach to image set classification. In: CVPR, pp (2012) 11. Turaga, P., Veeraraghavan, A., Srivastava, A., Chellappa, R.: Statistical computations on grassmann and stiefel manifolds for image and video-based recognition. IEEE PAMI 33, (2011) 12. Hamm, J., Lee, D.: Grassmann discriminant analysis: a unifying view on subspace-based learning. In: ICML, pp (2008) 13. Harandi, M.T., Sanderson, C., Wiliem, A., Lovell, B.C.: Kernel analysis over riemannian manifolds for visual recognition of actions, pedestrians and textures. In: WACV, pp (2012) 14. Hayat, M., Bennamoun, M., An, S.: Reverse training: an efficient approach for image set classification. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp Springer, Heidelberg (2014) 15. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE PAMI 24, (2002) 16. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Reverse training: an efficient approach for image set classification. J. Artif. Intell. Res. 16, (2002) 17. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, (2008) 18. Lee, K., Ho, J., Yang, M., Kriegman, D.: Video based face recognition using probabilistic appearance manifolds. In: CVPR, pp (2003) 19. Viola, P., Jones, M.J.: Robust real-time face detection. IJCV 57, (2004) 20. Gross, R., Shi, J.: The CMU motion of body (mobo) database. Technical report CMU-RI-TR (2001) 21. Kim, M., Kumar, S., Pavlovic, V., Rowley, H.: Face tracking and recognition with visual constraints in real-world videos. In: CVPR, pp. 1 8 (2008) 22. Ross, D.A., Lim, J., Lin, R., Yang, M.: Incremental learning for robust visual tracking. IJCV 77, (2008)