Implementation of a Face Detection and Recognition System

Transcription

1 Implementation of a Face Detection and Recognition System Sanghyun Choi Johns Hopkins University 3400 N. Charles Street Baltimore, MD 21218, USA schoi60@jhu.edu Michael Flynn Johns Hopkins University 3400 N. Charles Street Baltimore, MD 21218, USA mflynn19@jhu.edu Abstract This paper details the implementation of a computer vision system for face detection and face recognition. For face detection, Viola- Jones algorithm is used for binary classification to indicate whether or not a face exists in a given image, and for face recognition the Eigenfaces algorithm is implemented for feature learning and dimensionality reduction, employing a k-nearest neighbors classifier is used for multi-class classification. This paper details these algorithms and their implementations, and results are analyzed to evaluate the systems s performance. 1 Introduction With the advent of robust machine learning techniques, the study of face detection and recognition in images has become an increasingly popular field in computer vision, especially because it has a wide range of applications. For example, it enables social media sites to automatically identify the people present in a photo (and make suggestions accordingly) and it also improves everyday security by aiding the analysis of surveillance videos. There are still many challenges that need to be addressed in this field because no single algorithm is perfectly robust to all of the factors required to detect and recognizes faces. For example, Osuna (1997) applied support vector machines, a learning algorithm that finds the maximal margin to separate different classes of data, to face detection. However, using raw image pixels as feature vectors gave rise to extremely high dimensionality, making computations on these vectors cumbersome. Moreover, training itself was an intractable task since there were just too many non-face images that had to be provided for the classification to become accurate. Factors such as lighting condition, poses, and facial expressions also make face detection and recognition a challenging task. Hence, significant amount of effort is required to construct a feature that represents an image in such a way that it guarantees high performance in terms of computational speed and classification accuracy. Further more, even if the right features are found, there is yet another challenge to construct an appropriate classifier. Here we present our implementation of Viola and Jones (2004) algorithm, which revolutionized the field of face detection, and a system for recognizing faces in images based on a lower dimensional feature learning technique called principal component analysis combined with k-nearest neighbors classifier. The Viola and Jones algorithm is implemented because it is a state-of-the-art algorithm that addresses many of the difficulties mentioned above. From a set of features that are very quick to evaluate, it automatically chooses the best features and, based on those features, constructs a classifier that can classify new images in real-time. The stateof-the-art for face recognition would be algorithms involving deep neural networks, such as the work done by Wolf (2014) for Facebook, but this is beyond our scope and we resort to simpler, yet robust, machine learning techniques used by Turk and Pentland (1991). The algorithms are covered in more

2 depth in section 2. 2 Algorithms Overview 2.1 Viola and Jones The key concept behind the algorithm presented by Viola and Jones for face detection is a learning technique called AdaBoost (i.e. shorthand for Adaptive Boosting). The reason why boosting is suitable for face detection is because it automatically selects the best features that help distinguish faces from nonfaces. Conceptually, boosting trains a set of weak learners (i.e. learners that are slightly better than random decision generators) to produce a classifier that consists of a weighted ensemble of those weak learners. The training is performed in multiple stages where at each stage a misclassified weak learner is given more weight so that during the next stage, more focus is given to this weak leaner in order to make it stronger. In the context of Viola and Jones algorithm, weak learners are defined as Haar features. They are rectangular patches divided into different orientations of white and black regions as depicted in Figure 1. Evaluation of these features simply involve the summation of pixel intensity values in the white region subtracted by that of black region. Figure 1: Four types of Haar features used in Viola and Jones algorithm. Adapted from Viola and Jones (2004). The significance of these features lies in the fact that human faces are characterized by certain regions of varying intensity. For example, usually the eye area is shadowed, meaning it is darker compared to other area, so Haar feature type B in Figure 1 would capture this characteristic if black region is placed in the eye area. Viola and Jones algorithm trains these rectangular features of varying sizes and orientations to output the most defining features for human faces. What is novel about this algorithm is the highly efficient evaluation of these features. A naive summation and subtraction strategy for image intensity values would require enormous amount of computation because this is performed for every sub-patch of an image. To resolve this, Viola and Jones introduced the concept of integral images. Given an image, computing its integral image involves assigning each pixel a cumulative sum of intensity values for the rectangular region formed by the origin (top-left corner) and the current pixel. With this readily computed information, feature evaluation could be performed in a few arithmetic steps (i.e. some additions and subtractions of these cumulative sums). 2.2 Principal Component Analysis Oftentimes the underlying function that machine learning tries to uncover can be more efficiently represented in a subspace of the data. Principal component analysis (PCA) is a statistical unsupervised learning technique that aims to find an optimal projection of the data such that its variance is maximized. Because PCA is finding an optimal subspace of the data, the projected data will require fewer dimensions to represent it that the original data, and because of this PCA is often used a dimensionality reduction technique to find more compact representations of data. For images, which represent incomprehensibly complex functions and fall prey to the curse of dimensionality problem because of their size, PCA can assist in object recognition by finding an optimal subspace to represent that object. By reducing the dimensionality of the image and transforming it into a compressed representation that still encodes the underlying information about the image, common machine learning algorithms such as k-nearest neighbors or support vector machines can be applied to the image that would otherwise be unfeasible.

3 maximize U subject to Trace(U T C xx U) U T U = I (1) PCA is first posed as a question of how to maximize the variance in a projected subspace, as shown in equation 1, where C xx is the covariance matrix of X. Solving for this optimization problem reveals that the desired U that will maximize the subspace are the eigenvectors of XX T (also known as the left singular values of X or its principal components), which are given by the singular value decomposition (SVD) X = USV T. Therefore, by stacking features for different samples into a data matrix and finding its left singular values, samples can be projected into the subspace using X = U T X. This is the underlying concept that fuels Eigenfaces, one of the earliest face recognition algorithms that was able to achieve staggering recognition accuracies of up to 90% in certain situations (Turk, 1991). Similar to what was described earlier, the algorithm works by first unrolling training images into one long feature vector and stacking the resulting vectors on top on one another into a data matrix. Next, SVD is performed on the data matrix to find its principal components, which is finally used to transform both training and test data before feeding it into another algorithm which will work with this new representation instead of working with the raw data directly Kernel PCA Because PCA is a linear method, it cannot effectively find the subspace to represent nonlinear relationships in the data. However, it can easily be augmented to accommodate for nonlinear data by applying the kernel trick. This trick is a way to work with data in high-dimensional space created by a kernel function that maps data in this space, but without ever explicitly computing this high-dimensional space. This is done by computing a Gram kernel matrix K that represents the result of the kernel function between all combinations of samples, and working with respect to this matrix instead. One such example of a kernel function that can be used to build a Gram matrix is the radial basis function (RBF), which represents an infinite-dimensional Gaussian distribution of the inner product between two feature vectors, showed in equation 2. ( x ) y 2 RBF(x, y) = exp 2σ 2 (2) To apply this trick to PCA, the kernel matrix of data is computed first before SVD is ran, and the principal components of this matrix is found instead. To project data onto these components, a kernel matrix between the data to project and the data that was used to find the principal components is found, which is then used to project onto the principal components. It should be noted that because it is required to compute a kernel matrix before projecting data that the complexity of this algorithm is in terms of the number of samples used to find the principal components, which makes this algorithm intractable for large training sets. 2.3 k-nearest Neighbors One of the most simplest, yet powerful supervised learning algorithms is k-nearest neighbors (knn), which works by taking a sample, finding k of its nearest neighbors using some similarity metric such as the l 2 -distance, and returning the label that is most common among its neighbors. While easy to implement, the algorithm represents a powerful nonlinear function class, allowing it to be a good first-resort when approaching many new machine learning problems. However, knn fails when tacking high-dimensional data, as the complexity of feature vectors explode which makes it more difficult to get a reliable distance metric. However, since PCA will be used to transform the data into a lowerdimensional subspace, knn will be a suitable algorithm for classifying images for face recognition. 3 Implementation The face detection and recognition system is implemented in Python and is hosted in public GitHub repository Face Detection The design of the system takes the form given in Figure 2. It is designed so that different models (other than Viola and Jones algorithm) that could be implemented in the future can be readily adapted to 1

4 FaceDetector.py which is responsible for holding the detection model. line interface. Upon executing the driver file, a designated percentage of training data is fitted into the Viola and Jones model. When training is done, the threshold for classification is decreased until desired accuracy is achieved with respect to tuning data. Then the classification accuracy is computed for test data. The outcome of executing this file is a userfriendly report of the resulting accuracy and a short description of learned Haar features as shown in Figure 3. Figure 2: Design of the face detection system. The core modules comprise of ViolaJones- Model.py and HaarFeature.py. The former code implements the boosting algorithm while the latter implements the four types of features depicted in Figure 1. Implementing the Haar features requires numerous trial and error tests on simple synthetic data because it involves manipulations of different pixel coordinates within the image. To speed up the evaluation of Haar features using integral images as described in section 2.1, OpenCV 2, which is the most widely used computer vision open source library, is used to compute the integral values. To implement Viola and Jones algorithm, multiple external sources have to be referenced for details since the original paper does not give clear instructions on how to implement some parts of the algorithm. For example, the criterion for setting the value of polarity, which dictates the direction of inequality in Haar features, could only be determined by referencing an external source. To perform face detection on the data set, a driver file called FaceDet.py is provided. It takes input parameters such as type of data set (e.g. MIT Face Data), data directory, percentage of data allocated for training/tuning/testing, number of boosting iterations, and the minimum desired accuracy for face detection (at the cost of false positives, which is explained in section 4) from the user via command- 2 Figure 3: Sample Report of Face Detection System 3.2 Face Recognition Similar to the face detection system, the face recognition system was also designed for modularity and to allow for other models to be swapped in place of PCA and knn, shown in figure 4. FaceRecognizer.py is the high-level interface for training and classifying face images, with PCAModel.py handling PCA and projecting data onto principal components, and KNNClassifier.py handling the implementation of the knn algorithm. A command-line interface is also provided in FaceRec.py, which allows users to specify options to facilitate experimentation such as which data set to use, whether or not to use a kernel, the range of parameters to tune over, and how to partition the data into training, tuning, and testing sets. This program will also generate a report detailing the results of tuning and testing, making it relatively easy for a user to experiment with the system and explore its results.

5 of features by giving more variability in width and height of those features which leads to an increase in accuracy for detecting faces. Features Face (%) Non-Face (%) Time (s) Figure 4: Design of the face recognition system. 4 Results 4.1 Face Detection The face detection system uses data set acquired from MIT s Center for Biological and Computation Learning 3 (originally by Heisele et al. (2000) and further processed by Alvira and Rifkin (2001)). The data set has 2,901 cropped face images and 28,121 non-face images that have some characteristics of a face (e.g. a darker area in one region of the image similar to an eye area) Weak Learners The Viola and Jones algorithm suffers from significant amount of training time and training time is most affected by the number of Haar features (i.e. weak learners) we introduce, since at each stage of the iteration, all these features have to be evaluated to find the best one. So the experiments are performed based on a small number of features only. For each number of features, separate accuracies for classifying faces and non-faces are given in Table 1. In the first run, only feature types A, B, and D (refer to Figure 1) are introduced. Then in the second run, we introduce type C. Notice how in this run, although there is a slight decrease in accuracy for detecting faces, there is a dramatic increase in detecting non-faces (i.e. reduction in false positives). This is because the new feature, type C, is chosen as a good feature for differentiating faces from non-faces as it captures the intensity gradient around the nose area. In the final run, we simply increase the number 3 software-datasets/facedata1readme.html Table 1: Accuracies for classifying faces and non-faces for each number of Haar features. Training time increases with number of features but in a linear manner. As shown in Figure 3, for each run we obtain a list of learned Haar features with information on their type, position, width, and height. Based on this information, we can visualize the learned features as shown in Figure 5. Figure 5: From left to right: original face image, image with type C feature that captures nose area, image with type A feature that captures eye area Iterations The result of accuracies depending on the number of boosting iterations is given in Table 2. As evident from the results, blindly increasing the number of iterations does not necessarily yield a dramatically better accuracy. It is as if the performance is saturated at some point. Although we have perfectly labelled data for this project, increasing the number of iterations for cases where the data contains a lot of noise will actually decrease the performance significantly. The fact that the number of iterations is not directly proportional to accuracy is related to overfitting. At each successive iteration, the boosting algorithm will try harder and harder to fit misclassified training data (e.g. certain orientation of face). This eventually leads to a poor generalization to unseen images.

6 Iterations Face (%) Non-Face (%) Although OpenCV s built-in face detector cannot be applied to the MIT s face data that we are using due to incompatible image size (i.e. the images are too small), we still present the result of testing it on a real-world sample image as shown in Figure 6. Notice that even heavily trained open source detector cannot detect all of the faces. Those with sunglasses or those staring somewhere else at an angle seem particularly hard. In fact, a major limitation of the Viola and Jones algorithm is that it can only detect frontal faces. Table 2: Accuracies for classifying faces and non-faces for each number of iterations Data Partition Tuning for Viola and Jones algorithm is performed by adjusting the binary classification threshold. It is reduced until the desired accuracy on face data is achieved. Theoretically, this can yield 100% accuracy on face data but this comes at a cost of increase in false positives (i.e. non-face image classified as face image). Hence it is important to obtain a robust classifier in the first place using training data and tune the threshold by a minute amount. One way to do so is to find a good partition scheme of data. The effect of partition scheme (e.g. 70% for training, 20% for tuning, and 10% for testing) on accuracies is summarized in Table 3. With a mix of as the benchmark, you may notice that adding a portion of tuning data to training data slightly decreases performance since it is likely that the model is over-fitting the training data and thus performing poorly on training data. On the other hand, if we add a portion of training data to the test data, the model under-fits the training data and does poorly on test data, especially for non-face images. Partition Face (%) Non-Face (%) Figure 6: Result of applying OpenCV s Viola-Jones face detector on Hopkins commencement image. 4.2 Face Recognition Data and Methodology To evaluate face recognition with PCA and knn, two data sets from the Yale Face Database were used. The first data set used, which we ll refer to unofficially as Yale Faces A, consists of 15 subjects with different expressions and lighting conditions for a total of 165 images total (Belhumeur, 1997). The second data set, Yale Faces B+, is much larger, consisting of 39 subjects under various poses and lighting conditions for a total of 16,128 images (Georghiades, 2001). Example images from Yale Faces B+ can be seen in figure 7 Table 3: Accuracies for classifying faces and non-faces for each partition scheme (training-tuning-testing in %) Use of Open Source Figure 7: Example images from the Yale Faces B+ data set. Data was partitioned randomly into training, tuning, and testing sets such that they represented 70, 10, and 20% of the data respectively, and partitioning was done such that each set had an equal distribution of subjects. Five trials were run for each

7 data set with linear PCA and kernel PCA with a radial basis function (RBF), although kernel PCA was only run on Yale Faces A as computing the kernel became intractable for the much larger Yale Faces B+ to tune and evaluate in a reasonable amount of time. In each trial parameters were tuned for the number of neighbors for knn, the number of principal components to keep from PCA, and the variance for the RBF kernel. The results of these experiments can be seen in table 4. Generally, tuning resulted in k = 1 for nearest neighbors classification and all dimensions kept for PCA, although the variance for RBF kernel varied among trials for kernel PCA. Method A (%) B+ (%) Linear PCA Kernel PCA images with lighting conditions drastically different from the rest of the images, an example of which is shown in figure 8. In these images with drastic lighting, while the subject s faces are desirably lit in a different way from all of the others to challenge a recognition system, the background is also textured and shows the subject s shadow cast along the wall. In all other images, the background is whitened out, leaving just the subject. This merits criticism of the data set, as PCA would have projected these images into a much different subspace than the other images due to the new information present in the background, and is thus an unfair example to include in the data set (and may have been a motivator for the updated B+ set). Table 4: Average accuracy on Yale Faces data sets for PCA with a knn classifier using random partitioning Discussion The results obtained by this experiment meet expectations, in line with other baseline results for this method (Turk, 1991). For linear PCA, Yale Faces B+ fared worse than A, although this can be attributed to the fact that there were many more classes in this data set and the set is more challenging in terms of lighting and background scenes. Kernel PCA did not seem to have any benefit, and in fact fared worse than linear PCA on Yale Faces A while taking much more time to run. This reudction in accuracy may be attributed to the fact that finding an effective variance for the RBF kernel was difficult to do efficiently for each trial, requiring scanning over a range of 1,000 to 10,000 to find an optimal value. Because of this, a large step had to be taken and the true optimum value may have been skipped over in doing so, providing a worse result. However, because there were no obvious benefits and the algorithm took much more time to run (also being unable to run on Yale Faces B+), there seems to be little benefit to exploring this method further. When performing the experiments, certain situations in the random partitioning for Yale Faces A resulted in accuracies as low as 30%. In these situations, it was found that the test set was given Figure 8: Two example images from Yale Faces A showing drastically different lighting conditions. However, this brings attention to an important point about PCA that has so far been ignored in this paper. Because PCA is not resistant to affine transformations, an optimal application of PCA for face or object recognition requires that the object be cropped and in the center of the image. This fact has largely been ignored in the implementation so far, and future work could include determining the location and scale of a person s face in an image, and then cropping it out such that the face recognizer performs only on the subject s face, free of any distracting information present in the background. An ideal goal for this project was to be able to combine face detection with recognition to be able to do so, but because our face detection implementation only performs binary classification, that functionality was unable to be integrated into our final implementation. 5 Conclusion This paper demonstrates successful implementations of a face detection and recognition system using Viola-Jones algorithm and Eigenfaces for detec-

8 tion and recognition, respectively. Now that both components have been built and evaluated, further work can begin that moves towards integrating the components together to improve performance for each and allow it to be integrated into a larger application. 6 Appendix: Comparison to Proposal In our original proposal, we proposed the following objectives: Binary classifcation for face detection with Viola-Jones algorithm. Multiclass classification for face recognition with PCA and k-nn. Performance analysis based on hyperparameter tuning with reported accuracies. As an additional would like to achieve objective, we also proposed to compare our results with other open source algorithms such as OpenCV. Overall, all of our main proposed objectives were met. We were able to implement and measure the performance of both algorithms and found each to be successful. We also went a step beyond our proposal, exploring kernel methods with PCA to see how accuracies would change in the face recognizer. The only part of our project that we would have liked to achieve more of is the comparison with open source algorithms, but we were unable to do so because it was discovered that the data and methodologies we used for face detection was incompatible with OpenCV, the standard computer vision library that we would have liked to compare against. However, we did test OpenCV on other real-world sample images and one of the result is provided in section Since this performance comparison was originally proposed as an extraneous objective and we were able to meet all other objectives that we proposed, we consider this project an overall success. E. Osuna, R. Freund, and F. Girosi Training support vector machines: an application to face detection, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. L. Wolf Deepface: Closing the gap to humanlevel performance in face verification, Computer Vision and Pattern Recognition (CVPR). B. Heisele, T. Poggio, and M. Pontil Face Detection in Still Gray Images, CBCL Paper#187/AI Memo #1687. M. Alvira and R. Rifkin An Empirical Comparison of SNoW and SVMs for Face Detection, Technical Report A. I. Memo No , C.B.C.L. Memo No M. Turk and A. Pentland Face Recognition Using Eigenfaces, Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp P. Belhumeur, J. Hespanha, D. Kriegman Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, pp A.S. Georghiades and P.N Belhumeur, P.N. and D.J. Kriegman From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose, IEEE Trans. Pattern Anal. Mach. Intelligence, pp References P. Viola and M. J. Jones Robust real-time face detection, International Journal of Computer Vision 57.