A Framework for Long Distance Face Recognition using Dense- and Sparse-Stereo Reconstruction

Transcription

1 A Framework for Long Distance Face Recognition using Dense- and Sparse-Stereo Reconstruction Ham Rara, Shireen Elhabian, Asem Ali, Travis Gault, Mike Miller, Thomas Starr, and Aly Farag CVIP Laboratory, University of Louisville, KY, USA {hmrara01,syelha01,amali003,travis.gault,wmmill06,tlstar01, Abstract. This paper introduces a framework for long-distance face recognition using both dense- and sparse-stereo reconstruction. Two methods to determine correspondences of the stereo pair are used in this paper: (a) dense global stereo-matching using maximum-a-posteriori Markov Random Fields (MAP- MRF) algorithms and (b) Active Appearance Model (AAM) fitting of both images of the stereo pair and using the fitted AAM mesh as the sparse correspondences. Experiments are performed regarding the use of different features extracted from these vertices for face recognition. A comparison between the two approaches (a) and (b) are carried out in this paper. The cumulative rank curves (CMC), which are generated using the proposed framework, confirms the feasibility of the proposed work for long distance recognition of human faces. 1 Introduction Automatic face recognition is a challenging task that has been an attractive research area in the past three decades (for more details see [1]). At the outset, most efforts were directed towards 2D facial recognition which utilizes the projection of the 3D human face onto the 2D image plane acquired by digital cameras. The face recognition problem is then formulated as follows: given a still image, identify or verify one or more persons in the scene using a stored database of face images. The main theme of the solutions provided by different researchers involves detecting one or more faces from the given image, followed by facial feature extraction which can be used for recognition. Challenges involving 2D face recognition are well-documented in the literature. Intra-subject variations such as illumination, expression, pose, makeup, and aging can severely affect a face recognition system. To address pose and illumination, researchers recently are focusing on 3D face recognition [2]. 3D face geometry can either be acquired using 3D sensing devices such as laser scanners [3] or reconstructed from one or more images [4-6]. Although 3D sensing devices have been proven to be effective in 3D face recognition [7], their high cost, limited availability and controlled environment settings have created the need for methods that extract 3D information from acquired 2D face images. Recently, there has been interest in face recognition at-a-distance. Yao, et al. [8] created a face video database, acquired from long distances, high magnifications, and

2 both indoor and outdoor under uncontrolled surveillance conditions. Medioni, et al. [9] presented an approach to identify non-cooperative individuals at a distance by inferring 3D shape from a sequence of images. To realize our objectives and the current lack of existing facial stereo databases, we constructed our own passive stereo acquisition setup [10]. The setup consists of a stereo pair of high resolution cameras (and telephoto lenses) with adjustable baseline. It is designed such that user can remotely pan, tilt, zoom and focus the cameras to converge to the center of the cameras field of views on the subject s nose tip. This system is used to capture stereo pairs of 30 subjects at various distances (3-, 15-, and 33-meter ranges). The paper is organized as follows: Section 2 discusses stereo reconstruction methods (dense and sparse), Section 3 shows the experimental results, Section 4 validates the best method in Sec. 3 using the FRGC database, and later sections deal with discussions and limitations of the proposed approaches, conclusions and future work. 2 Stereo Matching-Based Reconstruction Dense, Global Stereo Matching: The objective of the classical stereo problem is to find the pair of corresponding points p and q that result from the projection of the same scene point (X, Y, Z) to the two images of the stereo-pair. Currently, the state-ofthe-art in stereo matching is achieved by global optimization algorithms [11], where the problem is formulated as a maximum-a-posteriori Markov Random Field (MAP- MRF) scenario. Given the left and right images, the goal is to find the disparity map D, where at each pixel p, the disparity is d p = p x q x. To correctly solve this problem, the constraints of the visual correspondence should be satisfied: (a) uniqueness, where each pixel in the left image corresponds to at most one pixel in the right image and (b) occlusion, where some pixels do not have correspondences. To achieve these constraints, similar to Kolmogorov s approach [12], we treat the two images symmetrically by computing the disparity maps for both images simultaneously. The disparity map D is computed by minimizing the energy function E D = E data D + E smoot h D + E vis D. The terms refer to the penalty, smoothness, and visibility constraint terms [13][14]. To fill the occluded regions, we propose to interpolate between the correctly reconstructed pixels of each scan line using the cubic splines [15] interpolation model. Finally, after getting a dense disparity map from which we get a set of correspondence points, we reconstruct the 3D points of the face [10]. To remove some artifacts of the reconstruction, an additional surface fitting step is done [16]. Sparse-Stereo Reconstruction: The independent AAM version of [17] is used to find sparse correspondences of the left and right images of the stereo pair. The shape s can be expressed as the sum of a base shape s 0 and a linear combination of n shape vectors s i, s = s 0 + i p i s i, where p i are the shape parameters. Similarly, the appearance A x can be expressed as the sum of the base appearance A 0 x and a linear combination of basis images A i x, A x = A 0 x + i λ i A i x, where the pixels x lie on the base mesh s 0. Fitting the AAM to an input image involves minimizing the error image between the input image warped to the base mesh and the appearance

3 A x = A 0 x + i λ i A i x, that is, A 0 x + i λ i A i x I W x; p 2 xεs 0. For this work, the error image is minimized using the project-out version of the inverse compositional image alignment (ICIA) algorithm [17]. To facilitate a successful fitting process, the AAM mesh is initialized according to detected face landmarks (eyes, mouth center, and nose tip). After detecting these facial features, the AAM base mesh is warped to these points. The detection of facial features starts with identifying the possible facial regions in the input image, using a combination of the Viola-Jones detector [18] and the skin detector of [19]. The face is then divided into four equal parts to establish a geometrical constraint of the face. The face landmarks are then identified using variants of the Viola-Jones detector, i.e., the face detector is replaced with the corresponding face landmark (e.g., eye detector) detector [20]. False detections are then removed by taking into account the geometrical structure of the face (i.e., expected facial feature locations). 3 Experimental Results The 3D acquisition system in [10] is used to build a human face database of 30 subjects at different ranges in controlled environments. The database consists of a gallery at 3 meters and three different probe sets at the 3-, 15-, and 33-meter ranges. Table I shows the system parameters at different ranges. Table I: Stereo-based acquisition system parameters Dense, Global Stereo 3D Face Reconstructions: The gallery is constructed by capturing stereo pairs for the 30 subjects at the 3-meter range. We reconstruct the 3D face of each subject using the approach that is described in Section 2. Fig. 1(a) illustrates a sample from this gallery for different subjects. This figure shows the left image of each subject and two different views for the 3D reconstruction with and without the textures. For the dense, global 3D reconstruction approach, only the images from the probe sets 3-meter and 15-meter ranges are considered. The reason behind this is that the methodology from Sec. 2.1 fails to determine acceptable correspondences of the stereo-pair images of the 33-meter range, leading to unacceptable 3D reconstructions. This result has led the authors to propose the second method (sparse-stereo) to deal with stereo-pairs that are difficult to extract dense correspondences. Fig. 1(b) and 1(c) illustrate the samples from these probe sets. Sparse-Stereo 3D Face Reconstructions: The gallery and probe sets are similar to above, except that the 33-meter images are now included in the probe set. The training of the AAM model involves images from the gallery. The vertices of the final

4 Figure 1: Dense 3D reconstructions: (a) 3-meter gallery, (b) 3-meter, and (c) 15-meter probes. AAM mesh on both left and right images can be considered as a set of corresponding pair of points, which can be used for stereo reconstruction. To further refine the correspondences, a local NCC search around each point is performed, using the left image as the reference, to get better correspondences. Fig. 2 shows stereo reconstruction results of three subjects, visualized with the x-y, x-z, and y-z projections, after rigid alignment to one of the subjects. Notice that in the x-y projections, the similarity (or difference) of 2D shapes coming from the same (or different) subject is enhanced. This is the main reason behind the use of x-y projections as features in Sec. 3 (Recognition). Figure 2: Reconstruction results. The 3D points are visualized as projections in the x-y, x-z, and y-z planes. Circle and diamond markers belong to the same subject, while the square markers are that of a different subject. Recognition: For face recognition, we use five approaches for using the 3D face vertices derived from dense- and sparse-stereo (3D-AAM) to identify probe images against the gallery: (a) moment-based approach for dense 3D reconstructions, (b) feature vectors derived from Principal Component Analysis (PCA) of 3D-AAM vertices, (c) goodness-of-fit criterion (Procrustes) after rigidly registering the 3D-AAM vertices of a probe with that of a gallery subject, (d) feature vectors from PCA of x-y plane projections of the 3D-AAM vertices, and (e) the same procedure as (c) but using the x-y plane projections of the 3D-AAM vertices of both probe and gallery, after frontal pose normalization.

5 Moment-based Recognition: For the dense 3D reconstructions, to compare between gallery and probe sets, feature vectors are derived from moments [21] derived from the 3D vertex coordinates. The moments are computed as η rst = X Y Z X r Y s Z t. Principal Component Analysis (PCA): To apply PCA [22] for feature classification, the primary step is to solve for the matrix P of principal components from a training database, using a number of matrix operations. The feature vectors Y can then be determined as follows: Y = P T X, where X is a centered input data. The similarity measure used for recognition is the L 2 norm. Goodness-of-fit (Procrustes): The Procrustes distance [23] between two shapes is a least-squares type of metric that requires one-to-one correspondence between shapes. After some preprocessing steps involving the computation of centroids, rescaling each shape to have equal size and aligning with respect to translation and rotation, the squared Procrustes distance between two shapes x 1 and x 2 is the sum of squared point distances, P d 2 = x 1 x 2 2. Figure 3: Cumulative match characteristic (CMC) curve of the: (a) 3-meter probe set, (b) 15- meter probe set, and (c) 33-meter probe set. Note that only the 3-m and 15-m probe sets use moment-based recognition (see Fig. 3) Discussion of Results: Fig. 3 shows the cumulative rank curves (CMC) curves for the five types of feature extraction methods in the previous section (Sec. 3), using 3-, 15-, and 33-meter probes. We can draw four conclusions: (a) both 2D Procrustes (i.e., x-y projection of 3D-AAM) and 2D PCA outperform both 3D Procrustes and 3D PCA, (b) goodness-of-fit criterion (Procrustes) slightly outperforms PCA in both 2D and 3D, (c) degradation of recognition at increased distances, and (d) the moment-based methods perform poorly at lower ranks but shoots up quickly to 100% at rank-5 and rank-8 for the 3-m and 15-m probe sets, respectively.

6 The conclusion in (a) can be explained with the help of Fig. 4. The diagram shows the top view of a simple stereo system. O l and O r are centers of projection, and p l, p r, q l, q r are points on the left and right images. Figure 4: Simple stereo illustration. 3D reconstruction is sensitive to the correspondence problem but projection to the x-y plane minimizes the error. Assume that the y-coordinates of the four image points are equal. p l and p r will reconstruct P, q l and q r will reconstruct Q, and so on. Notice that a small change in the correspondence affects the xyz reconstructions hugely, i.e., their Euclidean distance between each other is huge. But when they are projected to the x-y plane, their 2D Euclidean distances with each other are considerably lesser. This scenario is possibly happening with the correspondence of the AAM vertices between the left and right image. The conclusion in (b) is related to the primary purpose of PCA, which is optimal reconstruction error, in the mean-square-error (MSE) sense. Projection to a lowdimensional space may remove the classification potential of the vectors. There is no dimensional change with rigid alignment using Procrustes; similar shapes are expected to have less Procrustes distance after rigid alignment and geometric information of faces (e.g., distance ratios between face parts) are maintained. Results are expected to degrade with distance since the captured images are at less ideal conditions (although recognition using 2D x-y projections remain stable). The work in Medioni [9] deals with identification at a distance using recovered dense 3D shape from a sequence of captured images. The results at the 15- and 33-meter range (Figs 5(b,c)) are comparable (and slightly better) than their 9-meter results; however, their experimental setting may be less controlled than ours. The authors cannot find any concrete reason behind conclusion (d), except the fact that the dense reconstructed shapes may be overfitted by the gridfit procedure of [16]. This part of the framework is still a work-in-progress and more elaborate 3D recognition methods will be incorporated as future work (see Conclusions).

7 Sensitivity Analysis: This section performs a sensitivity analysis of the recognition performance with respect to errors in the AAM fitting of the stereo pair of images. To simulate errors introduced to the system, the fitted AAM vertices of the stereo pair are randomly perturbed with additive white Gaussian noise of a certain variance. Fig. 5 shows the plot of rank-1 recognition rates versus point sigmas, for the 3-, 15-, and 33- meter ranges. The recognition method used is the 2D Procrustes approach of Fig. 3. These findings reinforce the known fact that acceptable AAM fitting is necessary to get satisfactory correspondence, which leads to suitable recognition. Figure 5: Sensitivity analysis of recognition with respect to errors in the AAM fitting of the stereo pair of images. Notice that the recognition rates are fairly stable across various values of σ for the 3- and 15-meter probe sets. However, for the 33-meter probe set, the recognition performance severely deteriorates after σ = 5. 4 Validation with FRGC Database The main purpose of this section is to test if the 2D (x-y projection of 3D-AAM) Procrustes results of Fig. 5 carry over to the much larger FRGC database. Since the FRGC database contains close-to-perfect dense 3D information, the advantage of 2D Procrustes over its 3D equivalent would also be investigated. The section of the FRGC database [24] with range (3D) data is used. Each range is accompanied by a corresponding texture image. 115 subjects (with three images each, for a total of 345) were chosen out of the total number of subjects, the number being restricted by the manual annotation of AAM training data. Fig. 6 may provide some insights regarding the better performance of 2D Procrustes over 3D Procrustes. Note that the 2D+3D (range, depth) partition of the FRGC database contains 2D video images with corresponding range values for each pixel. After the manual/automatic fitting of AAM vertices, the corresponding range (depth) value is extracted for each vertex. In Fig. 6, the red dots represent the extracted depth values using the 2D coordinates of the fitted vertices. Notice that some depth values are undesirable; they do not contain the intended depth of the facial feature points (e.g., nose area of Fig. 6(a) and face boundary of Fig. 6(b)). The 2D coordinates of the AAM vertices (except for the face boundary) are adjusted accord-

8 Recognition Rate ing to the COSMOS framework [25]; specifically, the vertices are adjusted along a local neighborhood in the horizontal (x) direction, according to some extremum values defined by [25]. The green dots represent the adjusted 3D vertices. The face boundaries are adjusted using ad-hoc methods that investigate the most acceptable face vertex depth of the whole face boundary. Fig. 7 shows both the 2D and 3D Procrustes methods using the FRGC database. Similar to Fig. 3, the 2D approach outperforms the 3D version. Figure 6: Using the original 2D coordinates of the fitted AAM vertices in the range image do not give the desired feature locations; hence, the adjustment of the vertices: movement of the (x) dot to the ( ) dot at the (a) nose location and (b) face boundary, indicated by the arrow. The vertex movement is not noticeable in the corresponding 2D video image D Procrustes 3D Procrustes Rank Figure 7: Cumulative match characteristic (CMC) curve of the FRGC experiment. Notice the superiority of the 2D Procrustes approach over the 3D version. Classification is performed using the leaving-one-out methodology of [22]. The reason behind this is related to Fig. 6; the movement from the red to green dots represents a large variation in the 3D coordinate system, but when projected to the 2D x-y plane, the planar movement is much smaller, e.g., there is no noticeable pixel difference in the 2D image after the vertex movement in the nose of Fig. 6(a).

9 5 Conclusions and Future Work This paper described a framework for identifying faces remotely, specifically at distances 3-, 15- and 33-meters. The best approach used AAM to get correspondences between the left and right images of the stereo pair. This study found out that recognition using the point-to-point distance between 2D x-y projected shapes (after rigid registration, Procrustes) outperforms the others. The advantage of the 2D Procrustes approach over its 3D version carried over to the much larger FRGC database (with chosen 115 subjects). Using our database, we have illustrated the potential of using these few vertices, as opposed to the whole set of points of the human face (dense reconstruction). The authors are aware of more elaborate methods of 3D shape classification related to face recognition (even with the presence of face expression), such as [7]. However, for this application (of identifying faces at far distances), a close-to-perfect dense 3D scan of the face is difficult to obtain; therefore, this study currently deals with sparse 3D points related to the AAM vertices. The next step of this work is to densify the AAM mesh to have a reconstruction that has a close semblance to a dense 3D scan but still contain lesser vertices than conventional 3D scans. This study does not currently consider face expression (since the 3D sparse reconstruction can only do so much) but will be considered as future work once the densification of the AAM vertices is taken care of. Additionally, the authors plan to increase the database size (for better statistical significance) and capture images at further distances (with the help of state-of-the art equipment). References 1. Zhao, W., Chellappa, R., Rosenfeld, A.: Face recognition: a literature survey. ACM Computing Surveys 35 (2003) Kittler, J., Hilton, A., Hamouz, M., Illingworth, J.: 3D assisted face recognition: A survey of 3D imaging, modelling and recognition approaches. In: Proc. of CVPR. (2005) 3. Pan, G., Han, S.,Wu, Z.,Wang, Y.: 3D face recognition using mapped depth images. In: Proc. of CVPR- Workshop on Face Recognition Grand Challenge Experiments. (2005) 4. Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. on PAMI 25 (2003) Hu, Y., Jiang, D., Yan, S., Zhang, L., Zhang, H.: Automatic 3D reconstruction for face recognition. In: Proc. of Sixth IEEE International Conference on Face and Gesture Recognition. (2004) Chowdhury, A.R., Chellappa, R., Krishnamurthy, S., Vo, T.: 3D face reconstruction from video using a generic model. In: Proc. of IEEE International Conference on Multimedia and Expo. (2002) Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Three-dimensional face recognition. In: Intl. Journal of Computer Vision (IJCV). (2005) Yao, et al..: Improving long range and high magnification face recognition: Database acquisition, evaluation, and enhancement. In: Computer Vision and Image Understanding (CVIU). (2008)

10 9. Medioni, G., Jongmoo, C., Cheng-Hao, K., Choudhury, A., Li, Z., Fidaleo, D.: Noncooperative persons identification at a distance with 3D face modeling. In: Proc. of IEEE International Conference on Biometrics: Theory, Applications, and Systems (BTAS 07). (2007) 10. Rara, H., Elhabian, S., Ali, A., Miller, M., Starr, T., and Farag, A.: Face recognition at-adistance based on sparse-stereo reconstruction. In: Proc. of CVPR- Biometrics Workshop. (2009) 11. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M.F., Rother, C.: A comparative study of energy minimization methods for markov random fields. In: Proc. of ECCV. (2006) Kolmogorov, V., Zabih, R.: Multi-camera scene reconstruction via graph cuts. In: Proc. of ECCV. (2002) Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. IEEE Trans. on PAMI 20 (1998) Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. on PAMI 26 (2004) Knott, G.D.: Interpolating Cubic Splines. Springer-Verlag (2000) 16. D'Errico, J.: Surface Fitting using gridfit. In: Matlab Central File Exchange ( (2006) 17. Matthews, I., Baker, S.: Active Appearance Models Revisited. In: International Conference on Computer Vision (ICCV). (2004) 18. Viola, P., Jones, M. J.: Robust real-time face detection. In: International Journal of Computer Vision (IJCV). (2004) Jones, M., Rehg, J.: Statistical color models with application to skin detection. In: International Journal of Computer Vision (IJCV). (2002) Castrillon-Santana, M., Deniz-Suarez, O., Anton-Canalıs, L., Lorenzo-Navarro, J.: FACE AND FACIAL FEATURE DETECTION EVALUATION: Performance Evaluation of Public Domain Haar Detectors for Face and Facial Feature Detection. In: VISAPP. (2008) 21. Elad, M., Tal, A., Ar, S.: Content based retrieval of vrml objects: an iterative and interactive approach. In: Proceedings of the sixth Eurographics workshop on Multimedia. (2001) Belhumeur, P., Hespanha J., Kriegman, D.: Eigenfaces vs. Fisherfaces: Recognition using Class Specific Linear Projection. In: IEEE Trans. PAMI. (1998) 23. Cootes, T.F., Taylor, C.J.: Statistical Models of Appearance for Computer Vision. In: Technical Report, University of Manchester, UK. (2004) 24. Chang, K., Bowyer, K., Flynn, P.: An Evaluation of Multimodal 2D+3D Face Biometrics. In: IEEE Trans. PAMI. (2005) 25. Dorai, C., Jain, A.K.: COSMOS - A Representation Scheme for 3D Free-Form Objects. In: IEEE Trans. PAMI. (1997)