The International Workshop on Automatic Face and Gesture Recognition, Zurich, June 26-28, 1995 TASC. 55 Walkers Brook Drive

Transcription

1 The International Workshop on Automatic Face and Gesture Recognition, Zurich, June 26-28, 1995 Face Recognition from Frontal and Prole Views Gaile G. Gordon TASC 55 Walkers Brook Drive Reading, MA 1867 Abstract This paper presents a unique face recognition system which considers information from both frontal and prole view images. This system represents the rst step toward the development of a face recognition solution for the intensity image domain based on a 3D context. In the current system we construct a 3D face centered model from the two independent images. Geometric information is used for view normalization and at the lowest level the comparison is based on general pattern matching techniques. We also discuss the use of geometric information to index the reference database to quickly eliminate impossible matches from further consideration. The system has been tested using 2 subjects from the FERET program database and has shown excellent results. For example, we consider the problem of identifying the 5% of the database which is most similar to the target. The correct match is included in this list 85% of the time in the system's fully automated mode and 98% of the time in the manually assisted mode. 1 Introduction This work focuses on face recognition applications domains which have not been successfully addressed in previous research: uncooperative orunaware subjects matching against large subject databases. These issues add important new dimensions to an already challenging problem. The major problem introduced by uncooperative subjects is that, unlike face recognition in controlled environments, there can be no assurance of view consistency. Previous systems have approached face matching as a 2D problem which, because faces are not at, requires good consistency in view from one image of the subject to another. If there is not good view consistency, recognition errors are introduced by confusing similarity inview with similarity iniden- tity. These facts dictate that a 3D representation is required [1]. Our approach is based on the computation of a face centered 3D model which will be independent of view. The computation of a 3D model requires multiple views of the subject. We beginbyinvestigating recognition from two images of each subject: frontal and prole. It is interesting to note that although many researchers have investigated recognition from frontal view, and to a smaller degree, recognition This work was supported by the FERET program of the Army Research Lab under contract DAAL1-93-C-118. from prole view, no automated system has combined these information sources. Our future work will expand these ideas for video sequence input. Large subject databases introduce twokey problems. The rst problem is combinatorical. As database sizes grow, it becomes impractical to compare the target with each entry in the database. Our approach involves precomputing database indexing information based on face geometry. With this index information only a fraction of the total database must be compared one-to-one. The second problem in large databases is that, independent of the similarity metric used, as the number of subjects increases, the distance between \neighboring" subjects will decrease. Thus, more accurate subject descriptions are required to maintain the same recognition performance. By reducing the sources of error due to variations in view, our approach also addresses this issue. 2 Algorithm Overview The TASC system processes pairs of independently acquired frontal and prole images. These image pairs t into two categories: subjects to identify, and subjects who will be used for reference in the identication. The rst set will be referred to as probe or query subjects, and the second as the gallery or reference subjects. The overall organization of the algorithm is summarized in Figure 1. The recognition process begins by computing a 3D representation or model for the gallery images and the probe images. The models constructed from the gallery images form the reference database. The contents and construction of the models are discussed in more detail in Section 3 including feature extraction. An index is created for the reference database which groups the models based on basic geometric measures of the face. This index allows quick identication of those subjects in the database who fall within a given range in geometry. The potential of this process to reduce search time is examined in Section 5.3. The nal step is to compare each remaining reference model with the query model. The bases for this comparison are normalized image regions extracted from the original frontal and prole images. Any robust pattern matching algorithm could be used at this point our system is based on normalized cross correlation. This matching process is addressed in Section 4. Each comparison results in a similarity score. The output of the process is a ranked list of reference subjects, ordered by similarity to the query subject. Section 5 presents the results of algorithm evaluation,

2 and a discussion of the potential benets of video stream input on algorithm performance. Section 6 provides a nal summary of results and conclusions. Gallery Images Feature Extraction and Model Generation Database Query Images Feature Extraction and Model Generation 3D Database Pruning 3D Refined Analysis Using Normalized Pattern Information Figure 1: System block diagram 3 Contents of the Model Similarity Ranking The 3D model contains the location of key points on the face in a 3D coordinate system and a set of normalized image chips. Figure 2 shows a schematic example of a face model. Feature points are extracted using morphological operators, pattern matching, and other low level image analysis algorithms. The image chips are subimages taken from normalized versions of the original frontal and prole images. The rst normalization step adjusts the intensity range in the head region to the full dynamic range available, to 255. The second step adjusts the scale and rotation for each view. This process is based on the location of two key features within each image. For the frontal view the pupil centers are used, and for the prole view the tip of the nose and the tip of the chin are used. The feature extraction for the frontal view is discussed in detail in Section 3.2 and for the prole view in Section 3.3. The normalization process itself is discussed in Section 3.4. Figure 2: Components of the model Frontal and prole view planes provide the most complete 3D information available from two independent views. However, because the views are independent, scale or pose information computed from one image can not be applied to the other. E.g. it is impossible to compare distances measured in the XY plane (frontal view plane) with distances measured in the YZ plane (prole view plane). With two independent images we can only normalize for one degree of freedom in rotation in each image, which is about the optical axis of the view. This corresponds to the rotation about the Z axis for frontal view images, and about the X axis for prole views. Future work will focus on video sequence input because the constraints which exist on the relative pose between frames should allow a more complete specication of the model information. 3.1 Identifying the Head Bounds The the head bounding box is used both in the normalization of intensity and in the feature detection process which follows. The rst step in estimating the head bounding box is the segmentation of the subject from the background. In the FERET database the background is relatively homogeneous. The background is identi- ed based on detection of locally connected areas of low gradient magnitude in the vicinity of the image boundary. Once the subject silhouette has been identied, the next task is to identify the subregion associated with the head. This process is straightforward except for the inuence of dierent hair styles. For the frontal view images, we consider horizontal cross sectional area to identify the top and bottom of the head region, with the bottom of the head distinguished by the increase in area approaching the shoulders. Vertical cross sectional area is used to identify the sides of the head increase in areas between shoulder and head indicates a left or right bound of the head region. When considering prole images, the process is similar. However, because hair style and posture will have a strong eect on this portion of the image, we divide the silhouette vertically and discard the half containing the back of the head. Long hair or large scale images, in which the shoulders are not completely visible, complicate the process of head bound estimation. In these two cases, which are easily identied, general guidelines based on area of the foreground silhouette and location of the top of the head are used to set bounds. See Figures 3 for typical examples of the detection of the foreground, the location of the head region, and in the last two images, the adjusted intensity mapping. 3.2 Location of the Eyes The current normalization process for scale and view in the frontal view depends on the location of the pupil centers. The eye detection module nds likely candidates for individual eyes based both on similarity to a small sample of eye templates (this method has also been used by [2] and later [3]), as well as eye pupil detector based on circular image valleys (dark intensity regions) of the expected scale (this method has been discussed to a limited extent in [4]). A rough idea of the expected scale of the eyes, based on the width of the head region, is used to guide this

3 process. Both sets of candidates are combined and evaluated to identify the most likely eye pairs (set of two eye candidates). This process includes the following detailed steps: Identify individual eye candidates using template match with general eye templates Add candidates identied via valley based pupil detection Select likely eye pairs from the combined set using geometric constraints on separation (given scale range) and angle Normalize image view for each candidate pair selected Eliminate candidates with no evidence for nose and mouth features Select from remaining candidates (if any) based on bilateral symmetry, relative position within head region, eye candidate scores, and scale An example of eye location in the frontal view is given in Figure 3. The upper left image shows the detection of the foreground and the head extents, and the lower left image shows the nal normalized image with the eye centers and template regions marked. Figure 3: Processing of frontal and prole views. 3.3 Processing the Prole The description of prole data and its use in face recognition has been discussed since Sir Francis Galton's article in Nature in 1888 [5], although the work of Harmon in 1977 [6] is probably better known. Despite these encouraging recognition results, face recognition from frontal view images has dominated the attention of the research community recently, and little attention has been paid to the combined use of frontal and prole data. Our processing of the prole view is similar to processing of the frontal view. The current normalization process for scale and view in the prole view depends on the location of the tip of the nose and the tip of the chin. We chose these features because they can be identied a high percentage of the time. Also, because we locate positions to the nearest pixel, the relatively wide separation of these points reduces the errors due to spatial resolution. The steps to extract these points are as follow: Rene foreground border Extract prole line Extract approximate location of feature points from prole contour line using tangency constraints and a priori knowledge of head structure Rene position of feature points using bitangency constraints. Each step is discussed below. An example of the processing of the prole image is given in Figure 3. The upper right image shows the detection of the foreground, head extents, and rened prole line, and the lower right image shows the nal normalized image with the nose tip, chin tip, and template regions marked. The original contour of the subject identied during foreground background segmentation is rough because its edge is based on a specic threshold of the gradient magnitude. Generally, this contour will fall outside of the \ideal" edge. The renement ofthe contour is performed using a hierarchical watershed algorithm [7]. This process guarantees a single completely connected outline and makes ecient use of local and global information. A traditional watershed operation has also been used independently by researchers at Thomson-CSF [8] for extracting prole lines. Example of the rened prole contour is shown in the second image of Figure 3 juxtaposed with the original rough foreground segmentation. Contour following is used to extract the resulting outline as a connected list of points. Typical approaches used to extract the nose and chin tip include: computation of local curvature extrema, which has not been satisfactorily stable [8, 9], and use of the rightmost point in the subject's silhouette [6] which has problems with many hair styles and is not invariant to nodding of the head. Most previous work used a more limited subject database than the current work e.g. all male caucasian subjects with manually extracted silhouettes [6], or subjects of only asian descent who were all asked to \comb their hair back" [9]. Our approach is based on geometric tangency constraints and has been shown empirically to be quite robust with respect to head position and contour noise. We rst identify estimates of the chin and nose location which are rened by moving the points in a local neighborhood on the contour until a bitangent line can be identied. Bitangents are used in recognition of planar shapes or surfaces of revolution because they are perspective invariants []. Even though the 3D shape of the face doesn't t these criteria in general, bitangents do provide stability over

4 small left-right changes in face pose, and are invariant to motion in the prole plane. The last image in Figure 3 shows the normalized view of the head, and identies the location of the tip of the nose and chin. In the normalized position of the prole, the line connecting the nose and chin tip has a 2 degree oset with respect to the vertical axis. This position is close to the original pose in this example. 3.4 Normalizing for scale and rotation After the location of the eye pupil centers and the tip of the nose and chin, it is then possible to normalize the images and other feature points to conform to standardized positions. The frontal view is dened such that the optical axis is parallel to the Z axis, and the viewing distance places the eye centers at a specic xed separation. The prole view is dened such that the optical axis is parallel to the X axis (looking toward the positive direction - nose faces right), and the viewing distance places the tip of the nose and chin at a specic xed separation. These standardized views are computed by transforming the original frontal and prole images including a rotation about the optical axis, a general scale factor, and a translation. In the case of the prole images, an additional reection about the Y axis is added for left proles. 4 Template processing The nal comparison step is based on the ve image chips contained in each model. These regions are: left eye, right eye, nose, mouth, and prole. At aba- sic level, this process is similar to systems developed in [2, 3] in that it uses cross correlation. Key dierences include the introduction of prole image data, and the fact that the complete face is not used as one of the pattern inputs, only the smaller image chips corresponding to designated feature regions. Limited positioning errors in the selection of the template region are allowed by shifting the two patterns over a small neighborhood during comparison. The image is locally normalized to a zero mean intensity to minimize the eects of general lighting dierences. In most cases, a weighted sum of the 5 individual scores is used to produce a nal similarity score. The mouth region is weighted at less than half that of the eye and nose region because expression causes this area to be less reliable in identication. The region from the prole plane is weighted higher than the eyes and nose because it contains more information than the smaller individual templates from the frontal view. An additional aspect of the scoring involves compensation for errors in extraction of the features used for normalization. In cases where normalization features were not well located in one of the views, it may be more eective to base the similarity measure on only one view. In order to assist with this decision, we compute 3 similarity scores for each comparison: the weighted score from frontal view templates only, prole view only, and the normal score including both views. 5 Results 5.1 Current SystemPerformance The FERET program database was used for algorithm development andevaluation. It includes subjects photographed in four dierent time periods. There is very little overlap of subjects between the four sets. Three out of the four data sets included two frontal and two prole (left and right) images of each subject, enabling recognition testing. For each of these subjects, one of the two frontal view images and the left prole image was used to build a reference database model. The second frontal view image, paired with the right prole image, was used to build a query model. There is a wide variety of image scale and lighting and good representation of age, ethnic background, and gender in the database. Many of the subjects have glasses, several have moustaches and/or beards. The set of images available for testing included 236 subjects. A small number presented poor imaging conditions which our algorithms were not designed to accommodate. These conditions included eye glasses with glare spots covering the pupil, dark sunglasses, or prole images in which the prole line (nose to chin) was obscured by clothing, shoulder, or hair. Some of these problems will be more easily addressed from video data, but currently these cases were not included in the test group. The nal test pool contained 194 possible subjects. Results reported in the following section use a full 194 subject gallery and 194 subject probe set. It is expected in many face recognition applications that several of the top candidates, not just the top candidate, are presented for further human consideration. In the presentation of results we use the concept of rank threshold, R, which is the percentage of the best database entries to be presented as the result of the comparison. If the \ideal" match is contained within the rst R candidates in the ranking list, we consider this a successful recognition result. We report system performance by giving error rate as a function of rank threshold. This provides error rates for all choices of R. Figure 4 shows this result for the 194 subject test which we call Test 1. Error Rate (%) Rank threshold (% of gallery) frontal and profile Figure 4: Test 1: Error rates vs. rank threshold. To test the hypothesis that the major source of error was the extraction of normalization features we

5 used two data sets: 22 subject gallery vs. 22 subject probe with models based on manual feature extraction Test 1: results from subjects with good automatic normalization and total dataset The 22 subject database was constructed by entering the location of the normalization features (pupil centers, nose and chin tip) using a mouse. The 22 subjects included all of the 194 used in Test 1 plus a few of the subjects omitted from Test 1 because of poor imaging conditions for the automatic algorithm. The remainder of the model construction and comparison operation was exactly the same as in Test Test 1 Test 1 subset (subjects with good normalization) Using manually selected normalization points with two dierent data sets: 1) Test Scenario 1 (Figure 4) and 2) Similar models built with manual feature extraction (Figure 5(a)). Figure 6 shows this comparison graphically for the rst scenario. Our results support our thesis, showing that the addition of the prole information substantially reduces error rates, in these cases by between 45% (rst scenario) to 33% (second scenario) at R =5%. Error Rate (%) frontal only frontal and profile Figure 5: Performance with and without error in location of the 2 normalization features. It is clear from Figure 5 that the recognition results in cases where normalization feature detection was correct are very high. In Figure 5(a), where the normalization points are detected manually, the error rates are below 2%atR = 5%. The dotted line in Figure 5(b) shows that when the points are detected correctly by the automatic system, error rates are below 5%at R = 5%. Feature points were considered to be correctly located if the automatically identied position was within 4 pixels of the manually identi- ed location. Normalization error in the automatic feature detection increased system recognition errors by approx. % across most values of R. The residual error in the system (2% to 5% at R = 5%) were due to other sources of error including primarily large expression variations and view variation. 5.2 Evaluation of the Role of 3D Akey point of our approach is that the addition of 3D data should improve system performance by disambiguating eects of view from identity. The second view provides data about the 3rd dimension, that is, the relief or depth of the face, which is not available from the single frontal view. In the future we hope to enrich the model further through the use of video sequences. We tested this thesis by comparing recognition performance when only frontal images were used as input to the performance when both frontal and pro- le image pairs were used. The test was performed Rank threshold (% of gallery) Figure 6: Comparison with frontal only system. 5.3 Database Indexing Template based comparison systems must compare every probe subject with every gallery subject. Precomputed geometric features enable us to sort quickly to reduce the full gallery to the fraction of the database which is geometrically compatible with the query before the nal comparison process begins. We investigated the potential of this process to decrease search times using a subject dataset which was built with manually detected feature points. A vector of 12 geometric features consisting of distances and angles in the normalized frontal and prole images was computed from points identied in the image. The rst evaluation investigated the value of each feature individually in recognition as indicated by the Fisher's linear discriminate criteria: (Between Subject Variance) This criteria formalizes (Within Subject Variance) the intuition that a measure will be useful in discriminating between subjects if it varies very little over dierent instances of the same subject, but greatly between subjects. To compute the value of this criteria, we assembled a dataset of 3 subjects with 4-5 frontal/prole pairs per subject. These images contained some examples from the FERET database and some examples (14 subjects) which were taken at TASC over a series of 5days. The dierent features are shown in Figure 7 ordered from most discriminating to least discriminating. Interesting to note is that Kanade's work [11] this ratio was typically 2 or 3. Brunelli reports similar scores for his automated feature detectors [3]. These values indicate good potential for use in identication. The second evaluation performed was the use of a simple feature based classier which judged similarity on the basis of normalized distance in feature space.

6 G 3837 Discrimination Nose depth Nose height Nose ridge length Chin height Septum length Septum to nose ridge angle Nose length Face width Nose width Lower lip thickness Figure 7: Feature discrimination scores. The features used were: Nose depth, nose height, nose ridge length, chin height, septum length, septum to nose ridge angle, face width and nose width. This type of classier considers much less information than the template based system and therefore runs much faster (more than 2 times faster in our tests). The classier was tested with a database of subjects (1 pair of images per subject in the reference database, a dierent pair used as query). The error rates are shown in Figure 8, and indicate a % recognition rate at R = 38%. These results indicate that we can use geometric features to reduce the database to 38% of its size without introducing any further recognition errors. Results for classication using only those features derived from the frontal view are also shown, to point out the positive value of the prole features for this classier as well. Error Rate (%) Philtrim height Upper lip thickness Rank threshold (% of gallery) frontal only frontal and profile Figure 8: Error rates: feature based classication. In summary, the combination of feature based pruning with traditional template based comparison has the potential to dramatically reduce search times without eecting recognition error rates. We recommend the use of geometric features for coarse, conservative pruning, and localized pattern comparison for detailed dierentiation. 6 Conclusions We have discussed the design and implementation of a face recognition system which processes frontal/prole image pairs. We reported system per formance with respect to Rank Threshold, R, which is the number of most similar reference database subjects considered when judging correct identication. At R = 5% the automated system performed at a recognition rate of 85% for a test involving 194 subjects. The largest source of error was found to be the detection of two image feature points used during the normalization process. If wechange the recognition process only in the sense that these points are identied exactly (e.g. via manual intervention), the performance of the system is 98% for the same test. We showed that the addition of 3D information, in the form of the prole view in this case, substantially reduces error rates. In our tests, the use of the prole reduced errors by between 45% and 33%. It is likely that the benets of considering the prole view would be on a similar scale for any existing face recognition system which is based on frontalviewimagesonly. We also showed that the use of geometric features has the potential to substantially reduce search times. A feature based classier was shown to have the potential to reduce search times by 6% without aecting recognition rates. Again, this pruning process could be applied as a front end to any existing face recognition system. Our future work will address subject modeling from motion sequences. Motion sequences will enable improved performance through a more robust 3D model. References [1] G. G. Gordon, Face Recognition from Depth and Curvature. PhD thesis, Harvard University, Division of Applied Sciences, [2] R. Baron, \Mechanimsms of human facial recognition," International Journal of Man Machine Studies, vol. 15, pp. 137{178, [3] R. Brunelli and T. Poggio, \Face recognition: Features versus templates," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 15, pp. 42{ 52, October [4] A. L. Yuille, D. S. Cohen, and P. W. Hallinan, \Feature extraction from faces using deformable templates," in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (San Diego, CA), pp. 4{9, June [5] F. Galton, \Personal identication and description 1," Nature, pp. 173{177, June [6] L. D. Harmon and W. F. Hunt, \Automatic recognition of human face proles," Computer Graphics and Image Processing, vol. 6, pp. 135{156, [7] L. Vincent and P. Soille, \Watersheds in digital spaces: an ecient algorithm based on immersion simulations," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, pp. 583{598, June [8] L. Najman, R. Vaillant, and E. Pernot, \From Face Sideviews to Identication," in Image Processing: Theory and Applications, (Gianni Vemazza, ed.), pp. 299{32, Elsevier, June [9] C. Wu and J. Huang, \Human face prole recognition by computer," Pattern Recognition, vol. 23, pp. 255{259, 199. [] A. P. Zisserman, D. A. Forsyth, J. L. Mundy, and C. A. Rothwell, \Recognizing general curved objects eciently," in Geometric Invariance in Computer Vision, (J. Mundy and A. Zisserman, eds.), pp. 228{251, Cambridge, MA: MIT Press, [11] T. Kanade, Picture Processing System by Computer Complex and Recognition of Human Faces. PhD thesis, Kyoto University, Department of Information Science.