Pupil Detection under Lighting and Pose Variations in the Visible and Active Infrared Bands

Transcription

1 Pupil Detection under Lighting and Pose Variations in the Visible and Active Infrared Bands Thirimachos Bourlai #1, Cameron Whitelam #2, Ioannis Kakadiaris 3 # LDCSEE, West Virginia University, Evansdale Drive, Morgantown, WV , U.S.A. 1 Thirimachos.Bourlai@mail.wvu.edu 2 cwhitela@gmail.com Dept. of Computer Science, University of Houston, 4800 Calhoun, Houston, TX , U.S.A. 3 ioannisk@central.uh.edu Abstract We propose a novel and efficient methodology for the detection of human pupils using face images acquired under controlled and difficult (large pose and illumination changes) conditions in variable spectra (i.e., visible, multi-spectral, and short wave infrared (SWIR)). The methodology is based on template matching, and is composed of an offline and an online mode. During the offline mode, band-dependent eye templates are generated for each eye from the face images of a pre-selected number of subjects. Using the eye templates that are generated in the offline mode, the online pupil detection mode determines the locations of the human eyes and the pupils. A combination of texture- and template-based matching algorithms is used for this purpose. Our method achieved a significantly high detection rate, yielding an average of 96.38% pupil detection accuracy across all datasets used. Based on a comparative analysis on different databases, we concluded that: (i) a single methodological approach can be used to efficiently detect human eyes and pupils of face images (with strong pose and illumination variations) acquired in the visible and hyper-spectral bands, and (ii) the use of texture-based matching and normalized band-specific templates significantly increases detection accuracy. To the best of our knowledge, this is the first time in the open literature that the problem of multi-band pupil detection on face images in the presence of lighting and pose variations, is being investigated using a unified algorithm. I. INTRODUCTION Most of the commercial and research face recognition (FR) algorithms currently available require the determination of eye center (pupil) coordinates as part of the authentication process and their performance can be heavily impacted by an inefficient eye detection algorithm [1]. There are three main eye detection methodologies: Feature-based methods that focus on eye characteristics (e.g., eye shape and color distribution). Appearance-based methods that use the photometric appearance of the eyes for detection through classification approaches. Template-based methods that slide a pre-designed eye model (template) across a face image to obtain the best eye matches. Park et al. [2] determined the eye candidates by using texturebased eye filtering, and then detected the eye locations using /11/$26.00 c 2011 IEEE. face geometry. The face images used were acquired under controlled conditions with some variability in terms of facial expressions. Xingming et al. [3] proposed a pose-independent Adaboost method to detect faces, a heuristic rule to filter the non-eye candidates, and a Support Vector Machine (SVM) classifier to verify eye pairs. The method was successfully tested on a small dataset of visible spectrum images acquired under three different illumination conditions. Asteriadis et al. [4] proposed a method for eye localization based only on the geometrical information of human eyes. After an edge map is extracted from a detected face, a vector is assigned to every pixel. Then, the length and slope information for these vectors is used to find the eyes. The authors reported Fig. 1. Results of preliminary experiments that highlight the challenges. They were obtained by applying a benchmark [5] and a commercial eye detection technique (G8 provided by L1 Systems) on the original UHDB11 database (non-synthetic data with strong pose and illumination variations), as well as on the synthetic (where strong pose variations were imposed) WVU multispectral and SWIR datasets.

2 Fig. 2. Examples of face samples from the four databases used, before (top) and after (bottom) illumination normalization. 96.7% and 98.6% on the detection of eye centers when using visible band face images from both the XM2VTS and the BioID database, respectively. Chen and Liu [6] proposed a method to detect eye centers that uses color information and wavelet features. SVM was used for classification. The method achieved a 94.92% eye detection accuracy based on experiments performed on the Facial Recognition Grand Challenge (FRGC) database. Morimoto et al. [7] proposed a robust pupil detection technique that uses two near infrared time-multiplexed light sources synchronized with the camera frame rate. A common characteristic of the aforementioned techniques is that they are not designed to simultaneously deal with images acquired under multiple bands. However, in most operational scenarios (e.g., in military and law enforcement applications) [8], face images may have to be acquired not only under variable illumination conditions and poses, but also under variable spectra. The main advantages of using hyper-spectral images for recognition are that they are suitable for covert applications [9] and can be useful in a nighttime environment [10]. An efficient hyper-spectral eye detection algorithm (benchmark technique) that can perform eye detection at variable bands was proposed by Whitelam et al. in [5]. However, the limitation of the approach is that it was not designed to be pose and illumination invariant. In this work, we address the aforementioned limitations and propose a methodology that is capable of detecting human pupils from multi-band face images acquired under various realistic unconstrained situations, or when the original face datasets were manually rotated (adjusting the roll descriptor - Fig. 4) right and left resulting in pose variations. First, our proposed method was tested on the original data from the UHDB11 database (the face images of the dataset were acquired under strong light and pose variations) [11]. The proposed method was also tested on the WVU database [5], where face images were acquired at variable spectra, as well as a subset of the FRGC database [12] (acquired in the visible spectrum). WVU and FRGC were also used to generate two synthetic visible and hyper-spectral databases, where face images were randomly rotated to the right and left (at variable degrees), and thereby imposing strong pose variations. Preliminary experiments highlighted the problem of applying a benchmark [5] and G8 on the above datasets (Fig. 1). We evaluated the performance of the proposed method by computing its detection accuracy on the UHDB11, WVU, and FRGC datasets, and compared the performance of the algorithm against a commercial and a benchmark eye detection method (designed to operate on face images acquired in both the visible and active infrared bands). The experimental results demonstrated that the proposed approach can be used to efficiently detect human eyes and pupils on multi-spectral face images yielding a detection accuracy that ranges from 92.89% to 99.30%. The detection process is computationally feasible ( 5 s per image in a Matlab environment). The rest of the paper is organized as follows: Section II presents the proposed pupil detection system. The experimental procedure used to evaluate the proposed method and the databases used are described in Section III. Conclusions and future work are presented in Section IV. II. METHODOLOGY In this section, we outline the technique used for eye and pupil localization. The salient stages of the proposed method are described below, and the overall process is illustrated in Fig. 3. (i) Photometric Normalization (PN): PN was applied to all visible and hyper-spectral face images. As conventional techniques (e.g., histogram-equalization and homomorphic filtering) did not perform well, we followed the approach proposed by Tan and Triggs [13] that incorporates a series of algorithmic steps. The steps were chosen in a certain order to eliminate the effects of illumination variations, local shadowing and highlights, while still preserving the essential elements of

3 Fig. 3. Overview of the process used (eye and pupil detection). visual appearance for more efficient face recognition. The approach is based on the following steps: - Gamma Correction: A nonlinear gray-level transformation was used to replace gray level I with I γ (for γ > 0) or log(i) (for γ = 0), where γ [0, 1] is a user-defined parameter (here, γ = 2). This step enhances the local dynamic range of the image in dark or shadowed regions, while compressing it in bright regions. - Difference of Gaussian (DoG) Filtering: A blurred version of an original gray scale image was subtracted from another less blurred version of the original image using two Gaussian kernels (σ 1 = 1 and σ 2 = 2) and image convolution. - Masking: This process was used to suppress facial regions that are considered to be irrelevant. - Contrast Equalization: This step was used to globally rescale the image intensities to standardize a robust measure of overall contrast or intensity variation. Estimation is a two-stage process where the image is first smoothed, and a non-linear function is then applied to compress large intensity values. The resulting images for samples acquired from each of the databases used are shown in Fig. 2. (ii) Generation of Multi-Pose Eye Templates: We randomly selected ten subjects from each dataset. Then, for each subject, we performed face registration, i.e., we loaded a face image, manually marked the coordinates of the eye centers, geometrically normalized the image (using rotation and scaling of the positions onto two fixed points), and cropped the left and right eye templates at a resolution of Finally, we generated database-specific left and right average eye templates. Note that we empirically determined that 10 subjects were sufficient enough to generate the eye templates that can achieve satisfactory eye detection results. (iii) Detection of Eye Regions: We applied template convolution on the normalized face images by, first, centering each of the generated eye templates on the top left corner of each face image, and then, computing the Pearson Product Moment correlation (PPMC) coefficient. After rotating the original face image to various angles (i.e., they were rolled as illustrated in Fig. 4) ranging from -45 o to +45 o in 5 o increments, the procedure was repeated, using both the left and right average eye templates (sample eye templates are illustrated in Fig. 5), for the entire image. The position where each (left or right) of the generated eye templates best matches (i.e., highest correlation coefficient in the image domain) the face image was the estimated position of the template within the image. The PPMC measure is illustrated in Equation 1, where X and Y are the image and template pixel intensity values, respectively, σ X and σ Y are their respective standard deviations, and µ x and µ y are the expected values of x and y, respectively (N is the total number of pixels). N i=1 P P MC = (X i µ X i )(Y i µ Y i ) (1) Nσ X σ Y (iv) Feature Extraction: We used two different feature Fig. 4. Illustration of the three rotational descriptors on face images acquired under variable conditions using different sensors (visible spectrum, SWIR and multi-spectral). Note that the synthetic face images were generated adjusting only the roll descriptor.

4 Fig. 6. (a) Illustration of the relations between true (C l and C r) and estimated pupil positions ( C l and C r); (b) The relative error with respect to the right eye (Fig. (b)). A circle with a radius of 0.25 relative error is drawn around the eye center. This figure is based on the one provided in [14]. Fig. 5. Description of the (a) left and right average eye templates generated from the FRGC face database, and used for testing eye detection on FRGC, and (b) the left and right average eye templates generated from the WVU (SWIR) face dataset, and used for testing eye detection on the same dataset. descriptors, namely the Local Binary (LBP) and the Local Ternary Patterns (LTP) [13]. The LBP patterns in an image were computed by thresholding 3 3 neighborhoods based on the value of the center pixel. Then, the resulting binary pattern was converted to a decimal value. Even though LBP is invariant to monotonic gray-level transformations, one of its drawbacks is that it tends to be sensitive to noise in nearuniform image regions. This is because the binary code is computed by thresholding the exact center of the pixel region. Alternatively, Local Ternary Patterns overcome the abovementioned limitation, and were used in the proposed method. LTPs are 3-valued codes extended from LBP. In LTPs, gray-levels f p in an image region of width ±t (user defined threshold) around f c (the gray level value at the central pixel c of a 3 3 neighbor) were quantized to zero. If the gray-levels were above ±t, they were quantized to +1, and if they were below ±t, they were quantized to -1. Thus, the indicator S (output from LTP) is defined as follows: S(f p f c ) = 1 if f p f c t; 0 if f p f c t; 1 if f p f c t. The difference when using LTPs instead of LBPs is that t can be adjusted to produce different patterns. This threshold also makes the LTP code more resistant to noise. (v) Eye Detection: Each face image was rotated 18 times (right and left, from -45 o to +45 o in 5 o increments), and 18 different eye region candidates were determined for each eye. However, the problem was to determine which left and right detected eye region candidates (out of a total of 36) were the best eye matches. The experimental results showed that the PPMC measure alone cannot be used to solve that problem, and therefore, the LTP feature descriptor was used. First, LTP patterns were computed using (i) the multi-pose eye templates and (ii) the 18 different eye region candidates. Then, we computed the similarity between the extracted features (2) of the eye templates against the eye region candidates. The measure used for that purpose was the chi-squared distance that is defined as follows: χ 2 (n, m) = 1 2 l 1 h n (k) h m (k) h n (k) + h m (k), (3) where h n and h m are the two histogram feature vectors, l is the length of the feature vector, and n and m are two sample vectors extracted from an image in the gallery and probe sets, respectively. (vi) Pupil Detection: The pupil coordinates of the left and right eyes were determined as the lowest pixel intensity values per region within each of the right and left eye regions. To validate the performance of our pupil detection system we used the relative error measure (D p ) based on the distances between the expected (true pupil coordinates acquired by manual annotation), and estimated pupil positions [14]. First, for each eye, we computed the distance between the manually annotated eye center C l, C r R 2, and the estimated eye center C l, Cr R 2 (Fig. 6), i.e., d l and d r for the left and right eye, respectively. Then, we determined the maximum distance between d l and d r (max(d l, d r )). The distance is normalized by dividing it by the distance between the annotated eye centers, denoted as C l C r. The measure is shown in the following equation: D p = max(d l, d r ) C l C r In an average human face, the distance between the inner eye corners equals the width of a single eye. Thus, a relative error of 0.25 equals a distance of half an eye width (Fig. 6(b)). In this paper, a pupil is considered detected if D p 0.25, and is rejected, otherwise. III. EXPERIMENTS A. Face Image Databases In this section, we present a description of the following databases: the UHDB11 (visible spectrum; unconstrained data), the WVU (Near-IR and SWIR spectra; constrained data), and the FRGC2 [12] (visible domain; constrained data). The WVU and FRCG were used to generate the pose-variable (4)

5 were resized to have the same spatial resolution as the IR and Red images. The XenICs camera was used for the acquisition of SWIR face images. The camera has an Indium Gallium Arsenide (InGaAs) Focal Plane Array (FPA) with 30 µm pixel pitch, 98% pixel operability and three-stage thermoelectric cooling. It has a relatively uniform spectral response from nm wavelength (lower SWIR band) across which the InGaAs FPA has a largely uniform quantum efficiency. The spectral response of the camera falls rapidly at wavelengths lower than 950 nm and near 1700 nm. (c) FRGC: We used a randomly selected subset from the controlled dataset of the FRGC database that consists of full frontal facial images acquired under two lighting conditions (two or three studio lights) and with two facial expressions (smiling and neutral). In total, we used the face images of 408 subjects, resulting in 1,632 images. The images were acquired using a 10 MP Canon PowerShot G2 camera [15]. Fig. 7. Depiction of face images from the UHDB11 database (Left Column). Eye/Pupil detection results using the proposed algorithm (Right Column). WVU and FRGC datasets that are composed of random pose variations of the original images (i.e., each image from the original databases was randomly rotated around z axis to different angles ranging from -45 o to +45 o in 5 o increments (Fig. 2). The UHDB11 database was used unaltered (realistic scenario). (a) UHDB11: This database consists of 1,602 face images that were acquired from 23 subjects under variable pose and illumination conditions. For each illumination condition, the subjects faced four different points inside the room (their face was rotated on the Y axis that is the vertical axis through the subjects head). For each Y-axis rotation, three images were also acquired with rotations on the Z axis (that extends from the back of the head to the nose). Thus, the face images of the database were acquired under six illumination conditions, with four Y and three Z rotations. Figure 7 depicts the aforementioned variations for a single subject [11]. (b) WVU: WVU database consist of images that were acquired using a DuncanTech MS3100 multi-spectral and a XenICs camera. The MS3100 was used to create the mutlispectral dataset of the database. The camera consists of three Charge Couple Devices (CCDs) and three band-pass prisms behind the lens in order to simultaneously capture four different wavelength bands. The IR and Red (R) sensors of the multi-spectral camera have spectral response ranges from 400 nm to 1000 nm. The Green (G) channel has a response from 400 nm to 650 nm, and the Blue (B) channel from 400 nm to 550 nm. Note that the IR and Red sensor outputs an image of size The Green and Blue images were recorded on a RGB Bayer pattern sensor and are therefore, one-third the resolution of the other images. Then, Green and Blue images B. Experimental Results The proposed method was evaluated using two different experiments. In the first experiment, we investigated the pupil detection rate and pixel distance accuracy of our method when using the UHDB11, the pose variable FRGC and WVU (multi-spectral and SWIR) databases. Pupil detection performance was computed for each eye and was measured as the number of accurately detected pupils divided by the total number of pupils in each dataset employed. Pupil pixel distance accuracy was measured as the Euclidean pixel distance between the true positions of each pupil center (by manual annotation) and the detected pupil (Table I). In the second experiment, the efficiency of our pupil detection algorithm was tested against the eye detection method proposed in [5] and a commercially available software (G8) provided by L1 Systems ( The manually annotated eye centers were used as ground truth and were compared with the automatically detected pupils. Experimental results are summarized in Table I. When the UHDB11 database was used, the proposed method has an increase in average performance of 48.25% and 20.44% over the benchmark eye detection method and L1 s G8, respectively. Similar results were obtained when the synthetic FRGC database was used, i.e., we had a 52.08% and 12.80% improved performance when compared to the benchmark and G8 methods, respectively. When the synthetic WVU Multispectral database was used, we achieved an increase of the average performance by 4.5% (over the benchmark) and 5% (over G8). When the synthetic WVU-SWIR was used, an increase of 18.5% over the benchmark and 2% over the L1 s G8 methods was achieved. IV. CONCLUSIONS AND FUTURE WORK We studied the problem of multi-band pupil detection using real and synthetic face images captured under controlled and uncontrolled conditions. Our method was designed to work well with challenging data and achieved a significantly high

6 TABLE I COMPARISON OF ALGORITHMS ON UHDB11 AND SIMULATED DATASETS. EUCLIDEAN DISTANCE IS THE DISTANCE IN PIXELS BETWEEN THE MANUALLY ANNOTATED PUPIL CENTERS TO THE AUTOMATICALLY DETECTED ONES. NOTE THAT THE WVU-MULTISPECTRAL DATASET CONSISTS OF 100 FACE IMAGES. SIMILARLY, THE WVU-SWIR DATASET CONSISTS OF 100 FACE IMAGES. Datasets Detection Accuracy Original [5] (%) G8 (%) Proposed (%) UHDB11 Left Eye Right Eye FRGC Left Eye Right Eye Multi-spectral Left Eye Right Eye SWIR Left Eye Right Eye Datasets Euclidean Distance Original [5] (pixels) G8 (pixels) Proposed (pixels) UHDB11 Left Eye Right Eye FRGC Left Eye Right Eye Multi-spectral Left Eye Right Eye SWIR Left Eye Right Eye detection rate. In particular, it yielded an average of 96.38% detection accuracy across all datasets, which is a 49.2% increase in average detection performance when compared to the method proposed by Whitelam et al. [5] (designed to work well only with frontal face images collected under constrained conditions). The commercial software (G8) performed well on data acquired under controlled conditions. However, our method performed consistently better than G8 across all datasets, achieving a 14.4% increase in average pupil detection performance. Another important achievement of our work was its efficiency when using the original face images of the UHDB11 dataset, where none of the face images were synthetically altered with pose and illumination variations (Table I). This was the most challenging scenario to test our method, and we obtained the highest increase in pupil detection accuracy over both the benchmark and G8 algorithms, i.e., the pupil detection accuracy was (on average) above 92%, and outperformed G8 by over 20%. ACKNOWLEDGMENT This work is sponsored in part through a grant from the Office of Naval Research (N ) and the Eckhard Pfeiffer Endowment Fund. We are grateful to all faculty and students at WVU and UH for their valuable participation and assistance with the WVU and UHDB11 databases, respectively. REFERENCES [1] P. Wang, M. Green, Q. Ji, and J. Wayman, Automatic eye detection and its validation, in Proc. on Computer Vision and Pattern Recognition, San Diego, CA, USA, June 2005, pp [2] C. W. Park, J. M. Kwak, H. Park, and Y. S. Moon, An effective method for eye detection based on texture information, in Proc. International Conference on Communications and Information Technology, Washington, DC, USA, November 2007, pp [3] Z. Xingming and Z. Huangyuan, An illumination independent eye detection algorithm, in Proc. International Conference on Pattern Recognition, Hong Kong, China, August 2006, pp [4] S. Asteriadis, N. Nikolaidis, A. Hajdu, and I. Pitas, An eye detection algorithm using pixel to edge information, in Proc. International Symposium on Communications, Control and Signal Processing, Marrakech, Morocco, March [5] C. Whitelam, Z. Jafri, and T. Bourlai, Multispectral Eye Detection : A Preliminary Study, in Proc. on International Conference on Pattern Recognition, Istambul, Turkey, August 2010, pp [6] S. Chen and C. Liu, Eye detection using color information and a new efficient SVM, in Proc. International Conference on Biometrics: Theory, Applications and Systems, Washington, DC, USA, September [7] C. H. Morimoto, D. Koons, A. Amir, and M. Flickner, Pupil detection and tracking using multiple light sources, Image and Vision Computing, vol. 18, no. 4, pp , [8] N. Kalka, T. Bourlai, B. Cukic, and L. Hornak, Cross-spectral Face Recognition in Heterogeneous Environments: A Case Study on Matching Visible to Short-Wave Infrared Imagery, in Proc. International Joint Conference on Biometrics, Washington, DC, USA, [9] T. Bourlai, N. Kalka, D. Cao, B. Decann, Z. Jafri, F. Nicolo, C. Whitelam, J. Zuo, D. Adjeroh, B. Cukic, J. Dawson, L. Hornak, A. Ross, and N. A. Schmid, Ascertaining Human Identity in Night Environments. DVSN, Springer, 2011, pp [10] T. Bourlai and Z. Jafri, Eye detection in the Middle-Wave Infrared Spectrum: Towards Recognition in the Dark, in International Workshop on Information Forensics and Security, Foz do Iguaçu, Brazil, [11] G. Toderici, G. Passalis, S. Zafeiriou, G. Tzimiropoulos, M. Petrou, T. Theoharis, and I. Kakadiaris, Bidirectional relighting for 3D-aided 2D face recognition, in Proc. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, June 2010, pp [12] P. Phillips, P. Flynn, T. Scruggs, K. Boyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek, Overview of the face recognition grand challenge, vol. 1, San Diego, CA, USA, June 2005, pp [13] X.Tan and B. Triggs, Enhanced local texture feature sets for face recognition under difficult lighting conditions, Transactions on Image Processing, vol. 19, no. 6, pp , [14] O. Jesorsky, K. J. Kirchberg, and R. W. Frischholz, Robust face detection using the Hausdorff distance. Springer, 2001, pp [15] P. J. Phillips, W. T. Scruggs, A. J. O Toole, P. J. Flynn, K. W. Bowyer, C. L. Schott, and M. Sharpe, FRVT 2006 and ICE 2006 largescale experimental results, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, pp , 2010.