TRADITIONAL optical imaging devices require appropriate

Transcription

1 2436 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 6, JUNE 2014 Common Feature Discriminant Analysis for Matching Infrared Face Images to Optical Face Images Zhifeng Li, Senior Member, IEEE, Dihong Gong, Yu Qiao, Senior Member, IEEE, and Dacheng Tao, Senior Member, IEEE Abstract In biometrics research and industry, it is critical yet a challenge to match infrared face images to optical face images. The major difficulty lies in the fact that a great discrepancy exists between the infrared face image and corresponding optical face image because they are captured by different devices (optical imaging device and infrared imaging device). This paper presents a new approach called common feature discriminant analysis to reduce this great discrepancy and improve optical-infrared face recognition performance. In this approach, a new learning-based face descriptor is first proposed to extract the common features from heterogeneous face images (infrared face images and optical face images), and an effective matching method is then applied to the resulting features to obtain the final decision. Extensive experiments are conducted on two large and challenging opticalinfrared face data sets to show the superiority of our approach over the state-of-the-art. Index Terms Face recognition, infrared face, face descriptor. I. INTRODUCTION TRADITIONAL optical imaging devices require appropriate illumination conditions to work properly, which is difficult to achieve satisfactorily in practical face recognition applications. To combat low illumination at nights or indoors, infrared imaging devices have been widely applied to many automatic face recognition (ARF) systems. The task of infrared-based ARF systems is to match a probe face image taken with the infrared imaging device to a gallery of face images taken with the optical imaging device, which is considered to be an important application of heterogeneous Manuscript received July 4, 2013; revised November 22, 2013 and March 5, 2014; accepted March 10, Date of publication April 8, 2014; date of current version April 28, This work was supported in part by the Natural Science Foundation of China under Grant and Grant , in part by the Shenzhen Basic Research Program under Grant JC A, Grant JCYJ , and Grant JCYJ , in part by the 100 Talents Program, Chinese Academy of Sciences, in part by the Guangdong Innovative Research Team Program under Grant D , and in part by the Australian Research Council Project under Grant FT and Grant DP The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Bulent Sankur. Z. Li, D. Gong, and Y. Qiao are with the Shenzhen Key Laboratory of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen , China ( zhifeng.li@siat.ac.cn; dh.gong@siat.ac.cn; yu.qiao@siat.ac.cn). D. Tao is with the Centre for Quantum Computation and Intelligent Systems, Faculty of Engineering and Information Technology, University of Technology, Sydney, NSW 2007, Australia ( dacheng.tao@uts.edu.au). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TIP Fig. 1. Examples of optical and corresponding infrared face images for (a) HFB dataset and (b) CUHK VIS-NIR dataset. face recognition [1] (also known as cross-modality face recognition). The most challenging issue in heterogeneous face recognition is that face images associated with the same person but taken with different devices may be mismatched due to the great discrepancy between the different image modalities (optical and infrared), which is referred to as modality gap. As illustrated in Figure 1, the infrared photos are usually blurred, low contrast, and have significantly different gray distribution compared to the optical photos. To combat the great modality gap, a variety of algorithms have been proposed in the literature which can be summarized as belonging to the following three categories. The first category of approaches is to convert an image from one modality to the other by synthesizing a pseudo-image from the query image such that the matching process can be done within the same modality. For instance, [2] applies a holistic mapping to convert a photo image into a corresponding sketch image, and in [3] [5] the authors used local patch-based mappings to convert images from one modality to the other for sketchphoto recognition. In [6], Xiong et al. synthesized VIS face images from NIR face images with pose rectification. The second category of approaches is to design an appropriate representation that is insensitive to the modalities of images. For example, Klare et al. [7] used SIFT feature descriptors and multi-scale local binary patterns to represent both the sketch and photo images. Zhang et al. [8] proposed a learningbased algorithm to capture discriminative local face structures and effectively match photo and sketch. In [9], Liu et al. designed a multi-scale common feature descriptor to combat the large intra-class difference incurred by the modality (VIS-NIR) difference. The third category of approaches is to compare the heterogeneous images on a common subspace IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 LI et al.: CFDA FOR MATCHING INFRARED FACE IMAGES TO OPTICAL FACE IMAGES 2437 where the modality difference is believed to be minimized [10] [19]. For example, Tenenbaum et al. [10] applied the Bilinear Model (BLM) by Singular Value Decomposition (SVD) to develop a common content (associated with identity) space for a set of different styles (corresponding to modalities). In [11], Yi et al. used the Canonical Correlation Analysis (CCA) technique to construct a common subspace where the correlations between infrared and optical images can be maximized. In [12], Li et al. applied the CCA to cross-pose face recognition. In [13], Sharma et al. applied the Partial Least Squares (PLS) method to derive a linear subspace in which cross-modality images are highly correlated, while at the same time preserving variances more effectively than the previous CCA method. In [16], Lei et al. proposed an effective subspace learning framework called coupled discriminant analysis for heterogeneous face recognition. In [17], a generic HFR framework was proposed in which both probe and gallery images are represented in terms of non-linear kernel similarities to a collection of prototype face images to enhance heterogeneous face recognition accuracy. Motivated by the challenging task of VIS-NIR face recognition, in this paper, we propose a new approach called common feature discriminant analysis (CFDA) for matching infrared face images to optical face images. Although we only demonstrate the application in VIS-NIR face recognition, the basic idea behind our approach is based on general heterogeneous face recognition, thus it can be easily extended to many other similar applications like matching sketch to photography, or matching thermal to photography. In the CFDA approach, a new learning-based face descriptor is first proposed to reduce the modality discrepancy in the feature extraction stage. The key idea is to learn an encoding mechanism which will turn the optical and infrared face images into their encoded images to reduce the modality gap between them. The proposed new descriptor maximizes the correlation of the encoded images of the same object, and is thus beneficial to performance improvement. Extensive experiments are conducted on two large and challenging optical-infrared face datasets to show the significant performance improvement of our new approach over the state-of-the-art. The major contributions of this work lie in the proposed new feature descriptor that is well suited for matching infrared face images to optical face images. From experimental results, we can clearly see that our feature descriptor significantly outperforms the existing ones, regardless of which matching scheme is adopted. Moreover, this work focuses on feature representation, so it can be easily combined with the existing classification algorithms to produce better performance. Hence it is very easy to implement. The rest of this paper is organized as follows. In Section II, we describe the proposed CFDA approach for optical-infrared face recognition. Experimental results and analysis are given in Section III. Finally, we conclude this paper in Section IV. II. PROPOSED APPROACH In this section, we propose a new approach called common feature discriminant analysis (CFDA) for optical-infrared face recognition. In the CFDA approach, a new learning-based feature descriptor is first developed to learn a set of optimal hyperplanes to quantize continuous vector space into discrete partitions for common feature extraction, and an effective discriminant analysis technique is then applied for feature classification. The details are elaborated in the following subsections. A. Our Descriptor Vector quantization is an effective technique in mapping vectors of continuous space into discrete code representations, and has been widely used to create discrete image representations for object recognition. An image can be turned into an encoded image by converting each pixel into a specific code using the vector quantization technique. Various algorithms such as mean shift [20], k-means, random projection tree [21], [22], and random forest [23], [24] have been proposed to quantize a continuous space to form discrete partition cells for vector quantization. In this section, we design a hyperplane-based encoding method for effective feature representation for heterogeneous face images. Mathematically, a hyperplane divides a feature space R D 1 into two partitions, namely the positive part and the negative part. A vector x R D 1 will be classified to the negative part if: h( x) = w T x + b 0. (1) Otherwise, the vector will be classified to the positive part. Here the w is a unit normal vector and b is the offset. We encode face images by turning each of the pixel into a decimal code representing which spatial partition the pixel falls into, as illustrated in Figure 2. Each pixel is corresponding to a point in a R D 1 space, and thus any face image can be represented as a set of points I = { x 1,..., x M },where M = H W is the total number of pixels of the image. The detailed procedure in converting a pixel into its corresponding vector is elaborated as follows. Suppose we are given K pairs of training persons which can be represented as χ = {( x i 1, x i 2 ) }, i = 1,...,N where superscript 1 represents optical faces and 2 for infrared faces, subscript i indicates the i-th pixel vector, and N = H W K is the total number of vector pairs. Then we can compute the projections to their corresponding hyperplanes using (1), which is denoted as: γ = {( y 1 i, y2 i ) }, i = 1,...,N, (2) where yi 1 ( ) ( ) = h 1 x 1 i and y 2 i = h 2 x 2 i. Denote that X1 = [ x 1 1,..., x N 1 ], X 2 = [ x 1 2,..., x N 2 ], Y 1 = [y1 1,...,y1 N ],and Y 2 =[y1 2,...,y2 N ]. Since the vector pairs are associated with the same subject, it is expected that the projections on their hyperplanes will be highly correlated. Formally, we define a Kullback Leibler (KL) divergence [25], [26] to measure the correlation: KL P (Y 1, Y 2 ) P (Y 1 ) P (Y 2 ). (3) The KL divergence measures the distance between the joint distribution and the distribution of the independent assumption. If the pairs (Y 1, Y 2 ) are statistically independent, the KL divergence is zero. The greater the correlation between

3 2438 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 6, JUNE 2014 Fig. 2. The pipeline for extracting the feature descriptor. For each pixel, we first sample its five d-neighbor (Radii = d) pixels for each direction (the figure shows one of four directions, with blue arrows), and then subtract the center pixel value. Finally the centered vector is normalized into the unit L2-norm to form the associated pixel vector of that direction. Each pixel is associated with four vectors, forming four sets of training vectors that are used to train four encoders. Each encoder consists of two sets of mutually orthogonal hyperplanes (we only show one for illustration), which divide the vector space into four partitions. Vectors of each direction are encoded into a 2-bit value, according to the partition in which the vector lies (i.e. 00 for the first partition, 01 for the second partition, and so forth). Finally, the four 2-bit values are concatenated to form an 8-bit value that will be converted into a decimal value (from 0to 255) as the code. (Y 1, Y 2 ), the more the joint distribution will diverge from the independent assumption and the KL value. Our aim is to derive two hyperplanes H1 and H 2, such that the KL divergence can be maximized. Observed that the projections of the optical vectors and infrared vectors to their hyperplanes approximately follow joint Gaussian distribution (Empirically, we have tested that the projections are approximately Gaussian distributed), then the KL divergence can be simplified, as below: KL = ln cov (Y 1, Y 1 ) cov (Y 2, Y 2 ), (4) where is the determinant of the covariance matrix given by: [ ] cov (Y1, Y = 1 ) cov (Y 1, Y 2 ). (5) cov (Y 2, Y 1 ) cov (Y 2, Y 2 ) Substituting (5) into (4), we can obtain the equivalent optimization formula: w T max 1 1,2 w 2 w 1, w 2 w T 1 1,1 w 1 w T 2 2,2, (6) w 2 s.t. w 1 = 1, w 2 = 1, where 1,2 = X 1X T 2, 1,1 = X 1X T 1,and 2,2 = X 2 X T 2. X 1 and X 2 are the mean-centered data matrix whose columns consist of centered observations. The equation (6) is the CCA problem [27], [28], which can be solved with the following generalized eigen-decomposition equation: [ w ] [ w ] λc 1 D = C 1 w O, 2 w 2 where [ C D = 0 2,2 1,1 0 ] [ 0 and C O = T1,2 0 1,2 ]. In solving the generalized eigen-decomposition problem, we pick up the eigenvector corresponding to the largest eigenvalue as the eigenvalue λ is corresponding to the value that we want to maximize. The solution of (6) provides the normal vectors to the optimal hyperplanes that maximize the correlation of the distances between the optical image and infrared image feature vectors. To determine the offset values b 1 and b 2, we apply the KL criteria again on the quantized values of the feature vectors. Formally, Ŷ 1 = sign(y 1 ), (7) Ŷ 2 = sign(y 2 ), (8) and b 1 and b 2 are solved in a brute-force manner: choose (b 1, b 2 ) where b 1 Y 1 and b 2 Y 2 such that the following KL value can be maximized: ) ) ) KL P (Ŷ1, Ŷ 2 P (Ŷ1 P (Ŷ2. (9) One thing to note is that the computational cost of (b 1, b 2 ) in a brute-force manner is rather low. To show this point, we compute the KL distance defined in (4) for each pair of (b 1, b 2 ) taken uniformly from b 1 [min (Y 1 ) max (Y 1 )]and b 2 [min (Y 2 ) max (Y 2 )]. We found that dividing the solution regions into 1000 intervals can achieve a good approximation to the problem. For the computation of each KL distance, we only need to compute three variances which is O ( n 2),where n is the number of training pixel vectors. With the above algorithm, we obtain a pair of hyperplanes H1 and H 2. One thing to note is that we can introduce more pairs of hyperplanes to further divide the vector space if necessary. The k-th pair of hyperplanes can be obtained by solving the same eigen-decomposition problem but taking the eigenvectors corresponding to the k-th largest eigenvalue. For instance, if we already have k-1 pairs of hyperplanes {( H 1, j, H 2, j ) j = 1, 2,...,k 1 },

4 LI et al.: CFDA FOR MATCHING INFRARED FACE IMAGES TO OPTICAL FACE IMAGES 2439 Fig. 3. Illustration for encoded face images of (a) CUHK VIS-NIR face dataset and (b) HFB VIS-NIR face dataset. The first row shows the original optical face images as well as their encoded images of different radii, and the second row shows the corresponding infrared images. then the k-th pair of hyperplanes (H1,k, H 2,k ) can be obtained by selecting the eigenvectors corresponding to the k-th largest eigenvalues. The property of the eigenvectors guarantees that H1,k is orthogonal to } {H1, j j = 1,.., k 1, and also for H2,k {H and } 2, j j = 1,.., k 1. With M pairs of hyperplanes, we can divide each vector space (optical or infrared) into 2 M partitions, and any optical or infrared face image can be turned into a discrete coded representation by assigning a code to each of its pixels according to the partition in which the associated vector lies. To regularize the partitions with the existing geometry information, we encode each of the pixels in any face image with four associated vectors, as illustrated in Figure 2. Each pixel is associated with four vectors, one for each direction. For each direction, we split the vector space into four partitions with two orthogonal hyperplanes. The partitions of each direction are trained independently, yielding four sets of hyperplanes, with each set dividing the space of that direction into four partitions. For each direction, there are two orthogonal hyperplanes, yielding four encoded statuses, i.e. 00, 01, 10, 11. The encoded statuses of each direction are concatenated to form a final encoded status; thus, each pixel of the image has 4 4 = 256 discrete statuses, and we assign one of these status values (by converting the 8-bits binary value into decimal value, i.e. ( ) 2 (131) 10 ) to each pixel to form an encoded image. In our study, we sample the vectors of each pixel with multiple scales, i.e. sampling radii = {3, 5, 7, 9}. Hyperplanes are trained independently on each of the sampling patterns, and the final feature representation of the input image is derived by concatenating the feature representation of each of the encoded images. In Figure 3, we illustrate some examples of encoded face images. With face images encoded, we can extract feature representations using a densely sampling technique as follows. 1. Divide the whole encoded image into a set of overlapping patches with size c c pixels (the step between adjacent patches is s). 2. Compute the histogram, over each patch, of the frequency of each code occurring which gives a feature vector for the patch. 3. Concatenate the outputs of each patch into a long vector to form the final face feature. B. Matching Framework By applying the aforementioned feature descriptor, we can obtain a high-dimensional feature vector for each optical or infrared image. In our study, each face image is divided into 165 patches with each one represented by a 256-dimensional feature vector. We used four scales in our experiments, i.e. sampling radii ={3, 5, 7, 9}. So the feature dimension for the resulting feature vector is in our experiments. Direct processing such a high-dimensional feature vector might incur problems. First, high dimensionality places a huge demand on computer memory. Second, a more serious problem is the lack of sufficiently large datasets to train the high dimensional feature, which leads to overfitting [29]. In order to address these issues, we design a two-level matching framework. The matching framework involves two levels of subspace analysis. In the first level, the large feature vector is first divided into multiple segments of smaller feature vectors. Discriminant analysis is performed separately on each segment to extract the discriminant features. The goal for the first level is to generate more discriminative projections to reduce intraclass variations and avoid over-fitting. In the second level, projected features from all the segments are then combined, with PCA for efficient recognition. The two-level matching framework is based on the local feature-based discriminant analysis (LFDA) approach in [7]. The choice of such a matching framework is based on the reported success in heterogeneous face recognition [7]. Figure 4 presents an illustration of this algorithm. The details of this algorithm are described as follows. The first level of subspace analysis is conducted as follows: 1. Divide the original feature vector equally into several feature segments. Project each segment S to its PCA subspace using the transform matrix T 1, which is computed from the corresponding segment in the training set, F 1 = T T 1 S.

5 2440 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 6, JUNE 2014 Fig. 4. Illustration of the two-level matching framework in Section II.B. 2. Compute the intrapersonal subspace using the withinclass scatter matrix in the reduced PCA subspace. Adjust the dimensions of the intra-class subspace to reduce the intra-class variations. 3. Compute the class center of each person on the training set. Project all the class centers onto the intrapersonal subspace and then normalize the projections by intrapersonal eigenvalues to compute whitened feature vectors, F 2 = T2 T F 1, where T 2 is the normalized projection matrix. 4. Calculate the between-class scatter matrix based on the whitened feature vectors. 5. Compute the final discriminant feature vector using the PCA transform matrix T 3 based on the between-class scatter matrix calculated in step 4. F 3 = T3 T F 2. Procedures in the second level of subspace analysis are: 6. Combine the extracted discriminant feature vectors from each segment to form a new feature vector V. 7. To remove the redundant information, apply PCA on V to obtain the final feature vector F for classification, F = T4 T V, where T 4 is the corresponding PCA projection matrix. C. Discussions The CFDA approach is proposed specifically for handling the optical-infrared face recognition problem. In the feature extraction stage, a learning-based feature descriptor is developed to maximize the correlations between the optical face images and corresponding infrared images. In this way, the modality gap between the two kinds of face images can be significantly reduced; hence, it is expected that the resulting features will be well-suited to the optical-infrared face recognition problem, as supported by the experimental results in Section III. Our new feature descriptor differs significantly from state-of-the-art descriptors, such as the widely used HOG [30] and LBP [31], [32], in the literature. Instead of encoding the images using a handcrafted encoding scheme, our feature descriptor learns a new encoding scheme to encode the common micro-structure of both the optical and infrared face images for effective feature representation. Our experimental results also support the effectiveness of our new descriptor over state-of-the-art descriptors. Our new feature descriptor also differs significantly from the CITE in [8]. The major difference between them are summarized as follows. (1) CITE is inherently a treebased encoding method, while our feature descriptor is inherently a binary encoding scheme. (2) Unlike CITE, we encode a pixel with four directions to make full use of the geometry information. (3) From experimental comparison, we can clearly see that our new descriptor significantly outperforms CITE in [8]. This also reflects the significant difference between them. III. EXPERIMENTS In this section, we conduct extensive experiments on two large and challenging optical and infrared face datasets to investigate the effectiveness of our new approach. The first dataset is collected by CUHK, called CUHK VIS-NIR face dataset. It consists of 2876 different persons with each one having one optical photo and one corresponding near infrared photo. Some examples from this dataset are shown in Figure 1(b). In our experiments, we divide the dataset into two parts: the images of 1438 persons are used for training, and those of the remaining 1438 persons are used for testing. There is no overlap between the training set and the testing set. The second dataset is a public-domain heterogeneous face dataset,

6 LI et al.: CFDA FOR MATCHING INFRARED FACE IMAGES TO OPTICAL FACE IMAGES 2441 Fig. 5. Illustration of normalized optical face images and the corresponding infrared face images for (a) CUHK VIS-NIR face dataset; and (b) HFB VIS-NIR face dataset. TABLE I COMPARISON OF OUR DESCRIPTOR WITH STATE-OF-THE-ART DESCRIPTORS USING THE NEAREST NEIGHBOR MATCHING SCHEME Fig. 6. The illustration for parameter exploration of the sampling radii. called heterogeneous face biometrics (HFB) [33]. It consists of 200 subjects with probe images captured in the near-infrared spectrum ( 780 1,100 nm) and gallery images captured in the visible spectrum. For fair comparison, we use the same split scheme with [17] where 133 persons are used as training and the rest 67 persons are used as testing. The dataset is split randomly, and the results are reported as the average over 10 random runs. To better evaluate the recognition performance, we preprocess the face images through the following steps: 1) rotate the face images to align the vertical face orientation; 2) scale the face images so that the distance between the centers of eyes is the same for all images; 3) crop the face images (into pixels) to remove the background and the hair region. Figure 5 shows example images of infrared and optical data after preprocessing. We use FaceVACS (a commercial face recognition engineer) [34] to help detect the faces and locate the eyes. We further manually aligned a small set of training faces (98 pairs of training faces) with a face mask (which are divided into many triangle regions based on which we warp the faces into frontal ones) to help establish a good one-to-one correspondence of pixels. We use the 98 pairs of training faces to learn our encoding model and use all the 1438 pairs of training faces to learn the matching framework. For testing faces, we do not need this mask to warp the face to build pixel correspondence. A. Parameter Exploration The proposed algorithm has three parameters: the sampling radii r, the size of patch c, and the step of patches s. In our system, we empirically set the size of patch as pixels with step 8 pixels, which is a very popular configuration in the literatures. We explore the effect of sampling radii r by reporting the averaged 5-fold rank-1 recognition accuracies of different sampling radii on the HFB face dataset, as shown in Figure 6(a). We can see that the feature descriptor achieves the optimal performance around radii = 5, thus we choose a fusion of radii ={3, 5, 7, 9} in our system. B. Experiment on the CUHK VIS-NIR Face Dataset We first investigate the effectiveness of our descriptor and compare it with several state-of-the-art descriptors in heterogeneous face recognition: local binary pattern (LBP) [31]

7 2442 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 6, JUNE 2014 TABLE II COMPARISON OF OUR DESCRIPTOR WITH STATE-OF-THE-ART DESCRIPTORS USING THE PCA+LDA SCHEME Fig. 7. Comparison of our descriptor with state-of-the-art descriptors using the nearest neighbor matching scheme. TABLE III COMPARISON OF OUR DESCRIPTOR WITH STATE-OF-THE-ART DESCRIPTORS USING THE TWO-LEVEL MATCHING FRAMEWORK Fig. 8. Comparison of our descriptor with state-of-the-art descriptors using the PCA+LDA matching scheme. with 59-bits U2 encoding, multi-scale local binary pattern (MLBP) [32] with scales {1, 3, 5, 7}, histogram of orientation gradient (HOG) [30] with 12 discrete orientations, and a newly developed descriptor for heterogeneous face recognition called coupled-information tree encoding CITE [8]. We use three matching schemes for extensive performance evaluation: (i) nearest neighbor matching using Euclidean distance, (ii) PCA+LDA [35], and (iii) the two-level matching scheme presented in Section II.B. The comparative results for both face identification and face verification are reported in Tables I III and Figures 7 9. From these results, we make the following observations. First, it is encouraging to see that our new face descriptor significantly outperforms the state-of-the-art descriptors by a clear margin in terms of both face identification and face verification tasks, regardless of which matching scheme is used. This clearly shows the superiority of our new descriptor over existing descriptors. Second, the two-level matching scheme significantly boosts recognition performance, regardless of which descriptor is used. Finally, by integrating the proposed descriptor and the two-level matching scheme, we obtain the best identification Fig. 9. Comparison of our descriptor with state-of-the-art descriptors using the two-level matching scheme. rate, 80.19%. Considering the difficulty of this dataset, this is a very encouraging result. Next, we investigate the performance of our CFDA framework and also compare our approach with several newly developed algorithms in the literature for heterogeneous

8 LI et al.: CFDA FOR MATCHING INFRARED FACE IMAGES TO OPTICAL FACE IMAGES 2443 TABLE IV COMPARISON OF OUR NEW APPROACH WITH THE STATE-OF-THE-ART HETEROGENEOUS FACE RECOGNITION ALGORITHMS ON THE INFRARED-OPTICAL DATASET.LISTED ARE THE RANK-1 IDENTIFICATION ACCURACIES TABLE V COMPARISON OF OUR APPROACH WITH THE STATE-OF-THE-ART RESULTS ON HFB TABLE VI COMPARISON OF OUR FOUR-DIRECTION ENCODING METHOD AGAINST THE ENCODING METHOD BASED ON A DECISION TREE [8]. LISTED ARE THE RANK-1 IDENTIFICATION ACCURACIES Fig. 10. data set. Illustration of the rank-1 results by changing the size of the training The comparative results are reported in Table V. The parameters for the algorithms in Table V are tuned to the best settings according to their papers. It is encouraging to see that our approach also substantially outperforms the state-of-the-art on the HFB dataset. face recognition. The comparative results are reported in Table IV. It is very clear that our approach achieves superior performance over the state-of-the-art methods in the literature. Finally, we evaluate the performance of our approach by gradually increasing the number of training samples and at the same time keeping the testing dataset unchanged. We compare our algorithm against the approach in [17] that is the best-performing method among the existing methods in Table IV. The rank-1 recognition accuracies are reported in Figure 10, from which we can clearly see that our approach consistently outperforms the approach in [17] by a clear margin. This shows the robustness of our algorithm. C. Experiment on the HFB Face Dataset In this subsection, we compare our approach with several newly developed methods in heterogeneous face recognition on a public-domain database, the HFB face dataset [33]. Following the configuration of training and testing split for HFB dataset in [17], we use 133 persons as training and the remaining 67 persons as testing. The dataset is split randomly, and the results are reported as the average over 10 random runs. D. Additional Experiment A major difference between our encoding method and that in CITE is that our method encodes a pixel with four directions (in order to take use of geometry information). It is desirable to design an additional experiment to show the advantage of our four-direction encoding method. The comparative results are shown in Table VI. We compare our four-direction encoding method against the encoding method based on a decision tree [8] that has 256 partitions (the same as ours). For fair comparison, we use the same training and testing split on the CUHK dataset. One thing to note is that the feature length of our four-direction encoding is the same as that of the decision tree method. From Table VI, we can clearly see that our fourdirection encoding method notably outperforms the encoding method in [8]. IV. CONCLUSIONS AND FUTURE WORK In this paper, we have proposed a new approach called common feature discriminant analysis (CFDA) for matching infrared face images to optical face images. In CFDA, a new descriptor is first developed to effectively represent both

9 2444 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 6, JUNE 2014 optical and infrared face images to reduce the modality gap, and a two-level matching method is subsequently applied for fast and effective matching. Extensive experiments on two large and challenging optical-infrared face datasets show the significant improvement of our new approach over the state-ofthe-art. In the future, we will extend our approach to general cross-modality face recognition problems such as sketch-photo and high-low resolution face matching. ACKNOWLEDGMENT The authors would like to thank Prof. Anil K. Jain and Dr. Brendan Klare for their useful suggestions and comments. REFERENCES [1] S. Z. Li, Heterogeneous face biometrics, in Encyclopedia of Biometrics. New York, NY, USA: Springer-Verlag, [2] X. Tang and X. Wang, Face sketch recognition, IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 1, pp , Jan [3] B. Xiao, X. Gao, D. Tao, Y. Yuan, and J. Li, Photo-sketch synthesis and recognition based on subspace learning, Neurocomputing, vol. 73, pp , Jan [4] X. Wang and X. Tang, Face photo-sketch synthesis and recognition, IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 11, pp , Nov [5] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, Nonlinear approach for face sketch synthesis and recognition, in Proc. IEEE Comput. Soc. Conf. CVPR, Jun. 2005, pp [6] P. Xiong, L. Huang, and C. Liu, A method for heterogeneous face image synthesis, in Proc. 5th IAPR ICB, Apr. 2012, pp [7] B. Klare, Z. Li, and A. K. Jain, Matching forensic sketches to mug shot photos, IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 3, pp , Mar [8] W. Zhang, X. Wang, and X. Tang, Coupled information-theoretic encoding for face photo-sketch recognition, in Proc. IEEE Conf. CVPR, Jun. 2011, pp [9] S. Liu, D. Yi, Z. Lei, and S. Z. Li, Heterogeneous face image matching using multi-scale features, in Proc. 5th IAPR ICB, Apr. 2012,pp [10] J. B. Tenenbaum and W. T. Freeman, Separating style and content with bilinear models, Neural Comput., vol. 12, no. 6, pp , [11] D. Yi, R. Liu, R.-F. Chu, Z. Lei, and S. Z. Li, Face matching between near infrared and visible light images, in Proc. Int. Conf. Adv. Biometrics, 2007, pp [12] A. Li, S. Shan, X. Chen, and W. Gao, Maximizing intra-individual correlations for face recognition across pose differences, in Proc. IEEE Conf. CVPR, Jun. 2009, pp [13] A. Sharma and D. W. Jacobs, Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch, in Proc. IEEE Conf. CVPR, Jun. 2011, pp [14] W. Yang, D. Yi, Z. Lei, J. Sang, and S. Z. Li, 2D 3D face matching using CCA, in Proc. 8th IEEE Int. Conf. Autom. Face Gesture Recognit., Sep. 2008, pp [15] S. Liao, D. Yi, Z. Lei, R. Qin, and S. Li, Heterogeneous face recognition from local structures of normalized appearance, in Proc. 3rd ICB, 2009, pp [16] Z. Lei, S. Liao, A. K. Jain, and S. Z. Li, Coupled discriminant analysis for heterogeneous face recognition, IEEE Trans. Inf. Forensics Security, vol. 7, no. 6, pp , Dec [17] B. Klare and A. K. Jain, Heterogeneous face recognition using kernel prototype similarities, IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 6, pp , Jun [18] F. Nicolo and N. A. Schmid, Long range cross-spectral face recognition: Matching SWIR against visible light images, IEEE Trans. Inf. Forensics Security, vol. 7, no. 6, pp , Dec [19] Z. Lei, C. Zhou, D. Yi, A. K. Jain, and S. Z. Li, An improved coupled spectral regression for heterogeneous face recognition, in Proc. 5th IAPR ICB, 2012, pp [20] F. Jurie and B. Triggs, Creating efficient codebooks for visual recognition, in Proc. 10th IEEE ICCV, vol. 1. Oct. 2005, pp [21] Z. Cao, Q. Yin, X. Tang, and J. Sun, Face recognition with learningbased descriptor, in Proc. IEEE Conf. CVPR, Jun. 2010, pp [22] J. Wright and G. Hua, Implicit elastic matching with random projection for pose-variant face recognition, in Proc. CVPR, 2009, pp [23] F. Moosmann, B. Triggs, and F. Jurie, Fast discriminative visual codebooks using randomized clustering forests, in Proc. NIPS, 2007, pp [24] J. Shotton, M. Johnson, and R. Cipolla, Semantic texton forests for image categorization and segmentation, in Proc. IEEE Conf. CVPR, Jun. 2008, pp [25] S. Kullback and R. A. Leibler, On information and sufficiency, Ann. Math. Statist., vol. 22, no. 1, pp , [26] S. Kullback, Letter to the editor: The Kullback Leibler distance, Amer. Statist., vol. 41, no. 4, pp , [27] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, Canonical correlation analysis: An overview with application to learning methods, Neural Comput., vol. 16, no. 12, pp , [28] D. Weenink, Canonical correlation analysis, in Encyclopedia of Statistics in Behavioral Science. New York, NY, USA: Wiley, 2003, pp [29] S. Raudys and A. Jain, Small sample size effects in statistical pattern recognition: Recommendations for practitioners, IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 3, pp , Mar [30] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, in Proc. IEEE Comput. Soc. Conf. CVPR, vol. 1. Jun. 2005, pp [31] T. Ahonen, A. Hadid, and M. Pietikainen, Face recognition with local binary patterns, in Proc. ECCV, 2004, pp [32] T. Mäenpää and M. Pietikäinen, Multi-scale binary patterns for texture analysis, in Proc. SCIA, Gothenberg, Sweden, 2003, pp [33] S.Z.Li,Z.Lei,and M.Ao, The HFB face database for heterogeneous face biometrics research, in Proc. 6th IEEE Workshop OTCBVS, CVPR, Miami, FL, USA, Jun. 2009, pp [34] Cognitec Systems GbmH, Dresden, Germany. FaceVACS Software Developer Kit [Online]. Available: [35] P. Belhumeur, J. Hespanda, and D. Kriegman, Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection, IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp , Jul Zhifeng Li (M 06 SM 11) received the Ph.D. degree from the Chinese University of Hong Kong in After that, he was a Post-Doctoral Fellow with the Chinese University of Hong Kong and Michigan State University for several years. He is currently an Associate Professor with the Shenzhen Institutes of Advanced Technology, Chinese Academy of Science. His research interests include computer vision, pattern recognition, and multimodal biometrics. He is a regular member of the program committee for the most important computer vision conferences, including the International Conference on Computer Vision, the International Conference on Computer Vision and Pattern Recognition, and the European Conference on Computer Vision. He also serves as a reviewer for a number of major journals, including the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, International Journal of Computer Vision, IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, and IEEE TRANSAC- TIONS ON INFORMATION FORENSICS AND SECURITY. Dihong Gong received the B.E. degree in electrical engineering from the University of Science and Technology of China in He is currently an Inter-Student with the Shenzhen Institutes of Advanced Technology, Chinese Academy of Science. His current research interests include machine learning methods for face recognition, image retrieval, and image-based human age estimation.

10 LI et al.: CFDA FOR MATCHING INFRARED FACE IMAGES TO OPTICAL FACE IMAGES 2445 Yu Qiao (S 05 M 07 SM 13) received the Ph.D. degree from the University of Electro- Communications, Japan, in He was a JSPS Fellow and Project Assistant Professor with the University of Tokyo from 2007 to He is currently a Professor with the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences. His research interests include pattern recognition, computer vision, multimedia, image processing, and machine learning. He has authored more than 90 papers. He was a recipient of the Lu Jiaxi Young Researcher Award from the Chinese Academy of Sciences in Dacheng Tao (M 07 SM 12) is a Professor of Computer Science with the Centre for Quantum Computation and Information Systems and the Faculty of Engineering and Information Technology with the University of Technology, Sydney. He mainly applies statistics and mathematics for data analysis problems in computer vision, machine learning, multimedia, data mining, and video surveillance. He has authored and co-authored more than 100 scientific articles at top venues, including the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, TRANSACTIONS ON IMAGE PROCESSING, Artificial Intelligence and Statistics, Computer Vision and Pattern Recognition, and International Conference on Data Mining (ICDM), and was the runner-up for the Best Theory/Algorithm Paper Award at the IEEE ICDM 07.