Face Synthesis from Near-Infrared to Visual Light Via Sparse Representation

Transcription

1 Face Synthesis from Near-Infrared to Visual Light Via Sparse Representation Zeda Zhang, Yunhong Wang, Zhaoxiang Zhang Laboratory of Intelligent Recognition and Image Processing, Beijing Key Laboratory of Digital Media, School of Computer Science and Engineering, Beihang University, Beijing, , China Abstract This paper presents a novel method for synthesizing artificial visual light (VIS) face images from near-infrared (NIR) inputs. Active NIR imaging is now widely employed because it is unobtrusive, invariant of environmental illuminations, and can penetrate glasses and sweats. Unfortunately, NIR imaging exhibits discrepant photic properties compared with VIS imaging. Based on recent results of research on compressive sensing, natural images can be compressed and recovered with an overcomplete dictionary by sparse representation coefficients. In our approach a pairwise dictionary is trained from randomly sampled coupled face patches, which contains sparse coded base functions to reconstruct representation coefficients via l 1 -minimization. We will demonstrate that this method is robust to moderate pose and expression variations, and is efficient in computing. Comparative experiments are conducted with state-ofthe-art algorithms. 1. Introduction In the recent years, face recognition systems has seen remarkable improvements in accuracy and robustness [1], yet the factor of illumination variance remains a challeng for systems based on visible band mugshots. For acquisition of illumination invariant face images Li et al. [6] present an active NIR imaging system. Satisfactory classification result can be attained when both enrollment and query sets are composed of NIR images. Unfortunately, as NIR face photos are rarely captured or stored in social life, the problem to match NIR query images to a VIS gallery is arisen. This paper presents a framework to synthesize VIS face images from NIR inputs, which can be applied for recognition. Potential applications of this work include legitimate or security tasks when we need artificial VIS images for automatic or manual recognition, or to be kept as records. Images captured from different modalities are heterogeneous images. To match them many methodologies are proposed. Homomorphic filtering operations like HOG and DOG are employed to bridge the gap between VIS and NIR face photos [19] [15] [7]. Tang et al. [1] adopt principal component analysis (PCA) for eigentransformation. Lin et al. [8] propose Common Discriminant Feature Extraction, in which two transforms are simultaneously learned to transform the samples in both modalities respectively to a common feature space. Yi et al. [0] employ canonical correlation analysis (CCA) to project the two extracted linear discriminant analysis (LDA) feature vectors into a common space. Lei et al. [] propose Coupled Spectral Regression to seek two projections, which are derived from the view of graph embedding and spectral regression. Shao et al. [11] perform tensor analysis on heterogeneous image ensembles. Brendan et al. [4] perform LDA on a collection of random subspaces of feature vectors to learn discriminative projections. In these methods, the heterogeneous face photos are transformed into a common feature space for comparison. Heterogeneous synthesis of facial images is a process of generating a face image from an input which is taken from another modality, e.g. synthesizing a VIS photo from a s- ketch, or a 3D face, or a NIR photo, etc. and vice versa. Heterogeneous synthesis based methods provide a framework that we generate a synthesized VIS facial image from the NIR input before we apply the synthesized image for recognition. Inspired by the idea of local linear embedding (LLE), Liu et al. [9] estimate each image patch by approximately representing the NIR patches as weighted sum of their K nearest neighbors (KNN), and embed these patches by their neighbors as local linear mappings (LLM). Chen et al. [3] propose to decide the embedding coefficients of KNN patches by their multi-resolution local binary pattern (LBP) similarities. These KNN-based methods are efficient when the images do not show much expression and pose variations, in which case the images of both training and testing are well registered. When this condition no longer holds, in some regions of the reconstructed images may appear bad blocks or severe blurs, for the synthesized patches are reconstructed exclusively by the training patches at exactly the same locations. Especially for the LBP-KNN method [3], adjoining or even overlapping patches may come from /11/$ IEEE

2 Figure 1. The framework of our algorithm. training images with distinct poses and expressions, because LBP features are inherently robust to rotations and expressions. In this paper, we present a framework for heterogeneous synthesis from the perspective of sparse signal representation []. The general framework of our approach is presented in Fig. 1. Similar to the aforementioned KNN-based methods, our method also operates on image patches. In our methods, a single overcomplete dictionary is compressed from randomly sampled patch couples in the training stage, which is used to reconstruct the synthesized patches by l 1 - minimization in the test stage. As the patches at all locations are reconstructed from a common dictionary, it is no longer matter if the images are not strictly registered because of varied poses and expressions.. Background Tissues at human skin surface can be modeled as many layers according to their varied ratios of substances with distinct photic properties, i.e. melanin, water, chromophores, etc. These layers display different absorption rates and reflectance rates under different spectral bands. The effects of multiple scattering within tissues and reflections at tissue boundaries are taken into account and modeled as a Monte Carlo procedure [14], as shown in Fig.. As a result, the factor of spectral divergence may cause two main influences on the captured images: albedo variances due to reflectance and absorption discretion; and different s- cattering extents resulted from intra-layer reflections, which contribute to the diversity of local texture properties. Therefore, it is hard to recover visible band images directly from the NIR images. To solve this problem, the previous KNN-based approaches [9] [3] use the assumptions of local geometry preserving as prior knowledge. In these methods, all images are registered by the eyes and divided into patches. For each patch location a pair of dictionaries are constructed, which are composed of coupled patches in the training images. When an input NIR image is to be Figure. Monte Carlo modeling of light transport in multilayered tissues. synthesized, its patches are decomposed and reconstructed exclusively by the dictionaries at corresponding locations before pieced together. Specifically for an input patch x nir i at the i th location, the KNN coefficients β i are obtained (by Euclidean distances or by multi-resolution LBP similarities) to satisfy x nir i = k β i,idx(j) x nir i,idx(j) + ϵ, (1) where x nir i,idx(j) represents the idx(j)th training patch stored in the corresponding dictionary Di nir, idx(j) stands for the index of the j th nearest neighbor, k is a predefined parameter, and ϵ is the residue. With the solved KNN coefficients β i the synthesized patch is reconstructed as x syn i = k β i,idx(j) x vis i,idx(j), () where x vis i,idx(j) represents the idx(j)th training patch stored in the dictionary Di vis. 3. Synthesis from sparsity According to the theories of compressive sensing [], natural images contain redundant information, and can be recovered by a small group of base functions, which resembles the receptive fields of neurons in the visual cortex

3 [10]. That natural images are sparse and compressible gives the Sparsity prior: Both the VIS patches x vis and the NIR patches x nir can be represented as a sparse linear combination α vis or α nir in a dictionary D vis or D nir, respectively, which are trained from the patches sampled from the training images of the corresponding spectra. i.e. x vis D vis α vis, α vis 0 K, (3) x nir D nir α nir, α nir 0 K, (4) where K is the dimension of α. Liu et al. [9] introduced a Local geometry preserving prior: Corresponding VIS and NIR image patches are assumed to form manifolds with similar local geometries in two different image spaces. Based on this prior they proposed the LLM-KNN method, as described in Section. Now consider we have a pair of dictionaries D vis and D nir, in which all patches of the training images can be represented as a sparse linear combination of bases in the corresponding dictionary. For example, the patch x nir i,idx(j) in Eq. 1 can be represented as x nir i,idx(j) = Dnir α nir i,idx(j). (5) Combining this equation with Eq. 1, we can infer that the test image x nir i can be expressed as a linear combination of the bases in the dictionary D nir, decided by the coefficients α and β: x nir i = k β i,idx(j) D nir α nir i,idx(j). (6) Similar linear relationship between the synthesized patches and the VIS training patches can be derived in the same manner. As a matter of fact, the KNN coefficients β are inherently sparse and similar with the coefficients obtained by sparse constraints [17]. In this paper, we extend this prior to provide the assumption that VIS and NIR images share similar sparse coefficients α vis and α nir if the bases of the pairwise dictionary are coupled trained. The decomposition and reconstruction via sparse coefficients will be described in subsection 3.1, and the dictionary training process will be discussed in subsection Reconstruction via l 1 minimization As previously mentioned, a pair of dictionaries D vis and D nir are trained to have the same sparse representations for each pair of VIS and NIR images. For each input NIR patch x nir, we look for a sparse representation α with respect to D nir, which is used to reconstruct the VIS patch x syn by x syn = D vis α. (7) Constrained by the sparsity prior, the sparsest representation of x nir can be formulated by minimizing the l 0 -norm of representation coefficients. min α α 0 s.t. D nir α x nir < ε. (8) This optimization problem is NP-hard, and can be transformed into minimizing l 1 -norm instead: min α α 1 s.t. D nir α x nir < ε. (9) In the statistics literature of the Lasso [13], an equivalent formulation is constructed with Lagrange multipliers: min{λ α α D nir α x nir }, (10) where λ is a pre-defined parameter, which controls the tradeoff between fidelity and sparsity of the presentation coefficients. By solving this optimization problem the input patch is represented by the sparse coefficients α as a linear combination of base functions stored in the overcomplete dictionary D nir. The whole procedure is analyzed in Algorithm 1. Algorithm 1: Synthesis via Sparse Representation Input: The training dictionaries D vis and D nir, a test NIR face image X nir. FOR each patch x nir in X nir, Solve the sparse coefficients α of x nir in D nir by the Lasso, which is formulated in Eq. 10. Appliy the sparse representation α to generate the synthesized patch x syn using Eq. 7. END Piece together the synthesized patches x syn into the synthesized VIS face image X syn. The value of a pixel covered by overlapping patches is assigned to the average of its values in these patches. Output: The synthesized VIS face image X syn. 3.. Training an overcomplete dictionary In this section, we employ a strategy to train the coupled dictionaries D vis and D nir while keeping the correspondence of their bases, which has been used by Yang et al. for super-resolution [18]. The training set Y used to train the dictionaries consists of coupled patches randomly sampled from the training images, as shown in Fig. 3. By referring to the i th pair of patches as x nir and x nir, Y is constructed as Y = [ w nir x nir 1, w nir x nir w vis x vis 1, wvis x vis i i,, ], (11) where the weights w nir and w vis are used to balance the tradeoff between the fidelity to NIR training data and the

4 fidelity to VIS training data. In our work, the weights are used for normalization. They are assigned to the reciprocals of mean grayscale values of all training data in each spectra. (a) (b) Figure 3. (a) VIS and (b) NIR training images and their correspondent randomly sampled patches. The unknown dictionary D is formulated as the concatenation of the dictionary pair D nir and D vis, i.e. [ ] w D = nir D nir w vis D vis. (1) To reduce the redundant energy stored in Y, Sparse coding algorithms are employed to seek for overcomplete basis sets, in which the number of bases is larger than the input dimension, but much less than the number of original patches. To seek this basis set to form the dictionary D, the optimization problem is founded as min { Y D,S DS F } s.t. S 0 dim(s), (13) where S refers to the sparse representation coefficients matrix. This problem can be equivalently formulated as min { 1 D,S σ Y DS F + γ i,j Φ(S i,j)} s.t. i D i,j c, j = 1,..., n, (14) where σ is the covariance of the assumed reconstruction error distribution, γ is a pre-defined parameter, and Φ( ) is a sparsity penalty function, which could be: s (L 1 penalty function) Φ(s) = (s + ε) 1 (ϵ L 1 penalty function). (15) log(1 + s ) (log penalty function) Many sparse coding algorithms are presented to solve this optimization problem. In our work we adopted the method presented by Lee et al. [5] based on Lagrange dual. This method is based on the optimization of Eq. 14 with the l 1 penalty function Φ(s) = s. First, consider the Lagrangian: L(D, λ) = trace((y DS) T (Y DS)) + n λ j ( k Di,j c), (16) i=1 where each λ 0 is a dual variable. Minimizing over D analytically, the Lagrange dual can be obtained as: D( λ) = min L(D, λ) = trace(y Y T D Y S T (SS T + Λ) 1 (Y S T ) T cλ), (17) where Λ = diag( λ). This Lagrange dual can be optimized using Newton s method or conjugate gradient. After maximizing D( λ), the optimal bases D can be obtained as D T = (SS T + Λ) 1 (Y S T ) T. (18) 4. Experimental results and analysis Comparative experiments are presented in this section. In subsection 4., we exhibit samples of the synthesized images to see if they conform to human visual custom. We also present their SSIM values as a similarity measurement. In subsection 4.3, we evaluate the fidelity and discriminative efficiency of the synthesis results by their recognition accuracy when matched with visible band face images. In subsection 4.4, computing cost of different methods are compared in terms of time and space consumptions Setup The database is collected by a multispectral camera, which captures co-registered coupled VIS/NIR photos simultaneously. It consists of 700 face images taken from 150 subjects, including 100 males and 50 females. For each subject, 9 pairs of VIS and NIR frontal face images are collected, with great expression variance and modest pose variance. In both spectral bands the lighting conditions are controlled and provided in frontal directions. In our experiment, all images are downsampled to 7 7 pixels. We use 1 1 patchs, with an 9 pixels overlap in subsection 4. and 6 pixels overlap in subsection images of 50 subjects are used as training set, while the other 1800 images of 100 subjects are used for reconstruction and testing. Our experiments are conducted in comparison with the KNN-based methods introduced in Section, in this paper we refer to them as LLM- KNN [9] and LBP-KNN [3]. For LLM-KNN and LBP- KNN, the coefficients K are set to 10 and 15 respectively, as recommended in their papers. In LBP-KNN, only (4,1),(8,1),(1,1.5),(8,),(16,),(16,3) LBP features are used, without using (4,3) because it is too time expensive. This may cause certain degradation to its performance in our experiment. For our method 10 patches are sampled per training image. The dictionary is compressed to 104 dimensions with the sparsity parameter λ set to Synthesis results Some samples of NIR images, VIS images and the synthesized images by LLM-KNN, LBP-KNN and our method

5 are displayed in Fig. 4. The figure shows that although the KNN-based methods are very efficient in handling frontal face images with neutral expressions, when there is dramatic change of expressions or modest change of poses, the quality and fidelity of images synthesized by the KNNbased methods would drop inevitably. This phenomenon is aroused by the following reasons. First of all, some expressions are rare in the training images, and it is different to find similar patches in the training dataset. Secondly, even though all face images are registered by eye locations, other face region will still be badly aligned with varied expressions and poses, if we do not use additional technics like AAM (Active Appearance Model). In addition, as LBP features are relatively robust to expression variations and rotations, some patches from different expressions and poses may be used as similar patches for reconstruction, and cause blur in the overlapping regions. On the other hand, the synthesized images of our method are more proof to expression and pose variations. This is probably because the patches of our synthesized image are reconstructed from bases in the overcomplete dictionary, which make our method less demanding for alignment of patch locations. (a) (b) (c) (d) (e) Figure 4. (a) Samples of the input NIR images, their synthesized images by (b) LLM-KNN [9], (c) LBP-KNN [3], (d) our approach, and the corresponding (e) ground truth VIS images. To provide a quantifiable measurement to evaluate the fidelity of the synthesized images, their structural similarity (SSIM) [16] index values with the ground truth VIS images are presented in Fig. 5. This figure shows that all the methods can significantly enhance the similarity with real scene images, and our method is competitive compared with LLM-KNN and LBP-KNN. Figure 5. The SSIM values of (a) the input NIR images, their synthesized images by (b) LLM-KNN, (c) LBP-KNN and (d) our approach with the ground truth VIS images Recognition performance In this section, we report the recognition accuracy when the synthesized images are matched with ground truth VIS images. In our experiments, PCA and LBP histogram [1] are used to extract features, while the nearest neighbor method is used as classifier. Here in the LBP histogram method we use multi-resolution features (8,1), (8,), (16,) and (16,3) in 1 1 patches with no overlappings. The gallery set consists of ground truth VIS images. The experimental results are presented in Table 1 in terms of cumulative match scores. In this table, PCA refers to directly matching the NIR images with the VIS images by PCA, which is used as baseline. LLM-KNN, LBP-KNN and our method refer to using the images synthesized by LLM- KNN, LBP-KNN or our method as the probe set. PCA and LBP refer to the feature extraction methods. Table 1. Cumulative match scores (%) for different methods. Rank PCA LLM-KNN + PCA LBP-KNN + PCA Our method + PCA LLM-KNN + LBP LBP-KNN + LBP Our method + LBP The experimental results show that when the face images in the dataset have varied expressions and poses both in the training set and the testing set, the discriminative efficiency of the images synthesized by the KNN-based methods would drop. Especially, though the LBP-KNN method have high performance without large change in expressions, it seems very vulnerable to drastic expression variations. In contrast, the performance of our method is not influenced.

6 4.4. Computing cost In Table, the time and space cost of LLM-KNN [9], LBP-KNN [3], and our method are compared. The test is conducted with MATLAB code on a Core Duo.13 GHz and GB RAM PC. Here computing complexity is measured by running time of synthesizing a 7 7 test image by 1 1 patches, with 6 pixels overlapping (I) and 1 pixels overlapping (II). It should be noted that for the LBP-KNN method, here we only stored the original images instead of all multi-resolution LBP features, which would reduce the cost for storage, while greatly increase time expenses. This table shows that our method is competitive in terms of time expenses, and much superior in saving storage spaces. In the KNN-based approaches all training data have to be stored, while in our approach we used a more compact dictionary. Accordingly, our method exhibits high efficiency in dealing with large training datasets. Table. Time and space consumptions. Method LLM-KNN LBP-KNN Our method Running time I(s) Running time II(s) Dictionary size(mb) Conclusion This paper presented a novel framework for synthesizing VIS facial images from NIR inputs. We proposed to compress a pairwise overcomplete dictionary via l 1 - minimization from coupled face patches randomly sampled from the training images, which can be used to uncover common representations of bases. Experimental results showed that our method is discriminative for human vision and recognition, efficient for computing, and robust to expression variations and moderate pose variations. 6. Acknowledgment This work is funded by the National Basic Research Program of China (No. 010CB3790), the National Natural Science Foundation of China (No , No , No ) and the Fundamental Research Funds for the Central Universities. References [1] T. Ahonen, A. Hadid, and M. Pietikainen. Face description with local binary patterns: application to face recognition. PAMI, 8: , [] E. Candes. Compressive sensing. Proc. International Congress of Mathematicians, 006. [3] J. Chen, D. Yi, J. Yang, G. Zhao, S. Z. Li, and M. Pietikainen. Learning mappings for face synthesis from near infrared to visual light images. CVPR, pages , ,, 4, 5, 6 [4] B. Klare and A. K. Jain. Heterogeneous face recognition: Matching nir to visible light images. ICPR, [5] H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algorithms. NIPS, pages , [6] S. Z. Li, S. R. Chu, Liao, and L. Zhang. Illumination invariant face recognition using near-infrared images. PAMI, 9:67 639, [7] S. Liao, D. Yi, Z. Lei, R. Qin,, and S. Z. Li. Heterogeneous face recognition from local structures of normalized appearance. ICB, [8] D. Lin and X. Tang. Inter-modality face recognition. ECCV, [9] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma. A nonlinear approach for face sketch synthesis and recognition. CVPR, 1: , ,, 3, 4, 5, 6 [10] B. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381: , [11] M. Shao, Y. Wang, and Y. Wang. A super-resolution based method to synthesize visual images from near infrared. ICIP, pages , [1] X. Tang and X. Wang. Face sketch synthesis and recognition. ICCV, 1: , [13] R. Tibshirani. Regression shirinkge and selection via the lasso. J. Royal Statist. Soc B., 58:67 88, [14] L. Wang, S. L. Jacques, and L. Zheng. Mcml monte carlo modeling of light transport in multi-layered tissues. Computer Methods and Programs in Biomedicine, 47: , [15] R. Wang, J. Yang, D. Yi, and S. Z. Li. An analysis-bysynthesis method for heterogeneous face biometrics. Advances in Biometrics, [16] Z. Wang, B. A.C., S. H.R., and S. E.P. Image quality assessment: from error visibility to structural similarity. TIP, 13:600 61, [17] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma. Robust face recognition via sparse representation. PAMI, 31():10 7, [18] J. Yang, J. Wright, T. Huang, and Y. Ma. Image superresolution as sparse representation fo raw image patches. CVPR, pages 1 8, [19] D. Yi, S. Liao, Z. Lei, J. Sang,, and S. Z. Li. Partial face matching between near infrared and visual images in mbgc portal challenge. ICB, [0] D. Yi, R. Liu, and R. Chu. Face matching from near infrared to visual images. Advances in Biometrics, [1] W. Zhao, R. Chellappa, and A. Rosenfeld. Face recognition: A literature survey. ACM Computing Surveys, 35: , [] L. Zhen and S. Z. Li. Coupled spectral regression for matching heterogeneous faces. CVPR, pages ,