Sparse Representation based Face Recognition with Limited Labeled Samples

Transcription

1 Sparse Representation based Face Recognition with Limited Labeed Sampes Vijay Kumar, Anoop Namboodiri, C.V. Jawahar Center for Visua Information Technoogy, IIIT Hyderabad, India Abstract Sparse representations have emerged as a powerfu approach for encoding images in a arge cass of machine recognition probems incuding face recognition. These methods rey on the use of an over-compete basis set for representing an image. This often assumes the avaiabiity of a arge number of abeed training images, especiay for high dimensiona data. In many practica probems, the number of abeed training sampes are very imited eading to significant degradations in cassification performance. To address the probem of ack of training sampes, we propose a semi-supervised agorithm that abes the unabeed sampes through a muti-stage abe propagation combined with sparse representation. In this representation, each image is decomposed as a inear combination of its nearest basis images, which has the advantage of both ocaity and sparsity. Extensive experiments on pubicy avaiabe face databases show that the resuts are significanty better compared to state-of-theart face recognition methods in semi-supervised setting and are on par with fuy supervised techniques. Keywords Face recognition; semi-supervised earning; sparse representation. I. INTRODUCTION Face recognition techniques over the years have empoyed a variety of representations of the face such as Eigenface, Fisherface, Lapacianface, etc. [1]. Each of these techniques, tries to earn a basis or feature space by optimizing an objective function with a specific goa. Eigenface utiizes PCA to maximize the variance of the training images, whie Fisherface tries to maximize the separabiity of the casses in the feature space. In the recent years, ideas from Sparse Representation and Compressed Sensing have been appied to face recognition, where the representation is obtained by an objective function that eads to sparsity [2] in an over-compete basis. The representation over this basis is obtained using 1 -reguarized east square approximation. The east square approximation tends to reduce reconstruction error, whie 1 -reguarization gives rise to sparsity of representation. The test image is thus represented in terms of a few basis images, and is assigned a abe of the cass that gives minimum reconstruction error. The approach has proved to be very effective and achieves state-of-the-art resuts. Motivated from the success of sparse coding, researchers have deveoped various modes for face recognition and other appications [3]. One significant direction is earning dictionaries for reconstructive tasks such as denoising [4], [5] and discriminative dictionaries for cassification tasks [6]. Pate et a. [7] and Zhang et a. [8] expained how the dictionaries can be earned and appied for face recognition using sparse representation. Sparse representation based face recognition methods need a arge amount of training data to have an over-compete dictionary, which is essentia to obtain a sparse representation. The sparse representation is found to be discriminative [2] as the non-zero weights are mosty at ocations corresponding to training images simiar to a query image, thus heping in cassification. Sparsity, thus is the key for the success of these agorithms, but this has a dependency on the size of the dictionary. However, in many practica situations we wi have imited number of training sampes. For exampe, in automatic face abeing/tagging of photo coections and abums, it is necessary to recognize the faces with imited user-tagged photos for a better user experience. Simiary, in scenarios such as depoying a monitoring system for access contro in factories or attendance checks in cassrooms, if we can work with a coupe of abeed sampes that are aready coected for other purposes such as ID cards or empoyee registration, we can avoid the costy phase of manua abeing of faces. Designing an accurate cassifier with imited abeed training sampes is the focus of this paper. A possibe soution to dea with the ack of abeed sampes is to earn a mode using both the abeed and unabeed sampes using semi-supervised earning techniques [9]. There are attempts made to dea with ack of training sampes. In [10], sef-taught earning was proposed based on sparse representation, where a dictionary is earned using unabeed sampes. Labeed sampes are coded sparsey over the earned dictionary and a cassifier is earned using SVM. Projection based methods, [11], [12], [13] have been proposed for face recognition with few training sampes. Roi et a. [11], computed the eigenspace using the abeed sampes. Unabeed sampes that are coser to the projected mean tempates of each cass are seected and augmented to the abeed set and the procedure is repeated ti a the unabeed sampes are abeed. In Semi-supervised discriminant anaysis [13], abeed data was used to infer the discriminative structure of the data whie the intrinsic geometric structure of the data is inferred from both abeed and unabeed sampes. Zhao et a. [12] used an approach simiar to [11] except that feature space is computed using LDA. In this paper, we propose a graph based muti-stage abe propagation agorithm based on sparse representation for semisupervised face recognition. In each stage, unabeed sampes are abeed using the abe propagation agorithm [14] and ony highy confident sampes are seected. We propose a nearest neighbor based sparse coding (NNSC) agorithm to obtain the graph weights. NNSC is simiar to [15], [2] and represents each sampe as a inear combination of its nearest neighbors (see Fig. 1). However, it is many times faster than sparse representation based cassifier (SRC) with

2 Labeed dictionary abe= arg min i q Xw i 2 Feature extraction and projection Labeed dictionary X in the feature space w i is a k x 1 vector whose nonzero entries are the entries of w corresponding to cass i Test q Seect k nearest neighbors and construct X Feature extraction and projection Reconstruct q using w = arg min w q Xw 2 + λ w 2 Fig. 1: Overview of our proposed NNSC agorithm. Test image is represented as a inear combination of its nearest neighbors and assigned a abe of the cass which gives minimum reconstruction error. simiar performance. In our representation, a connection is formed ony within a neighborhood for each node, whie SRC might give connections between far away nodes. We aso show an extension of our agorithm for out-of-cass sampes using a cassifier simiar to SRC, which tries to minimize the reconstruction error. The query image is assigned the abe of the cass whose sampes minimize the reconstruction error. Experiments on AR and CMU PIE data sets ceary demonstrate the superiority of the approach compared to the existing methods. II. SPARSE MODELS FOR FACE RECOGNITION Given an image x R d, aong with an over-compete basis or dictionary A R d n with d eements, n d, the image is represented as a inear combination of few eements of the dictionary A. In other words, x is approximated as: x Aw, where w is sparse and is computed as: arg min w w 0 subject to x Aw, (1) where. 0 denotes 0 -pseudo norm and indicates the number of non-zero eements in w. Soving the above 0 minimization probem is NP-hard and is usuay done with greedy methods such as Matching Pursuits (MP) or by 1 -convex reaxation. Dictionary A coud be a pre-specified, such as waveets or training images itsef [2], or they coud be earned [4], [6] using the foowing objective function. arg min A,w w 0 subject to x Aw (2) Sparse representation thus coud invove two tasks, earning a dictionary, and finding a sparse representation over the dictionary. However, when there are ess training sampes, it may not be possibe to earn a good dictionary that gives a sparse decomposition for the training sampes. A. Supervised and Unsupervised Methods We have a sma set X ={x (1), x (2),..., x (m) } of m abeed exampes with their abes y (i) {1,2,...,C} and a arge set of n unabeed exampes X u = {x (1) u, x (2) u,..., x (n) u }. The subscripts and u indicate abeed and unabeed images. In unsupervised sparse coding methods, image is decomposed as a inear combination of a few eements of unabeed basis. Any image x can be represented sparsey over the dictionary A, which is not abeed as shown in Eq. (2). Dictionary A coud be pre-specified or earned using training images. These methods are suitabe for reconstructive tasks such as denoising [5], [4] or to understand the structure of the data [10]. In supervised sparse coding, abe information of the training sampes X is used to buid modes that wi hep cassification tasks. In this approach, an image x can be coded sparsey over a dictionary A that is abeed as shown beow. n arg min w 0 subject to x a (i) i.w i, (3) A,w where a (i) i s are basis eements beonging to cass i. In supervised methods, a discriminative basis can be earned such that sampes of each cass is best represented by the basis earned for that cass [6], [8]. SRC [2] used a simiar technique, where a given query image is decomposed over a abeed dictionary (training sampes in SRC). B. Semi-supervised Loca Coding When the number of training sampes in X are imited, semi-supervised methods that expoits the unabeed sampes X u to understand the structure of the data can be usefu. Unabeed sampes are abundant and easy to coect and better accuracies can be achieved if they are used propery aong with the abeed training sampes. i=1 Sparse coding in genera, requires arge number of sampes whether in supervised or unsupervised mode. If the images themseves are used as dictionaries [2], then a arge number of training images are required to make sure that it is overcompete to ensure sparsity. Aso, earning the dictionary using a few images may not give optima resuts. In such cases where the number of sampes is ess, semi-supervised methods that uses both abeed and unabeed sampes can be empoyed to improve the cassification performance. There are many semisupervised methods that are used in practice and the reader may refer to [9] for a detaied survey. III. SEMI-SUPERVISED LEARNING FOR FACE RECOGNITION We wi consider an exampe to understand the scenario of our probem. Assume that the training set consists of a singe image for each of the 100 subjects in the AR database [16]. We wi use SRC [2], a popuar and effective technique to serve as a baseine. The first row of Fig 2 shows a query (Fig. 2) and the first ten training sampes (Fig. 2). SRC correcty identifies the given query as beonging to the second cass, which is quite simiar to the query. However, the query in Fig. 2(c) was not recognized as cass two due to the significant expression difference. Now we increase the training set by adding another exampe per subject that incudes some expression variation to the training set (see Fig. 2(d)) and again use SRC to recognize the abe of sampe in Fig. 2(c). This time SRC correcty identifies the sampe. Whie this is a specific case, the observations hods true across databases as indicated by our experiments given in ater sections. In practica situations, when we have imited training sampes, supervised methods do not perform we as they are unabe to account for the kind of variations encountered in practica situations. Semi-supervised methods can be empoyed in such cases that uses unabeed sampes to improve the performance of the recognition system. We now ook into the proposed semi-supervised face recognition agorithm based on oca sparse coding and its extension to out-of-sampe data.

3 where i, j {1, 2,.., n} and ŵ i (p) denotes the p-th eement of vector ŵ i. Weights obtained by this method may not be symmetric i.e w ij w ji. We make the fina weights symmetric with the operation: w ij = w ji = (w ij +w ji )/2. We normaize the weight matrix symmetricay as done in the spectra custering in order to ensure convergence of abe propagation agorithm as shown beow. (c) Fig. 2: Demonstration about the effect of ack of training sampes. SRC identified the first image correcty but faied to identify the (c) second image with sma dictionary d 1 (one sampe per subject). Using (d) arge dictionary (two sampes per subject), it identified both images correcty. A. Transductive earning Given a set X={x 1, x 2,.., x, x +1..x n } of training sampes. x i R d, i = {1, 2,.., } are a set of abeed sampes beonging to cass {1,2,..,c} and remaining sampes, x i, i = {+1, +2,.., n} are unabeed. The objective is to predict the abes of the unabeed sampes and subsequenty use them to get a better representation of a nove test image to improve the cassification performance. Construct an undirected graph V, E with simiarity matrix W, using both abeed and unabeed points. Each node in the graph corresponds to a face and the edges E represents simiarities between them. Large edge weights w ij indicate that corresponding nodes/faces are very simiar. One coud use simpe k-nearest neighbor method to compute the weights where w ij =1 if x i is among the k-nearest neighbors of x j or vice-versa and 0 otherwise. Another option for simiarity measure is Gaussian function: w ij = exp( x i x j 2 /2σ 2 ) where σ contros the spread of the Gaussian function. For face recognition, the above mentioned simiarity measures may not be accurate as they are sensitive to iumination and expression variations. Representing a face as a inear combination of training sampes is found to be robust to these variations as reported in SRC. Such a method obtains better weights to revea the reation among different face sampes. However, SRC ignores the neighborhood information and thus gives non-zero weights even for the far-away sampes in the process of reducing the reconstruction error. Inspired by (SRC) and ocaity constrained inear coding (LLC), we propose a nearest neighbor based sparse coding (NNSC) that considers both ocaity and sparsity. In this representation, each sampe is represented as a inear combination of its nearest neighbors. We use the NNSC representation to construct the simiarity matrix of the graph. ŵ i = arg min w i x i B i w i 2 + λ w i 2 s.t k w ik 0 (4) where the coumns of B i are the k-nearest neighbors, N(x i ) of x i. We add a positive constraint since graph weights are supposed to be greater than 0. We add a reguarization term inspired by CRC [17], which resuts in a representation that is discriminative. λ is a Lagrangian constant that contros the trade-off between the two terms. Construct the matrix W R n n as: {ŵi (p), if x W ij = j N(x i ) 0, otherwise, (d) L = D 1/2 W D 1/2 where D ii = j W ij (5) Let F R n c be a matrix from which the abe y i of a sampe x i, {i = 1, 2,.., n} can be obtained as y i = arg max F ij, where j = {1, 2,.., c}. For the abeed sampes j x i, {i = 1, 2,.., } we define Y ij =1 if y i =j and 0 otherwise. For unabeed sampes Y ij = 0 for a j where j ={1,2,..,c}. We assume that the abe of a sampe can be computed as a inear combination of the abes of other sampes with the weights being computed through sparse coding of the sampe. We propagate the abes of the abeed sampes to the unabeed ones using the constructed graph weights. Using the abe propagation framework, we et unabeed sampes receive some amount of abe information from its neigbours and retain a part of its initia information at every iteration. The amount of information it receives depends on the corresponding normaized weights. We begin the iteration with F (0) = Y, and for any t 1, the abeing matrix F is given by, F (t + 1) = αlf (t) + (1 α)y, (6) where Y is the initia abeing of the sampes and α is a parameter that decides the amount of information a sampe receives at each iteration. It is we known from the abe propagation iterature [14] that the above iterative method converges to F = (1 α)(i αl) 1 Y. The abes of unabeed sampes can be predicted using y i = arg max Fij. j B. Muti-stage abe propagation In the idea case, each row of F contains a singe nonzero component corresponding to the true cass. The ratio of two argest components in a row in such case wi be infinite. However in practice, the ratio wi be arge ony when majority of the sampes contributing to the reconstruction beong to a particuar cass. We can use the ratio to measure how confident is the abeing decision after the convergence of abe propagation. We propose a muti-stage abe propagation agorithm, where we seect ony highy confident abeings after the convergence of each stage of the abe propagation agorithm. If the ratio of two argest abeing components of a sampe i in Fij j = {1, 2,.., c} exceeds a threshod, the abeing of the sampe is considered as confident. Such sampes are considered to be abeed for the next stage of abe propagation. We wi show in our experimenta resuts, how this muti-stage approach gives a significant improvement over singe stage abe propagation, especiay when there are very few abeed sampes and the dataset contains arge intracass variations. Another advantage of this method is that we can reject the sampes such as outiers or out-of-database cass images, which might otherwise reduce the performance on test images.

4 (c) Fig. 3: Exampes from Yae AR and (c) CMU PIE databases. and (c) have iumination and ighting variations whie has varying expression and iumination. C. Extension to Out-of-Sampe data The goa of any cassification agorithm is to predict the abes of nove test images correcty. To cassify a test image, we can use the reconstruction error as the criterion for cassification as done in [2], [17]. Given a test image q R d, its representation w over its k nearest neighbors, B q = [x 1, x 2,..., x k ], is obtained using Eq. (4) (without positive constraint). For each cass i, we construct a function δ i : R k R k which gets the coefficients associated with the i-th cass in B q. δ (j) i = ŵ j if x j N(q); j = 1, 2,..., k (7) where the non-zero entries of δ i correspond to entries beonging to cass i from w. Test image is then assigned the abe of the cass that minimizes the reconstruction error. abe(q) = arg min i q B q δ i 2 (8) IV. RESULTS AND DISCUSSIONS We use Extended Yae B, AR and CMU PIE (shown in Fig. 3) data sets to carry out our experiments. We seect the error toerance, e = 0.05 for SRC in a the experiments and choose L1LS [18] 1 -reguarized east squares sover to sove the minimization probem in SRC. A. NNSC-LP Semi-supervised Method Extended Yae B database [19] consists of 2414 fronta face images of 38 individuas captured under various ighting conditions. Each image is of size We resize the image to. We randomy seect haf of the images for training (32 images per cass) and other haf for testing. Few images are shown in Fig. 3. To create a semi-supervised setting, we keep the abes of ony few training sampes per cass (3 to 24) and remove the abes for the rest of training sampes. We seect the maximum number of stages to 10 and the ratio of two argest abeing component Fij (first and second argest abeing component ) to 1 : 2.5. We choose k, that indicates number of nearest neighbors in NNSC to 120. We use both abeed and unabeed sampes to find the feature space using PCA. We seect the dimension of eigenface to 504 (to compare with resuts reported for SRC), α = 0.9 and λ = For different trias, we keep the abes of 3, 5, 10, 16 and 24 training sampes in our experiments and abes of rest of the sampes are removed. To measure the performance of mutistage method, test accuracy is cacuated after every stage. Fig. 5, shows the recognition rates of the test set after every stage for initia abeed sampes 3, 5, 10, 16 and 24. It is cear from the figure that for every stage there is an improvement in the accuracy. Number of unabeed sampes seected Yae AR Fig. 4: Number of unabeed sampes seected after every stage on Yae and AR database for three abeed exampes and threshod ratio=1.5. Effect of convergence and accuracy for various vaues of ratio threshod on Yae database with three abeed sampes Train 5 Train 10 Train 16 Train 24 Train ratio=3.0 ratio=2.0 ratio=1.5 ratio= Fig. 5: Recognition rates on Extended Yae B database and AR database vs. NNSC-LP stages for varying abeed sampes Train 2 Train 3 Train 4 Train When the number of abeed sampes considered are 16 and 24, gain in accuracy is not significant as there are enough abeed sampes aready avaiabe. The significant gain in accuracy can be observed for the trais with 3 and 5 abeed sampes where there is amost 20% increase in the recognition rates that indicates the advantage of semi-supervised methods when there are imited training sampes. Fig. 4 shows the number of unabeed sampes seected after every stage for Yae and AR database with the ratio of two argest abeing component Fij = 1.5 and 3 abeed exampes per subject. It is cear that, arge number of training sampes are seected in the first few stages and hence arger gain during these stages. These highy confident sampes seected in the first few stages wi hep in abeing the hard sampes in the ater stages. The seection of threshod ratio to seect highy confident sampes is important for performance. It decides the growth rate of abeed training set at every stage. There is a trade off between accuracy and maximum number of stages as can be seen in Fig. 4. B. Comparison with Other Methods AR database: We consider a subset of AR face data base [16] consisting of 50 mae and 50 femae subjects. For each subject there are 14 images with varying expressions and iuminations. Each image is of size We convert the images to grayscae and resize to. Images are taken in two different sessions. We seect seven images from session 1 for training and remaining seven images from session 2 for testing. Few images from this database are shown in Fig. 3. We seect the dimension of eigenspace to 504, α = 0.9 and λ = 0.1. For SRC, we seected a eigenspace dimension that

5 TABLE I: Recognition rates [%] of various methods on AR database for different number of abeed exampes. Method 1 Train 2 Train 3 Train SRC [2] CRC [17] PCA sef-training [11] LDA sef-training [12] NNSC NNSC-LP TABLE II: Recognition rates on CMU PIE database in (mean±std-dev%). Method Unabeed set Test set Eigenface 25.3± ±1.6 Lapacianface 56.1± ±2.4 LapSVM [21] 56.5± ±2.6 LapRLS [21] 57.5± ±2.6 SDA [13] 59.0± ±2.7 LDA sef-training [12] 84.5± ±6.5 SRC [2] 74.7± ±1.3 CRC [17] 74.9±1.41.1±1.32 NNSC.0± ±1.3 NNSC-LP 92.1± ±1.5 maintain a % overcompete dictionary. We set the number of nearest neighbors k for NNSC to 100, the ratio of two argest abeing component Fij to 1 : 1.5 and maximum number of stages to 10. We conduct three trias with 1, 2 and 3 abeed sampes. The recognition rates of the test set at various stages is shown in Fig. 5. Tabe I shows the comparison of our muti-stage NNSC-LP accuracy with other methods. It is cear from the tabe that, the performance of our NNSC-LP agorithm is superior than other mentioned methods. C. Singe Training Image Face Recognition CMU PIE consists of 68 subjects with 41, 368 face images with varying iumination, pose, expression and ighting. As reported in [13], [12], we choose a subset of ony fronta faces (C27) with ony iumination and ighting variations which resuts in 43 images per subject. The images are cropped to We used PCA to reduce the dimension of the image to 504 and k is set to 50, α = 0.9 and λ = For SRC, dimension is reduced to 50 to have an overcompete dictionary. We set the maximum stages to 15 and the ratio of two argest abeing component Fij to 1 : 1.5. For any tria, 30 images are seected for training and remaining 13 are seected for test. Among the 30 training images, ony one image is randomy seected and abeed and remaining 29 sampes remain unabeed. The experiment is carried out 20 times and the resuts are averaged over 20 trias. The resuts in Tabe II show superiority of our NNSC-LP agorithm compared to other methods. D. Effect of parameters: k, λ and α We observed that the parameters λ and α affected the performance of the resuts. For a the experiments, we obtained best resuts with α = 0.9 and λ in the range We empiricay seected the vaue of k for NNSC. We found that agorithm is not very sensitive to k and a vaue of seemed to work we in practice. V. CONCLUSIONS We demonstrate the effect of imited abeed training sampes on the accuracy of sparse coding based recognition techniques and how it can be overcome through a semi-supervised approach. The proposed NNSC-LP agorithm accuratey abes the unabeed sampes and utiizes them for recognition. NNSC combines the concepts of sparsity and ocaity. Unike, SRC, which uses 1 norm to get the sparse representation, NNSC represents a sampe over a few seected neighbors and thus is faster. Experimenta resuts ceary demonstrates the superiority of the proposed method over existing methods when ony a few abeed sampes are avaiabe. VI. ACKNOWLEDGEMENTS This work is party supported by the MCIT, New Dehi. Vijay Kumar is supported by TCS research feowship. REFERENCES [1] W.-Y. Zhao, R. Cheappa, P. J. Phiips, and A. Rosenfed, Face recognition: A iterature survey, ACM Comput. Surv., [2] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, Robust face recognition via sparse representation, PAMI, [3] J. Wright, Y. M. Y. Ma, J. Maira, G. Sapiro, T. S. Huang, and S. Y. S. Yan, Sparse representation for computer vision and pattern recognition, Proc. of the IEEE, [4] M. Aharon, M. Ead, and A. Bruckstein, K-svd: An agorithm for designing overcompete dictionaries for sparse representation. IEEE Trans. on Signa Processing, [5] J. Maira, M. Ead, and G. Sapiro, Sparse representation for coor image restoration, IEEE Trans. on IP, [6] J. Maira, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Discriminative earned dictionaries for oca image anaysis, in CVPR, [7] V. M. Pate, T. Wu, S. Biswas, P. J. Phiips, and R. Cheappa, Dictionary-based face recognition under variabe ighting and pose, IEEE Trans. on IFS, [8] Q. Zhang and B. Li, Discriminative k-svd for dictionary earning in face recognition, in Proc. CVPR, [9] X. Zhu, Semi-supervised earning iterature survey, Computer Sciences, University of Wisconsin-Madison, Tech. Rep. 1530, [10] R. Raina, A. Batte, H. Lee, B. Packer, and A. Y. Ng, Sef-taught earning: transfer earning from unabeed data, in Proc. ICML, [11] F. Roi and G. L. Marciais, Semi-supervised pca-based face recognition using sef training, in Proc. SSPR/SPR, [12] X. Zhao, N. W. Evans, and J.-L. Dugeay, Semi-supervised face recognition with LDA sef-training, in Proc. IEEE ICIP, [13] D. Cai, X. He, and J. Han, Semi-supervised discriminant anaysis, in Proc. ICCV, [14] X. Zhu and Z. Ghahramani, Learning from abeed and unabeed data with abe propagation, Carnegie Meon University, Tech. Rep., [15] J. Wang, J. Yang, K. Yu, F. Lv, T. S. Huang, and Y. Gong, Locaityconstrained inear coding for image cassification, in CVPR, [16] A. Martinez and R. Benavente, The AR face database. CVC technica report #24, June [17] L. Zhang, M. Yang, and X. Feng, Sparse representation or coaborative representation: Which heps face recognition? ICCV, [18] S.-J. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, An interiorpoint method for arge-scae 1 -reguarized east squares, J-STSP, [19] K. Lee, J. Ho, and D. Kriegman, Acquiring inear subspaces for face recognition under variabe ighting, PAMI, [20] T. Sim, S. Baker, and M. Bsat, The CMU pose, iumination, and expression (PIE) database, in Proc. FG, [21] M. Bekin, P. Niyogi, and V. Sindhwani, Manifod reguarization: A geometric framework for earning from abeed and unabeed exampes, JMLR, 2006.