Texture Features in Facial Image Analysis Matti Pietikäinen and Abdenour Hadid Machine Vision Group Infotech Oulu and Department of Electrical and Information Engineering P.O. Box 4500, FI-90014 University of Oulu, Finland {mkp, hadid}@ee.oulu.fi http://www.ee.oulu.fi/mvg Abstract. While features used for texture analysis have been successfully used in some biometric applications, only quite few works have considered them in facial image analysis. Texture-based region descriptors can be very useful in recognizing faces and facial expressions, detecting faces and different facial components, and in other face related tasks. This paper demonstrates this issue by considering the local binary pattern (LBP) as an example of texture-based approach and showing its efficiency in facial image analysis. 1 Introduction Texture is an important characteristic for the analysis of many types of images. Among the traditional application areas of texture analysis are industrial inspection, biomedical image analysis, analysis of remotely sensed images, and content-based retrieval from image databases. A textured area in an image can be characterized by a non-uniform or varying spatial distribution of intensity. The intensity variation reflects some changes in the scene being imaged. The specific structure of the texture depends on the surface topography and albedo, the illumination of the surface, and the position and frequency response of the camera. A wide variety of techniques for describing image texture have been proposed [1]. The methods can be divided into four categories: statistical, geometrical, model-based and signal processing. Among the most widely used approaches are statistical methods based on co-occurrence matrices of second order gray level statistics, signal processing methods based on multi-channel Gabor filtering or wavelets, and model based methods using Markov random fields. Texture could play an important role in many biometric applications [2]. The most notable example of recent success is iris recognition, in which approaches based on multi-channel Gabor filtering have been highly successful. Multi-channel filtering has also been widely used to extract features e.g. in fingerprint and palmprint analyses. The financial support of the Academy of Finland and the National Technology Agency of Finland is gratefully acknowledged.
2 Matti Pietikäinen and Abdenour Hadid Some works have also considered texture features in facial image analysis. For instance, the well-known Elastic Bunch Graph Matching (EBGM) method is using Gabor filter responses at certain fiducial points to recognize faces [3]. Gabor wavelets have also been used in facial expression recognition yielding in good results [4]. A problem with the Gabor-wavelet representations is their computational complexity. Therefore, simpler features like Haar wavelets have been considered in face detection resulting in a fast and efficient face detector [5]. Recently, the local binary pattern (LBP) texture method has provided excellent results in various applications. Perhaps the most important property of the LBP operator in real-world applications is its better tolerance against illumination changes than most of the other texture methods have. Another equally important property is its computational simplicity, which makes it possible to analyze images in challenging real-time settings [6 8]. In this paper, we will consider LBP features as examples to demonstrate the usefulness of texture-based approach in facial image analysis. 2 The Local Binary Pattern Approach to Texture Analysis The LBP texture analysis operator is defined as a gray-scale invariant texture measure, derived from a general definition of texture in a local neighborhood. For each pixel in an image, a binary code is produced by thresholding its value with the value of the center pixel. Fig. 1 shows an example of an LBP calculation. Fig.1. Example of LBP calculation A histogram is created to collect up the occurrences of different binary patterns. Each bin of the histogram (LBP code) can be regarded as a micro-texton. Local primitives which are codified by these bins include different types of curved edges, spots, flat areas etc. Fig. 2 shows some examples. The basic version of the LBP operator considers only the eight neighbors of a pixel (Fig. 1)[6]. Later, the operator has been extended to consider different neighborhood sizes [7]. For example, the operator LBP 4,1 uses only 4 neighbors while LBP 16,2 considers the 16 neighbors on a circle of radius 2. In general, the operator LBP P,R refers to a neighborhood size of P equally spaced pixels on a circle of radius R that form a circularly symmetric neighbor set.
Texture Features in Facial Image Analysis 3 Fig. 2. Examples of texture primitives which can be detected by LBP (white circles represent ones and black circles zeros) The LBP method has already been used in a large number of applications all over the world, including visual inspection, image retrieval, remote sensing, biomedical image analysis, face image analysis, motion analysis, environment modeling, and outdoor scene analysis. For a bibliography of LBP-related research, see [9]. 3 Texture Features in Facial Image Analysis: Case Studies Applying a texture operator and considering only the response distributions as facial representation may yield in loss of spatial information. For efficiently representing the facial images, one should codify the texture information while retaining also their locations. One way to achieve that goal is to divide the face image into several regions and then extract (texture) features from each of them. Using the local binary pattern (LBP) operator, for instance, the face image can be divided into several regions (or blocks) from which the local binary pattern histograms are computed and concatenated into a single histogram (see Fig. 3.a). In such a representation, the texture of facial regions is encoded by the LBP while the shape of the face is recovered by the concatenation of different local histograms. The idea of using LBP for face representation is motivated by the fact that face images can be seen as a composition of micro-patterns, such as those shown in Fig. 2, which can be well described by LBP. 3.1 Face Recognition To analyze the performance of the texture based representation shown in Fig. 3.a, a face recognition system was built in [10]. The system uses a nearest neighbor classifier for recognition. A comparison against well known methods such as PCA, EBGM and Bayesian Intra/extrapersonal (BIC) was also done. To achieve a fair comparison, the FERET frontal face database and protocol were used. In the LBP-based representation, each face image is represented by a texture feature histogram. In the nearest neighbor classifier, the χ 2 (Chi-square) dissimilarity metric is adopted for comparing a target face histogram S to a model histogram M: χ 2 (S, M) = l i=0 (S i M i) 2 S i+m i, where l is the length of the feature
4 Matti Pietikäinen and Abdenour Hadid Fig. 3. (a)an LBP description. (b) The assigned weights: black squares indicate weight 0.0, dark grey 1.0, light grey 2.0 and white 4.0 vector used to represent the face image. Note that the choice of the χ 2 measure is motivated by the experimental findings [10] which showed that χ 2 gives better results than other dissimilarity measures such as the histogram intersection and Log-likelihood statistic. When dividing the facial images into several regions, it can be expected that some of the regions contain more useful information than others in terms of distinguishing between faces. Therefore, one may use different weights, depending on the importance of the given regions in recognition. For instance, since the eye regions are important for recognition, a high weight can be attributed to the corresponding regions. Fig. 3.b shows the weights that were assigned to the different regions [10]. The extensive experiments clearly showed the superiority of the LBP-based approach over all considered methods (PCA, BIC and EBGM) on the FERET tests which include testing the robustness of the methods against different facial expressions, lighting and aging of the subjects. Additional experiments on the ORL face database (Olivetti Research Laboratory, Cambridge) have also showed a relative robustness with respect to alignment. One may wonder whether the very good results are due to the division of the face images into local regions (instead of an holistic approach) or to the discriminative power of LBP features. This issue has been investigated in [11], by comparing the performance of four texture descriptors (gray-level difference histogram, homogeneous texture descriptor, texton histogram, and LBP) extracted from local regions. The performance of LBP was shown to be superior to the comparison algorithms. This confirms the validity of using LBP for face description. The main explanation for the better performance of the local binary pattern operator over other texture descriptors is its tolerance to monotonic gray-scale changes. Moreover, no gray-scale normalization is needed prior to applying the LBP operator to the face image. Directions for future research include using learning algorithms (such as AdaBoost) for selecting the optimal subset of prominent LBP features extracted with different parameters. Some work on this issue has already been done in [12], in which the AdaBoost learning algorithm is used for finding optimal windows
Texture Features in Facial Image Analysis 5 for recognition. Using this method promising recognition results are achieved with a smaller feature vector length. Recently, Zhang et al. proposed another direction consisting of combining texture operators. The approach [13] named multi-resolution histograms of local variation patterns, first computes a multiresolution and multi-orientation description of an image using Gabor decomposition. Then, LBP histograms are computed from the Gabor features for small non-overlapping regions and concatenated into a feature histogram. Excellent results are reported for the FERET database. 3.2 Face Detection In [14], a variant of the LBP-based facial representation shown in Fig. 3.a is proposed for detecting frontal faces. A specific of this new representation is the use of overlapping regions and a 4-neighborhood LBP operator LBP 4,1 to avoid statistical unreliability due to long histograms computed over small regions. Additionally, the holistic description of a face is enhanced by including the global LBP histogram computed over the whole face image. Investigating the performance of the new LBP representation in detecting frontal faces, training sets of face and non face images were first collected [14]. Then, the LBP facial representations were extracted from the training data and used to train an SVM classifier. After training, the system was run on several images from different sources to detect faces. Fig. 4 shows some detection results. The experimental results on 80 test images from the MIT-CMU sets have showed that the LBP-based face detector compares favorably against the comparative approaches (Bayesian Discriminating Features, Schneiderman & Kanade method, and using normalized pixel values as inputs to an SVM classifier). Fig.4. Detection examples using LBP approach Another variant of LBP-based facial representation, called Improved LBP, has been used in [15]. Instead of using the extracted features as inputs to an
6 Matti Pietikäinen and Abdenour Hadid SVM like in [14], the authors have considered a Bayesian framework for classifying the LBP representations. The face and non-face classes were modeled using multivariable Gaussian distributions while the Bayesian decision rule was used to decide on the faceness of a given pattern. The reported results are very encouraging. As an extension to the reported work, it would be of interest to consider the LBP features with learning algorithms, such as AdaBoost and its variants, in building a fast and robust multi-view face detector. 3.3 Facial Expression Recognition In [16], an approach to facial expression recognition from static images was developed using LBP histograms computed over non-overlapping blocks for face description. The Linear Programming (LP) technique was adopted to classify seven facial expressions: anger, disgust, fear, happiness, sadness, surprise and neutral. During the training, the seven expression classes were decomposed into 21 expression pairs such as anger-fear, happiness-sadness etc. Thus, twenty-one classifiers are produced by the LP technique, each corresponding to one of the 21 expression pairs. A simple binary tree tournament scheme with pairwise comparisons is used for classifying unknown expressions. Good results (93.8%) were obtained for the Japanese Female Facial Expression (JAFFE) database used in the experiments. The database contains 213 images in which ten persons are expressing three or four times the seven basic expressions. Some sample images are shown in Fig. 5. Fig.5. Samples from original Japanese Female Facial Expression images Another approach to facial expression recognition using LBP features is proposed in [17]. Instead of the LP approach, template matching with weighted Chi square statistic and SVM are adopted to classify the facial expressions using LBP features. Extensive experiments on the Cohn-Kanade database confirmed that LBP features are discriminative and more efficient than Gabor-based methods especially at low image resolutions. 3.4 Other Facial Image Analysis Tasks The applicability of texture features in facial image analysis is not limited to the tasks presented above. A similar methodology could also be used, for instance, in
Texture Features in Facial Image Analysis 7 detecting different facial components such eyes, mouth, nose etc. Fig. 6 shows an example of an LBP-based scheme for detecting the eyes. In addition, LBP fea- Fig.6. Using LBP features for detecting facial components such as eyes tures have also been considered for localizing and representing facial key points. An accurate localization of such points of the face is crucial to many face analysis and synthesis problems such as face alignment. In [18], it was shown that using extended LBP features in the Active Shape Model (ASM) approach enhances the face alignment accuracy compared to the original method used in ASM. 4 Discussion The results presented in this paper show that a texture-based approach can be very useful in various tasks of facial image analysis. Approaches based on Gabor filtering or wavelets measuring the frequency contents of facial image points or regions at different resolutions and orientations are often powerful, but computationally quite complex. The idea of using computationally simple LBP operator for face description is motivated by the fact that faces can be seen as a composition of micro-patterns (edges, lines, flat areas, spots etc.) which are well described by the operator. Combination of different texture approaches, like using Gabor filtering with LBP [13], could be one way to go ahead. Using the dynamics of facial images could improve the accuracy of face and facial expression recognition. New approaches based on dynamic texture could be useful in this kind of problems. Naturally texture-based approach has also limitations. Images taken at different times with different sensors in different illumination conditions are problematic, because under these changes the surface texture is likely to change. Proper methods for image preprocessing and features which are more robust against these kinds of transformations are needed. References 1. Tuceryan, M., Jain, A.K.: Texture Analysis. In: Chen, C.H., Pau, L.F., Wang, P.S.P. (eds.): Handbook of Pattern Recognition and Computer Vision, 2nd edn. World Scientific (1999) 207-248
8 Matti Pietikäinen and Abdenour Hadid 2. Wayman, J., Jain, A.K., Maltoni, D., Maio, D.: Biometric Systems: Technology, Design and Performance Evaluation. Springer (2005) 3. Wiskott, L., Fellous, J.-M., Kuiger, N., von der Malsburg, C.: Face Recognition by Elastic Bunch Graph Matching. IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (1997) 775-779 4. Tian, Y.-L., Kanade, T., Cohn, J.F.: Facial Expression Analysis. In. Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition. Springer (2004) 247-275 5. Viola, P.A., Jones, M.J.: Robust Real-time Face Detection. International Journal of Computer Vision 57 (2004) 137-154 6. Ojala, T., Pietikäinen, M., Harwood, D.: A Comparative Study of Texture Measures with Classification Based on Feature Distributions. Pattern Recognition 29 (1996) 51-59 7. Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (2002) 971-987 8. Mäenpää, T., Pietikäinen, M.: Texture Analysis with Local Binary Patterns. In: Chen, C.H., Wang, P.S.P. (eds.): Handbook of Pattern Recognition and Computer Vision, 3rd edn. World Scientific (2005) 197-216 9. http://www.ee.oulu.fi/research/imag/texture/ 10. Ahonen, T., Hadid, A., Pietikäinen, M.: Face Recognition with Local Binary Patterns. In: Computer Vision, ECCV 2004 Proceedings, Lecture Notes in Computer Science 3021 (2004) 469-481 11. Ahonen, T., Pietikäinen, M., Hadid, A., Mäenpää, T.: Face Recognition Based on the Appearance of Local Regions. In: 17th International Conference on Pattern Recognition (2004), Cambridge, UK, 3:153-156 12. Zhang, G., Huang, X., Li, S.Z., Wang, Y., Wu, X.: Boosting Local Binary Pattern (LBP)-Based Face Recognition. In: Advances in Biometric Person Authentication, SINOBIOMETRICS 2004 Proceedings, Lecture Notes in Computer Science 3338 (2004), 179-186 13. Zhang, W., Shan, S., Zhang, H., Gao, W., Chen, X.: Multi-resolution Histograms of Local Variation Patterns (MHLVP) for Robust Face Recognition. In: Audio- and Video-Based Biometric Person Authentication, AVBPA 2005 Proceedings, Lecture Notes in Computer Science 3546 (2005), 937-944 14. Hadid, A., Pietikäinen, M., Ahonen, T.: A Discriminative Feature Space for Detecting and Recognizing Faces. In: IEEE Conference on Computer Vision and Pattern Recognition (2004) II: 797-804 15. Jin, H., Liu, Q., Lu, H., Tong, X.: Face Detection Using Improved LBP Under Bayesian Framework. In: Third International Conference on Image and Graphics (ICIG 04), Hong Kong, China, 18-20 Dec 2004, 306-309 16. Feng, X., Pietikäinen, M., Hadid, A.: Facial Expression Recognition with Local Binary Patterns and Linear Programming. Pattern Recognition and Image Analysis 15 (2005) 550-552 17. Shan, C., Gong, S., McOwan, P.W.: Robust Facial Expression Recognition using Local Binary Patterns. In: IEEE International Conference on Image Processing (2005) 18. Huang, X., Li, S.Z., Wang, Y.: Shape Localization Based on Statistical Method Using Extended Local Binary Pattern. In: Third International Conference on Image and Graphics (ICIG 04), Hong Kong, China, 18-20 Dec 2004, 184-187