Color-based Face Detection using Combination of Modified Local Binary Patterns and embedded Hidden Markov Models

Transcription

1 SICE-ICASE International Joint Conference 2006 Oct. 8-2, 2006 in Bexco, Busan, Korea Color-based Face Detection using Combination of Modified Local Binary Patterns and embedded Hidden Markov Models Phuong-Trinh Pham-Ngoc and Kang-Hyun Jo Graduate School of Electrical Engineering, University of Ulsan, Korea (Tel : ; {equation/jkh2005}@islab.ulsan.ac.kr) Abstract: This paper presents an improved face detection method for color images. We propose a boosted skin-color model in RGB space which can reduce more effectively noises forming from similar skin colors. With our solution, we receive more reasonable skin detection for different human races. We modified Local Binary Pattern (LBP) by adding a set of spatial templates. This LBP considers both principal local shapes and spatial textures of facial components. Human face is represented by LBP histogram. Moreover, the grayscale image of human face is changed to Discrete Cosine Transform (DCT) coefficients used in embedded Hidden Markov Models (ehmms). A modified LBP (mlbp) histogram matching and ehmms are composed to hierarchical classifier to determine whether skin regions are faces or not. The experiments show that our method performs a better capability for face detection in complex environments than using separately ehmms or LBP histogram. The correct face detection rate of proposed system is over 94% among our test database which consists totally 485 single and multi-face color images of 429 persons in different lighting conditions, face rotations, occlusions and complex backgrounds from different sources: Caltech face database, Sumgmug image library, family photos, personal digital images and world wide web. Keywords: Face detection, skin segmentation, Local Binary Patterns, embedded Hidden Markov Models.. INTRODUCTION Face detection is a key step of the automatic face analysis system. In recent years, many face detection methods have been published and have achieved some encouraging results. A comprehensive survey on face detection in images can be found in [9]. In general, we can divide those approaches into two categories: model-based technique and feature-based technique. The first one assumes that a face can be represented as a whole unit. Several statistical learning mechanisms are explored to characterize face patterns, such as neural network, Bayes classifier and boosting algorithm. The second category considers a face as a collection of components. Important facial features such as eyes, nose and mouth are extracted, and by using their locations and relationships, the faces are detected []. Among feature-based face detection methods, using skin color as a detection cue is very popular. Skin color is important and powerful information for human face. It can be used as the first step in face detection process in color images. Many researchers use skin color models to locate potential face regions and then examine the locations of faces by analyzing each face candidate s shape and physical geometric information []. In order to represent to human face, what features are invariant for face detection is still an open problem. Previous researchers presented many representations for facial feature extraction, such as edges, rectangle features and Gabor [,8,0]. In this paper, we propose an improved face detection method in color images. A boosted skin color model performs more reasonable skin segmentation. Unreasonable skin regions are discarded by physical geometric conditions of human face. Reasonable ones are prevented by replacing them with elliptic skin regions. Finally, we receive the most potential ones considered as face candidates. Next, we present an improved LBP considering not only local spatial textures but also principal local shapes. The human face is divided into two parts and each of them gives us one developed LBP histogram. The mixed histogram formed from these two histograms is considered as facial representation. A hierarchical classifier combining proposed LBP histogram matching and ehmms is used to identify whether face candidate as human face or not. This paper is organized as follows: face localization with proposed skin-color model is introduced in section 2. In the next section, we will present an improved LBP. A discriminant analysis of this LBP histogram is described in section 4. Then we introduce a hierarchical classification for face detection system in section 5. The experimental results are given and analized in section 6. Finally, we will give conclusions in section FACE LOCALIZATION The use of color information can simplify the task of face location in complex environments. It allows fast processing and is highly robust to geometric variations of the face pattern. Therefore, we use skin color detection as the first step in detecting faces. 2. Skin color model For skin color detection task, many colorspaces with different properties have been applied. Many researchers have achieved some results with RGB, normalized rgb, HSI, YCrCb and RGB-space ratios. A survey of skin color detection can be found in [7]. In order to obtain better skin detection, we create an improved skin-color model in RGB space. It can reduce

2 effectively similar-skin colors causing noises in skin segmentation. These colors can be yellow, white, orange, pink or red. Other skin models such as the works of Lin [3] and Peer [5] show nice skin segmentation but they are still sensitive to retain those non-skin colors. Lin model often retains similar yellow color and Peer model is sensitive to keep red color. This weak point sometimes becomes a serious problem because non-skin regions are retained too many to determine correct face regions. Our way is to build a skin classifier to define explicitly the boundaries of skin cluster in RGB space. This method is simple to lead a rapid classifier. Decision rules of our skin modeling are as follows:, if a set of conditionsissatisfied δ ( P( x, y)) = 0, otherwise where P(x,y) is an image pixel and a set of conditions are listed in table. Table. A set of conditions defining skin pixel. These conditions should be satisfied simultaneously. R (R-G) (G-B) B G [70,85] [30,55] [-5,35] [20,255] [30,255] [86,00] [30,60] [-5,40] [30,255] [40,255] [0,50] (R-G) (G-B) (R-B) (R+B-2G) G [0,7] [-2,0] [5,75] - - [8,30] [-255,-0] or [25,45] - [-5,285] - [3,70] [-5,90] [-255,20] [-20,285] [50,255] [7,75] [-5,0] [-255,70] - [50,255] [5,200] (R-G) (G-B) (R-B) (R+B-2G) B [5,20] [0,40] [20,255] [-20,285] - [2,30]] [-5,0] [20,255] [35,285] - [3,85] [-5,70] [20,255] [0,285] [40,255] [20,255] (R-G) (G-B) (R+B-2G) [5,25] [40,70] [-30,285] [26,00] [0,70] [-5,285] In fact, the strongest component among R, G, B defines the color. For skin color, generally R component is always the strongest one because human skin has the special expression of blood color. According to [5] and our experiment, in general, if R value is smaller than 70, the effect of R component is not enough to describe skin color; the color can be dark red, brown, green or black. If R is greater than 70 but G and B are smaller than 40 and 20 respectively, the color is strong red or dark red. Besides, if R is too greater than G or B, the color can be real red, light red, pink, yellow or orange. And the color is white or light yellow, if R value is too close to G and B. It means the color is not skin color if the differences between R, G and B are too big or small or R value is smaller than 70. The level of red color affects the decision rule of our skin model. Approximately, we divide R value larger than 70 into five main ranges. So our work is adjusting reasonable differences between R, G and B according to these ranges. The proposed solution shown in table gives more reasonable skin detection for different human races. Some skin detection results shown in Fig. prove the advantage of our skin model. (a) (b) (c) (d) Fig. Comparison of skin detection results: (a) original colored image; (b), (c) and (d): skin detection results of [3], [5] and our skin model respectively. 2.2 Face candidate localization An overview of face detection system is described in Fig. 2. Fig. 2 Face detection system scheme After skin segmentation, we label connected skin regions and erase the regions whose areas are smaller than the threshold. In our experiment, this threshold is 08 pixels considered as half of the smallest face to be detected. We call this step reducing small noise. Besides, because skin segmentation is effected by different light conditions, maybe we lose some face regions. We will recover those necessary regions by labeling connected non-skin regions in each skin regions and change them into skin ones. This process ignores non-skin regions connecting directly to boundaries of their skin regions and non-skin regions whose areas are bigger than selected thresholds. With skin regions having properties of human faces, we preserve them by covering their areas with skin ellipses. Those works are very important for our face localization step because we will use strong conditions of intensity histogram of skin pixels to separate connected faces. In some cases, human faces in images can be connected together or connected with other things such as hand, arm. This is one of challenges for face detection problem. Without those processes above, we may reduce or lose correct faces under strong separation. After dividing connected objects, we reject non-face skin regions by some

3 physical geometric conditions. Finally, we get the most reasonable skin regions considered as face candidates. Those candidates are changed to 72x93 grayscale images used in hierarchical classification. We will introduce it in next sections. 3. A MODIFIED LBP FOR FACE REPRESENTATION Human face is a near-regular texture pattern generated by facial components and their configuration. By considering facial components such as eyebrow, eye, pupil, nose and face boundary, we select 8 main different spatial templates shown in Fig. 3 to preserve information about shape of facial component. (0) () (2) (3) In fact, mlbp gives us information about both local shapes through 8 spatial templates and local textures. We retrieve more information to distinguish face and non-face objects. We use this mlbp histogram to represent a face. Because of occlusion by sun glasses, human body parts or other complex objects, some parts of human faces can be occluded. If we only use single mlbp histogram for the whole face candidate image, occlusion will affect matching algorithm seriously. In general, human face has two most important parts: the upper part from nose up to forehead and the lower part from nose down to neck which includes the top of nose, mouth, lips, chin and neck. So we should calculate each part by individual histogram. And these two histograms are connected sequentially to create one mixed 255x2 bin histogram representing to face candidate image. By this way, we can reduce effectively the influence of occlusion. Fig. 5 shows gray image sample, its mlbp image and mixed histogram. (4) (5) (6) (7) Fig. 3 Spatial templates. With only those spatial templates, we can describe all facial components; for example, eyebrow can be described by a union of templates 3, and 2. However, we combine both those spatial informations and local texture informations to improve the capacity of describing faces. We improved LBP algorithm with new rules according to proposed spatial templates. The original LBP [6] is a grayscale irrelevant texture operator with powerful discrimination. From original algorithm, LBP has been improved to different goals such as overcoming illumination problem [2]. In our method, instead of considering the central pixel P C only with its each neighborhood pixel as original LBP operator did, we use each pair of two neighborhood pixels (P i,p i2 ) according to spatial templates to compare with the central pixel P C. Eight spatial templates form 8 binary digits of mlbp number. So mlbp operator produces 256 different mlbp values. Function () gives the computation of mlbp number. mlbp = 7 i= 0 S i ( x) 2 i where S i is the i th binary digit of mlbp number;, ( PC > Pi ) and ( PC > Pi 2 ) Si ( x) = 0, otherwise An example of mlbp operation is given in Fig. 4. Fig. 4 An example of modified LBP operation. () (a) (b) (c) Fig. 5 (a) Image sample, (b) mlbp image and (c) mixed mlbp histograms (255x2 bins). Given an image I, one mlbp 255x2 bin histogram is denoted by H mlbpmix (I). There are many methods of measuring similarity between two histograms. In our experiments, we adopt error measurement because it is simple and fast computation. A distance measurement is defined as: D ( H( I), H( I2)) = Hi n i= mlbpmix ( I ) H mlbpmix i ( I ) where H mlbp (I ) and H mlbp (I 2 ) are two mixed mlbp 255x2 bin histograms, and n is the number of bins. 2 (2) 4. DISCRIMINANT ANALYSIS OF MIXED MLBP HISTOGRAM 4. Face and non-face class selection Face detection will become an easy problem if we have clearly face and non-face class modeling. However, it is difficult to model non-face class because anything which is not a face belongs to non-face class. In our method, because of boosted skin color model having advantage of reducing non-skin color, the results of face localizations are mostly human body parts such as face, hand, arm, shoulder, ear, leg. We collect 90 frontal and profile face images to create face samples. With non-face class, we choose three main non-face objects creating non-face samples: arm (0 samples), hand (3 samples) and noise (56 samples). All samples are 72x93 size color images. In our experiments, those samples are enough to represent face and non-face classes. Fig. 6

4 shows some randomly chosen non-face samples. Fig. 6 Non-face samples; (a) noise; (b) arm; (c) hand. All face and non-face samples are used for discriminant analysis of mixed mlbp histogram to distinguish face and non-face objects. We will introduce it in detail in the next part. 4.2 Discriminant analysis of mixed mlbp Histogram matching is a direct approach for object recognition. In this approach, we apply histogram error as a distance measure to object recognition. Given a face database with m samples, for any sample P, we change it from color image to grayscale one and define its histogram-matching feature as the average distance to face training samples as follows: m f face( P) = D( H ( P), H ( X i )) (3) m i= where X i is a face training sample. This mixed mlbp histogram has the discriminating ability between face and non-face patterns. To demonstrate this property, we can see Fig. 7 which shows the positive and negative distance measure distribution over 56 face samples and 2 non-face samples. distance measure non-face face images Fig. 7 Distribution of distance measure. With this feature f face, we use thresholds to classify face and non-face objects. Those thresholds T face and capacity of this classification is described in table 2. Table 2. Face detection rate following T face thresholds T face Face detection rate [0,800] 99% [80,3500] 65% >3500 % In range [800,3500] of T face, the classification capability is not good and we improve it by the following feature. With non-face database, for any sample P, we also define its histogram-matching feature as the minimum of three average distances to three non-face object training samples, given by f ( P) min( f ( P), f ( P), f ( P)) (4) nonface = arm hand noise where f arm, f hand, f noise are calculated following to (3). The difference between f nonface and f face shown in Fig. 8 also has the discriminating ability between face and non-face patterns. We call this difference D f, given by D ( P) = f ( P) f ( P) (5) f difference feature Df nonface face non-face face images Fig. 8 Distribution of difference feature D f. We use D f to improve the face detection rate when T face is in [800,3500]. We can define the explicit thresholds T D for D f to distinguish face and non-face patterns. In addition, ehmm is also added to our classification and we will present it in the next part. 5. HIERARCHICAL CLASSIFICATION FOR FACE DETECTION SYSTEM We apply a hierarchical classification scheme for face detection, shown in Fig. 9. Fig. 9 Hierarchical classification. A face candidate is checked under 3 stages and face detection system for this candidate can be stopped in whatever stage which gives face result. The first stage is mixed LBP histogram matching as fine detection stage giving strictly correct face detection. Because of the strict mlbp histogram matching, some human faces can be ignored by the first stage. In next stage, a difference feature D f is used to retain these lost faces. After these stages, face detection rate can reach over 80%. In order to gain the effect of our system, ehmm algorithm is used finally to find lost human faces after the second stage and ignore non-face objects. 5. Mixed mlbp histogram and D f matching As demonstrated in the previous section, our mixed mlbp histogram matching feature f face and difference feature D f have the ability to identify face and non-face. We specify matching conditions for both f face and D f and use them jointly for two first matching stages. The decision rules are as follows:

5 , if ( fface 800) or ( ffaceanddf are as δ ( fface, Df ) = listed intable 3). 0, otherwise Table 3. A set of conditions defining human face f face D f Aspect ratio 0.85 Aspect ratio< [ 800,2500) 0 ~ 400 ~ [ 2500,2900) 300 ~ 500 ~ [ 2900,3500) 300 ~ 400 ~ where aspect ratio is a ratio between the height and the width of one face candidate. After those two stages, if face candidate is still decided as non-face, ehmm is used to check it last time to give the final conclusion. and sizes. On the color image database, finally our system correctly detects 643 faces and produces 69 false detections. The correct face detection rate for our proposed system is 94.32%, which proves that our method is effective in detecting faces. Some frontal face detection examples are shown in Fig Embedded Hidden Markov Models (ehmms) Face detection is regarded as a multi-class pattern classification problem. In our algorithm, we define non-face class as three different sub-classes: arm, hand and noise. It means our face detection is changed to four class pattern classification problem. EHMMs classifier [4] performs pattern recognition for a four-class problem by determining the maximum likelihood to find the most similar class for candidate object. Given training sets of positive and negative samples, we will have four ehmm models corresponding to four classes: face, arm, hand and noise. A face candidate which was ignored by the two first stages of face detection system is checked by ehmm matching. If the result of this face candidate under ehmm stage is non-face, it means finally this is not face and our system stops. 6. EXPERIMENTS We implement the proposed method and conduct experiments to evaluate its effectiveness. Many face databases for testing face detection results are used by researchers such as FERET and CMU face dataset which have gray-scale images only. Because our work is done with color images, we built a color face test dataset. Our face test dataset consists totally 485 color images from different sources: Caltech face database, Smugmug image library, family photos, personal digital images and world wide web. It includes single and multi face images from 429 persons (809 men, 597 women and 23 babies) who are European, Asian, American and African. There are 40 frontal faces and 289 profile ones with variations in rotation, color, position, size and expression. Our face test database also gets images with total 86 occluded faces, 329 faces with glasses and 52 connected faces. These images are taken under complex environments and different lighting conditions. We use this test database to estimate the proposed method. Our system can detect frontal and profile faces under complex backgrounds. With modified skin color model, our method can detect dark skin-tone and bright skin-tone faces in different lighting conditions. It also can detect single and multi faces with different colors Fig 0. Examples of perfect frontal face detection. The results shown in table 4 prove that our method has better result than using only original ehmms or mlbp histogram matching algorithm. Table. 4 Comparison of face detection results Algorithms Correct detected faces Detection rate (%) Fault detection Proposed method mlbp histogram matching ehmms where total faces in test dataset are 742 ones. Moreover, as shown in Fig., the face test dataset demonstrates that our system can solve effectively problems of in-plane face rotation (examples g), occlusion with occluded part less than 50% region of

6 face (examples b and c), sun-glasses (example a), connection (example e) and profile face with the view angle rotation less than 75 o by the vertical axes (examples d, f and g). The results of those problems are described in table 5. Fig. Some successful face detection results. Table 5. Summary face detection results Problem of face detection Face detection rate In-plane face rotation 94% Occluded faces 7% Profile faces 9% However, our method sometimes can fail or miss to detect human faces as shown in Fig. 2. Fig.2 (a) Missing matching; (b) Missing detection; (c) False detection. Faces whose areas are smaller than 26 pixels are often missing detected because of being ignored in face localization step. Missing matchings can be caused by shadows, occlusions Non-face objects similar to face patterns in color and shape cause false detection. One of our future works is overcoming these limitations. 7. CONCLUSIONS In this paper, we proposed an improved face detection system for color images. The advantage of our method is the improved skin color model reducing more noises to give us more potential face candidates. We modified the original LBP feature by combining spatial templates into it to make a developed LBP feature which has more effective information to represent patterns. Our method of using hierarchical scheme combining feature histogram matchings and ehmms for classification performs a good capability for face detection in color images with complex background. Comparing to original ehmms or LBP histogram matching, our method has better face detection results. In future, we plan to apply and develop this system to face recognition task. ACKNOWLEDGEMENT The authors would like to express both financial supports from NARC (Network-based Automation Research Center) and post-bk (Brain Korea) 2 program which are sponsored by both of Ulsan Metropolitan City and MOCIE (Ministry of Commerce, Industry and Energy), and MOE (Ministry of Education and Human Resources Development), respectively. REFERENCES [] R. L. Hsu, M. A. Mottaleb and A. K. Jain, Face Detection in Color Images, IEEE Trans. on PAMI, vol. 24, no. 5, pp , [2] H. Jin, Q. Liu, H. Lu and X. Tong, Face Detection using Improved LBP under Bayesian Framework, Proc. of ICIG, [3] C. Lin and K. C. Fan, A Color-Triangle-Based Approach to the Detection of Human Face, BMCV 2000, vol. 8, pp , May [4] A. Nefian and M. Hayes, Face Recognition using an embedded HMM, Proc. IEEE Conference on Audio and Video-based Biometric Person Authentication, pp. 9-24, 999. [5] P. Peer, J. Kovac and F. Solina, Human Skin Colour Clustering for Face Detection, EUROCON, [6] L. G. Shapiro and G. C. Stockman, Computer Vision, Prentice Hall, New Jersey, 200. [7] V. Vezhnevets, V. Sazonov and A. Andreeva, A Survey on Pixel-Based Skin Color Detection Techniques, Proc. Graphicon, pp , [8] P. Viola and M. Jones, Robust Real-time Object Detection, Proc. IEEE Workshop on Statistical and Theories of Computer Vision, 200. [9] M. H. Yang, D. J. Kriegman and N. Ahuja, Detecing Faces in Images: A Survey, IEEE Trans. on PAMI, vol. 24, no., pp , [0] P. Yang, S. San, W. Gao, S. Z. Li and D. Zhang, Face Recognition using Ada-Boosted Gabor Features, Proc. 6 th IEEE International Conference on AFGR, pp , May [] K. C. Yow and R. Cipolla, Feature-based Human Face Detection, Image and Vision Computing, vol. 5, no. 9, pp , 997.