Face Detection and Recognition with Multiple Appearance Models for Mobile Robot Application

Transcription

1 Face Detection and Recognition with Multiple Appearance Models for Mobile Robot Application Taigun Lee, Sung-Kee Park and Munsang Kim Advanced Robotics Research Center, Korea Institute of Science and Technology, Hawolgok-dong 39-1, Sungbuk-gu, Seoul , Korea (Tel : ; Fax : ; {dungding, skee, munsang}@kist.re.kr ) Abstract: For visual navigation, mobile robot can use a stereo camera which has large field of view. In this paper, we propose an algorithm to detect and recognize human face on the basis of such camera system. In this paper, a new coarse to fine detection algorithm is proposed. For coarse detection, nearly face-like areas are found in entire image using dual ellipse templates. And, detailed alignment of facial outline and features is performed on the basis of view- based multiple appearance model. Because it s hard to finely align with facial features in this case, we try to find most resembled face image area is selected from multiple face appearances using most distinguished facial features- two eyes. In face recognition step, probabilistic density of above image is estimated from a view of maximum-likelihood. Multi-modal Gaussian densities from expectation-maximization algorithm is applied and multiple appearance model of person is also used, which can cope with limitations such as a variety of training samples, varying illumination conditions. Finally, we experiment with these methods within the traveling range of mobile robot and results of face detection and recognition are reported. Keywords: face detection, face recognition, view-based appearance model, mobile robot application, PCA, facial color 1. Introduction This paper presents face detection and recognition algorithms using mobile robot vision system originally designed for visual navigation. This mobile robot is developed towards indoor personal services. It has various sensors, such as stereovision camera, ultrasonic sensors, and laser slits. Specially, stereovision that equipped on the top of robot is used to obtain information of random indoor-environment for navigation. Incorporated with navigating works, we try to perform task of human-face detection and recognition in one vision system. This is useful for the common operations of indoor personal robot, for example, helping people s work and interacting with human at the same time. However, this vision system is developed for visual navigation, so camera s field of view is large and focal length and focus of lens are suitable to long distance (e.g. about m). So, the part of facial area from entire camera image is very small (below 10%), and, due to these reason, it is difficult to find facial features (e.g. two eyes, nose, mouth, et al.) precisely and obtain normalized face image (with aligned features) for recognition, rather than conventional approaches that used relatively large face image. Considering these characteristics of our camera, this paper mainly focuses on the detection and recognition of faces obtained from moving camera. So, a new algorithm for estimating normalized face images and recognizing human face is proposed. For this work, we apply multiple face appearance models in both face detection and recognition step. This algorithm is significant for natural process of coarse to fine face detection and recognition, especially small face images under various environment. Previous researches are presented in section 2. Our algorithms of face detection and recognition are followed in section 3. And we show experimental results in section 4, and conclusion in section Previous researches In the human-computer interaction, face interface technique is excellent by reason that minimal help of user is needed for identification. Recently, many efforts on practical technology have reported for more reliable and robust human identification. Main goal of face detection (FD) is locating position of face in uncontrolled environment, whether face detection is considered as a prior step of face recognition (FR). Generally, many FD researches are classified as two approaches, feature-based or image-based approach. The considerations of characteristic geometry of face using low-level vision algorithms [1,2] are typical feature-based approaches, which are advantageous for real-time detection. Next, representative image based approach is template matching methods. With converting face image into face vector (array), location of face is obtained by calculating correlations between face array and standard face template. Some researches present good results from some of these techniques, including principal component analysis (PCA)[3], linear/fisher discriminant analysis (LDA/FDA) [4,13].

2 On the other hand, popular for pattern recognition problems, neural networks (N.N.) that use trained model information show very good results [5]. However, while its high precision rate of detection, these template matching or NN technique have time-consuming problem. In recent years, FR technology is an active field of study, and also many research and commercial results have reported. PCA, using eigenvectors of facial image, is also typical approach in FR. Until a recent date, many papers about PCA have been presented. Especially, subspace methods that obtain probabilistic measure of similarity using simple subspace-restricted norm of eigenface are a good example [7]. The proposed similarity measure is based on a standard Bayesian analysis of image differences of two categories, intra-personal variations and extra-personal variations, and also high-dimensional pdf functions are used to estimate likelihood, so performance improvements are achieved. FR systems using LDA/FDA, extended form of PCA, have also been successful [4,6]. LDA training is carried out via scatter matrix analysis and face image is retrieved by discriminant analysis of eigenfeatures [8]. Other group is FR based on Neural Network (N.N). Many approaches using NN are reported, as like HyperBF networks, Dynamic Link Architecture (DLA) [8,9]. The DLA architecture has been recently extended to Elastic Bunch Graph Matching (EBGM), good results are showed [10]. A set of jets are attached to NN node from various faces, its results are improved. Also, to cope with pose variation, the predefined face pose is trained using prior information. Besides, the improvement techniques of FR in video-based face processing, face recognition using infra-red images, and multi-modal based person recognition system incorporated with speech recognition are reported [8]. 3. Face detection and recognition methods 3.1 Outline of face detection and recognition algorithm In the step of face detection, first, coarse face location from entire image is found. Then, fine location and pose of face are decided by this method. That is, most resembled face image area is selected from multiple face appearances on the basis of most distinguished facial features- two eyes. Whether this face area (patch image) belongs to a certain appearance of FR steps is not yet determined. This area is regarded to criterion image which faces of people include this area, in next step. Following step of face recognition, multiple face appearances are considered once again. These multiple models are prepared as two or three appearances per each person. And, per each appearance of people, several images are included for training data images. For all appearances of multiple models, probabilistic measure of similarities are estimated based on Maximum Likelihood. Then, the model that has most probabilities greater than predefined threshold is adopted. The person who this model belongs to is decided to result of recognition. 3.2 Face detection step Through two-step processes coarse and fine detection of face [16], precise location of face in random images and standard face-image that has aligned facial features are obtained. In the first step of FD, segmentation of color image for making binary and gray image is performed on the basis of skin color characteristics. For segmenting facial colors from various and large range of skin appearance from different races to different light conditions, we use YCbCr color space[11] for simple and compact clustering. Transforming from RGB space to YCbCr space, we saw that the cluster of skin color is more compact. Fig.1 shows distribution of FCs(face colors) in 2 color space from various skin image patches. Fig.1. Facial color distribution of RGB and YCbCr space Such distributions of facial color are segmented using the bounding line conditions in Cb-Cr planes, Cr FC : 2 2 r i = Cri + Cbi, r i > 0, ri < 70 (1) C φ FC : φ i = A tan 2( Cr, Cb), 1.6 < φ i < 3.0 (rad) (2) [ I φ ] FCi = U r i i ( ri C r FC, ri Cφ FC ) (3) C FC Based on YCbCr distribution, binary facial color image is created as like Fig.2. Fig.2. Binary and gray image based on skin color The second step of FD, for real time computation, we use multi-resolution pyramids image using wavelet transform [12], for the reason of searching the scaled-down image quickly. Especially, low frequency image of wavelet multiscale representation have showed good experimental results that the locations of facial area lump in binary FC image are finely preserved in wavelet transform process. Next step is coarse detection of face. In above transformed image, double-ellipse templates are applied. First, by means of considering appropriate face size that can be detected in image size, multiple ellipse sets are predefined. Then, as Fig.3,

3 dual-ellipse templates are created like that inner ellipse whose rate of facial colors of pixels is greater than defined threshold and outer ellipse pixels whose rate of non-facial colors are greater than its threshold. square face image is normalized with histogram-equalization. Ei ( xi, yi ) pos Cellipse (4) where, C = Th < E < Th ) ( Th < E ellipse { ) } ( FC, min inner FC,max NFC, min outer With these templates scanned over entire image, multiple satisfied positions are obtained. Also, varying all scale of templates, most 10 satisfactory rates of positions are selected and then solve average face position and template size. Detected area is derived from these values, (5) 10 FD coarse = U E i (6) i=1 Fig. 3. Multiscale dual ellipse templates for the coarse detecting The fourth step of FD is fine detection of face. About previous obtained area, face similarity is estimated with also varying scale in detail. For calculating similarity of corresponding area, face-likelihood template function is defined. This function accounts for satisfactory rate of facial feature geometry. Typical facial features are bright elements (e.g. forehead, cheek, nose-tip, et al.) and dark elements (e.g. two eyes, eyebrows, nose, mouth, et al.). According to individual characteristics of people, some differences exist, but general dimension and geometric relations of facial features are same. So, our function finds location of maximum likely area in each scale by accumulating pixel values that satisfy criterion of facial feature intensity (brightness), most satisfactory region is determined. Specially, with focusing on most distinctive and well-observed feature two eyes, intensity criterion about the size and place of elements is applied to whole face features as nose, mouth, cheek, and forehead. The likelihood feature conditions and template function definition are like that, FFfacial feature = { Constraint s ; Rate _ location & Rate _ pixel int ensity } FFLikelihood = 5 FeyeR+ L + 3Feyebrows + Fnose + 2Fmouth + 3 Fcheek + 3F forehead (7) Let two eyes be located in standard position in square image patch. Also, to make the intensity of face images similar, Fig. 4. Fine detection of face by estimating position of two eye and feature geometry 3.3 Face recognition step As the condition that small face images captured from moving camera, our method of FR must be suitable for these frontal patches of face, and also must yield good recognition rate about small images. In this paper, we adopted the method of probabilistic subspace PCA [7], regarded as good for our condition and then formulated whole steps of face recognition using this method. Probabilistic distribution of subject face image is estimated from a view of Maximum-likelihood (ML). Common methods of PCA are performed in principal subspace, which means that the appearance image of certain object is decomposed into its eigenvectors and principal component of eigenvectors are only used for recognition. In the method of probabilistic subspace PCA, the motive that orthogonal complements of eigenvectors affects improvement of recognition rate is presented. In this study, the way that principal parts of complementary components are quickly calculated is also devised. The likelihood of an input pattern x is given by estimated mean x and covariance Σ of the distribution from the predefined training set. Using diagonalized form of covariance, the characterizing distance (numerator of likelihood) can be rewritten like this, 1 T T 1 [ ΦΛ Φ ] x = y Λ y T 1 T d ( x) = x Σ x = x (8) And then, divided into two parts- principal and complementary components, decoupled form is expressed as, M 2 2 y N i yi d( x ) = + ( = F + F ) (9) i= 1 λi i= M + 1 λi M means threshold for dividing between principals and complements, is adjusted according to facial image size. Incorporated with these two components, complete estimate of likelihood with ML shows good results of recognition [7]. In addition, we noticed that calculating orthogonal complements are complex for plenty of components and some parts of complements nearly amounted to all parts of eigenvalues. So, similar to threshold of principal components M, we introduce the threshold of complementary components N end rather than N. Then, d (x) is simplified to d * ( x) and it is also useful for reduced time to recognizing face.

4 At this point, we can represent the form of the likelihood estimated using simplified d * ( x), P ˆ ( x Ω) = P ( Ω) ˆ * F x PF ( x Ω) (10) For example, from all 900 eignevalues of image, range from 891 to 900 in Fig.5 is principal and range of 881 to 890 is meaningful (most part of) components of complements over threshold, and those values are adopted for computing. is that P ˆ ( x Ω) = P( y Θ * ) Pˆ F ( x Ω) (13) In our experiments, three models per each class (person) is set, and several images are allocated to each model. About all models, likelihood estimations are performed using Maximum-Likelihood (ML). Then, the model which has most likelihood is adopted, and then the person who this model belongs to is decided to final result of recognition. Fig. 5. Choosing boundary of principals and complements On the other hand, previous researches assumed that probability density of the training images was Gaussian(unimodal). This is true in cases where the training images are accurately aligned frontal views of faces seen from a standard view-condition. However, if training faces show multiple views under varying illumination and scale condition, the distribution of training set is no longer unimodal. For analyzing such multimodal training set, we propose a multiple appearance-based approach. This approach is revised from view-based formulation where separate eigenspaces are used for each view [14] and from parametric mixture model [7]. This equation models arbitrary mixed Gaussian density p(y) estimated using parametric mixture model. Nc p( y Θ) = π i g( y; µ i, Σi ) (11) i= 1 ( π i : probability mixture rate, µ i : average of model image, Σ i : covariance of model ) Fig. 6. The process diagram of face recognition based on multiple appearance models 4. Experimental results At this time, our goal is detection and recognition of 510 peoples in the images from moving camera. In this environment, our target of recognition rate is about 90%. Face shape for FR is near-frontal image containing a little bit panning and tilting of camera. 4.1 FD test Based on 320*240 pixels image, single or plural face detection is performed. Real time detection is achieved (about 7 times per second), irrespective of similar face-color regions around face. * And, parameter Θ and its estimated form Θ specify the mixture. N N Θ = { π i, µ i, Σi } i= c 1, Θ * T = t arg max p( y Θ ) (12) t= 1 Fig. 7. Results of face detection (with a similar facial color regions) In [7], the problem of parametric estimation is solved using Expectation-Maximization (EM) algorithm [15]. Final estimated model of likelihood, based on mixed Gaussian density,

5 Fig. 8. FD at each template scales and final FD results 4.2 FR test Training images of people s face are obtained from face detection process. Fig. 10. Face recognition results in 0.6 m Fig. 9. Part of the training model images for face recognition Ten peoples for training database are chosen, and 20 patch images are obtained from face detection step, regarding the face-variation with scale and illumination. The face information - such as multimodal parameters, average face, and covariance of training images- are calculated in off-line process. These data are used to recognizing face in real time as basic face database. In our experiments, the time required for both face detection and recognition process is about 2 seconds, at P4-1.6Ghz. Pictures of FD and FR process in various distance, are showed in Fig.10,11. And the rate of detection and recognition for varying distance of observation are also presented. Especially, near 1m distances, our algorithms show good detection and recognition results, rather than any other cases. Fig. 11. Example of entire face detection and recognition steps in real-time computation 5. Conclusion We propose face detection and recognition algorithm that is suitable for characteristics of mobile robot vision system. This novel method deals with face images that obtained from a long distance, by using multiple face appearance models that performed in both face detection and recognition step. In face detection step, based on the coarse and fine detection skill, face locations are searched and normalized face image for FR is found. After that, in face recognition step, using this standard face image, probabilistic similarity of this image is estimated. From a view of maximum-likelihood estimation and parametric mixture model with multimodal distribution, recognition rate can be improved. We performed the face detection and recognition in mobile robot vision task, for single face, and showed and validated the results of our algorithms. Fig. 12. Comparison of face detection and recognition methods References [1] R. Brunelli and T. Poggio, Face recognition: Feature versus templates, IEEE Trans. Pattern Analysis and. Machine Intell., vol.15, 1993, pp [2] Cox, I.J.; Ghosn, J.; Yianilos, P.N., Feature-based face

6 recognition using mixture-distance, Computer Vision and Pattern Recognition, Proceedings CVPR '96, IEEE Computer Society Conference on, 1996, pp [3] G. Yang and T. S. Huang, Human face detection in a complex background, Pattern Recog. 27, 1994, pp [4] D.L.Swets and J.Weng, Using Discriminant Eigenfeatures for Image Retrieval, IEEE Trans. Pattern Analysis and Machine Intell., vol.18, pp , 1996 [5] H. A. Rowley, S. Baluja, and T. Kanade, Neural network-based face detection, IEEE Trans. Pattern Anal. Mach. Intell. 20, January 1998, pp [6] W.Zhao, R.Chellappa, N.Nandhakumar, Empirical performance analysis of linear discriminant classifiers, Computer Vision and Pattern Recognition, Proceedings. IEEE Computer Society Conference on, 1998, pp [7] B. Moghaddam, A.P. Pentland, Probabilistic visual learning for object detection", IEEE Trans. Pattern Analysis and Machine Intell., vol.19, July [8] W.Zhao, R.Chellappa, A.Rosenfeld, P.J.Phillips, "Face Recognition : A Literature Survey", TR 2000 [9] Lades, M.; Vorbruggen, J.C.; Buhmann, J.; Lange, J.; von der Malsburg, C.; Wurtz, R.P.; Konen, W., Distortion invariant object recognition in the dynamic link architecture, IEEE Transactions on Computers, Vol. 42 Issue 3, March 1993, pp [10] Wiskott, L.; Fellous, J.-M.; Kuiger, N.; von der Malsburg, C., Face recognition by elastic bunch graph matching, Pattern Analysis and Machine Intelligence, IEEE Transactions on, Volume: 19 Issue: 7, July 1997, pp [11] C. Garcia and G. Tziritas, "Face Detection using Quantized Skin Color Regions Merging and Wavelet Packet Analysis", IEEE Trans. Multimedia, vol.1, no.3, Sep [12] J. Karlekar, U.B. Desai, "Finding Faces in Color Images using Wavelet transform", Image Analysis and Processing, Int'l, Conf [13] E. Hjelmas, B.K.Low, "Face Detection : A Survey", Computer Vision and Image Understanding no.83, pp , 2001 [14] A. Pentland, B. Moghaddam, and T. Starner, View-Based and Modular Eigenspaces for Face Recognition, Proc. IEEE Conf. Computer Vision and Pattern Recognition, Seattle, June [15] Todd K. Moon, "Expectation-Maximization Algorithm", IEEE Signal Processing Magazine, Nov [16] Francois Fleuret Donald Geman, "Coarse-to-Fine Face Detection", Avant-Projet IMEDIA, June 2000