A REAL-TIME FACIAL FEATURE BASED HEAD TRACKER

Transcription

1 A REAL-TIME FACIAL FEATURE BASED HEAD TRACKER Jari Hannuksela, Janne Heikkilä and Matti Pietikäinen {jari.hannuksela, jth, Machine Vision Group, Infotech Oulu P.O. Box 45, FIN-914 University of Oulu, Finland ABSTRACT This paper presents a fast and efficient head tracking approach. Skin detection, gray-scale morphology and a geometrical face model are applied to detect roughly the face region and extract facial features automatically. A novel Kalman filtering framework is utilized for tracking and estimation of the 3-D pose of the moving head. An application to cursor control on a computer display is presented. Experiments with real image sequences show that the system is able to extract and track facial features reliably. Pose estimation accuracy was tested with synthetic data and good preliminary results were obtained. The real-time performance achieved indicates that the proposed system can be applied in platforms where computational resources are limited. 1. INTRODUCTION Proactive computing technology aims to the development of more intelligent and more friendly human-computer interaction (HCI) methods. In the future computers will interact with people in a more natural way. To achieve this kind of advanced interaction, the system should understand human behaviour and respond appropriately for example by observing one s facial expressions and gestures. Machine vision provides an excellent way of realizing such an intelligent human-computer interface that enables a user to control a computer without any physical contact with devices such as keyboards, mice and displays. The estimation of head position and orientation (pose) allows recognition of simple gestures such as nodding and head shaking. The pose also makes it possible to estimate roughly the human gaze direction and to determine which part of the screen the user is looking at. Instead of using a mouse, the user can control the desktop of the computer or a smart phone and activate the desired event using head movements. A number of real-time head tracking and pose estimation methods have been proposed such as [1, 2, 3, 4]. On the other hand, various methods have been proposed for facial feature extraction [5], which is a crucial stage in a feature based tracker. However, the current real-time methods need a lot of computational resources or special hardware, and certain functionalities such as automatic initialization are often missing. Also, there is still a need to improve the speed, robustness and accuracy of such methods. Our purpose is to develop a more flexible approach which can be applied in resource limited embedded systems such as smart phones but still retaining good pose estimation accuracy. We propose a fast real-time head tracker based on extraction of facial features and continuous estimation of head pose. The implementation consists of initialization and tracking stages. In the initialization facial features are extracted automatically whenever a human face is detected. For proper face candidate detection, a near-frontal view is needed. After success in the first stage, tracking is started. The head pose is initialized by a perspective three point (P3P) algorithm. During tracking, a Kalman filter estimates the head pose utilizing a simple head model containing the eyes and the mouth. The approximate gaze direction is also estimated from the orientation of the head. To demonstrate the practicality of the approach, gaze direction information is applied to cursor control on a computer display. 2. FACIAL FEATURE EXTRACTION Facial features are extracted in three steps. First, the facelike regions are found by skin color analysis and then the shapes of the regions are verified. Next, the facial feature candidates are searched for within those regions. Finally, the candidates are evaluated to find features of interest. a) Input image b) Skin detection result Figure 1: Skin detection. 267

2 2.1. Face-like region detection Color information has been widely used in face detection and tracking, because processing of color is faster than of any other features. We detect possible face-like regions in an input image (Fig. 1a) utilizing skin detection proposed by Martinkauppi et al. [6]. They have found Normalized Color Coordinates (NCC) combined with the skin locus most appropriate for skin detection under varying illumination. First, the image presented in the RGB color space is converted to the NCC space r, g and b. Because r+g+b = 1, we only use two chromaticities r and b for detection in a histogram where the skin color occupies a small cluster. The cluster is found by training with facial images in different illuminations. If r and b of a pixel fall into the area of the skin locus, the pixel belongs to skin. In practise detection is implemented using a look-up table. The result is enhanced using a morphological binary closing with a mask size of 3 x 3 pixels (Fig. 1b). After the skin regions have been detected, they are possible faces. To distinguish faces and other skin-like areas, connected component analysis is performed, and only large components of the skin color are kept. The size of the bounding rectangle for the skin region should be at least 1% of the image size Facial feature candidate extraction Once a possible face is detected, the features are searched for within the facial region. Based on the observation that facial features are usually darker than their surroundings, we use a morphological valley detector. The morphological operators are well suited for rough feature extraction due to their fast implementation and robust nature. For example, morphology has been utilized for eye region detection [7]. We apply an operation called black top-hat to gray scale image. At first, a closing operation is performed and then the original image is subtracted from the result. The valley image V is defined as V (x, y) = f(x, y) B f(x, y), (1) where f(x, y) is the image tone at a point (x, y) and B is a small structuring element. The size of B is 3 x 3 pixels. A pixel at (x, y) belongs to a feature candidate if the following equations are true f(x, y) < T L and V (x, y) > T V, (2) where T L and T V are pre-defined thresholds. The detection result can be noisy and there exists some isolated pixels. Therefore we improve the result by utilizing morphological binary closing with a mask size of 3 x 3 pixels. Then, connected component analysis is performed and centers of mass of the objects are used as feature. Figure 2: Facial feature extraction Best constellation search After the possible facial features are detected, a similar method as proposed by Jeng et al. [8] is applied to evaluate feature constellations. Based on experimental evaluation we have modified their face model and the search procedure to be more appropriate for our purposes. The original model does not match very well for western people. Also, two nostrils are used instead of the nose. We propose a geometrical face model including eyes, eyebrows, nostrils and mouth as shown in Fig. 3. The line passing through the centers of the eyes is called the base line. The d mouth, d nostril and d eyebrow indicate the distances from features to the base line. Two features form a c 1 c 2 deyebrow d nostrill baseline ax + by + c = dmouth Figure 3: Geometry of the face model. possible eye pair if they locate at the upper half, and the distance D between them is under 8% of the width of the face bounding box. Also, the base line angle should be under 3 degrees. These conditions reduce significantly the number of potential eyes. Based on D, other reference distances D nostril =.7D, D eyebrow =.35D and D mouth = 1.1D are defined. The model distances are found from measurements with several real faces. For 268

3 each eye pair, the other facial features are searched for. The possible mouth for an eye pair is searched from a region that is located within the distance of 1.1D from the base line. If that region contains a feature which locates between the boundary lines c 1 and c 2, it is treated as a mouth for this possible eye pair. In order to rank many candidates, a special evaluation function is utilized for each facial feature: camera Y C v X C image (u, v) Z C u X D Z H N H P 2 P 3 X H P 1 Y H head E feature = exp( 1( d feature D feature ) 2 ), (3) D P D display where feature = {mouth, nostril, eyebrow}. The evaluation function for the eye pair is Z D Y D E eyepair = exp( 1( D.4BB width ) 2 ), (4) D where BB width is the width of the face bounding box. The total evaluation value is a weighted sum of the values for each facial feature. The weights for the eye pair, mouth, nostrils and eyebrows are.4,.3,.1 and.5, respectively. The weight is based on the importance of a given feature for face detetction. The eyes are considered the most important features. The constellation which has the largest evaluation value, is assumed to be a face. Fig. 2 shows an example of facial feature extraction where results for valley detection is on the left, candidates on the middle and the best constellation on the right. 3. HEAD TRACKING The pose of a human head can be estimated by extracting the facial feature correspondences between successive images. However, the measurements are usually affected by noise that may produce large estimation uncertainties. Visual tracking of the facial features in a video sequence with a proper filtering technique can improve the result significantly. We use Kalman filtering to estimate the 3-D pose of a moving head and to track facial features Head pose and motion model In order to track the head, we propose a simple rigid head model including the mouth (P 1 ) and the eyes (P 2 and P 3 ) as shown in Fig. 4. The 3-D pose of the head can be solved using only these three points if the corresponding 2-D image observations are known. The center of the head coordinate system is set to the center of gravity of these three points. The in the model are defined as follows: P 1 = [4α,, ] T, P 2 = [ 2α, 3, ] T, P 3 = [ 2α, 3, ] T, Figure 4: Coordinate systems involved. where the scaling factor α = d mouth /D. We model the head pose and motion using 1st order dynamics which allows prediction and filtering of the pose from a sequence of images. The head is treated as a rigid object that moves with constant velocity. Accelerations are modelled as system noise because the head motion is assumed to be smooth. The motion model is x k+1 = Φ k x k + Γ k ε k, (5) where the parameters to be estimated are presented by the state vector x k at time instant k, and Φ k is the state transition matrix. The state transition matrix relates the state at time instant k to the state at time instant k + 1. Γ k ε k models the uncertainty of the motion model. The process noise ε k is assumed to be Gaussian distributed with an expected value E{ε k } = and the covariance matrix Q k = E{ε k ε T k }. Γ k is the process noise transition matrix. The state vector x k consists of the position (x k, y k, z k ) and orientation (ω k, ϕ k, κ k ) of the head with respect to the camera and the corresponding velocities at time instant k. It is defined as follows x k = [x k, x k, y k, y k, z k, z k, ω k, ω k, ϕ k, ϕ k, κ k, κ k ] T. The initial 3-D pose of the head is determined by using the perspective-three-point (P3P) algorithm [9]. However, the solution is not unique and there exist up to four possible solutions for the pose. In order to evaluate all possible poses, we solve the system in closed-form [1]. We choose the pose where the z-axis of the head coordinate system intersects nearest to the center of the display. In the beginning, the velocities are set to zero. The time step between two successive images is normalized to 1. We approximate the variances of the process noise from the maximum accelerations allowed. 269

4 3.2. Measurements We use a perspective camera model to relate the object P i to image (u i, v i ) of three facial features during tracking as shown in Fig. 4. The camera is assumed to be pre-calibrated. The 3-D pose of the head is computed using these points. However, by using only three points the solution is not unique as mentioned in previous section. Our idea is to evaluate all the possible solutions and choose the one nearest to the previous pose to be the measurement at a particular time instant. Fig. 5 shows a block diagram of the tracker. The measurement model to map the state vector to the pose measurements is defined as z k = Hx k + η k, (6) where H is the observation matrix and it is defined as 1 1 H = The measurement noise η k models uncertainty in the pose measurements and it is assumed to be Gaussian distributed with an expected value E{η k } = and the covariance matrix R k = E{η k η T k }. The pose noise covariance R k can be derived from the image measurement noise that is assumed to be Gaussian distributed with zero-mean and a predefined variance. Because we use a nonlinear perspective camera model linearization is needed. Linearization is performed using the partial derivatives of the measurements with respect to the state variables of x k to obtain the Jacobian matrix J. R k is obtained as follows R k = J 1 C k J T, (7) where C k is the image measurement noise covariance matrix. The actual image measurement is performed in a small region around the predicted location of the feature, which decreases the computational load. The size for the region is adjusted dynamically by projecting the estimation uncertainty to the image. The features are extracted by utilizing valley detection similarly as in the initialization stage (Section 2.2). If there are several components within a search region, then the nearest one to the predicted position is chosen Tracker The Kalman filter algorithm estimates the pose recursively repeating two stages: prediction and correction [11]. In the first stage, the pose and locations of the features at Figure 5: Block diagram of the tracker. the next time instant are predicted based on the previous pose estimate and the dynamical model. In the correction stage the predicted pose is adjusted by using the measured pose. The feature measurements can occasionally cause errors that make tracking to fail. If the feature position is not within the uncertainty limits derived from the error covariance matrix or feature is lost then the predicted pose is used instead of the measurement. The error cannot exist longer than three frames or the initialization is started again. 4. EXPERIMENTAL RESULTS We evaluated the system s ability to track the facial features with several real image sequences containing large head motions and pose changes. A digital video camera was placed above the display of the computer. The length of recorded videos was 2 frames. The user in the image tries to control the cursor on the computer display. The pixel resolution was 32 x 24 and the frame rate 15 fps. Because the ground truth was not known, we assessed the operation visually. The tracking was successful with most of the sequences. Fig. 6 shows an example result of the feature locations during tracking. The tracked features; the mouth and the eyes are marked with red dots. Although, the sequence contains a man with a beard, the system still works properly. During testing we met major problems with only one sequence that contains a person with glasses. Certain types of spectacle rims and bad illumination can cause the system to fail in tracking. We evaluated the theoretical pose estimation accuracy with simulated data, due to the lack of ground truth for the head pose. The data of 1 frames generated contains head poses corresponding to a real situation where the user is trying to control the cursor on the monitor display. The 27

5 frame #1 frame #3 frame #5 frame #7 frame #9 frame #11 frame #13 frame #15 frame #17 frame #19 Figure 6: Facial feature tracking example. initial pose was [19.11, 8.65, 685.9, 2.34,.2, 1.59]. translation [mm] 5 translation in xaxi s 5 translation in yaxi s 6 translation [mm] translation [mm] 4 2 translation in zaxi s frame # rotation [rad] rotation [rad] rotation about xaxi s 2. 4 rotation about yaxi s rotation about zaxi s 1.8 rotation [rad] frame # Figure 7: Head pose estimation experiment. The focal length was f = 15 mm and the conversion factor from mm to pixels was 5. The measurements are generated by projecting the head model points to the image plane using the ground truth pose. We added random noise with a variance of 4 pixels to the measurements in order to simulate the true error. Fig. 7 shows a sample result. The ground truth is drawn with a solid line and the estimated pose with a dashed line. It can be clearly seen that the error in the z-translation is biggest. The obvious reason is that the depth information is more uncertain due to the perspective camera. In order to get more reliable results the average of 1 simulations was measured. The root mean square errors (RMS) obtained for the translations along x-, y-, and z-axes directions and rotations about x-, y-, and z-axes were 1.61 mm, 1.3 mm, 6.23 mm,.8 rad,.4 rad and.3 rad, respectively. The system was implemented using C++ to validate the real-time performance. The processing speed was measured on a PC with a 2.2 GHz CPU and 512 Mb memory. The times are the average for of 1 executions in typical situations. The total time for initialization was about 1 ms and for tracking 2 ms per frame. This provides theoretical maximum frame rate of 5 fps in tracking. However, due to the limitations of the video acquisition, we used a frame rate of 15 fps. It should be remembered that the processing speed also depends on the face sizes and the features. Based on the experiments, the system clearly fulfills the real-time demands. 5. APPLICATION: USER INTERFACE CONTROL The proposed approach was applied to cursor control on a computer display. The rough gaze direction is estimated from the orientation of the head. The direction of the gaze is the normal defined by three facial features extracted: the mouth (P 1 ) and the centers of the eyes (P 2 and P 3 ) as shown in Fig. 4. In order to solve the gaze position in the display, the gaze vector has to be transformed from the head coordinate system to the display coordinate system. First, the gaze vector is transformed to the camera coordinate system using the estimated 3-D pose. After that, transformation to display is performed. It is assumed that the position and orientation of the display in the camera coordinate system are known in advance. Then, the intersection of the gaze vector with the display can be solved. After the position P D on the display is obtained, the user can control the cursor in the graphical user interface. The correct operation of our gaze direction estimation method was verified with many real image sequences. In these sequences a similar setup has been used than in facial feature tracking tests. In these experiments the user makes some dedicated movements. Fig. 8 shows the trajectory when the user alternates his attention from the right side to the left side on the computer display. There are some- 271

6 times noisy feature measurements that cause errorneous gaze position estimates. In this case more filtering is actually needed. Experiments show that the system is capable of detecting gaze position on the display. The variation in the trajectory is quite big but it should be remembered that we cannot say what is the exact gaze position. However, the movement seems to be roughly correct Figure 8: An example trajectory obtained with real image sequence. 6. CONCLUSIONS We have presented a new facial feature based head tracking approach using skin detection, gray-scale morphology and a geometrical face model. The 3-D pose of the moving head is estimated using Kalman filtering. The advantage of the system is the low computational cost that makes it usable for platforms such as mobile devices where computational resources are limited. Although very good premilinary pose estimation results were achieved, it should be remembered that the accuracy values obtained are more or less theoretical, which apply under certain constraints. The main factors causing problems for the system are users with glasses, occlusions, and shadows. In the future work, the feature extraction methods will be further investigated in order to find more stable features, which do not necessarily have to be predefined facial features. 7. ACKNOWLEDGEMENTS The financial support of the Academy of Finland and the National Technology Agency of Finland is gratefully acknowledged. 8. REFERENCES [1] A. Azarbayejani, T. Starner, B. Horowitz, and A. Pentland, Visually controlled graphics, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp , June [2] M. La Cascia, S. Sclaroff, and V. Athitsos, Fast, reliable head tracking under varying illumination: An approach based on registration of texture-mapped 3d models, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 4, pp , Apr. 2. [3] T. S. Jebara and A. Pentland, Parametrized structure from motion for 3d adaptive feedback tracking of faces, in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, June 1997, pp [4] A. Gee and R. Cipolla, Fast visual tracking by temporal consensus, Image and Vision Computing, vol. 14, pp , [5] M. H. Yang, D. Kriegman, and N. Ahuja, Detecting faces in images: A survey, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 24, no. 1, pp , Jan. 22. [6] B. Martinkauppi, M. Soriano, and M. Pietikäinen, Detection of skin color under changing illumination: A comparative study, in Proc. of the 12th International Conf. on Image Analysis and Processing, Sept. 23, pp , Mantova, Italy. [7] K. W. Wong, K. M. Lam, and W. C. Siu, A robust scheme for live detection of human faces in color images, Signal Processing: Image Communication, vol. 18, no. 2, pp , Feb. 23. [8] S. H. Jeng, H. Y. M. Liao, C. C. Han, M. Y. Chern, and Y. T. Liu, Facial feature detection using geometrical face model: An efficient approach, Pattern Recognition, vol. 31, no. 3, pp , Mar [9] M. A. Fischler and R. C. Bolles, Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Comm. of ACM: Graphics and Image Processing, vol. 24, no. 6, pp , June [1] S. Linnainmaa, D. Harwood, and L. S. Davis, Pose determination of a three-dimensional object using triangle pairs, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 1, no. 5, pp , September [11] R. E. Kalman, A new approach to linear filtering and prediction problems, Trans. of the ASME- Journal of Basic Engineering, vol. Series D, no. 82, pp ,