Real Time Detection and Tracking of Mouth Region of Single Human Face

Transcription

1 2015 Third International Conference on Artificial Intelligence, Modelling and Simulation Real Time Detection and Tracking of Mouth Region of Single Human Face Anitha C Department of Electronics and Engineering B M Sreenivasaiah College of Engineering, Bangalore / The National Institute of Engineering, Mysore, India anitha.bmsce@gmail.com M K Venkatesha Department of Electronics and Communication R N Shetty Institute of Technology Bangalore, India B Suryanarayana Adiga Research and Development Tata Consultancy Services Ltd., Bangalore, India Abstract - For any real-time application, detection and tracking of features becomes very important. The detection and tracking algorithms have to be very robust and efficient with least or zero false positives and false negatives. We use a novel combination for detection and tracking purpose. In this paper we propose a robust mouth region extraction and tracking algorithm that works in real-time. The region of interest for our application requires the face and mouth regions. We propose a novel technique for extracting the mouth region automatically. The proposed technique detects and tracks the mouth in either closed or opened state. We use the color components for skin tone and lips extraction. Keywords - Color images; skin-tone detection; region of interest; feature extraction; feature tracking I. INTRODUCTION The features such as face and mouth region extraction is important in our application which involves tracking the driver s behavior while driving. The purpose is to alert the driver if he/she is not completely alert, feeling drowsy, yawning a lot or not completely focused. There are several ways through which a driver gets distracted which includes repeated talking to the co-passengers in the car, not always looking in front, repeated yawning and so on. This analysis requires continuous detection and tracking of the face and mouth regions. Automatic tracking of mouth region involving single face consists of two steps: Robust detection of human face is done using Viola-Jones face detection algorithm and tracking this facial region is done using Mean Shift algorithm. The output of tracking algorithm is a bounding box covering the facial region and when this region is subjected to detection of mouth using skin-tone detection algorithm it works well with few number of false positives and false negatives because the algorithm has to work on small region which to a great extent contains skin region associated with face. Probability of non-skin regions appearing as skin region is very less. The first step of face detection using Viola-Jones algorithm is essential to start the process automatically and also to recover the process in case of occlusions. Our skin tone detection algorithm is purely color space based and does not use any probabilistic approach. In this paper we mainly focus on mouth tracking algorithm based on skin- tone detection. The most commonly used color spaces are RGB, YCbCr, and HIS. However other color spaces [6] such as HSV, YUV and YIQ are also considered. Most algorithms use mixed color spaces for better performance at the cost of enhanced complexity. Use of RGB color space is generally not preferred because the components R, G and B are dependent on illumination. And hence is not suitable for illumination-varying face detection. However it can be used in combination with other color spaces. YCbCr color space is highly preferred because its Y (luminance) component can be used to eliminate any variations related to illumination (Light compensation). The Cb and Cr components are exclusively used for skin segmentation. Skin tone detection algorithm is based on heuristics. II. RELATED WORK We have considered totally 11 algorithms from literature for Skin Tone Detection purposes. All the algorithms are in transformed color spaces. Some algorithms use 2 or 3 combinations of colored spaces for better performance. The last algorithm considered is due to Peer [10] and exclusively works in RGB color space. But it is sensitive to light intensity variations. However since the algorithm is computationally less intensive, it is very popular and is in vogue even today. Algorithm 1 considers only the YCbCr color space. Only the Cr component is considered for the heuristics definition (10<Cr) & (Cr>45) to identify the skin tone. Algorithm 2 considers the YCbCr and HSI color spaces. The heuristics in this algorithm are (77<Cb & Cb<127), (133< Cr & Cr< 177), (0<H & H<0.09) and (0.9<H & H<1). Skin Color Pixels are identified if (Heuristics1 && Heuristics2) && (Heuristics3 Heuristics4) are true. The approach considered is pixel based. Algorithm 3 is also a pixel based approach. This algorithm operates only in the HSV color space. The heuristics considered are (0< H & H< 50), (0.20<S & S<0.68) and (0.35<S & S<1). The Skin Color Pixels are identified if (Heuristics1 && Heuristics2 && Heuristics3) are true /15 $ IEEE DOI /AIMS

2 Algorithm 4 uses the YCbCr color space in a pixel based approach. The heuristics considered here are (85<Cb & Cb<135), (135<Cr & Cr <180) and (Y> 80). The Skin Color Pixels are present if (Heuristics1 && Heuristics2 && Heuristics3) are true. Algorithm 5 uses YIQ color space. The heuristics considered are (44<Y & Y<223) and (0< I & I< 64). The Skin Color Pixels are identified if (Heuristics1 && Heuristics2) are true. Algorithm 6 is a pixel-based approach which uses both YIQ and HSV color spaces for skin tone detection. The heuristics are (20<I & I<90), (0.20< S & S <0.75), (0< H & H <2.5) and (V>0.35). If (Heuristics1 && Heuristics2 && Heuristics3 && Heuristics4) are true, skin tone is detected. Algorithm 7 is also a pixel-based approach which uses the YUV color space for skin tone detection with four heuristics (65<Y & Y<170), (85< U & U <140), (85< V) and (V <160). The Skin Color Pixels are detected if (Heuristics1 && Heuristics2 && Heuristics3 && Heuristics4) are true. Algorithm 8 uses YU V color space. Both U and V components are considered for the heuristics definition. If (30<Ch & Ch<220) is true, skin color is detected. Where, Ch = (U 2+V2)1/2. Algorithm 9 uses YUV and RGB color spaces. A number of heuristics are defined to detect the skin color in the input image. They include, (80<U & U<130), (136<V &V<200), (V>U), (R>80), (G>30), (B>15) and (abs(r-g) >15). Skin Color Pixels are detected if (Heuristics1 && Heuristics2 && Heuristics3 && Heuristics4 && Heuristics5 && Heuristics6 && Heuristics7) are true. Algorithm 10 uses YCbCr color space for skin tone detection. The heuristics defined are (10<Cr) and (Cr<45). Skin Color Pixels are found if (Heuristics1 && Heuristics2) are true. Algorithm 11 [10] uses RGB color space. The heuristics defined are (R > 95 &G > 40 & B > 20), Max(R, G, B) Min(R, G, B) > 15, Abs(R-G) >15 and (R>G & R >B). The Skin Color Pixels are detected if (Heuristics1 && Heuristics2 && Heuristics 3 && Heuristics 4) are true. The output of all the algorithms gives a binary image white ( 1 ) for skin pixels and black ( 0 ) for non-skin pixels. III. PROPOSED SYSTEM We propose two algorithms, for extracting the regions of interest associated with face and mouth. The algorithms are part of the driver s behavior monitoring system. The first step in the monitoring system is the detection and tracking of the driver s face while driving. We use a novel algorithm [5] which is a combination of Voila-Jones face detection [1] and the Mean Shift tracking algorithms [2]. After tracking the face, we need to extract the regions associated with face and mouth. The algorithms proposed in this section are computationally very simple, yet robust. The efficiency of these algorithms were evaluated using the YAWDD, the Yawn Data set [3]. The first subsection describes the extraction of face region and gives a comparison with the algorithms discussed in the previous section. The second subsection proposes the algorithm to extract the mouth region. A. ROI Associated with Face ROI associated with face is a binary image, white pixels ( 1 ) representing region interior to the face and black pixels ( 0 ) representing region exterior to the face. The size of the binary image is the size of the rectangular boundary (bounding box) covering the facial region appearing on the current tracked video sequence. The contour dividing the interior and exterior of facial region is the Face boundary. From each tracked video sequence (fig.1(b)), we extract image information covering the interior of the bounding box (generally rectangular) and subject it to further processing with a view to get ROI associated with face (fig(1(c)). The image sub region corresponding this ROI is used to do gesture analysis including yawning. A.1 Observation We make the following important observation. The interior of the bounding box if it covers facial region (it does this with high probability) contains to a large extent skin region associated with face and also sometimes other tiny skin regions related to non-facial parts of human body. We use the word tiny because non facial part of human body cannot occupy a large region comparable to the size of face inside the bounding box due to its relatively small size. This observation makes our algorithm distinct from other face detection algorithms discussed in section 2. Our application needs tracking of single human face and well known algorithms which are efficient and computationally not too expensive like Lucas Kanade [7] and Mean shift [2] are available for this purpose. Our task mainly involves obtaining ROI associated with face starting with application of Skin Tone Detection algorithm followed by Morphological and Connected Component Labeling algorithms on the image sub region inside the bounding box. We used Image Processing Toolbox provided by MATLAB to develop the prototype model. The output of skin tone detection algorithm is a binary image. White patches (pixels with value 1 ) represent skin regions where as black patches (pixel value 0 ) represent non skin regions. Under ideal conditions the output contains black blobs only in regions where facial features like eyes and mouth (when opened) exist. 298

3 Figure 1: (a) Typical input frame, (b) Tracked region, within rectangle, (c) Region of interest associated with face (to be extracted). We clear up these anomalies by applying Morphological operations like Dilation, Erosion and Flooding on the output of skin tone detection algorithm. From the cleaned up binary image ROI associated with the face is obtained by applying Connected Component Labeling technique. We describe these processes in detail in the following sections. We evaluated the above described algorithms (in section II) on the Yawning database the results are as shown in the figure 2. A.2 Morphological Operations used (a) Input Frame (b) Algorithm 1 (c) Algorithm 2 (d) Algorithm 3 (e) Algorithm 4 In this paper we mainly consider two operations flood-fill and dilation in order to clean up the binary image obtained after skin detection [9]. The motivation is to remove facial features. Sometimes especially eye regions may appear as backwater like regions instead of lake regions (blobs). Operation flood filling fails to remove backwater like regions. This is the reason we need to apply dilation operation after flood filling operation. A.2.1 Connected Component Labeling A connected component in a binary image is a set of pixels that form a connected group (figure 3). For example, the binary image below has three connected components. In our case the set of all disjoint skin and skin like regions (white patches) in the binary image form connected groups. (f) Algorithm 5 (g) Algorithm 6 (h) Algorithm 7 (i) Algorithm 8 Figure 3: Connected components illustration (j) Algorithm 9 Figure 2: Input frame and the corresponding ROI output of some of the algorithms discussed in section II. However because of imperfections in the skin tone detection algorithms, some black blobs may appear in skin regions pretending like non-skin regions. Connected component labeling is the process of identifying the connected components in an image and assigning each one a unique label [8], like figure 4. It is to be noted that in our case the connected component having largest pixel membership happens to be ROI associated with face (this claim is with high probability). So our task is to extract connected components from the binary image consisting of skin and skin-like regions and then identify the one with largest pixel membership. 299

4 The output of the algorithm is group of labelled connected components,. MATLAB built-in function does the operation of Connected Component Labeling on binary image. Value of implies Figure 4: Connected components labeling A.2.2 Extraction of a Connected Component in a Binary Image Let BW be a binary image of size (h x w). The set of 1 s in BW contain a group of connected components. We have to extract individual components and label them. We start with a pixel of BW say, BW(i,j)=1. Let Y represent the connected component associated with BW(i,j). Then the following iterative procedure yields all elements of Y. Here B is a suitable Structural Element (SE). Step 1: Define a binary matrix such that and rest of the pixels are 0. Step 2: Compute Define Step 3: Compute and Step 4: Repeat Steps 1, 2 and 3 till. Then A.2.3 Connected Component Labeling of a Binary Image Now we give a procedure for extracting and labeling group of connected components in a binary image BW. Step 1: Create a Marker Image,. Step 2: Scan row-wise and stop when a pixel is encountered. Step 3: Extract the connected component to which belongs using the algorithm of previous section. Label the member pixels of in as. Mark pixels of that correspond to the pixels of in as. Step 4: Repeat steps 1, 2 and 3 till becomes all zero matrix. At the end of each iteration, labeling number is incremented by 1. and implies. is the group of labelled connected components and A.3 Finding ROI Associated with Face is the group size. Step 1: Get Connected Components of Step 2: Get Pixel Membership of individual components Step 3: Identify the largest Connected Component ( ). Step 4: Find the and coordinates of (Candidate). Get the ROI of Face. Step 5: Make pixels of ROI of Face corresponding to (Candidate) equal to. ROI of Face is a binary image where pixels with value represent facial region (interior). B. ROI Associated With Mouth In this section we discuss the localization of Regions of Interest (ROIs) associated with prominent facial feature namely Lips. The input to the localization algorithm is the image of face enclosed inside the ROI associated with face obtained in the previous section. In a face there is one mouth comprising two lips. Lip detection means we have to identify both the lips. The most popular technique of lip detection is through snake contour. This technique finds a snake on the outline of the lips [11], [12]. The snake contour technique is computationally complex for real-time applications. Hence, we require an algorithm that is computationally less complex and produces faster detection rates. The algorithms used in the paper are pixel based and work in YC b C r [4] or RGB color space. We develop mathematical models for LIPMAP based on some characteristic features of color components in mouth region. These are based on key observations made by several researchers through extensive experiments. We highlight two such models in vogue one based on YC b C r and the other one based on RGB. The second algorithm is computationally fast but performance wise is slightly inferior to the first. This is due the fact that R, G, and B components are sensitive to light intensity variations. 300

5 B.1. Observations 1. Value of C r is greater than C b near mouth region. 2. Mouth has relatively low response to C r /C b 3. Mouth has high response for C r 2 4. Lip color contains a strong red (RED) component and weak green (G) and blue (B) components (in normalized domain) Based on these observations the mathematical model for LIPMAP in YC b C r space is formulated as: region ) over the image Similarly in RGB (rgb) space, LIPMAP is formulated as; We use the following steps to localize the lips (mouth): 1. Get the Face region mapped inside the ROI associated with face. 2. Apply one of the models stated above on the Face region to get LIPMAP. 3. Apply morphological operation Dilation on LIPMAP. We use the MATLAB structure to get the Structural Element and to perform Dilation operation. 4. Apply a mask of size with weight 1 on the dilated of 3. The is divided into blocks and every pixel inside the box is filled with the average of the values of the 25 pixels inside the box. The output is low-pass filtered. 5. Apply THRESHOLDING on the masked of 4 to get binary image consisting of Mouth region (two lips). This region is called associated with Lips. We arrange the pixels of masked in ascending order, according to their values and retain only top 3% of them. We construct a binary image of size of face where pixels corresponding to those retained pixels in masked are made 1 and rest of them are retained as zeros. 6. Get connected component labeling of associated with lips to identify mouth. We start with the first image sequence. We go through the steps 1 to 6 and get the connected components of the ROI with labeling. Connected component algorithm gives information about the centroids of the components along with a wide range of other useful information. Using the information about centroids we select the component that occupies the lowermost region in the ROI associated with lips. This means the y- axis value of the centroid of the selected component is the maximum of the y-axis values of the centroids of all components (Origin is on the top left corner). We keep the centroid information (namely y-axis and x-axis values) of the selected component as reference for tracking the ROI associated with lips in the next image sequence. If only one component is there then it implies the lips are closed. In case the mouth is open there will be two components and lower lip will be selected as reference. Worst case the component describing chin region will be selected as reference. For the second image sequence after step 6 we compute the Euclidian distances of centroids of all components with respect to the reference centroid obtained in the first frame. The component having the minimum Euclidian distance with respected to the reference (centroidwise) is logically selected as ROI associated with lips. For the third image the centroid of the selected component in the second image sequence is selected as the reference. This process continues for subsequent image sequences. We elaborate on step 6. Under ideal conditions i.e. when the mathematical modeling works properly we should get either one or two connected components. If there is only one component it implies lips are closed. Otherwise it implies lips are open. The second case may occur while the person is speaking or smiling or yawning. Under inappropriate lighting condition it may so happen that lip detection algorithm may output two more components one associated with nose and the other associated with chin. Thus legitimate number of components is maximum of 4 all related to mouth region. Sometimes even region associated one of the ears may appear as a connected component if the face is heavily turned. So depending on the application (mainly from gesture analysis point of view) identifying the mouth may mean detection of two lips or detection of lips along with chin and nose. The entire process is employed in the algorithm liptracking with functions lipcbcr1 and lipcbcr2. The first function is used for determining components associated with lips and determining the relevant component in the first image sequence while the second function is used for tracking relevant components in the 2 nd, 3 rd etc. image sequences. Algorithm lipcbcr1: This algorithm is used for the detection of the lip region in the first image sequence based on the heuristic that it is the lower most connected component of LIPBIN in the face region. Step 1: Conversion from unsigned integer to double 301

6 Step 2: Getting size of the image and R, G, B components Step 3: RGB TO YC b C r Conversion Step 4: Getting LIPMAP Step 5: Morphological operations Step 6: Masking Step 7: Thresholding Step 8: Getting Binary image of lip region Step 9: Connected component labeling of LIPBIN (b) (c) (d) Algorithm lipcbcr2: This algorithm is used for the detection of the lip region from the second image sequence onwards based on the heuristic that Euclidian distance of the selected component in the LIPBIN of the present image sequence is nearest to the centroid of the region selected in the previous image sequence. Step 1: Conversion from unsigned integer to double Step 2: Getting size of the image and R, G, B components Step 3: RGB TO YC b C r Conversion Step 4: Getting LIPMAP Step 5: Morphological operations Step 6: Masking Step 7: Thresholding Step 8: Getting Binary image of lip region Step 9: Connected component labeling of LIPBIN Step 10: Getting Centroids: Getting the Present as the centroid of the connected component nearest to the Previous Centroid. IV RESULTS The YAWDD The Yawn Database [3] was used for evaluating the proposed algorithms. Figure 5 below shows the step-wise outcome of the ROI associated with face algorithm. As seen in figure 5, the skin-tone or face region is extracted efficiently after enhancing the output of the skin-tone detection algorithm using morphological operations to remove the noise components. (a) (e) (f) (g) Figure 5: (a) Tracked image frame as input, (b) Input reduced to tracked region only, (c) Output of Skin tone detection, (d) After noise elimination, (e) After morphological operations, (f) Region of interest associated with face, (g) Face map of (f). The outcome of the lip detection algorithm s (lipcbcr1) important stages is shown in figure 6 for an example frame from the YAWDD database. (a) (b) (c) (d) (e) (f) Fig. 6: (a) Example input frame; (b) Output after step 2; (c) Output after step 3; (d) Output from step 4; (e) Possible lip regions; (f) Final lip region The concept of lip tracking takes a central role in detecting spontaneous gestures that we intend to apply on. The YAWDD yawning database was used for evaluation purpose. Some sequences from the yawn database with ROI associated with mouth detected and tracked are as shown in figure 7. (a) 302

7 of a driver such as talking, yawning, and smiling and so on. Most of these gestures are distractions while driving. Hence the algorithms involved in each stage of gesture detection have to work in real-time. The key parameter we have aimed to minimize is the time taken for computation. Another most important feature of this algorithm is it is very simple but robust at the same time. REFERENCES Fig. 7: First & Third Row: Original tracked sequence; Second & Fourth row: Corresponding Lip tracked outcome. V. CONCLUSION The proposed feature extraction technique for extracting the mouth region is a novel and efficient technique. This technique can be used in several applications that require detection and continuous tracking of mouth feature. In our application, the driver gesture detection, we need to continuously tracking the face and the mouth region. Our first step towards gesture detection was to detect and track the driver s face. We successfully could achieve this by the novel technique which combines the best aspects of Viola- Jones face detection algorithm and the Mean Shift object tracking algorithm. The novel, high speed face detection and tracking technique is very robust and is capable of tracking the driver s face even with occlusions and partial features. It can be referred in [5]. Once the face is tracked, we need to extract the face region. After the face region is extracted, the mouth region is extracted from it. The mouth feature s tracking helps us in detecting the common gestures [1] Viola P, Jones M, Snow D, Detecting pedestrians using patterns of motion and appearance, In IEEE International Conference on Computer Vision (ICCV) , [2] D. Comaniciu and P. Meer, Mean shift: A robust approach toward feature space analysis, IEEE Transaction Pattern Analysis and Machine Intelligence, 24(5), , [3] S Abtahi, M Omidyeganeh, S Shirmohammadi, B Hariri, YawDD: A Yawning Detection Dataset, in proceedings of ACM Multimedia Systems, Singapore, pg , March [4] Yu-Ting Pai, Shanq-Jang Ruan, Mon-Chau Shie, Yi-Chi Liu, A Simple and Accurate Color Face Detection Algorithm in Complex Background, In Proceedings of IEEE International Conference On Multimedia And Expo, [5] Anitha C, M K Venkatesha, B Suryanarayana Adiga, High Speed Face Detection and Tracking, in Proceedings of 40th IRF International Conference, [6] Rafael C Gonzalez, Richard E Woods, Color Image Processing, Digital Image Processing, 3rd Edition, PHI, pg , [7] Lucas B D, Kanade T, An iterative image registration technique with an application to stereovision, In Proceedings of International Joint Conference on Artificial Intelligence, [8] Rafael C Gonzalez, Richard E Woods, Object Recognition, Digital Image Processing, 3rd Edition, PHI, pg , [9] Rafael C Gonzalez, Richard E Woods, Morphological Image Processing, Digital Image Processing, 3rd Edition, PHI, pg , [10] Jure Kovac, Peter Peer and Franc Solina, Human Skin Color Clustering for Face Detection, in Proceedings of IEEE EuroCon 2003, Vol. 2, [11] M Barnard, E Holden and R Ownes, Lip Tracking using pattern matching snakes, in Proceedings of 5th Asian Conference on Computer Vision 2002, Melborne, Australia. [12] G Chiou and J Hwang, Lipreading from Color Video, IEEE Transactions on Image Processing, Vol.6, no. 8, pg ,