A novel information fusion based FFT algorithm for a driver fatigue monitoring system

Transcription

1 Int. J. Modelling, Identification and Control, Vol. 10, No. 3-4, A novel information fusion based FFT algorithm for a driver fatigue monitoring system Qiufen Yang 1, Huosheng Hu 2 *, Weihua Gui 1 1 School of Software, Central South University, Changsha, Hunan, China yqf5569@sohu.com 2 School of Computer Science & Electronic Engineering, University of Essex Colchester CO43SQ, UK hhu@essex.ac.uk * The author for correspondence Abstract: This paper presents a novel FTT (Feature Triangle Tracking) algorithm to track the eyes, mouth and pose of a fatigue driver. The proposed method uses the motion information to localize the face region, and the face region is processed in YCrCb color space to determine the locations of the eyes and mouth. According to geometrical relationships of facial components and the varied pose of the driver, the system derives from the isosceles feature triangle and right feature triangle. The FTT Algorithm removes the confusing triangle from the tracking of the video sequence effectively. The experimental results show that the proposed algorithm precisely tracks the varied pose of a fatigue driver in real-time video frames. Keywords: driver fatigue; FFT; information fusion; feature triangle Biography notes: QiuFen Yang received her B.Eng. and M.Eng. degrees degree from Nanjing University, Jiangsu, China and National Defense Science and Technology University, ChangSha, Hunan,China in 1996 and 2004,respectively. She is currently pursuing her Ph.D.degree in the School of software, Central South University, Changsha, Hunan, China. Her current research interests are in the areas of data fusion and computer vision. yqf5569@sohu.com Huosheng Hu received his Ph.D. degree from Oxford University, UK.He is a Professorin Computer Science at the University of Essex,UK,, leading the Human Centred Robotics Group.His research interests include biologically inspired robotics, service robots,human robot interaction, evolutionary robotics, data fusion, artificial life,embedded systems, pervasive computing and RoboCup.He is also a Chartered Engineer,a senior member of IEEE and a member of IEE,AAAI,IAS, IASTED and ACM. hhu@essex.ac.uk Weihua Gui received the degree of the B.Eng.( Automatic Control Engineering) and the M.Eng. (Control Science and Engineering) from Central South University, Changsha, China in 1976 and 1981, respectively. From 1986 to 1988 he was a visiting scholar at Universität-GH-Duisburg, Germany. He has been a full professor in the School of Information Science & Engineering, Central South University, Changsha, China, since His main research interests are in modeling and optimal control of complex industrial process, distributed robust control, and fault diagnoses. Guiwh@mail.csu.edu.cn. Copyright 2008 Inderscience Enterprises Ltd.

2 Int. J. Modelling, Identification and Control, Vol. 10, No. 3-4, Introduction In the transportation industry, about 57% fatal automobile accidents are due to driver fatigue and/or sleepiness [1], which makes this an area of great socio-economic concern. Many research groups are working on the possibility of providing the car with sensors for monitoring the steering wheel, brakes, accelerator, lane keeping and so on [yang,2005]. Some repeated experiments have shown that among all driver performance and bio-behavioral measures tested, the percentage of eyelid closure over time (Perclos) reliably predicts the most widely recognized psychophysiological index of loss of alertness [Park,2005]. Many commercial and experimental sensors [Q.Ji,2002] currently use the Perclos measure to evaluate driver fatigue and many research efforts are centered on the effects of warning systems on driver performance [Raw,2005]. In any Perclos system, the first thing is to locate the drivers eyes. Many research projects on eye detections assume that either eye windows have been extracted or rough face regions have already been located [Y.Li,2001-Feng,2001]. The first assumption that a visual system for automotive applications has to solve is to model the eye occurrence on video sequences to detect driver status. The second assumption is that a visual system for automotive applications doesn t use the drivers action such as his/her pose and mouth. The prevailing method for detecting driver s fatigue involves the tracking of the head and eyes of a driver. In this paper, a novel face tracking algorithm is firstly proposed to detect and track eyes, mouth and pose from video sequence in real time, such as head nodding and shaking. It is based on the Chiunhsiun s method [Chiunhsiun,2001] for the face detection in an image, which uses both isosceles triangle and right triangle to detect the potential face region in a frame. This method is shown to be robust to track the eyes, mouth under changing head gesture and light conditions. Secondly, we propose an effective method to deal with the arbitration of confusing eye mouth triangles in a video sequence. The main contributions of this paper are: (i) the arbitration of confusing eye lip triangles; (ii) the novel method to track the driver s eyes, mouth and gesture; (iii) Simple description of feature triangles. The tracking system runs at about 15 frames per second on a PC with 3.0 GHz CPU. The experimental results show that the system tracks the face correctly about 98.0% of the frames in the image sequence. The rest of the paper is organized as follows. Section 2 presents some background of image preprocessing. In Section 3, rules for forming a feature triangle are described. Section 4 describes the method to speed up the searching for feature triangles. The results of tracking eyes, mouth and pose are given in Section 6 to show the performance. Finally, a brief conclusion and future extension are given in Section 7. 2 Image processing In our method, the YCrCb color space is obtained from RGB color space using the following equations. Y 0.299R 0.587G 0.114B Cr R - Y Cb B- Y (1) where R, G, and B are the red, green, and blue components of the color image, respectively[kass,1987]. The RGB and YCrCb representation of the face region is shown in Figure 1. Figure 1 RGB and YCrCb representation of the face region. (a) from left to right red, green and blue components of the face region; (b) Y, Cr and Cb components of the face region. Figure 2 The construction of threshold face image The Y, Cr, and Cb values are normalized to the range [0, 255].The eye regions have low intensity (Y), low red chrominance (Cr), and high blue chrominance (Cb), when compared to the forehead region of the face. Using this fact, the face region is threshold to obtain the threshold face image U, given by 255 if Y( i, j) 1 and Cr ( i, j) 2 U ( i, j) and Cb ( i, j 2) (2) 3 I( i, j) otherwise where 1, 2,and 3 are the average Y, Cr and Cb values of the pixels in the forehead region, respectively. The Copyright 2008 Inderscience Enterprises Ltd.

3 232 N. Lu, J. Wang forehead region is determined from w1 and (cx, cy), where w is the width of the image.the relative positions of the centroids with respect to the rectangular bounding box enclosing the face region and the contrast information in the eyebrow region are used to determine the locations of the eyes. The eyebrow (E) contrast information is obtained using obtained as i<j<k. For example, as shown in Figure 4(b), if three points (i, j, k) satisfy the matching rules, they form an isosceles triangle. 1 if Y( i, j) 1 and Y( i, j 1) 1 (3) E( i, j) and Y ( i, j 2) 1 0 otherwise Figure 3(a) shows the centroids of the white blobs in the threshold face image, and Figure 3(b) shows the locations of the eyes. Figure 4 Form the isosceles triangle. (a) The isosceles triangle i j k; (b) Three points (i, j, k) satisfy the matching rules, which will form an isosceles triangle. Figure 3 Determination of face location. (a) centroids of the blobs ; (b) locations of the eyes and mouth. After binaration, the next task is to obtain 4-connected components, label them, and then find the center of each block. We label two eyes, mouth, ear etc. The details of connected component finding can be found in [Gonzalez, 1992]. 3 Rule for feature triangle forming During the driving, the driver moves his/her head left or right, as well as up and down from time to time. So we divide the frame images into frontal images and side view images (faces that tilt left or right for more than 45 ). For frontal images, we could search the potential face regions that are obtained from the criteria of the combination of two eyes and one mouth (isosceles triangle). For side-view images, we use the rationale of the combination of one eye, one ear hole, and one mouth (right triangle), which is called the triangle Feature Triangle. 3.1 Isosceles feature triangle In Figure 4(a), ijk is an isosceles triangle. Statistics shows that the Euclidean distance between two eyes (line ik) is about % of the Euclidean distance between the center of the right/left eye and the mouth. The first matching rule can be stated as (abs(d(i, j)-d( j, k))<(0.25max(d(i, j), D( j, k))) and the second matching rule is (abs(d(i, j)-d(i, k))<(0.25max(d(i, j),d( j, k))) Since the labeling process is operated from left to right and then from top to bottom, the third matching rule can be Figure 5 Assume that (Xi, Yi), (Xj, Yj) and (Xk, Yk) are the three center points of blocks i, j, and k, respectively. The four corner points of the face region will be (X1, Y1), (X2, Y2), (X3,Y3), and (X4, Y4). After we have found the isosceles triangle, it is easy to obtain the coordinates of the four corner points that form the potential facial region. Since we think the real facial region should cover the eyebrows, two eyes, mouth and some area below the mouth, the coordinates can be calculated as follows: Assume that (Xi, Yi), (Xj,Yj) and (Xk, Yk) are the three center points of blocks i, j, and k, which form an isosceles triangle.( X1,Y1), (X2, Y2),(X3,Y3), and (X4, Y4) are thefour corner points of the face region as shown in Fig. 5. They satisfy with the following matching rules: X1 X4 Xi -1/3D(i, k) X2 X3 Xk 1/3D(i, k) Y1 Y2 Yi 1/3D(i, k) Y3 Y4 Yj -1/3D(i, k) 3.2 Right feature triangle (4) ijk is a right triangle (30, 60, 90 ), as shown in Figure 6. The four rules are used to obtain the right triangle for right side view. The first matching rule is abs (D(i, k)-d( j, k))<(0.60d(i, k) and abs(d(i, k)-d( j, k))>0.40d(i, k)) the second matching rule is abs(d(i, k)-d(i, j))>0.13d(i, k) and

4 Uterine Electromyography Signal Feature Extraction and Classification 233 abs(d(i, k)-d(i, j))<(0.19d(i, k)), the third matching rule is abs(d(i, j)-d( j, k))<(0.44d(i, k) and abs(d(i, j)-d( j, k))>0.29d(i, k)). and the forth matching rule is i<j<k. Figure 6 The right triangle ijk; Three points (i, j, k) form a right triangle (30, 60, 90 ) According to the detected right triangle, it is easy to obtain the coordinates of the four corner points that form the potential face region. The coordinates of the four corner points shown in Figure 7 can be calculated as follows: incorrect one. To solve the ambiguity, we propose a method to remove the confusing triangles. A criterion [18] named skin color ratio (SCR), is defined for the arbitration of the confusing triangles. The SCR is defined as follows: Nskin ijk SCR( ijk)= A(ijk) (6) where A( ijk) denotes the area of the triangle ijk and skin ijk is the number of skin-color pixels within the triangle ijk. If SCR is very low, then ijk is very likely to be a triangle that contains three facial components from distinct or scattered skin color regions. For this kind of triangles, they are thus very possible to be unwanted because the higher ratio of non-skin pixels usually comes from the non-skin environment background that separates the contained multiple faces (refer to triangle T2 in Figure 8). Hence, SCR is a good indication for the arbitration of confusing triangles. N X1 X4 Xi -1/6D(i, j) X2 X3 Xi 1.2D(i, j) Y1 Y2 Yi 1/4D(i, j Y3 Y4 Yi -1.0D(i, j) (5) Figure 8 Case of confusing triangles in facial component verifications. T2 is a bad confusing triangle. Figure 7 Assume that (Xi,Yi), (Xj,Yj) and (Xk, Yk) are the three center points of blocks i, j, and k, respectively. (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4) are the four corner points of the side view region. These 4 rules aim to obtain the right triangle for left side view. The definition and matching rules for finding a right triangle of the left side view are similar to those for the right side view as described previously. 3.3 The arbitration of confusing eye-mouth triangle According to the verification on the above four geometrical and spatial relationship rules for eyes and mouth, the candidate facial triangles on the skin color regions can be extracted for further processing. Most noise components can be removed after the component verification process. However, it might be still possible that some confusing combinations need to be dealt with further. The confusing combinations result from those triangles that contain the lips from one face and the eyes from another. As shown in Fig.7, we see three possible triangles T1, T2 and T3 for facial components, where T2 is an In order to calculate N, we examine each detected skin skin pixel on the image to see if it is in the interior of the triangle ijk. If it is, then N skin ijk is increased by one. In this way, we need not to use time-consuming algorithms ijk fortracing the interior pixels of each confusing triangle. We just need to perform simple triangle interior checks over each confusing triangle for each skin pixel. According to the simple geometry mathematics, given the three vertices i(x1; y1), j(x2, y2), and k(x3, y3) of a triangle, the rule for the triangle interior check of any point p(x, y) is performed as follows: C1: x(y2- y1) C2 : x(y3- y2) C3: x(y1- y3) y(x1- x2) y(x2- x3) y(x3- x1) (x2y1- x1y2) 0, (x3y2- x2y3) 0, (x1y3- x3y1) 0, If (all C1, C2 and C3 are true) or (all C1, C2 and C3 are false) then p(x, y) is inside the triangle ijk. This method costs not very high because the number of confusing triangles is usually very small after the component-based verifications. 4 Fast searching for feature triangle extraction

5 234 N. Lu, J. Wang We are looking for any three centers to form an isosceles triangle or a right triangle by the matching rules as mentioned previously. The first matching rule can be stated as abs(d(i, j)-d( j, k))<(0.25max(d(i, j), D( j, k)) and the second matching rule is abs(d(i, j)-d(i, k))<(0.25max(d(i, j),d( j, k)) and the third matching rule is i<j<k. If the Euclidean distance between the centers of block i (right eye) and block j (mouth) is already known, then block center k (left eye) should locate in the area of % of Euclidean distance between the centers of block i (right eye) and block j (mouth), which will form a circle. The search area is only limited in the dark area instead of the whole area of the image as shown in Figure 9. In this way, the triangle based segmentation process can reduce the background part of a cluttered image up to 97%.This process significantly speeds up the subsequent face detection procedure because only 3-9% regions of the original image are left for further processing. Figure 9 The search area of the third block center k is only limited to the dark area instead of the whole area of the image. 5 Tracking strategy After speeding up to form feature triangles, according to either Equation (3) or Equation (4), we can derive a feature rectangle to acquire a potential face region. Then, the FTT (feature triangle tracking) algorithm is adopted to track the face of fatigue drivers. The steps are as follows: Table 1 the pseudo code for FTT algorithm 1. Initialization Initialization the input color image first to Grey level images then to binary (in YCrCb color space) 2. Forming Feature triangle A reference image, at index i, from an image sequence is obtained. The next i+1 frame does as 1) If both i+1 frame and i frame are the same tilting pose, constitutes feature triangle 2) If the tilting pose of i+1 frame are different from i frame a) If i frame is a frontal image, if both i+1 frame and i frame are the same tilting pose, then goes to 1) else do section b) if i frame is a side view image case 1: if i frame tilting from right to left and i+1 frame tilting from left to right(a frontal image) do section case 2: if i frame tilting from left to right and i+1 frame tilting from right to left(a frontal image) do section ). If i frame detects false,i+1 frame does section 3 respectively. Here, we describe the steps of the eye tracking process. In FTT algorithm, computing b) does not consume a lot of time. As a driver is not in a fatigue state, he or she can tilt his/her head frequently and quickly. 6 Experiment results The performance of the proposed tracking system was implemented on a PC with 3.0GHz Microprocessor with 516MB RAM running under the Windows XP operating system. A standard CCD camera with square pixels was employed to capture sequences of color images at the rate of 30 fps. The images are pixels in size with a pixel depth of 24 bits (eight bits in each red, green and blue channel). The data set obtained for evaluation consisted of 1,000 frames sequences from 40 people with different poses, facial expressions, and eyeglasses under varied lighting conditions. The number of false detection of lips is about 19 for the 1000 test images. Thus the system tracks the face correctly about 98.0% of the frames in the image sequence. Figure10 shows a number of examples to illustrate the process of how to get the verified face in frontal image, and the process of how to get the verified face in side views image. Figure 11 shows a part set of test images from an image sequence. As shown in Figure 12, the average of the x-coordinate is more than that of the y-coordinate; we can decide that it is the drivers head nodding (top or bottom). On the contrary, the average of the y-coordinate is more than that of the x - coordinate; we can decide that it is the drivers head shaking (left or right). Figure 13 shows the running average face pose estimation for a period of 6 min. As can be seen, most times during this period, face pose is frontal. But there are times when an extended period of time is spent on other directions (left or right, top or bottom), representing inattention. Figure 14 shows some examples for tracking results of testing video sequences. Face in frame 33 was detected wrongly since her ear cannot be detected. But the face in frame 34 is correct detection. The faces in both frame 105 and frame 106 are correctly detected although they have different tilting poses. But they implement the arbitration of confusing eye mouth triangles. Both frame 857 and frame 858 are correct detection. Note that faces looking up or down are not concerned in the tracking system.

6 Uterine Electromyography Signal Feature Extraction and Classification 235 (a) The process of obtaining the verified face in frontal images (b) The process of obtaining the verified face in side images Figure10 Example images of the process of acquiring the feature triangle. Figure11 A part set of testing images from an image sequence. Figure12 The variation of variation of the x-coordinate and the y-coordinate of head nodding. Figure 13 Face pose monitoring over time.

7 Int. J. Modelling, Identification and Control, Vol. 10, No. 3-4, References Frame 33 Frame 34 Frame 105 Frame 106 Frame 857 Frame 858 Figure 14 Tracking results on test video sequence. 7 Conclusion and future work In this paper, we have proposed a new method (FTT algorithm) to detect and track varied pose of fatigue driver in real time from video sequences. The FTT algorithm locates the face of a fatigue driver correctly, i.e. about 98%. The proposed algorithm exhibits satisfactory performance in terms of both accuracy and speed for detecting faces with wide variations in size, scale, orientation, color, and expressions. Our final aim is to propose a visual framework that, together with other kinds of sensors, can be used for issuing a warning signal once driver fatigue is detected. Our future work is to classify the fatigue level. That is to say, we must propose a method to distinguish opening eyes, closing eyes and blinking eyes. Moreover, the real-time performance is an important factor and should be realized in a driver fatigue monitoring system. Acknowledgements This research is partially supported by the National Grand Fundamental Research 973 Program of China under Grant No.G2002cb The authors would like to thank the CAS-PEAL Large-Scale Chinese Face Database,the subjects for participating in the experiments, as well as the anonymous reviewers for their helpful comments. K. Torkkola, N. Massey, C. Wood. Driver inattention detection through Intelligent Analysis of Readily Available Sensors. Proceedings of IEEE Conference on Intelligent Transportation Systems, Washington DC, October 2004, pp G. Yang, Y. Lin, P. Bhattacharya. A driver fatigue recognition model using fusion of multiple features. Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Hawaii USA, vol. 2, October 2005, pp S. Park, M. Trivedi. Driver activity analysis for intelligent vehicles: issues and development framework. Proceedings of IEEE Intelligent Vehicles Symposium, Las Vegas, USA, June 2005, pp Q. Ji, X. Yang. Real time eye gaze and face pose tracking for monitoring driver vigilance. Journal of Real Time Imaging, 8 (2002), pp P.S. Rau. Drowsy driver detection and warning system for commercial vehicle drivers: field operational test design, data analyses, and progress. Proceedings of the 19th International Conference on Enhanced Safety of Vehicles, Washington, DC, June 6 9, Chiunhsiun Lin, Kuo-Chin Fan. Triangle-based approach to the detection of human face. Journal of Pattern Recognition 34(2001), pp Y. Li, X.-l. Qi, Y.-j. Wang. Eye detection by using fuzzy template matching and feature-parameter based judgment. Journal of Pattern Recognition Letter, 22 (10) (2001), pp [S. Baskan, M. Bulut, V. Atalay. Projection based method for segmentation of human face and its evaluation. Pattern Recognition Letter 23 (14) (2002), pp Y. Ma, X. Ding, Z. Wang, N. Wang. Robust Precise eye location under probabilistic frame work. Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Seoul Korea, May 2004, pp S. Sirohey, A. Rosenfiled. Eye detection in a face image using linear and nonlinear filters. Journal of Pattern Recognition 34 (7) (2001), pp S. Sirohey, A. Rosenfiled, Z. Duric. A method of detection and tracking iris and eyelids in video. Journal of Pattern Recognition 35 (6) (2002), pp P. Smith, M. Shah, N. da Vitoria Lobo. Determining driver visual attention with one camera. IEEE Transactions on Intelligent Transport Systems, 4 (4) (2003), pp G.C. Feng, P.C. Yuen. Multi cues eye detection on gray intensity image. Journal of Pattern Recognition, 34 (5) (2001), pp R.C. Gonzalez, R.E. Woods. Digital Image Processing. Addison- Wesley Publishing Company, Reading, MA, USA, M.Balasubtamanian, S.Palanivel, V. Ramalingam, Real time face and mouth recognition using radial basis function neural networks. Expert System with Applications (2008),doi: /j.eswa E. Wahlstrom, O. Masoud, N. Papanikolopoulos. Monitoring Driver Activities ITS Institute. Research Reports No. CTS 04-05, University of Minnesota. C.H. Morimoto, D. Koons, A. Amir, M. Flickner. Pupil detection and tracking using multiple light sources. Journal of Image Vision Computation, 18 (4)(2000), pp Z. Zhu, Q. Ji. Eye and gaze tracking for interactive graphic display. Journal of Machine Vision Applications, 15 (3) (2004), pp Z. Zhu, Q. Ji, P. Lan. Real time non intrusive monitoring and prediction of driver fatigue. IEEE Transactions on Vehicular Technologies, 53 (4) (2004), pp A. Haro, M. Flickner, I. Essa. Detecting and tracking eyes by using their physiological properties, dynamics, and appearance. Proceedings of IEEE CVPR 2000, Hilton Head Island, South Carolina, June 2000, pp Copyright 2008 Inderscience Enterprises Ltd.

8 Uterine Electromyography Signal Feature Extraction and Classification 237 Mohamed Said Sayed Ahmed,Ping Zhang,Yun-Jie Wu. Position Control of Synchronous Motor Drive by Modified Adaptive Two-phase Sliding Mode Controller. International Journal of Automation and Computing, 05(4), October 2008, pp Edm ee Amstutz, Tomoaki Teshima, Makoto Kimura, Masaaki Mochimaru, Hideo Saito1. PCA-based 3D Shape Reconstruction of Human Foot Using Multiple Viewpoint Cameras. International Journal of Automation and Computing.,05(3), July 2008, pp Yong-Zhong Lu, A Novel Face Recognition Algorithm for Distinguishing Faces with Various Angles. International Journal of Automation and Computing 05(2), April 2008, pp Y. Tian, T. Kanade, J.F. Cohn, Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity,proceedings of FGR02, 2002, pp Wei, H-L, Billings, S.A., A comparative study of global wavelet and polynomial models for non-linear regime-switching systems, Int. J. Modelling, Identification and Control, Vol.2, No.4, pp , 2007.