Lane Departure and Front Collision Warning Using a Single Camera

Transcription

1 Lane Departure and Front Collision Warning Using a Single Camera Huei-Yung Lin, Li-Qi Chen, Yu-Hsiang Lin Department of Electrical Engineering, National Chung Cheng University Chiayi 621, Taiwan hylin@ccu.edu.tw, lichychenwailen@gmail.com, iopp5@gmail.com Meng-Shiun Yu Create Electronic Optical Co., LTD Taipei 23557, Taiwan kobeyu@create-eop.com.tw Abstract Improving the driving safety is one major concern for the design of intelligent vehicles. In this paper, we present a monocular vision based driver assistance system for dangerous traffic warning. The video sequences captured from a single camera mounted behind the windshield are used for lane detection and front vehicle identification. Two basic modules, lane departure warning system (LDWS) and front collision warning system (FCWS), are developed and then integrated on an embedded DSP platform for automotive electronics applications. Error analysis on system installation is carried out to verify the correctness of measurements. Experimental results have demonstrated that the proposed technique is able to achieve 97% of accuracy on dangerous traffic warning while maintaining the real-time processing requirement. Index Terms intelligent vehicle; driver assistance I. INTRODUCTION In recent years, traffic safety has becomes a more and more important issue due to the increased number of vehicles worldwide. Since a considerable fraction of traffic accidents is caused by fatigue or distraction of the drivers, design of a driver assistant system for traffic safety is an important and challenging problem with great practical values. Several related fields such as intelligent transport systems (ITS), vehicle active safety systems (VASS) and intelligent vehicles (IV) have attracted researchers in both academia and industry to put a tremendous amount of effort on the technology investigation. Among the existing approaches, lane departure warning and front collision warning are two of the initial driver assistant techniques towards the development of intelligent safety vehicles [1]. In the front collision warning system (FCWS), laser and radar are two commonly used sensors for distance measurements. Their advantages include the capability of detecting long range front vehicles even under poor illumination conditions such as in the rainy/cloudy days or at night. However, these approaches are not popular on the market mainly due to the high equipment and installation costs. The lane departure warning system (LDWS) aims to provide the drivers warning signals if the vehicles deviate too much from their driving lanes. Unless the transportation infrastructure is integrated with the ability of vehicle location sensing, the image-based methods are usually the only ways to detect the left and right lines as lane marks. To reduce the cost of driver assistant systems, many visionbased approaches using single or dual cameras have been proposed [2]. This kind of techniques not only can be realized with cheaper hardware for LDWS and FCWS implementations compared to the laser or radar based systems, the rich information provided by the image sensors can also be extended for traffic sign and pedestrian recognition. Thus, it has great potential to use image-based approaches as assistive technology for vehicular applications, even with the current limitation on the measurement precision under varying illumination conditions. The previous work on lane detection can be categorized to two different approaches: feature-based and model-based techniques. The feature-based methods take the road marks [5] or the lane dividing lines [6] as low level features to locate the lane position in the image. The model-based methods, on the other hand, use the pre-defined curves (e.g. line equations [7] or parabola equations [8], etc.) to approximate the lanes via parameter estimation. Hough transform is often used to combine with the road model for lane detection in a short distance range [1]. In addition to the above methods, there also exist some other techniques such as transforming the images to top-view images for straight line detection [3] or using deformable model for lane detection and tracking [4]. For the research on front collision warning systems, the existing techniques include feature-based, stereo-based, and motion-based approaches [14]. The shadow detected by greyscale image thresholding or edge detection is commonly used to locate the front vehicle. Since the shadow can be easily affected by illumination changes, it is required to verify the vehicle position using training data [12] or symmetry information [15]. In [16], the stereo vision system uses inverse projective mapping for obstacle and lane detection. As for the motion-based approaches, optical flow or regional color feature change can be used to identify the vehicle in the video sequence [17]. In this work, we propose an advanced driver assistance system (ADAS) which uses only a single camera. Our main objective is to develop an efficient and robust technique for LDWS and FCWS. Since these two two basic modules will be integrated in an embedded platform, the computational TABLE I ALGORITHM COMPARISON FOR THE FEATURE-BASED AND Feature-based approaches Fast processing in computation Low-cost hardware required Sensitive to illumination change More input parameters required LEARNING-BASED APPROACHES. Learning-based approaches High accurate detection results Less user-defined parameters High computational complexity More expensive hardware required IEEE 64

2 W W H v v H H p H (a) Installation with camera looking downward. (a) Vanish point. (b) Calibration object. Figure 2. Camera extrinsic parameters calibration. Figure 1. (b) Installation with camera looking upward. Camera installation and imaging geometry. cost is the key issue for our algorithm design. As illustrated in Table I for the comparison of learning-based and featurebased algorithms, the former approach usually provides better results but at the cost of high computational complexity and hardware requirement. Thus, the feature-based approach is adopted in this work. Our LDWS and FCWS modules are first developed on a PC for algorithm design and testing, and then implemented on a DSP platform. The experiments have shown that our technique is able to meet the real time constraint while maintain low error rate. II. CAMERA SETTING AND SYSTEM CALIBRATION The proposed ADAS adopts a single camera to capture the video sequence, and used for lane detection and distance estimation of the front vehicles. To make our system flexible, portable and easy to install, an auto-calibration mechanism for camera parameter estimation is developed. We consider two commonly used camera installation settings, looking upward and looking downward, and analyze the errors introduced by imprecise system setup. A. Camera Geometry and Calibration As shown in Figure 1, a camera with focal length f is mounted on a vehicle at the height β above the ground plane. We first consider the case that the camera looks downward with an angle θ v between the optical axis and the ground plane as depicted in Figure 1(a). Let the image size be W H and the vanishing point v located at H v from the bottom of the image (see Figure 2(a)). Then the relationship between H, H v, f, and θ v is given by tanθ v = H v H/2 f Now, suppose an object with vertical size p is imaged by the camera with height H p as shown in Figure 2(b). Then the angle (1) θ p between the optical axis and the line-of-sight to the top of the object can be written as tanθ p = H p H/2 f If we further assume that the object is located at the distance α in front of the camera, then it can be shown that (2) β = p α tan(θ p θ v ) (3) In the above equations, the camera parameters f, W and H are available from the manufacturer s data sheets. The vanishing point v can be estimated using parallel lines (such as lanes) on the ground plane. Thus, the camera s tilt angle θ v can be derived from Eq. (1). If the location α and height p of the calibration object are known (say, a pole or a person in front of the camera), then Eqs. (2) and (3) can be used to derive the camera s mounting position β. Finally, with the availability of the parameters α, θ v and f, the one-to-one correspondence between the ground plane and the image plane can be established. Thus, the object distance can be measured directly from the captured image. Similarly, for the case that the camera looks upward as shown in Figure 1(b), θ v, θ p, and β are given by and θ v = tan 1 H 2H v, θ p = tan 1 2H p H 2 f 2 f β = p α tan(θ p + θ v ) respectively. The camera s installation parameters (position and orientation) can then be obtained accordingly. B. Error Analysis on System Installation It is well-known that the correctness of measurements can be affected by imprecise system installation and calibration. For the proposed monocular vision system, the intrinsic parameters (e.g. focal length and image sensor dimension) can be fairly accuracy since they are obtained off-site. The extrinsic parameters (i.e. position and orientation), however, are relatively unstable since they have to be derived on-site for individual system installation. Thus, knowing the possible error introduced by system installation is informative for the distance estimation of front vehicles. In this work, the most commonly adopted forward looking camera installation setting is considered. We will analyze the 65

3 IEEE International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS 212) November 4-7, Error of Front Car Distance by Installed Error of Front Car Distance with angle(looking down) and Instaled Error of Distance (%) Installation Height (cm) Error of Front Car Distance (%) Installation Height (cm) θ Figure 3. Distance computation errors introduced by imprecise camera installation height between 8 18 cm. (a) Error introduced by camera looking downward offset 1.5 and installation height between 8 18 cm. error caused by using incorrect position and title angle of the camera for the measurement at a specific distance. This distance is reasonably chosen as the predefined dangerous range of a front collision warning system, which is set as 3 m in the error analysis. In our system, the camera s focal length and pixel size are 2.28 mm and 5.6µm, respectively. The distance estimation error is defined as Z err = Z est 1% (4) where Z err and represent the estimated and actual distance, respectively. For the case that the camera is looking forward as shown in Figure 1(a), the front car distance Z can be written as Error of Distance (%) Installation Height (cm) Error of Front Car Distance with angle(looking up) and Installed (b) Error introduced by camera looking upward offset 2 and installation hight between 8 18 cm. Figure 4. Distance computation errors introduced by imprecise camera installation height and tilt angle. θ Z = f H Y where H is the camera s installation height and Y represents the projected image point. If the assumed and true camera s installation heights are H ass and H tru, respectively. Then the measured Y is given by and the estimated distance is Y mea = H tru f Z est = H ass f Y mea = H ass H tru Finally, the distance estimation error defined by Eq. (4) can be written as Z err = H ass 1 H tru 1% If the camera s installation height is 13 cm, then the distance computation error for the true installation height between 8 cm and 18 cm is shown in Figure 3. We then take the camera s tilt angle error θ e into account In this more general case, Y mea and Z est can be derived as ( Y mea = f tan tan 1 H ) tru + θ e and Z est = H ass ( ) tan tan 1 H tru + θ e respectively, and θ e is positive for looking upward. Figures 4(a) and 4(b) illustrate the errors introduced by imprecise tilt angle settings ( 1.5 looking downward and 2 looking upward), respectively, with different camera s installation height. III. APPROACHES The proposed driver assistant system adopts a single camera installed behind the windshield of a vehicle. It consists of two modules, lane departure warning system (LDWS) and front collision warning system (FCWS). Prior to its use for lane detection and front vehicle identification, a system calibration stage as described in the previous section is carried out first. The parameters are then used to determine the image region for low-level processing and distance computation. After the lane detection result is obtained, it is used to further restrict the region of interest (ROI) for the distance estimation of front vehicles. A. Lane Departure Warning System The flowchart of our lane departure warning system is shown in Figure 5. Typically, the input image is only about half size of the original captured image after removing the unnecessary region (e.g. the sky and traffic signs in the upper part of the image). The Sobel operator G= G 2 x+ G 2 y 66

4 (a) Lane departure status. (b) Lane departure state machine. Figure 6. Robust lane departure warning using a finite state machine. Figure 5. The flowchart of our lane departure warning system. where G x = , G y = is first used to obtain a gradient map. It will serve as the input to detect the lane dividing lines and the vehicle shadows for the front collision warning system. To detect the driving lanes, the gradient image is binarized to an edge image for line detection. Since the image regions have different characteristics for the scenes associated with different distances (i.e. the intensity variation is larger for distant scenes), the input image is partitioned to three parts (upper, middle and lower regions), and different thresholds are used to derive the corresponding edge images. Hough transform for line detection with the orientation constraints specified by the vanishing point is then carried out to extract the lane dividing lines. Using the locations of detected lines in the images, the lane departure status left, right, and normal, can be defined as shown in Figure 6(a). However, there might be some false alarms due to subtle changes in the image or miss detection of the lanes. To increase the accuracy and robustness of the lane departure warning system, a finite state machine as illustrated in Figure 6(b) is implemented. In the proposed system, only the normal state can change to the right or left states, and generate a warning signal. The experimental results show that this approach can successfully reduce the false alarms due to the lane dividing lines occasionally occluded by other vehicles. B. Front Collision Warning System The front collision warning system to identify the front vehicle, estimate the distance to the vehicle, and generate a warning signal if the range is smaller than a safety threshold. The input to the system is the gradient image and lane dividing lines obtained from the lane departure warning system. To detect the front vehicle s location, we search the possible shadow areas in the image along three lines parallel to the (a) Shadow detection along the lines. Figure 7. (b) Front vehicle location. Front vehicle identification using shadow and gradient information. lanes as shown in Figure 7(a). The positions with large gradient changes are marked, and the candidate vehicle location is enclosed by a bounding box as illustrated in Figure 7(b). Since it is possible to obtain an incorrect vehicle location based on the shadow detected by the gradient information, we further incorporate the above method with a histogrambased verification technique. First, the candidate image region of a vehicle is extracted for verification. The histogram of its vertical projection for the intensity values larger than a threshold is computed by h t H(x)= [P(x,i)> th edge ], x [w l,w r ] (5) i=h b where P(x,i) is the intensity value of the pixel (x,i), th edge is an edge threshold, and w l, w r, h t and h b correspond to the left, right, top and bottom of the bounding box. A binary pattern B(x) is then derived by applying a threshold th curve on H(x), i.e., { 1 if H(x)> thcurve B(x)= else The sample results of H(x) and B(x) are shown in Figure 8. If the image region contains a vehicle, then there will be only a single white pattern, instead of the one shown in Figure 8(b). For the real scene images in the experiment, this technique for vehicle detection is also robust under different weather conditions. Once the front vehicle is identified, its vertical position in the image is used to calculate the distance to the camera system. This can be done by using the formula given in Section II, or using the lookup table recording the one-to-one mapping between the pixel location and the depth information on the ground plane. By subtracting the offset associated with the 67

5 (a) (b) Figure 8. The histogram-based verification for a vehicle s location. camera s installation, the distance between the two vehicles can be derived. In our front collision warning system, the distances for warning and danger are set as 5 m and 3 m, respectively. (a) Sunny noon. (b) Sunny day and dark road. IV. EXPERIMENTAL RESULTS The proposed techniques for LDWS and FCWS have been tested on both PC-based and DSP-based platforms. 1 A driver recorder CB7 developed by Create Electronic Optical is mounted behind the windshield of a sedan. The video sequences are captured during driving and processed off-line to verify the performance of our techniques. Figure 9 shows some test videos used in the performance evaluation. It covers a variety of illumination conditions such as in sunny or cloudy weather, or in the tunnel. Figures 1 and 11 illustrate some results of lane departure and front collision warnings under various situations. Figures 1(a) 1(d) shows the lane dividing lines detected with the presence of expansion joints and viaducts, near a tool booth area, and with unclear road markings, respectively. The solid square on the image indicates the waning signal when the vehicle shifts to the left or right. In Figures 11(a) 11(d), the detected vehicle is marked with a thick green line on the bottom. Its vertical position is used for distance computation. The other two horizontal lines (green and purple) indicate the distances of 5 m and 3 m, respectively. Some challenging situations such as the images containing expansion joints, driving in the tunnel, and an overexposed image capture are shown in Figures 11(b) 11(d), respectively. The performance of our LDWS and FCWS techniques is evaluated based on the detection accuracy and computation time. Since the evaluation of system accuracy for all possible situations is usually not necessary and mostly infeasible, some reasonable criteria listed below are used in this work: 1) The width of the lanes is 3 4 m. 2) The vehicle velocity is over 6 km/hr. 3) The curvature of the lanes is over 25 m in radius. 4) The visible range of the camera is over 12 m. 5) The toll booth areas are not considered. The accuracy is then defined by Accuracy= No. o f Correct Frames No. o f Testing Frames 1% where No. o f Correct Frames is the number of correctly detected frames in terms of system warning settings, and 1 Due to the space limit, we only show the performance analysis of DSPbased platform, which is designed for the on-board real-time processing. Figure 9. (c) Cloudy noon. (e) Sunny noon. (d) Afternoon and dark road. (f) Tunnel. Some test videos used in the performance evaluation. No. o f Testing Frames is the number of frames used for accuracy evaluation. The DSP-based platform used in our development is TMS32 DM6437 from Texas Instruments. It can process eight 32-bit instructions in a cycle at the clock rate of 597 MHz. Since the time consumption is the most important issue for real-time applications on a DSP platform, the EDMA memory and TI s optimized libraries are used whenever possible for parallel processing. The accuracy for the test videos shown in Figure 9 is tabulated in TABLE II. The computation time of our LDWS and FCWS is tabulated in TABLE III for each processing stage. It is shown that the proposed system is able to achieve the real-time constraint on the DSP-based platform. V. CONCLUSION In this work, we present a vision-based driver assistance system for dangerous traffic warning. A driver recorder with a single camera is used to acquire the traffic information for processing. Two subsystems, lane departure warning system (LDWS) and front collision warning system (FCWS), are developed and integrated on an DSP-based platform for automotive electronics applications. We have analyzed the correctness of measurements affected by imprecise system installation and calibration. In the performance evaluation on various illumination conditions, our techniques are able to achieve 97% of accuracy on lane departure and front collision 68

6 TABLE II ACCURACY OF THE PROPOSED LDWS AND FCWS TECHNIQUES EVALUATED ON THE TEST VIDEOS. (a) (b) Test video sequence LDWS (%) FCWS (%) Figure 9(a) Figure 9(b) Figure 9(c) Figure 9(d) Figure 9(e) Figure 9(f) Average TABLE III COMPUTATIONAL TIME OF THE PROPOSED LDWS AND FCWS TECHNIQUES ON THE DSP-BASED PLATFORM. Figure 1. (c) (d) Some results of our LDWS under various situations. Processing stage Time (ms) Acquire Y information, ROI setting 5.11 Sobel detection (IMLIB) 2.51 Hough transform 19.7 Display warning.48 Lane departure, Sobel detection 4.37 Front collision warning.2 Total 31.8 Figure 11. (a) (c) (b) (d) Some results of our FCWS under various situations. warnings while maintaining the real-time processing constraint at 3 frames per second. VI. ACKNOWLEDGEMENTS The support of this work in part by Create Electronic Optical Co., LTD, Taiwan, is gratefully acknowledged. REFERENCES [1] I. Gat, M. Benady, and A. Shashua, A Monocular Vision Advance Warning System for the Automotive Aftermarket, SAE Technical Paper , 25. [2] J. Cui, F, Liu, Z. Li, and Z. Jia;, Vehicle localisation using a single camera, Intelligent Vehicles Symposium (IV), 21 IEEE, vol., no., pp , June 21. [3] Y. Zhou, R. Xu, X. Hu, and Q. Ye, A robust lane detection and tracking method based on computer vision, 26 IOP Publishing Ltd, Vol. 7, pp.62-81, February, 26. [4] Y. Wang, E. K. Teoh, and D. Shen, Lane detection and tracking using B-Snake, Image and Vision Computing, Vol. 22, No. 4, pp , 24. [5] M. Bertozzi and A. Broggi, Parallel and local feature extraction: a real-time approach to road boundary detection, IEEE Trans. on Image Processing, Vol.4, No.2, pp , [6] R. Aufrere, R. Chapuis, and F. Chausse, EA fast and robust vision based road following algorithm, in Proc. IEEE Intelligent Vehicles Sym., Dearborn, MI, Oct.3-5, 2, pp [7] A. Gern, U. Franke, and P. Levi, EAdvanced lane recognition - fusing vision and radar, in Proc. IEEE Intelligent Vehicles Sym., Dearborn, MI, Oct.3-5, 2, pp [8] A. Kaske, R. Husson, and D. Wolf, Chi-square fitting of deformable templates for lane boundary detection, in Proc. IAR Annual Meeting, Grenoble, France, Nov. 1995, pp [9] T. H. Chang and C. J. Chou, Rear-end collision warning system on account of a rear-end monitoring camera, 29 IEEE Intelligent Vehicles Symposium, pp , 3-5 June 29. [1] A. Suzuki, N. Yasui, and M. Kaneko, Lane Recognition System for Guiding of Autonomous Vehicle, Intelligent Vehicle 92, pp , Sept. 2. [11] C. J. Chen, H. Y. Peng, B. F. Wu, and Y. H. Chen, A Real-Time Driving Assistance and Surveillance System, Journal of Information Science And Engineering, 25, pp , 29. [12] X. L. Shen, D. C. Tseng, C. W. Lin, T. Hu, and R. Liou, Versatile Visual Detection Techniques for Advanced Safety Vehicles, National Central University and Chung-Shan Institute of Science & Technology. [13] Z. Sun, R. Miller, G. Bebis, and D. DiMeo, A Real-time Precrash Vehicle Detection System, in Proceedings of the Sixth IEEE Workshop on Applications of Computer Vision, 22. [14] Z. Sun, G. Bebis, and R. Miller, On-road vehicle detection: a review, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.5, pp , May 26. [15] A. Broggi, P. Cerri, P. C. Antonello, Multi-Resolution Vehicle Detection using Artificial Vision, IEEE Intelligent Vehicles Symposium, pp , 24. [16] M. Bertozzi and A. Broggi, GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection, IEEE Trans. on Image Processing, vol.7, no.1,pp.62-81, Jan [17] T. Naito, T. Ito, and Y. Kaneda, The Obstacle Detection Method using Optical Flow Estimation at the Edge Image, Intelligent Vehicles Symposium, Istanbul, Turkey, June 13-15,