Stereo Vision Based Advanced Driver Assistance System

Transcription

1 Stereo Vision Based Advanced Driver Assistance System Ho Gi Jung, Yun Hee Lee, Dong Suk Kim, Pal Joo Yoon MANDO Corp ,Gomae-Ri, Yongin-Si, Kyongi-Do, , Korea Phone: (82) Fax: (82) This paper describes a stereo vision based obstacle detection algorithm, which is the core component of advanced driver assistance system incorporating lane departure warning, forward collision warning and avoidance. The proposed vision system recognizes the road lane, on which host vehicle is traveling, by template matching on the bird s eye view of forward scene. The recognition of road lane uses an assumption that a lane marking is a pair of neighboring rising and falling edge and a road lane is a pair of lane marking with a fixed distance. ROI (Region Of Interest) is established according to the recognized ego-lane because preceding vehicle on the ego-lane is expected to be a potential threat to host vehicle. After the establishment of ROI, vision system generates disparity histogram by feature based stereo matching. Because the preceding vehicle has a large amount of vertical edges with the same disparity, it forms a peak in the disparity histogram. Consequently, the preceding vehicle can be detectable by simple thresholding. The threshold of peak detection is designed to vary with respect to disparity, i.e. distance, considering the fact that obstacle appears smaller as its distance becomes further. Detected peaks are verified by the comparison of edge and color between left and right image. Ego-lane based ROI establishment and feature based stereo matching drastically reduce computational burden. Furthermore, disparity histogram based obstacle detection is proved to be robust because it captures big picture successfully ignoring the details. The effect of ego-lane based ROI and adaptive thresholding is verified by experiments with real vehicle. 1. INTRODUCTION Recently, there seems to be enormous needs of intelligent vehicle [1]. Economically practical solutions from fast evolving electronics and sensor industries are expected to realize what has been thought to be impossible in the near future. Introduction of environmental sensor such as radar, vision and GPS allows advanced driver assistance system to recognize surrounding situation. Furthermore, intelligent control algorithm utilizing recognized environment enables intelligent vehicle to automatically enhance the convenience and safety of driver. Among several kinds of environmental sensor, vision sensor is taking intensive attention because of its compatibility with human visual system. In other words, because current driving environment is designed for human without the consideration of automation, especially depending on human visual perception capability, vision sensor is thought to be ideal to interpret a driving situation. It can be easily approved by the plentifulness of recent vision applications such as traffic surveillance [2], license plate / traffic sign recognition [3, 9], free parking site localization [4], lane detection [5, 6] and obstacle / pedestrian detection [6-8]. Stereo vision system has advantages over range sensor system including laser radar and mm-wave radar: it can recognize visual cues such as road lane, traffic sign and detailed object shape. Although some companies are trying to develop monocular vision system that can recognize road lane and obstacles [7], it is certain that stereo vision is more intuitive than monocular vision for the recognition of 3D information. Fig. 1 shows the advantage of stereo vision over monocular vision. In monocular vision, points on a certain sight line are mapped onto one point. However, stereo vision can distinguish these points using displacement between left and right image. This paper explains our approach to stereo vision based collision avoidance system incorporating several vision based functions such as lane departure warning, collision warning and avoidance. X X X Fig. 1 Monocular camera vs. stereo camera 2. SYSTEM ARCHITECTURE The advanced driver assistance system proposed in this paper consists of 5 main components: stereo vision based obstacle distance measurement, dynamic model of ego-vehicle, collision avoidance algorithm, active x L x L X x R X x R

2 braking and HMI (Human Machine Interface). The dynamic model of ego-vehicle estimates required state variable of ego-vehicle utilizing the existing sensors already installed on the vehicle such as wheel speed, lateral acceleration, yaw rate and steering angle. The collision avoidance algorithm considers current vehicle state and obstacle state, and then sends required braking command to active braking system via CAN in order to avoid upcoming collision. The active braking system should be able to generate required braking force without driver s pedal maneuver. In the proposed system, the Mando MGH-Series ESP is used as an active braking actuator. Fig. 2 shows the brief architecture of the system. The stereo vision based obstacle distance measurement, the main focus of this paper, consists of 4 phases: ego-lane based ROI (Region of Interest) establishment, feature based stereo matching, peak detection by adaptive threshold in disparity histogram and verification of candidate peaks by edge/color similarity. The proposed stereo matching measures the distance of obstacle, i.e., preceding vehicle, using disparity histogram. It assumes that preceding vehicle will make a definite peak in the disparity histogram and peak detection can recognize the existence of the object. To make the disparity histogram emphasize the preceding vehicle and ignore potential disturbances effectively, vision system establishes ego-lane based ROI. To compensate many kinds of variation, adaptive threshold is used in the peak detection. Peak detection in disparity histogram does not need the recognition of object boundary, which may be the major challenge of object recognition system because of its complexity and high computational load. This paper shows that the effect of ego-lane based ROI and adaptive thresholding is proved by experimental comparisons. Finally, the proposed method is compared with laser radar in the aspect of detection range and response time. 3. ROI ESTABLISHMENT ROI (Region Of Interest) for the following operations is established according to recognized ego-lane. ROI establishment is important in two aspects: to reduce computational load and to improve the robustness of system by ignoring useless image portion. Furthermore, because the established ROI contains sufficient image portion to detect the distance of preceding vehicle, explicit object boundary detection can be omitted. In this paper, a template matching on the bird s eye view of forward scene is used for the detection of ego-lane. 3.1 The Bird s Eye View of Forward Scene The bird s eye view of an image is a virtual image taken from the sky as if a flying bird sees the same scene. In general, bird s eye view is introduced to eliminate or mitigate the effect of perspective distortion. When a man sees a scene with his own eye or camera, an object located further appears smaller than the nearer in spite of the same length. This kind of distortion is common in pinhole camera model and called perspective distortion. Fig. 3(a) shows the perspective distortion. Although the lane in front of host vehicle is expected to have a fixed width, the width decreases as the distance becomes further. In contrast to the captured image, the bird s eye view of the image, shown in Fig. 3(b), reveals the fixed width of road lane. Furthermore, lane marking also has a fixed width in the bird s eye view. However, it is noticeable that the bird s eye view of an image considers every pixel to be attached onto a plane surface and can not help a strange distortion caused by an object above the ground surface such as preceding vehicle. (a) Captured scene (b) Bird s eye view Fig. 3. The bird s eye view of a forward scene The bird s eye view of an image is generated with homography matrix H. Homography defines the one-to-one relation between pixels of two images, in this case, between captured image pixel and bird s eye view pixel. If X is the coordinate of image pixel and X' is the coordinate of corresponding bird s eye view pixel, homography matrix H interconnects them like equation (1). X, X' and H uses homogeneous coordinate system. X'= HX (1) Fig. 2 System block diagram Because X-X' pairs are available from test image and desired bird s eye view, homography matrix H can be estimated by SVD (Singular Value Decomposition). H generally has an inverse matrix H -1 corresponding to

3 the inverse mapping. 3.2 Ego-Lane Recognition A template matching technique is used to detect the lane marking on bird s eye view image. The template is implemented under the assumption that a lane marking is a neighboring falling and rising edge pair with a fixed distance w in the horizontal direction [6]. Equation (2) shows how the template matching result image T(x,y) is calculated. In this equation, H and W are the height and width of the bird s eye view, respectively. B(x,y) designates the intensity of a pixel (x,y). Consequently, T(x,y) measures how the pixel (x,y) is likely to be lane marking by examining the intensity differences between the pixel and left/right pixels respectively with distant ±w. Binarization of T(x,y) by applying thresholding generates a lane marking candidate image, as shown in Fig. 4(a). H W T ( x, y) = (2B( x, y) B( x w, y) B( x + w, y)) (2) y= 0 x= 0 Detecting lane marking pairs with a fixed distance in the horizontal direction recognizes road lanes in image row y. Because the ego-lane is only the image of interest, one road lane locating near to the vertical centerline is selected as an ego-lane from the image row y. Ego-lanes of each image row are expected to totally form ego-lane in the sense of LS (Least Squared). Therefore, curve fitting of ego-lanes of each image row can be a good estimate of ego-lane. Fig. 4(b) shows the result of ego-lane recognition and the central dotted line is the centerline of ego-lane. perspective view. ROI is established as an image portion under two curves, which is the recognized ego-lane boundaries moved upward with a fixed offset. 4. STEREO MATCHING In this paper, feature based stereo matching finds out the distance information of image pixels. In the case of automotive vision, it is known that vertical edges are sufficient to detect noticeable objects []. Consequently, stereo matching using only the vertical edges can drastically reduce the computational load. The feature based stereo matching consists of pixel classification and similarity based matching. 4.1 Feature Detection: Pixel Classification Pixel classification investigates the intensity differences between a pixel and 4 directly connected neighbors so as to assign the pixel class reflecting the intensity configuration. It is known that the feature based stereo matching with pixel class is fast and robust to noise []. Equation (3) shows how the relationship between g(x) and g(i) is encoded. g(i) is the gray value of neighboring pixel and Fig. 6(a) shows how the pixel index i is defined. Consecutively, equation (4) assigns a class to the investigated pixel by truncating 4 codes like Fig. 6(b). It is obvious that a pixel of smooth surface will be classified as zero class and a pixel of edge will be classified as non-zero class. To reduce the effect of threshold T, histogram equalization or adaptive threshold can be used. (a) Neighboring pixel index (b) Class encoding Fig. 6 Neighboring pixel indexing and class encoding (a) Lane marking candidates (b) Recognized ego-lane Fig. 4 Lane marking and ego-lane recognition result 3.3 ROI Setting 1 if g( i) g( x) > + T d( i) = 2 if g( i) g( x) < T (3) 0 else where, g(.) means the gray value 3 i d ( x) = 3 d( i) (4) i= 0 (a) Recognized ego-lane (b) Established ROI Fig. 5 ROI establishment based on recognized ego-lane ROI for following operations is established using the recognized ego-lane. Ego-lane on bird s eye view is transformed into ego-lane on perspective view by homography matrix. Fig. 5(a) shows the ego-lane on (a) Original image (b) Classification result Fig. 7 Feature detection result

4 Figure 7(a) shows original image and figure 7(b) shows feature detection result. 6.4% of total pixels are classified as non-zero class. 4.2 Stereo Matching Stereo matching is performed only on pixels classified as vertical edge. Furthermore, stereo matching is composed of step-by-step test sequences through class comparison, class similarity, color similarity and maximum similarity detection. Only correspondence candidates passing the previous test step will be investigated in the next test step S class ( x, ys, ) = f ( Lclass ( x+ u, y+ v), Rclass ( x+ u+ s, y+ v)) (5) 3 3u= 1 v= 1 where, 0, a = b f ( a, b) = 1, a b In general, object attached to the ground surface contributes to a wide range of disparities because it exists in a wide range of distance from host camera. In contrast to the case, object above the ground surface including preceding vehicle tends to contribute to a narrow range of disparities because it exists in a narrow range of distance from the camera. In other words, host camera can observe the backend of preceding vehicle, which is generally a plane parallel to the image plane of the camera. Therefore, vertical edge caused by the preceding vehicle s backend plane is supposed to exist at the same distance. Consequently, preceding vehicle reveals its existence in disparity histogram as a definite peak. []. Fig. 8 is the example of disparity histogram. occurrence S color 1 ( x, y, s) = u= 2v= 2 ( X left ( x + u, y + v) X 5 5 right ( x+ u + s, y + v)) 2 (6) where, X : [R, G, B] S( x, y, s) S ( x, y, s) S ( x, y, s) = (7) color class Assuming that the vertical alignment of stereo rig is correct, the search range of a pixel is limited to a horizontal line with a fixed displacement. First, correspondence test is performed on pixels with the same class as the investigated pixel. Class similarity defined by equation (5) is the measure of how the candidate pixel is similar to the investigated pixel in the sense of 3x3 class window. Color similarity defined by equation (6) is the measure of how the candidate pixel is similar to the investigated pixel in the sense of 5x5 color window. Total similarity defined by equation (7) is the product of the class similarity and the color similarity. If highest total similarity is lower than a certain threshold, the investigated pixel fails to find corresponding point and is ignored. The pixel with highest total similarity is corresponding point. 5. DETECTION OF OBSTACLE DISTANCE The proposed system detects the distance of preceding vehicle by the adaptive thresholding of disparity histogram and verification with respect to disparity. 5.1 Disparity Histogram Preceding vehicle is supposed to form a peak in disparity histogram []. Disparity histogram measures how many pixels have a certain disparity and is implemented as an accumulator array. While system scans whole disparity map, which is the result of stereo matching, each bin of the accumulator array counts up when a pixel has corresponding disparity value. Fig. 8 Disparity histogram disparity 5.2 Adaptive Threshold Adaptive threshold compensates the variation of peak height, which is supposed to be proportional to disparity value. Furthermore, adaptive threshold is designed to vary dynamically reflecting the characteristics of current scene. Although preceding vehicle certainly makes a peak in disparity histogram, the height of peak varies with respect to its disparity value. It can be naturally derived from the fact that distant object appears small and near object appears big. Because near object appears big, the probability of the occurrence of vertical edge pixel is high. Near distance means large disparity. Therefore, near object is expected to generate a peak with high height value at large disparity value. Inversely, because distant object appears small, the probability of the occurrence of vertical edge pixel is low. Distant object is expected to generate a peak with low height value at small disparity value. Constance threshold of the peak detection cannot reflect the disparity-peak height relation. If constance threshold is too low, false detection rate at large disparity value increases. If threshold is set too high to avoid the false detection, preceding vehicle, which is our target to detect, might be missed. Threshold line with respect to disparity is proved to overcome the drawback of constance threshold. Threshold line is a line passing through the origin and its slope is changed according to the characteristics of current scene. There are many factors that influence the occurrence of vertical edge on the vehicle s backend plane. The distance factor is already compensated by the introduction of threshold line. Other factors include

5 illumination condition, reflectance of the vehicle s surface, shape of the vehicle. Continuous length of disparity range over the threshold line is the bottom line of a peak. Long bottom line means too low threshold and short bottom line means too high threshold. Of course, few existence of peak also means the possibility of excessive threshold. Therefore, adjusting the slope of threshold line makes the peak detection adaptive to the current situation. Figure 9 depicts the concept of adaptive threshold. bottom line if too low if too high Fig. 9 Adaptive threshold threshold line 5.3 Verification Sometimes, the peak detection finds several peaks above the threshold line. These peaks are candidates of preceding vehicle. Among the candidates, best one is selected by edge similarity and color similarity. Because correct disparity is the displacement between left and right pixels belonging to the preceding vehicle, right image shifted by the correct disparity will be exactly overlapped on the left image in the area of the preceding vehicle. Edge similarity of a certain disparity d is defined in equation (8) and measures how much portion of edge pixels are overlapped if shifted by the disparity. Edge similarity is normalized measure to have a value between 0 and 1. Color similarity of a certain disparity d is defined in equation (9) and measures how much portion of edge pixels is similar in the sense of R, G, B color code value. Color similarity is also normalized measure to have a value between 0 and 1. Total similarity of a certain disparity d is defined as the product of the color similarity and the edge similarity like equation (). disparity d is able to select the correct disparity. Because disparity histogram might be thought of a kind of dimension reducing transformation, impatient determination of the best peak will make the false detection rate higher. Such a consideration justifies the acceptance of multiple candidates and verification based on the original pixel information. 6. EXPERIMENTAL RESULT Proposed system is validated by in-vehicle tests. Specially, proposed ROI establishment based on ego-lane boundary is validated by comparison with fixed ROI case. Adaptive thresholding is also validated by comparison with constant thresholding. Finally, comparing the distance sequence of cutting-in vehicle with the distance sequence measured by laser radar, I will discuss the advantage of proposed method over laser radar. 6.1 Experimental Setting Stereo camera used in experiments is made with two off the shelf CMOS cameras and its baseline is cm in order to be able to detect distant objects [11]. Figure (a) shows stereo camera module installed at the windshield of test vehicle. Caltech Matlab toolbox is used for stereo camera calibration and rectification [12]. Images used for tests and evaluations are acquired at highway. During the experiments, one laser radar is installed at the front end of test vehicle to record distance measurements, which is used in the validation and performance comparison. (a) Stereo camera installed on the vehicle H W 1 SE ( d) = f ( E ( x, y), E ( x + d, y)) (8) edge left right N y = 0 x = 0 where, 0, a = b f ( a, b) =, E : edge image, 1, a b N: the number of edges I (, xy) I ( x+ dy, ) H W left right 1 RGB,, color ( ) = left (, )(1 ) N y = 0 x = 0 256*3 SE d E x y where, I(x,y)=[R,G,B] at point (x,y) (9) SE( d) = SE ( d) SE ( d) () edge color Correct disparity is expected to have maximum value in the similarity evaluation. It means how much left and right pixels can be overlapped with a certain (b) Laser radar installed on the vehicle Fig. Forward looking sensors on the vehicle 6.2 Effect of Ego-Lane based ROI Distance measurements of an acquired image sequence are calculated by two methods: object detection with fixed ROI and object detection with

6 ego-lane based ROI. Fig. 11(a) is the distance measurements calculated with the fixed ROI method. It is observed that measured distances have large noise. Fig. 11(b) is the distance measurements calculated with the ego-lane based ROI method. Table 1 shows the comparison of two cases. It is obvious that ego-lane based ROI method has small average error compared with fixed ROI method. Table 1. Error mean and variance of each case Fixed ROI Lane ROI Error mean Error variance (a) Fixed ROI (b) Wrongly detected vehicle Fixed ROI Reference (a) Distance measurements using fixed ROI Ego-Lane based ROI Reference (b) Distance measurements using proposed ROI Fig. 11 Fixed ROI vs. ego-lane based ROI By investigating the curved road situation, it is analyzed how the proposed ego-lane based ROI improves the performance of preceding vehicle detection. Fig. 12(a) shows the example of fixed ROI and Fig 12(b) shows pixels having the same disparity as the output of preceding vehicle detection. Because the road is curved, the detected vehicle is located at adjacent road lane. Correct preceding vehicle, which should be used for longitudinal control, is rejected because of comparatively small peak height. In other sides, Fig. 12(c) shows the ROI established according to the recognized ego-lane. It is obviously shown that the ROI successfully captures the image of preceding vehicle ignoring the adjacent vehicle. (c) Adaptive ROI (d) Correctly detected vehicle Fig. 12 Comparison of ROI methods in case of road with curvature 6.3 Effect of Adaptive Threshold Preceding vehicle detection is tested with two thresholding methods: constant thresholding and adaptive thresholding. Fig. 13(a) shows the distance measurements calculated with constant thresholding. It can be observed that the distance measurements have very large error. Fig. 13(b) is the distance measurements calculated with adaptive thresholding. Table 2 shows that adaptive thresholding method has smaller average error than constant thresholding method. During experiments, constant threshold is fixed at a small value because only small threshold value can guarantee correct peak is included in candidate peaks. Therefore, a pixel belonging to background portion, which generally forms small peak at small disparity value, is detected as the preceding vehicle and causes false detection result. Error of constant threshold case includes background pixels and traffic markings. Table 2. Error mean and variance of each case Constant thresholding Adaptive thresholding Error mean Error variance Constant Threshold Reference (a) Distance measurements by constant thresholding

7 Adaptive Threshold Reference (c) Adaptive thresholding and detected peaks (b) Distance measurements by adaptive thresholding Fig. 13 Constance threshold vs. adaptive threshold By investigating a critical situation, it is analyzed how the proposed adaptive thresholding improves the performance of peak detection. Fig. 14(a) is the example of constance thresholding and detected peaks. A peak in circle denotes a peak recognized as preceding vehicle finally. In Fig. 14(b), it can be observed that traffic sign on the ground surface is pixels corresponding to the detected peak. Fig. 14(c) is the example of adaptive thresholding and detected peaks. Circled peak also denotes a peak recognized as preceding vehicle finally. In this case, thanks to the adaptive thresholding, correct peak, which is missed in the constance thresholding, can be successfully detected. Fig. 14(d) shows that the preceding vehicle causes pixels corresponding to the detected peak. (d) Correct detection result and corresponding pixels Fig. 14 Comparison of two thresholding methods 6.4 Comparison with Laser-Radar Comparing its output with the distance measured by laser radar validates the proposed method. It is also confirmed that large FOV (Field Of View) of vision system could improve the response time compared with the narrow FOV of laser radar Stereo Vision Laser Radar (a) Constant thresholding and detected peaks Fig. 15 Measured distances with laser radar and proposed method (b) Wrong detection result and corresponding pixels In an open space, test vehicle approaches a preceding vehicle standing still then returns to the initial position in backward direction. For the sake of safety, we fulfilled the experiment at our test track and could not use the ego-lane based ROI method. Fig. 15 shows the two distance sequences measured by proposed system and laser radar. Proposed system seems to

8 measure the distance correctly in the range of ~60m. Although the measured distance is not perfect, it looks sufficient for forward collision warning and avoidance. Furthermore, in the comparison between laser radar and proposed method, it should be noticed that vision based system can simultaneously detect ego-lane, which is necessary for lane keeping and target resolution on a curved road. Another major difference between stereo vision and laser radar is FOV. Laser radar should select narrow FOV to cover far distance because it uses TOF (Time Of Flight) principle without means of bearing angle measurement. In the contrast, vision system selects wider FOV because it can measure the bearing angle of object. Furthermore, Ego-lane based ROI eliminates the disturbance of vehicles on adjacent road lane. Fig.16 shows the distance measurements when a vehicle is going to cut-in. It is noticeable that vision system detects the cutting-in vehicle early compared with laser radar. Such a fast response time is crucial for the successful management of cut-in vehicle (a) Detected cutting-in vehicle Stereo Vision Laser Radar (b) Fast response of vision based system Fig. 16. Effect of vision system s wide FOV 7. CONCLUSION This paper proposes a stereo vision based obstacle detection, which consists of ego-lane based ROI establishment, feature based stereo matching, peak detection in disparity histogram by adaptive thresholding, edge/color based verification. Experiment results shows that the proposed method is valid in the range of ~60m. Effects of ego-lane based ROI and adaptive thresholding are analyzed. Comparison with laser radar shows that vision system has several advantages because of the capability of lane detection and wide FOV. REFERENCES [1] Bishop, R.: A Survey of Intelligent Vehicle Applications Worldwide, Proceedings Of IEEE Intelligent Vehicle Symposium, vol. 5, pp. 0-8, Dearborn, MI, USA (00). [2] Li, X., Liu, Z.Q., Leung, K.M.: Detection of vehicles from traffic scenes using fuzzy integrals, Pattern Recognition, vol. 35, no. 4, pp , (02). [3] Yu, M., Kim, Y.D.: An approach to Korean license plate recognition based on vertical edge matching, IEEE International Conference on Systems, Man, and Cybernetics, vol.4, pp , Nashville, TN, USA (00). [4] Ho Gi Jung, Dong Suk Kim, Pal Joo Yoon and Jai Hie Kim, Stereo Vision Based Localization of Free Parking Site, LNCS Vol (CAIP 05), pages , Sep. 05. (05) [5] Gern, A., Moebus, R., Franke, U.: Vision-based lane recognition under adverse weather conditions using optical flow, Intelligent Vehicle Symposium, vol. 2, pp , France (02). [6] Bertozzi, M., Broggi, A.: GOLD: A Parallel Real-Time Stereo Vision System for Generic Obstacle and Lane Detection, IEEE Transactions on Image Processing, vol. 7, no. 1, pp , (1998). [7] Dagan, E., Mano, O., Stein, G.P., Shashua, A.: Forward Collision Warning with a Single Camera, Intelligent Vehicles Symposium, pp , (04). [8] Paia, C.J., Tyanb, H.R., Liangc, Y.M., Liaoc, H.Y.M,: Pedestrian detection and tracking at crossroads, Pattern Recognition, vol. 37, pp , (04). [9] Escalera, A., Armingol, J.M., Mata, M.: Traffic sign recognition and analysis for intelligent vehicles, Image and Vision Computing, vol. 21, pp , (03). [] U. Rranke and I. Kutzbch, Fast Stereo based Object Detection for Stop&Go Traffic, Intelligent Vehicle, pp , [11] ml [12]