Vision-based Real-time Road Detection in Urban Traffic

Transcription

1 Vision-based Real-time Road Detection in Urban Traffic Jiane Lu *, Ming Yang, Hong Wang, Bo Zhang State Ke Laborator of Intelligent Technolog and Sstems, Tsinghua Universit, CHINA ABSTRACT Road detection is the major tas of autonomous vehicle guidance. We notice that feature lines, which are parallel to the road boundaries, are reliable cues for road detection in urban traffic. Therefore we present a real-time method that etracts the most liel road model using a set of feature-line-pairs (FLPs). Unlie the traditional methods that etract a single line, we etract the feature lines in pairs. Woring with a linearl parameterized road model, FLP appears some geometrical consistenc, which allows us to detect each of them with a Kalman filter tracing scheme. Since each FLP determines a road model, we appl regression diagnostics technique to robustl estimate the parameters of the whole road model from all FLPs. Another Kalman filter is used to trac road model from frame to frame to provide a more precise and more robust detection result. Eperimental results in urban traffic demonstrate real-time processing abilit and high robustness. Kewords: Autonomous vehicle, lateral visual guidance, road detection, feature-line-pair, Kalman filter 1. INTRODUCTION Autonomous vehicle guidance has been a hot research area in the past 20 ears [1][2]. Among the comple and challenging tass that have received the most attention, road following, which is composed of road detection and obstacle detection, is the most important one. Recentl, due to the low cost of camera and well-developed algorithm in computer vision, visual guidance for autonomous vehicle has been highlighted research field, which focuses on machine vision techniques that detect particular features in images of the road ahead of the vehicle, and determine the desired vehicle position with respect to the road boundar based on these features. In the past decades, several road following sstems were proposed and demonstrated successfull. Most of them aimed at lateral vehicle guidance on highwa or other well-structured road. GOLD (Generic Obstacle and Lane Detection) sstem reduced road detection to the localization of specific structured features painted on the road surface, such as lane marings [3]. RALPH sstem at CMU introduced the planar road assumption and traced the parallel lines on the road, performing robust road following despite degradation of lane marings [4]. PVR III sstem at Pohang used similar method to implement road detection tas [5]. Some other sstems etended the detection of lane marings to the detection of road boundaries, tpicall using gradient operators. All these sstems usuall have ver fast processing speeds and are ver well suited for structured roads with good conditions. Although lateral visual guidance for automatic vehicle on highwa has been thoroughl eplored, few attention has been put on urban traffic, which is characterized b degradation of lane marings, random road geometr, tight curves and wea traffic participants lie biccles and pedestrians. It is obvious that such traffic scene becomes ver comple and it is much more difficult to reliabl interpret. SCARF and UNSCARF sstems at CMU have been built to deal with complicated scenes. The have achieved impressive accurac and robustness on fairl unstructured roads with shadows, leaves ling on the road and lighting changes [6]. But the are too slow to meet the real-time requirement, which is important in urban traffic. * lujiane00@mails.tsinghua.edu.cn; phone ; State Ke Laborator of Intelligent Technolog and Sstems, Tsinghua Universit, Beijing , CHINA

2 Usuall there are man lines parallel to the road boundaries in the urban traffic scene, such as lane marings and vehicle tracs. Such lines, called feature lines in this paper, mae up of a great part of all the edge lines in the image and demonstrate some geometrical consistenc as a whole. In this paper, we propose a real-time road detection method based on FLPs in urban traffic. Unlie the traditional methods that etract a single line, we etract the feature lines in pairs with global criterions. Woring with a linearl parameterized road model, we can detect FLP with a Kalman filter tracing scheme. Since each FLP determines a road model, we appl regression diagnostics technique to robustl estimate the parameters of the whole road model from all FLPs. Another Kalman filter is used to trac road model from frame to frame to provide a more precise and more robust detection result. Eperiments in urban traffic on THMR-V (Tsinghua Mobile Robot V), an autonomous vehicle developed b Tsinghua Universit, demonstrate real-time processing abilit and high robustness of our method. The rest of this paper is organized as follow. Firstl, several commonl used road models along with three reasonable assumptions are introduced in section 2. Secondl, we give the definition of FLP and a fast, robust FLP detection algorithm with Kalman filter in section 3. Thirdl, we introduce the robust estimation of the road model parameters on the basis of FLP and road model tracing in section 4. Then we present some eperimental results and detailed analsis in section 5. Finall, we end this paper with some conclusions and description of future wor in section ROAD MODEL A complete description of the road geometr in the image can be comple since the road ma var in width and curvature. The more parameters used in the road model, the greater the chance of error in estimating those parameters and the more computation required. Some road models have been proposed, such as clothoid curve, polnomial curve with low order, parabola, and the simplest triangular model [7]. The perform differentl in processing time and detection accurac. In our method, a linearl parameterized road model is used with the following assumptions: Let ( *, *, z * ) be the point in real world coordinate. 1. The road is planar, i.e. z * =0 with each point on the road plane; 2. The road boundar is approimated b a d-order polnomial: d = = * * i a i i (1) 0 where a i is the coefficient of polnomial; 3. The road is equall wide at each point, which ma be locall approimated as: * * * 2 = 1 + w (2) where w * is the constant width between road boundaries. According to the assumption 1, transformation between the road plane ( *, * ) and the image plane (, ) can be described as [8]: 1 = l * * = l (3) * where (, ) denotes the point in the image plane; l and l depend onl on the camera calibration parameters. We set the origin of the coordinate sstem to a point on the sline, which can be computed from prior camera calibration. Combining Eq. (2) and Eq. (3), we ma describe one of the road boundaries as follow in the image plane: d i 0 = 1 = b i i i1 b i = l ai l (4) According to assumption 3, another boundar ma be described in the real world as * 2 = Σ d i=0 a i *i + w * and ma be described in the image as d 1 i b w 2 = i = i + 0 b = l i a l l i l i1 w = w * (w is also a constant) (5)

3 On the basis of the above pair of curves, road model as A = ( b i, w ), 0 i d is defined, which is called linearl parameterized road shape [8]. In this road model, if d=1, the model turns to be the classical triangular model, which is an acceptable approimation in most cases. In other words, we ma mae an assumption that the road is locall straight. Under the same circumstance, [6] simpl describe the road geometr with a four-parameter model as shown in Fig. 1.a, or we ma modif it to the one shown in Fig. 1.b. V V Vanishing Point θ V V Vanishing Point W 1 2 Figure 1. Four-parameter road model (a) and (b) 3. DETECTION OF FEATURE-LINE-PAIR 3.1 Feature-line-pair In urban traffic, the road is generall well-constructed, locall flat and equal width, which satisf the assumptions described in section 2. Furthermore, we notice that feature lines, which are parallel to road boundaries, are reliable cues to detect the road. Therefore, we define feature-line-pair (FLP) as pair of local parallel lines, which are also parallel to the road boundaries. Although feature lines ma be split into pieces b occlusion of other vehicles or degradation of shadow, all these FLPs appear to contribute to the same road model. Fig. 2 shows the FLPs in a tpical traffic, where line 2 and line 6 are road boundaries. Besides line 2 and line 6, there are man edge lines parallel to them. Each two of them, lie line 3 and line 4, line 1 and line 7, compose different FLPs. Obviousl each FLP is a part of a triangle, which corresponds to a road model described in Fig. 1. We ma easil prove that all triangles corresponding to an FLP share the same verte, which is called vanishing point. Owing to robust FLP detection introduced below, we can appl simple and fast edge detector to the image before FLP detection. Once to all the FLPs are detected, we can reconstruct the whole road model easil. 3.2 Feature-line-pair detection with Kalman filter Figure 2. FLPs on the road Traditionall, we can detect the FLP b local connectivit or correlation tracing method [9]. When we wor with a specific road model described in section 2, it is possible to detect FLP with the Kalman filter. First, we define each edge point pair (EPP) of an FLP as (width, L ), where width denotes the difference between left point and right point, and L denotes the -location of the left point in the image. Combining the Eq. (4) and Eq. (5), we get (width, L,) = (w, Σ d i=0 b i 1-i ). We ma search EPP row b row, beginning from the bottom of the image. As the decrease, i.e. we search for the upper row, width is epected to decrease b a constant w. Since width linearl depends on, we ma trac it with a normal Kalman Filter, which will give out more accurate detection result with high speed. If we let d=1, then L =b 0 +b 1 and it also linearl depends on, which allow us to trac both width and L within an integrated Kalman filter. Thus we wor out an effective and robust method to detect all of the FLPs.

4 If we simpl tae d=1, we can trac all the EPP with one integrated Kalman filter. Let the measurement vector z =[width, L ] T. The processing state vector can be described as a four-dimension vector =[width, L, w, v ] T, where w and v represent the velocit of width and L, respectivel. All above variables are define in the (width, L ) coordinates. The processing equation and measurement equation of our Kalman filter can be described as = A 1 + w 1 (6) z = H + v where the subscript denotes the time instance, A is matri of linear dnamics sstem, H is matri of linear measurement. For the EPP tracing in our application, we ma define A and H as A = H = (7) where time interval is assume to be 1. In Eq. (7), the random variables w and v represent the process and measurement noise respectivel. The are assumed to be independent of each other, white, and with normal probabilit distributions p( w) ~ N(0, Q) (8) p( v) ~ N(0, R) In practice, the process noise covariance Q and measurement noise covariance R matrices are constant, and the can be measured with some off-line sample measurement. Although Kalman filter can give real-time performance and reduces computation greatl, it has its own defects. Kalman filter provides a recursive solution of the least-square method, and it is not a robust estimator. It is incapable of detecting and rejecting the outliers, which ma cause collapse of tracing. Besides, Kalman filter records not the data ever measured but onl the combination states at time -1, which means that the final detection result is sensitive to the order of measurement. Sometimes such properties will worsen the accurac of the detection result. To alleviate these problems, we define a measurement describing the reliabilit of current Kalman tracing. The reliabilit of the net measurement appended to current Kalman filter is the ratio between the correction increment S B and the prediction increment S A, as shown in Fig. 3. SB 2 R = ( ) where S A 1 R * = l R + (1 l) R (0<l<1) (9) S S ˆ A = 1 = ( ˆ B K z H ˆ After careful stud of some cases, we find that such measurement is reliable reflection of the reliabilit of FLP we are tracing with, and we ma easil determine tracing criterions b off-line eperiments. On the other hand, we attach a point list to the Kalman filter, recording all its measurements. Once the Kalman filter breas down due to the outliers, we ma estimate the tracing result from all the measurement point in the point list. ) (10) S A ˆ ˆ = 1 z S B K ( z H ˆ ) = R = ( S B S A ) 2 ˆ S B ˆ 1 S A ˆ (a) definition of the reliabilit of Kalman filter (b) reliabilities while EPP tracing Figure 3. Introducing Reliabilit to Kalman Filter

5 There is another problem deserves further discussion here. While we are tracing multi-flp in the same (width, L ) coordinate, we meet with the corresponding problem, which lies that in multi-target tracing. To achieve more robust detection result as well as reduce the necessar computation, we mae some improvements to the standard Kalman tracing method. Before the Kalman filter is reliable, which ma be judged b Reliabilit of the Kalman filter introduced above, we follow all EPP nearb in order to avoid missing an possible tracs. When the Kalman filter is reliable enough, we follow onl the EPP that matches the Kalman prediction, and follow the whole road model criterions, such as the location of the vanishing point in the image. Such improvements allow us to detect all the FLPs more accuratel and more quicl. 4. ROAD DETECTION BASED ON FEATURE-LINE-PAIR Compared with etracting single feature line in traditional sstems, it is more robust and effective wa to detect the whole road model on the basis of FLPs. Traditional sstems usuall use searching method to determine the accurate shape and position of the current road. UNSCARF sstem evaluates all possible roads that could appear in the image, loos at the difference between the road model and the region edges in the image, and select the one with lowest cost. SCARF sstem, using the same four road parameters as UNSCARF, build an accumulator. Each piel is assigned a value indicating how probable this piel belongs to the road region. The interpreter searches for the road having the highest combined lielihood using a voting scheme similar to the Hough technique [6]. Both these method need to search a whole sub-space of the road parameter space, which ma causes man problem. The most important is how to quantize the high dimension road model space. If the grid is too small, we have to search a comparativel large space, which will cost more computation and more memor. On the contrar, if the grid is too large, we can onl get a coarse result, which ma not be adequate for vehicle guidance. Since road model detection problem ma be regarded as a parameter estimation problem, we can use robust estimation method to determine the accurate shape and position of the road. Compared to searching scheme, robust estimation can give more precise result with much less computation cost. Among classical robust estimation methods, we appl regression diagnostics technique to our method. It tries to iterativel detect possibl wrong data and reject them through analsis of globall fitted model. The technique wors as follows [10]: 1. Determine an initial fit to the whole set of data through least squares; 2. Compute the residual for each datum; 3. Reject all data whose residuals eceed a predetermined threshold; if no data has been removed, then stop; 4. Determine a new fit to the remaining data, and go to step Road detection based on feature-line-pair As mentioned in section 2, we wor with a linearl parameterized road model (b i, w), 0 i d. Since each FLP detected in current image contributes to a unique road model, we ma reconstruct all possible road models, and estimate the proper one with regression diagnostics technique. Fig. 4 gives out a simple eample. Suppose d=1, i.e. the simple four-parameter road model shown in Fig. 1.b is used to describe the road model. After FLP detection, road models corresponding to each FLP are reconstructed (Fig. 4.a). Instead of estimating four parameters as a whole, we divide the estimation process into two stages. Firstl, we estimate the position of the vanishing point (v, v ) in road model, because it is the most robust propert in the estimation process. Fig 4.b focuses on the neighborhood of the vanishing point, which is estimated b regression diagnostics technique. In Fig. 4.c, dots represent the vanishing points given b FLPs and the star smbol represents the estimation of vanishing point (v, v ). Then the ( L, R ) pair is combined into a single parameter in order to eliminate the spurious edge line due to processing errors and get more accurate result. Some classical clustering method is applied to classif all data. Similarl, we use regression diagnostics technique to compute the core of each class. Star smbols in Fig. 4.d represent the estimations of data over the histogram. Estimations of road models are shown in Fig. 4.e. Furthermore, if we have prior nowledge about (width, L ) of the whole road model, which ma come from real world measurement and projective matri, or from detection result in last frame, we can select the proper and give a determinate detection result of the road model parameters (Fig. 4.f).

6 (a) road model reconstructions (b) neighborhood of the desired (c) robust estimation of (v, v ) with all FLPs vanishing point 4.2 Road model tracing (d) robust estimation of i (g) road model reconstruction (f) road model detecion result Figure 4. Robust estimation of road model parameters In order to achieve real-time performance and accurate detection result, man sstems use Kalman filter to trac road model and location from frame to frame. In our method, we appl such technique in a few aspects. While tracing the EPP in FLP detection, we assign a proper initial value to the Kalman filter b the location of the first EPP and the FLP detected in the last frame. While detecting the FLP, we ma use the road model detected in last frame to give some global criterions, lie the location of vanishing point, throwing off the spurious FLP in time and alleviating the computation. While road model estimation, (width, L ) derived from last frame will direct the validation of the road model in current image. Once the road model detection fails in the current image, we ma use the previous detection result to give a rational estimation. Road model tracing with Kalman filter in image sequence not onl accelerate the detection processing but also provides us more precise and more robust road model, especiall in dense traffic and in bad weather situation. 5. EXPERIMENTS The method proposed in this paper has been implemented in C++ on a commercial AMD 1.33Ghz PC with single camera and a Matro Meteor RGB/PPB digitizer. Our method has been successfull tested on THMR-V (Tsinghua Mobile Robot V), an intelligent vehicle developed in Tsinghua Universit, and in various inds of environment. Letting d=1, we wor with a simple triangular road model described in Fig. 1.b, and we gives a general eample under such circumstance in Fig. 5. The original gra image captured in general urban traffic scene is shown in Fig. 5.a. Due to the robustness and accurac of EPP tracing with Kalman filter, we ma appl simple edge detector on the image before FLP detection (Fig. 5.b). In order to alleviate the high computation, we onl perform the rest operation to the lower 1/3 of the whole image. Eperiment results show that it is reasonable for real-time application. There are man EPPs in the original edge image. When we search the EPP row b row from the bottom of the image, EPPs belonging to the same FLP leave tracs in the (width, L ) coordinate lie the those of a moving target with constant velocit (Fig. 5.c). The all move from right to left, which means two lines of a FLP is getting closer and closer until finall the

7 meet at a point in the image. Such point is just the vanishing point in the projective image. From the EPP tracs, we can recover all the FLPs in the image (Fig. 5.d). Net, since each FLP determines a road model, we appl regression diagnostics technique to robustl estimate the parameters within the linearl parameterized road model (Fig. 5.e). Finall, with the information from the prior nowledge and detection results of previous frames, a Kalman filter is used to trac road models in images sequence, and the precise road model is detected.. In this image, onl the smmetrical feature lines are selected as the road boundaries (Fig. 5.g). (a) original gra image (b) edge image (c) EPP tracing (d) FLP detection within (width, L ) coordinate (e) reconstruction from all FLPs (f) road parameters estimation (g) road model detection Figure 5. Simple road shape detection When d>1, things become more comple. Let us simpl tae d=2 for eample, then (width, L ) = (w, b 0 +b 1 +b 2-1 ). Since there is a -1 term in the equation, we can no longer trac all the EPP within a standard Kalman filter. However, we notice that width is still linearl dependent on and independent of d. Therefore we ma use a hbrid method combining correlation tracing and Kalman filter tracing. In the (width, L ) coordinate, we select and record all EPP point with a correlation criterion of L and trac onl the width propert with a Kalman filter. When an FLP is detected, its parameters of (b 0, b 1, b 2 ) can be robustl estimated from all its recorded EPP members. Then FLPs in the image plane are detected. In road model estimation, since different FLPs belonging to the same feature-line-pairs vote for the same v but not the same v an longer, we ma firstl estimate the v parameter, then classif all the FLPs according to their width properties and use robust estimation method again to determine the final road model. We ma tacle it in the similar wa when d>2. However, we find in practical eperiments that comple road model seems so unstable that even small noises in the edge image will greatl affect the detection result. Currentl we are woring on such problems. (a) v tracing in images sequence (b) v tracing in images sequence Figure 6. Tracing of measurements of the four-parameter road model

8 Our method has also been tested with several images sequences. Fig. 6 shows the measurements of the road model parameters in one of the tests. Owing to the Kalman filter tracing, some errors are corrected and we get quite stable detection results. Our method also shows real-time performance in all the eperiments due to the use of Kalman filter tracing. For the sequence with images in Fig. 10, the whole processing taes 4 ms to detect the edge piels, 80 ms to detect all the FLPs, 11 ms to estimate the road model parameters, and no more than 1 ms to determine the eact road model with Kalman filter tracing. Time costs add up to no more than 96 ms, which satisfied the real-time requirement in urban traffic. 6. CONCLUSIONS AND FUTURE WORK A robust detection and tracing of road shape via on-broad camera becomes more and more important for autonomous vehicle guidance. For lateral vehicle guidance, road boundaries detection must at least provide at a good rate estimates of the relative orientation and of the lateral position of the vehicle with respect to the road [8]. In this paper we proposes a real-time road detection method in urban traffic. Etracting feature-line-pair instead of single feature line maes gives more robust detection result despite degradation and occlusion of lane marings. Woring with a linearl parameterized road model, FLPs detection with EPP tracing ma achieve effectivel. Regression diagnostics technique is used to robustl estimate road parameters from all FLPs, which gives us more accurate and more robust road shape. Eperiments on THMR-V demonstrate robust and real-time performance. Currentl, we are woring on optimizing this method, and etending the technique for detecting more comple road model in more generic urban traffic scene. We will begin more research on robust tracing in image sequence, and improve the real-time performance and robustness in dense traffic. ACKNOWLEDGEMENTS This research was supported in part b Chinese High Technolog Development Program and Portugal-China Science and Technolog Cooperation Project. We wish to than Bin Dong, Qian Yu for their valuable help on this paper. REFERENCES 1. R. Chapuis, A. Potelle, J.L. Brame, et al, Real-time vehicle trajector supervision on the highwa, International Journal of Robotics Research, 14, pp , C. Thorpe, M. Herbert, T. Kanade, et al, Vision and navigation for the Carnegie-mellon navlab, IEEE Transactions on Pattern Analsis and Machine Intelligence, 10, pp , M. Bertozzi, A. Broggi, GOLD: A parallel real-time stereo vision sstem for generic obstacle and lane detection, IEEE Transaction on Image Processing, 7, pp , D. Pomerleau, RALPH: rapidl adapting lateral position handler, IEEE Smposium on Intelligent Vehicles, pp , K.I. Kim, S.Y. Oh, S.W. Kim, et al, An autonomous land vehicle PRV III, in proceedings of the IEEE Intelligent Vehicles Smposium, pp , C.E. Thorpe, Vision and navigation - the Carnegie-Mellon Navlab, Kluwer Academic Publishers, M. Yang, J.Y. Lu, H. Wang, et al, Vision based road following, Chinese Journal of Pattern Recognition and Artificial Intelligence, 14, pp , (In Chinese) 8. F. Guichard, J. -P. Tarel, Curve finder combining perceptual grouping and a alman lie fitting, in International Conference on Computer Vision, pp , Z. Li, et al, Dnamic Image Analsis, China National Defence Industr Press, Z. Zhang, Parameter estimation techniques: a tutorial with application to conic fitting, in Image and Vision Computing Journal, 15, pp , 1997.