Noise-resilient Road Surface and Free Space Estimation Using Dense Stereo

Transcription

1 2013 IEEE Intelligent Vehicles Symposium (IV) June 23-26, 2013, Gold Coast, Australia Noise-resilient Road Surface and Free Space Estimation Using Dense Stereo Jae Kyu Suhr, Member, IEEE, and Ho Gi Jung, Senior Member, IEEE Abstract This paper proposes a noise-resilient road surface and free space estimation method using dense stereo. The proposed road surface estimation method selects 3D points expected to compose a road surface using YZ-plane accumulation, and then finds the road surface by sequentially estimating a piece-wise linear function based on a RANSAC framework. This makes our method insensitive to 3D points on obstacles and stereo matching errors on textureless road regions. The proposed free space estimation method is based on the fact that disparities from roads and obstacles should be equal at the free space boundary. This method calculates disparity consistency between road and obstacle surfaces, and finds free space that gives the best disparity consistency and depth smoothness using dynamic programming. This approach achieves robustness against stereo matching errors on obstacle surfaces and objects located in the air since its estimation process is independent of disparity accumulation unlike previous occupancy grid-based method. The experimental results show that the proposed method is able to estimate road surfaces and free spaces in various severe situations. I. INTRODUCTION Stereo cameras have been widely used for driving environment analysis since it can provide both image and depth information. A system for this task typically starts with the extraction of the road surface and free space boundary. Road surface estimation is used for pitch angle compensation [1] and improving the accuracy of obstacle detection [2], and free space estimation is utilized for vehicle navigation [3] and region of interest selection for pedestrian [4] and vehicle detection [5]. Road surfaces have been estimated in the v-disparity domain by using linear [6] or piece-wise linear functions [1]. Since v-disparity has a drawback in that the resolution decreases with increasing distance [2], many methods that estimate road surface in the YZ-plane domain using multivariate polynomial [7] and B-spline functions [2] have been proposed. Even though the models that have high degrees of freedom are able to describe complex road surfaces, their performances could be affected by 3D points produced by obstacles and stereo matching errors on textureless road regions. To alleviate this drawback, this paper proposes a noise-resilient road surface estimation method. This method first samples the 3D points expected to compose a road surface to reduce the effect of 3D points produced from obstacle surfaces. Then, a Random Sample Consensus (RANSAC)-based line estimator is applied to the sampled points for robust estimation of the piece-wise linear function. To achieve robustness against situations where stereo matching partially fails on textureless road surfaces, the interval of the piece-wise linear function is adaptively changed according to the number of inliers found by RANSAC. Finally, the location of the last inlier is determined as a limit of the reliable road surface. Free space has been computed using inverse perspective mapping [8] and disparity space image [9] without generating a dense disparity map to reduce computational cost. As efficient stereo matching algorithms and their hardware implementations have achieved real-time processing [10], dense disparity map-based methods have been suggested [7], [10]. Among them, a polar occupancy grid-based method [10] has been widely used in various applications [5], [11]. This method works properly with well-estimated disparity maps, but it could produce unstable results when stereo matching fails on obstacle surfaces and objects (e.g. tree branches) are located higher up. This is because this approach depends on disparity accumulation. To overcome this problem, this paper proposes a disparity consistency-based free space estimation method utilizing the fact that disparities from roads and obstacles should be equal at the free space boundary. This method measures disparity consistency between road and obstacle surfaces, and determines an optimal path that gives the best disparity consistency and depth smoothness as a free space using dynamic programming. This approach is robust against stereo matching errors because it does not depend on the number of correctly matched disparity pixels. Since this measure is calculated on obstacle areas that are attached to the road surface, its performance is not affected by objects that are higher up. In experiments, the proposed method was tested using a publicly available stereo database [12] produced under urban situations. The results show that this method reliably estimates both road surface and free space even in various complex road and obstacle environments, and in particular, achieves better free space estimation results compared with the previous occupancy grid-based method in [10]. * This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology ( ). J. K. Suhr is with the Research Institute of Automotive Electronics and Control at the Department of Automotive Engineering, Hanyang University, Seoul, Korea. H. G. Jung is with the Department of Automotive Engineering, Hanyang University, Seoul, Korea ( hogijung@hanyang.ac.kr). II. DENSE STEREO MATCHING This paper utilizes a dense disparity map for road surfaces and free space estimation. Although a dense disparity map requires a high computational cost, its real-time embedded implementation is available thanks to recent research on hard-wired stereo matching. For instance, Daimler /13/$ IEEE 461

2 (a) (a) Figure 1. (a) Left image acquired by a stereo camera. Dense disparity map calculated by SGM. Stereo matching fails in a large portion of road surfaces due to strong reflections. implemented semiglobal matching (SGM) using a field-programmable gate array (FPGA) and achieved a frame rate of 25Hz [10]. For this paper, dense disparity maps are calculated using an SGM algorithm as implemented in [13]. Figs. 1(a) and show a left image acquired by a stereo camera and a dense disparity map calculated by SGM, respectively. It can be seen that stereo matching fails in a large portion of road surfaces due to strong reflections even though SGM is one of the most reliable stereo matching algorithms. III. ROAD SURFACE ESTIMATION A. Point Sampling One of the major factors that hinder road surface estimation is 3D points produced from obstacle surfaces. This may not be a major problem in highway situations where wide road regions and few obstacles are present. However, it becomes a serious factor when dealing with urban environments that include narrow roads and various obstacles. To prevent 3D points on obstacles from disturbing road surface estimation, we utilize a sampling method that selects the 3D points that the road surface is expected to be composed of. The proposed sampling method first calculates the 3D positions of all pixels in a dense disparity map using parameters estimated during the stereo camera calibration procedure. Then, all 3D points are accumulated on the YZ-plane. Fig. 2(a) shows an accumulation result of 3D points produced by the dense disparity map in Fig. 1. This is depicted in a log-scale for visualization purposes. In this figure, the horizontal axis indicates Z-coordinates from 0m (left) to 100m (right), and the vertical axis indicates Y-coordinates from -10m (bottom) to +10m (top). The camera is located at (0m, 0m). Bin size is set to 10cm 10cm. In Fig. 2(a), it can be seen that there are many 3D points produced from obstacles such as parked cars and buildings. Once 3D points are accumulated onto the YZ-plane, those points expected to be a part of the road surface are selected. For this sampling task, we select a location that gives an accumulated value larger than a predetermined threshold (T sample ) and the least Y-coordinate for each column (Z-coordinate). Since the accumulated value is inversely proportional to the Z-coordinate, the threshold (T sample ) is adaptively selected as (1). Tsample Z W Z (c) Figure 2. (a) Accumulated 3D points on YZ-plane in a log-scale. Sampled points (yellow dots). (c) Estimated road surface (red line) and inliers (yellow dots). Horizontal axis indicates Z-coordinate from 0m (left) to 100m (right), and vertical axis indicates Y-coordinate from -10m (bottom) to +10m (top). f (1) where W and f indicate a minimum width of the valid road surface and the focal length in terms of pixel dimensions, respectively. W is set to 0.5m, which means that this method samples a location accumulated by properly matched 0.5m road surfaces. The minimum value of (T sample ) is set to 10 to discard erroneous locations far away from the camera. Fig. 2 shows a sampling result of Fig. 2(a). In this figure, yellow dots indicate the locations sampled by the proposed approach. It can easily be noticed that most sampled locations are from road surfaces. Although this sampling method is similar to the method in [14] that selects a location of maximum accumulated value for each column, the proposed method produced better performance in our experiments. B. Piece-wise Linear Function Estimation Since real world roads usually include non-flat surfaces, it is more appropriate to model them using multivariate polynomial and B-spline functions rather than a simple straight line. However, the road surface estimation performances of these functions could be affected by stereo matching errors due to their high degrees of freedom. For instance, a B-spline function with high degrees of freedom could give an unexpected fitting result in a range where there is no or are few sampled points in the middle of Fig. 2. Thus, this paper models a road surface with a piece-wise linear function that consists of multiple lines since each line is defined by only one or two parameters and robust estimators can easily be applied. A robust estimator (i.e. RANSAC [15]) is sequentially applied to estimate multiple lines that compose a piece-wise linear function since the sampling result includes points from both road surfaces and obstacles as shown in Fig. 2. When estimating a line in each interval, the line estimation interval and slope constraint are adaptively adjusted according to the number of inliers counted by RANSAC. In other words, if the number of inliers is less than a predetermined value (MIN_INLIER), the estimated line is discarded and a line is 462

3 ALGORITHM I. PIECE-WISE LINEAR FUNCTION ESTIMATION Z = 0; INTERVAL = DEFAULT_INTERVAL; ANGLE = DEFAULT_ANGLE; while Z+INTERVAL <= MAX_DISTANCE end Select points between Z and Z+INTERVAL; Estimate a line using RANSAC with ANGLE; If number of inliers >= MIN_INLIER else end Save the estimated line parameters; Z = Z+ INTERVAL; INTERVAL = DEFAULT_INTERVAL; ANGLE = DEFAULT_ANGLE; INTERVAL = INTERVAL + DEFAULT_INTERVAL; ANGLE = ANGLE + DEFAULT_ANGLE; Select the farthest inlier as the road end; re-estimated by increasing the line estimation interval (INTERVAL) and slope constraint (ANGLE). This procedure is used to achieve robustness against situations where stereo matching fails on textureless or reflective road regions as shown in Fig. 1(a). Finally, the location of the last inlier is determined as a limit of reliable road surface. In this paper, INTERVAL, ANGLE, and MIN_INLIER are set to 5m, 1, and 50% of the bin number within INTERVAL, respectively. This procedure is summarized in Algorithm I. Fig. 2(c) shows the road surface estimation result. In this figure, a red line and yellow dots indicate the estimated road surface and sampled points determined as inliers, respectively. It can be seen that a road surface is properly estimated despite points from obstacles and ranges with no sampled points. The estimated road surface ends at the location of the farthest inlier. IV. FREE SPACE ESTIMATION A. Cost Matrix Calculation The proposed free space estimation method utilizes a disparity consistency (DC) to measure how a certain location is likely to be a free space boundary. DC is calculated by measuring the absolute difference between the disparity of the road and that of the obstacle. This is based on the fact that the disparity of the road surface (d) and potential obstacle disparity (POD(u,d)) at the free space boundary should be equal by assuming that obstacles stand perpendicularly on the road surface. In other words, DC should be zero at the free space boundary in an identical case. This can be described as (2).,,,, : DC u d d POD u d where POD u d median D u v d h d v d where u and d are the horizontal coordinates of the image and (2) disparity value, respectively, and POD(u,d) indicates a potential obstacle disparity at (u,d). A function, POD(u,d), is calculated by a median operator from the disparity map (D) within an obstacle region whose horizontal location is determined at u and the vertical range is determined from v(d)-h(d) to v(d). v(d) indicates a road surface function which links between disparity (d) and the vertical coordinates of the image (v). This function is estimated during the road surface estimation procedure. h(d) is a height of the obstacle in terms of pixel dimensions that depends on the disparity value (d) and can be calculated as (3). h d (a) Figure 3. (a) Obstacle regions at various image locations. Calculated cost matrix. Horizontal and vertical axes of cost matrix are horizontal coordinate of image (u) and disparity (d), respectively. d Hobstacle (3) B where H obstacle indicates a real world obstacle height, and is set to 1.0m. This means that the proposed method is able to detect obstacles approximately taller than 0.5m since the median operator is used to extract potential obstacle disparity. This setting is enough to detect pedestrians, vehicles, and other obstacles that can appear in front of the ego-vehicle. Vertical red lines in Fig. 3(a) show several obstacle regions at various image locations. It can easily be noticed that the heights of obstacle regions are adaptively selected according to distances from the camera. If there is an insufficient number of valid disparity pixels in an obstacle region (e.g. less than 30% of the obstacle region), a default value (=2) is assigned. Fig. 3 shows a cost matrix calculated by (2) at every (u,d) location. Horizontal and vertical axes of the cost matrix are a horizontal coordinate of image (u) and disparity (d), respectively. Compared with the previous occupancy grid-based method in [10], the proposed disparity consistency-based method has the following advantages. The occupancy grid-based method estimates free space using accumulated values. Thus, it could produce unstable results when an insufficient number of disparities are accumulated due to stereo matching errors caused by reflective vehicle surfaces and vegetation. However, the proposed method is more robust against the same situation since it relies on a potential obstacle 463

4 disparity value rather than an accumulated value. The median operator used to extract the potential obstacle disparity value makes it even more robust against outliers produced by stereo matching errors. An average operator has been tested for the same purpose, but it gives a less reliable result than the median operator. Furthermore, the proposed method is not affected by obstacles higher up because potential obstacle disparity values are calculated only in the obstacle regions attached to the road surface. Example results on these situations will be presented in the experimental section. B. Free space computation using dynamic programming To find a free space that minimizes the cost matrix with spatial smoothness, dynamic programming [16] is utilized. The cost of dynamic programming consists of three terms: data term L(u,d i ), smoothness term S(d i,d j ), and background term B(d i ) as in (4)., i, j, i i, j i C u d d L u d S d d B d (4) In (4), data term (L(u,d i )) is defined by a cost matrix calculated by DC as in (5). If DC(u,d i ) is larger than T L, L(u,d i ) is assigned by T L which is empirically set to 10. L u, d i,,, DC u di if DC u di TL (5) T, otherwise L The smoothness term (S(d i,d j )) is established in terms of the metric depth distance between d i and d j (dist(d,d j )). If dist(d,d j ) is larger than T s, S(d i,d j ) is set to w s T s. w s and T s are set to 2 and 5m exactly the same as in [10] for better performance comparison. S d, d i j S,,, dist di d j if dist di d j TS w TS, otherwise Background term (B(d i )) is used to select the nearest location at each column (u) when multiple locations have similar costs. This is because free space is a boundary between the road surface and the first obstacle. In the previous occupancy grid-based method, this task is achieved by using the background subtraction procedure which marks all occupied cells behind the first maximum cell as free. Since the threshold used to find the first maximum cell is sensitive to stereo matching error, this paper conducts this procedure as a part of dynamic programming as in (7). B d i d d 1 (6) i max wb (7) dmax where d max indicate the maximum disparity value. w B is empirically set to 0.1. The red lines in Figs. 4(a) and show an optimal path calculated by dynamic programming and the final free space boundary depicted on left image, respectively. It can easily be found that free space is correctly estimated even under serious stereo matching errors. V. EXPERIMENTAL RESULTS The proposed road surface and free space estimation method was tested using a publicly available stereo database (a) Figure 4. (a) Optimal path calculated by dynamic programming. Estimated free space. [12]. This database was acquired by a stereo camera whose baseline length and height are 0.25m and 1.17m, respectively, and consists of pixels rectified grayscale stereo images mostly acquired in urban environments. Figs. 5(a),, and (c) show road surface and free space estimation results of the proposed method in various environments. In these figures, the first, second, and third columns show the disparity map, estimated road surface (lower) with sampled points (upper), and estimated free space (red lines) on the left image, respectively. Fig. 5(a) shows an uphill situation. In this case, road surface estimators that have no mechanism to determine the end of the road tend to wrongly recognize the area beyond the top of the hill as the road surface because obstacles are located not at the top of the hill but far beyond it. However, the proposed method correctly estimated the location of the end of the road by using the farthest inlier point as shown in the second column of Fig. 5(a). This makes the free space end at the location of the top of the hill as shown in the third column of Fig. 5(a). Fig. 5 shows a situation where the road slope severely changes and stereo matching fails in half of the road surface, and Fig. 5(c) shows a situation where only a small portion of the road surface is visible due to vehicles and buildings located near the camera. It can be seen that the proposed method successfully estimates the road surface and free space even in these unfavorable situations. Figs. 6(a),, and (c) show a qualitative performance comparison between the occupancy grid-based method [10] and the proposed method. The method in [10] was implemented by us for this comparison. In these figures, the first, second, and third columns show the disparity map, free space estimated by the occupancy grid-based method, and free space estimated by the proposed method, respectively. In both cases, the road surfaces are estimated by the proposed method. Fig. 6(a) shows a case where the disparity map is fairly well-estimated. In this situation, both methods produce similar and reliable free space estimation results. Figs. 6 and (c) 464

5 (a) (c) Figure 5. Results of road surface and free space estimation. (a) Uphill situation. Road slop changes and stereo matching fails. (c) Small portion of road surface is visible. The first, second, and third columns show disparity map, estimated road surface (lower) with sampled points (upper), and estimated free space (red lines) on the left image, respectively. show situations where severe stereo matching errors are presented on vehicle surfaces and vegetation, respectively. In these cases, the proposed method outperforms the occupancy grid-based method. Unlike the proposed method, the occupancy grid-based method miss-estimates free space in the middle of the road in Fig. 6(c). This is because it wrongly recognizes tree branches located higher up as obstacles. VI. CONCLUSION This paper proposes a road surface and free space estimation method that is robust against various noise factors presented in dense stereo. Point sampling and RANSAC-based sequential calculation of a piece-wise linear function enable a robust estimation of road surface against 3D points on obstacles and stereo matching errors on textureless road regions. The disparity consistency measure based on the median operator makes free space estimation insensitive to stereo matching errors on reflective vehicle surfaces and vegetation and objects located higher up. In the future, we are planning to extend the proposed method by adopting temporal information, and to conduct quantitative performance evaluation. REFERENCES [1] R. Labayrade, D. Aubert, and J.-P. Tarel, Real Time Obstacle Detection in Stereovision on Non Flat Road Geometry Through v-disparity Representation, In Proceedings of the Intelligent Vehicle Symposium, 2002, pp [2] A. Wedel, H. Badino, C. Rabe, H. Loose, U. Franke, and D. Cremers, B-Spline Modeling of Road Surfaces With an Application to Free-Space Estimation, IEEE Transactions on Intelligent Transportation Systems, vol. 10, no. 4, pp , Dec [3] M. Muffert, T. Milbich, D. Pfeiffer, and U. Franke, May I Enter the Roundabout? A Time-To-Contact Computation Based on Stereo-Vision, In Proceedings of the Intelligent Vehicle Symposium, 2012, pp [4] J. K. Suhr, K. M. Kang, and H. G. Jung, Dense stereo-based critical area detection for active pedestrian protection system, Electronics Letters, vol. 48, no. 19, pp , Sep

6 (a) (c) Figure 6. Performance comparison between occupancy grid-based method and proposed method. (a) Disparity map is fairly well-estimated. Stereo matching fails on vehicle surfaces. (c) Stereo matching fails on vegetation and tree branches are located in the air. The first, second, and third columns show disparity map, free space estimated by occupancy grid-based method, and free space estimated by proposed method, respectively. [5] M. Enzweiler, M. Hummel, D. Pfeiffer, and U. Franke, Efficient Stixel-Based Object Recognition, In Proceedings of the Intelligent Vehicle Symposium, 2012, pp [6] R. Benenson, R. Timofte, and L. V. Gool, Stixels estimation without depth map computation, In Proceedings of the International conference on Computer Vision Workshops, 2011, pp [7] S. Nedevschi, R. Danescu, D. Frentiu, T. Marita, F. Oniga, C. Pocol, T. Graf, and R. Schmidt, High accuracy stereovision approach for obstacle detection on non-planar roads, In Proceedings of the Intelligent Engineering Systmes, 2004, pp [8] P. Cerri and P. Grisleri, Free Space Detection on Highways using Time Correlation between Stabilized Sub-pixel precision IPM Images, In Proceedings of the International Conference on Robotics and Automation, 2005, pp [9] S. Kubota, T. Nakano, and Y. Okamoto, A Global Optimization Algorithm for Real-Time On-Board Stereo Obstacle Detection Systems, In Proceedings of the Intelligent Vehicle Symposium, 2007, pp [10] D. Pfeiffer and U. Franke, Modeling Dynamic 3D Environments by Means of The Stixel World, IEEE Interlligent Transportation Systems Magazine, vol. 3, no. 3, pp , Fall [11] F. Erbs, A. Barth, and U. Franke, Moving Vehicle Detection by Optimal Segmentation of the Dynamic Stixel World, In Proceedings of the Intelligent Vehicle Symposium, 2011, pp [12] C. Keller, M. Enzweiler, and D. M. Gavrila, A New Benchmark for Stereo-based Pedestrian Detection, In Proceedings of the Intelligent Vehicle Symposium, 2011, pp [13] OpenCV 2.1, [14] A.D. Sappa, F. Dornaika, D. Ponsa, D. Gerónimo, and A. López, An Efficient Approach to Onboard Stereo Vision System Pose Estimation, IEEE Transactions on Intelligent Transportation Systems, vol. 9, no. 3, pp , Sep [15] M.A. Fischler and R.C. Bolles, Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, vol. 24, no. 6, pp , Jun [16] R. Bellman, Dynamic Programming. New Jersey: Princeton University Press,