A New Stereo Matching Approach for Real-time Road Obstacle Detection for Situations with Deteriorated Visibility

Transcription

1 2008 IEEE Intelligent Vehicles Symposium Eindhoven University of Technology Eindhoven, The Netherlands, June 4-6, 2008 A New Stereo Matching Approach for Real-time Road Obstacle Detection for Situations with Deteriorated Visibility Mohamed El Ansari, Stéphane Mousset, Abdelaziz Bensrhair Abstract This paper presents a fast stereo matching approach for road obstacle detection under foggy weather conditions. The stereo matching process can be treated as the problem of finding an optimal path on a 2D search plane. To obtain this path, we propose a new cost function. This last is derived from the variance values of the intensities on the right hand sides of the matched declivities. The matching process is executed independently for each scanline. In order to reduce the false matches and speed up the matching process, we propose to exploit the relationship between successive stereo images. So, the disparity map computed for one stereo pair will be used to find the disparity range for the next stereo pair. The disparity range is deduced for each scanline. The proposed approach has been tested on synthesized and real images under foggy weather conditions. The new method gives satisfactory results. I. INTRODUCTION Obstacle detection is one of the most active research areas in the field of intelligent transportation systems. Several stereovision-based approaches for obstacle detection have been proposed in the literature [1], [2], [3], [4], [16]. These approaches exploit the stereo-correspondence between features extracted from the left and right images. The key problem in these methods is the matching process, which consists in comparing each feature extracted from one image with a number, generally large, of features extracted from the other image in order to find the corresponding one, if any [5], [6]. Most of the vision-based methods, proposed in the literature, for road obstacle detection fall when the visibility is poor, i.e., foggy weather conditions. Some methods have been proposed to estimate the visibility distance [17], [18]. However, there is no vision-based method for obstacle detection has been presented to operate when the visibility is deteriorated. In this paper, we propose a new fast stereo matching method for road obstacle detection when the visibility is poor. The proposed method is devoted to stereo image sequences. First, the so-called declivities are extracted from the stereo images by the mean of the method described in [9]. Second, the stereo matching problem is considered as a path finding problem in a 2D plane [8], [7] and M. El Ansari is with LabSIV, Department of Mathematics and Computer Science, Faculty of Science, University of Ibn Zohr, Agadir, Morocco. He is also an associate member at the LITIS Lab., INSA-Rouen, France. m.elansari@univ-ibnzohr.ac.ma S. Mousset is with the LITIS Lab., INSA Rouen, B.P. 08, F Mont-Saint-Aignan Cedex, France stephane.mousset@insa-rouen.fr A. Bensrhair is with the LITIS Lab., INSA Rouen, B.P. 08, F Mont-Saint-Aignan Cedex, France Abdelaziz.Bensrhair@insa-rouen.fr performed independently for each scanline. A new cost function is proposed to measure the similarity between pairs of declivities. Third, the novel method computes the possible disparity range for each scanline in order to reduce both the number of false matches and running time. To achieve this, the method takes advantage from the existing relationship between successive stereo images. The new method is tested on virtual sequence available in [13] and gives good results when compared with other methods. In our work, we are interested in obstacle detection under bad weather conditions. That s why we have made the virtual sequence as foggy one and apply the proposed method on. The new approach has been applied also to stereo data acquired under foggy weather conditions and the results obtained illustrate the effectiveness of the method. The remainder of the paper is organized as follows. Section II presents the method used to extract primitives. The new method is describe in section III. Experimental results are shown in section IV. Section V concludes the paper. II. FEATURE EXTRACTION The first step in stereo vision consists in extracting significant features from the stereo images. In our approach, we consider the so-called declivity [11], [9] as the primitive to be matched. In an image line, a declivity is defined as a set of consecutive pixels in an image line whose grey levels are a strictly monotonous function of their positions. As shown in Fig. 1, a declivity is limited by two end-points s i (start point) and e i (end point) which correspond to two consecutive local extrema of grey-level intensity. The greylevel intensities at the end-points are respectively I(s i ) and I(e i ). Each declivity is characterized by it amplitude defined by d i = I(e i ) I(s i ). Relevant declivities are extracted by thresholding these amplitudes. To be self-adaptive, the threshold value is defined by d t = 5.6σ (1) where σ is the standard deviation of the white Gaussian noise component in each image line, which is computed using the cumulative histogram of the absolute value of the gradient [9]. The position of a declivity is computed using the mean position of their points weighted by the gradients squared. ei 1 x=s X i = i [I(x + 1) I(x)] 2 (x + 0.5) ei 1 (2) x=s i [I(x + 1) I(x)] 2 where X i is the position of the declivity on the image line shown in Fig /08/$ IEEE. 355

2 Fig. 1. Characteristic parameters of a declivity. A declivity D is characterized by the following attributes (Fig. 1): its x-coordinate X i in the image line as defined in equation 2. their start point s i, end point e i, and the intensity values at these points. the set of points situated between its end point e i and the start point s i+1 of the following declivity. We name this set as the right hand side of the declivity D. We note these characteristics will be used in the matching algorithm. III. STEREO MATCHING ALGORITHM Let {D l i } i=1..m and {D r j } j=1..n be two sets of declivities ordered according to their coordinates in an arbitrary l right and l left epipolar scanline. M and N are the numbers of the declivities on the left and right sacanlines respectively. The problem of obtaining correspondences between declivities on right and left epipolar scanlines can be solved as a path finding problem on 2D plane [8]. Fig. 2 illustrates this 2D search plane. The vertical lines show the positions of declivities on the left scanline and the horizontal ones show those on the right scanline. We refer to the intersections of those lines as nodes. Nodes in this plane correspond to the stages in dynamic programming where a decision should be made to select an optimal path to that node. Optimal matches are obtained by the selection of the path which corresponds to minimum value of the global cost. The optimal path must goes from the upper left corner S to the lower right corner G monotonically due to the condition on ordering. Because of the nonreversal ordering constraint, starting from S, a path can be extended towards only one of the three directions: east, south, or southeast. A. Matching constraints In order to reject invalid nodes in the search plane, we consider some local constraints. The first one is a geometric constraint [10], resulting from the sensor geometry, which assumes that the declivities Di l and Dr j appearing in the left and right scanlines, respectively, represent possible match only if the constraint X i > X j is satisfied. X i and X j are the positions of Di l and Dr j respectively. The second constraint is the slope constraint, which means that only declivities with the same slope sign are considered to be matched. Fig. 2. 2D search plane. The horizontal axis corresponds to the left scanline and the vertical one corresponds to the right scanline. vertical and horizontal lines are the declivity positions and path selection is done at their intersections. Fig. 3. B. Disparity range Right stereo images of the frames 46 and 47, respectively. Most of the existing approaches use a maximum disparity value for the matching process [5], [7]. This value is considered for the whole image. However, this value can vary from an epipolar line to another. That s why we propose to compute the disparity range for each epipolar line independently. The proposed method exploits the link between the successive frames. Figs. 3 depicts the right stereo images of the frames 46 and 47 of the sequence used in our experiments 1. Let I L (t 1) and I R (t 1) be left and right stereo images acquired at time t 1 with estimated disparity map d(t 1), and I L (t) and I R (t) be left and right images acquired at time t. We would like to define the disparity range d r of the disparity map d(t) at time t based on d(t 1). For this purpose, we need to find for each declivity in the image I L (t 1) (resp. I R (t 1)) its associate one in the image I L (t) (resp. I R (t)). We note that the associate declivity in a frame is not necessary its corresponding one in the next frame. The associate declivities should belong to two curves in the two frames, which represent the projections of the same object into these frames. Let D(t 1) be a declivity in the l epipolar line in the image I(t 1) and X(t 1) be its coordinate. Its associate declivity D(t) in the image I(t) should belongs to the l epipolar line in the image I(t). The declivity D(t) 1 The sequence is available in [13] 356

3 should have its coordinate X(t) in the interval [X(t 1) dx,x(t 1) + dx] on the l epipolar line of the image I(t), where dx is a predetermined threshold. To reduce false associations, we use a slope sign constraint. Two associates declivities should have the same sign of their slopes. Finally, the gradient information is used to select the best associates. As a similarity criterion, the absolute difference of the declivities is used. The association process is done between the declivities of the frames F t 1 = (I L (t 1),I R (t 1)) and F t = (I L (t),i R (t)) to find for each declivity on the first frame its associate one, if any, in the second frame. Once we have the associations between the declivities of the frames F t 1 and F t, and the correspondences between the declivities of the stereo images of the frame F t 1, we can get the correspondences between the declivities of the frame F t, which have their associates ones in the preceding frame. let N be the number of the declivities in the image I R (t), the pre-estimated disparity map d p (t) of the frame F t can be computed as follows. FOR i=1 TO N Dec = D(i); IF associate(dec) exists adec = associate(dec); IF match(adec) exists MaDec=match(aDec); amadec=associate(madec); disparity(dec)=amadec-dec; ENDIF ENDIF ENDFOR The algorithm gives a pre-estimated disparity map pd(t) for the t frame. This disparity map is used in order to get a disparity range for each line to be used for the estimation of d(t). The so-called v-disparity image is constructed based on the method presented in [12]. The processing of the v- disparity image provides geometric content of road scenes. As illustrated in Fig. 4, detected objects in the scene are represented by vertical lines. To determine the possible disparity values, we look for the farthest object (left blue line), the closest object (right blue line), and the road map (red line). These three lines give an idea about the possible disparity values for each scanline. The possible disparity values is the area between the green lines. C. Cost function To fill in the search plane, we propose a new cost function which is defined based on the variance of the intensities of the pixels situated on the right hand sides of the matched declivities. Let Di l and Dr j be two declivities belonging to the same scanline on the left and right images respectively. We denote by S l = {p m } pm =e and l i,..,sl i+1 S r = {p n } pn=e r the sets of pixels situated on the j,..,sr j+1 right hand sides of the declivities Di l and Dr j, respectively. To measure the similarity between the stereo declivities we are interested in the intensity values at the pixels of the sets S l and S r. So, homologous declivities should have the same Fig. 4. V-disparity of the pre-estimated disparity map pd(t). The blue lines represents the farthest and closest objects in the scene. The red line represents the road map. The possible disparity values are delimited by the green lines. intensity values at their right hand sides, i.e., the two right sides represent the projections of the same 3D object into the stereo images. We consider a set S = S l US r formed by the pixels of the two sets S l and S r. Corresponding declivities should have low value of the variance of the intensities of the set S. Consequently, we define the cost function as follows. C(D l i,d r j) = variance(i(s)) (3) where I(S) is the intensity values at the pixels of the set S. IV. EXPERIMENTAL RESULTS In order to evaluate the performance of the proposed approach, it has been applied on virtual and real image sequences. First, we have tested our method on MARS/PRESCAN synthetic stereo images available from the internet [14], [13]. Fig. 6 illustrates the frames 46 and 47 of the synthetic stereo images. The extracted declivities are depicted in Fig. 7. The resulting disparity maps obtained by the new method are shown in Fig. 8. As mentioned before, the new method contains an interesting step which allows the computation of the disparity range (subsection III-B) for each scanline. To show the advantage of the proposed method, particularly, the disparity range computation step, we have applied our method with a disparity range [0,150pixel] for the whole image. The results obtained are depicted in Fig. 9. The improvement dues to the computation of the range disparity for each epipolar line is clear when we compare different disparity maps. We remark clearly that a lot of noised disparity values appeared in the disparity obtained by the last method (without disparity range computation step). To obtain dense disparity map, a simple interpolation is done as follows [7]. For each right epipolar line, every 357

4 Fig. 5. The interpolation procedure. lines, which represent the car and the building on the sides of the road. The road map is represented by the inclined line on the bottom of the v-disparity map. The same in the top-right image, the car and the buildings appeared as noised vertical lines because of the effect of fog. Finally, the four cars together with other objects are depicted as vertical noisy lines in bottom image. The v-disparity approach gives good results when the rad map is plane. To get the positions of the detected objects, Hough transform can be used as detailed in [12]. TABLE I MEAN ABSOLUTE WITH (FULL METHOD) AND WITHOUT DISPARITY RANGE COMPUTATION. frame 46 frame 47 Full method Without disparity range computation declivity is assigned to its disparity value. For points which do not correspond to declivities, disparities are computed as shown in Fig. 5. Figs. 10 and 11 show the resulting dense disparity maps by applying the interpolation to the sparse ones in Figs. 8 and 9, respectively. High disparities are represented by high intensities in the disparity map. We remark clearly the advantages of using the disparity range step in our method. As we know the ground truth of the virtual sequence, we have computed the mean absolute error (MAE) between the estimated disparities with the two methods and real disparities. The MAEs are shown in table I. By analyzing the above table, we conclude the effectiveness of the new method. In order to test the robustness of the proposed approach to foggy weather conditions, we have used the Koschmiedder s model [15] to make the virtual sequence as foggy one (see Fig. 12). Fig. 13 shows the extracted declivities. The results of the matching process are depicted in Figs. 14 and 15 for the new method with and without range disparity computation step, respectively. The interpolated disparity maps are illustrated in Figs. 16 and 17. The results show the effectiveness of the method against foggy conditions. Finally, the proposed method has been tested on real sequence depicted in Fig. 18. The sequence is acquired under foggy weather conditions. The corresponding declivities are shown in Fig. 19. Fig. 20 depicts the computed disparity map for the real sequence. The results are satisfactory when compared to other methods developed in LITIS laboratory 2. The matching process is performed within 80ms and 70ms for the synthetic and real stereo images respectively. The hardware used for the experiments is a Pentium IV 2.53GHZ running under Windows XP. For obstacle detection, the v-disparity [12] image is constructed for the resulting disparity maps. Figs. 21 illustrates the one frame v-disparities computed for the three sequences. In top-left image, we remark the presence of different vertical 2 V. CONCLUSION A real-time stereo matching approach is presented. The matching problem is formulated as a path finding in a 2D search plane. First, some constraints are used to reject the invalid nodes in the search plane. Second, a disparity range is computed for each epipolar line based on both the disparity map of the preceding frame and the v-disparity approach. This step allows more rejection of other invalid matches. Third, a new cost function is defined to measure the similarity between candidate matches. The proposed approach is executed independently for each scanline. The performance of the new method is evaluated for real-time obstacle detection in front of a moving car. The results show the effectiveness of the proposed method in particular when under foggy weather conditions. VI. ACKNOWLEDGMENTS We thank very much the Région Haute Normandie for its financial support. REFERENCES [1] G. Toulminet, M. Bertozzi, S. Mousset, A. bensrhair, and A. Broggi, Vehicle detection by means of stereo vision-based obstacles features extraction and monocular pattern analysis, IEEE Trans. Image Processing, vol. 15, NO. 8, 2006, pp [2] A. Bensrhair, M. Bertozzi, A. Broggi, P. Miché, S. Mousset, and G. Toulminet, A ccoperative approach to vision-based vehicle detection, in Proc. ITSC, Japan, October [3] U. Franke and A. Joos, Real-time stereo vision for urban traffic scene understanding, in IEEE Intelligent Vehicle Symposium, Dearborn, USA, October [4] Q. -T. Luong, J. Weber, D. Koller, and J. Malik, An integrated stereobased approach to automatic vehicle guidance, in Proc. of the 5th Int. Conf. on Computer Vision, Cambridge, MA, USA, June 1995, pp [5] U. R. Dhond and J. K. Aggarwal, Sructure from stereo - A review, IEEE Trans. on Syst. Man and Cybern., vol. 19, 1989, pp [6] S. Barnard and M. Fisher, Computational stereo, ACM Comput. Surveys, vol. 14, 1982, pp [7] A. Bensrhair, P. Miché, and R. Debrie, Fast and automatic stereo vision matching algorithm based on dynamic programming method, Pattern Recognition Letters, vol. 17, 1996, pp [8] Y. Otha and T. Kanade, Stereo by intra- and inter-scanline search using dynamic programming, IEEE Trans. Pattern Anal. Mach. Intell, vol. 7, NO. 2, 1989, pp [9] P. Miché and R. Debrie, Fast and self-adaptive image segmentation using extended declivity, Ann. Télécommun., vol. 50, NO. 3-4, 1995, pp [10] M. Hariti, Y. Ruichek, and A. Koukam, A voting stereo matching method for real-time obstacle detection, in Proc. of the 2003 IEEE Int. Conf. on Robotics & Automation, Taipei, Taiwan, September 14-19, 2003, pp [11] T. Quiguer, P. Miché, and R. Debrie, Segmentation method by self-adaptive thresholding, in Proc. Conf. of Image Analysis and Processing, Como, Italy, 1991, pp

5 [12] R. Labayrade, D. Aubert, and J. P. Tarel, Real time obstacle detection in stereo vision on non flat road geometry through v-disparity representation, In Proc. IEEE Intelligent Vehicle Symposium, Versailles, France, June [13] Stereo data for algorithms evaluation, available online: [14] Wannes van der Mark and Dariu M. Gavrila, Real-time dense stereo for intelligent vehicles, IEEE Trans. Intelligent Transportation Systems, Vol. 7, NO. 1, 2006, pp [15] W. Middleton, Vision through the atmosphere, University of Toronto Press, [16] M. Bertozzi, A. Broggi, and A. Fascioli, Vision-based intelligent vehicles: State of the art and perspectives, Robotics and Autonomous Systems, Vol. 32, 2000, pp [17] N. Hautire, J.-P. Tarel, J. Lavenant, and D. Aubert, Automatic Fog Detection and Estimation of Visibility Distance through use of an Onboard Camera, Machine Vision and Applications Journal, 17(1):8-20, April [18] N. Hautire, R. Labayrade, and D. Aubert, Real-Time Disparity Contrast Combination for Onboard Estimation of the Visibility Distance, IEEE Transactions on Intelligent Transportation Systems, 7(2): , June Fig. 9. Disparity map computed for the frames shown in Fig. 6, when a predetermined maximum disparity value is selected for the whole image. Fig. 10. Disparity maps when applying interpolation to those of Fig. 8. Fig. 6. Virtual left stereo images from the sequence available on the internet (frames nr. 46 and 47). Fig. 11. Disparity maps when applying interpolation to those of Fig. 9. Fig. 7. declivities of the images shown in Fig. 6. Fig. 12. Foggy virtual left stereo images (frames 290 and 291). Fig. 8. Disparity map computed for the frames shown in Fig. 6 by using the full proposed method (Including the disparity range computation step for each scanline). Fig. 13. Extracted declivities for the images shown in Fig

6 Fig. 14. Disparity map computed for the frames shown in Fig. 12 with the full new method. Fig. 20. method. Disparity maps for the real sequence obtained by the proposed Fig. 15. Disparity map computed for the frames shown in Fig. 12, when a predetermined maximum disparity value is selected for the whole image. Fig. 16. Disparity maps when applying interpolation to those of Fig. 14. Fig. 17. Disparity maps when applying interpolation to those of Fig. 15. Fig. 18. Real sequence acquired under foggy weather conditions by the stereo vision system of the LITIS Laboratory.. Fig. 21. V-disparity of the three sequences. Fig. 19. Declivity images of the images in Fig