Stereo Vision-based Subpixel Level Free Space Boundary Detection Using Modified u-disparity and Preview Dynamic Programming

Transcription

1 2015 IEEE Intelligent Vehicles Symposium (IV) June 28 - July 1, COEX, Seoul, Korea Stereo Vision-base Subpixel Level Free Space Bounary Detection Using Moifie u-isparity an Preview Dynamic Programming Ho Gi Jung, Senior member, IEEE, an Jae Kyu Suhr, Member, IEEE Abstract As free space bounary efine as the bounary between the closest obstacles an a roa surface provies information about rivable space an obstacles, it has been one of the most important research topics in the automotive fiel. This paper proposes a metho enhancing the accuracy of u-isparity-base free space bounary estimation to the subpixel level. The conventional metho etects peak series by exploiting ynamic programming in u-isparity mae by accumulating isparity maps eping on the u coorinate. It assumes that the bigger accumulation value of the u-isparity of a position an the smaller vertical coorinate change between the position an its previous column, the higher is the possibility that the position belongs to peak series. However, as the conventional metho consiers only vertical coorinate change with respect to the previous column, it has a problem that even a straight line will be penalize if it is not seen to be horizontal. To aress this problem, this paper proposes preview ynamic programming, which is moifie to consier the slope change with the previous an next column. Aitionally, to solve the problem that a slope smaller than 45 cannot be represente by integer coorinates, this paper proposes moifie u-isparity, which uses average coorinates weighte by u-isparity values within a fixe size winow. Through qualitative evaluations with an open ataset, it is confirme that the propose metho significantly mitigates the quantization noise of free space bounary estimate by the conventional metho. I. INTRODUCTION Thanks to the great progress of electronic harware an computer vision, application cases of ense stereo vision are increasing in the automotive fiel [1]. As forwar stereo vision provies information to multiple applications such as forwar vehicle etection, peestrian etection, roa curvature preiction, an sensor fusion-base localization, application-wise ata processing will be inefficient as a lot of common operations will be repeate multiple times. To solve this problem, meium level representation has been evelope [2] that processes common operations an provies the results to multiple application systems. It inclues an elevation map [3], occupancy gri [4], an stixel, which is being researche most actively. Stixel represents the forwar scene with sticks having a fixe with an consists of sequential phases such as roa profile estimation, free space bounary estimation, obstacle height estimation, static stixel generation, an ynamic stixel generation [2]. Roa profile estimation, assuming the H. G. Jung is with the Department of Automotive Engineering, Hanyang University, Seoul, Korea ( hogijung@hanyang.ac.kr). J. K. Suhr is with the Research Institute of Automotive Control an Electronics, Hanyang University, Seoul, Korea. horizontal height variation of a roa is relatively small, estimates the vertical height of the roa in the riving irection. It coul be implemente by etecting peak series in v-isparity. Free space bounary estimation etects the bounary between the roa surface an obstacles. It coul be implemente by etecting peak series in u-isparity. Obstacle height estimation etects the upper bounaries of obstacles an coul be implemente by a membership voting metho assuming that the isparity values of an obstacle in the vertical irection shoul be almost the same. Base on the etecte free space bounary an obstacle height, static stixel generation constructs stixels by bining areas of a fixe with an assigning their representative isparity values to them. Dynamic stixel generation as motion-relate information to the static stixels. u-isparity-base free space bounary estimation exploits two assumptions. First, as isparity values of a sie surface of obstacles, which is expecte to be vertical to the groun surface, are almost the same, they will form a peak in u-isparity, which consists of isparity histograms of each u coorinate. Secon, as the free space bounary of consecutive stixels is smooth, the vertical coorinate change of the peaks will be small. Generally, ynamic programming (DP) reflecting these two assumptions on the cost is use to etect peak series in the u-isparity. To etect the secon obstacle behin the first obstacle contacting the free space, multi-layer stixel is also propose [5]. It assumes that the vertical profile of each column has a shape consisting of an oblique line, a vertical line an another vertical line corresponing to the roa surface, the first obstacle, an the secon obstacle, respectively. This paper starts from a critical min that, in conventional u-isparity-base free space bounary estimation, the smoothness term of DP penalizes even a straight obstacle bounary if it is not seen as horizontal. As a solution, this paper proposes that a slope change in the consecutive three columns shoul be consiere instea of a vertical coorinate change (or Z-epth change) in the consecutive two columns in -u space (or u-isparity), of which the horizontal an vertical axis is about the u coorinate an isparity inex, respectively. To implement the iea, this paper proposes a preview DP which simultaneously consiers the previous an next columns of the current position. Aitionally, even when three columns are simultaneously consiere, if positions in u-isparity are represente by integer coorinates, the slope of the peak series cannot be smaller than 45 an the improvement by the preview DP is limite. To aress this problem, this paper proposes moifie u-isparity which uses the center of gravity (CG) of a winow with a fixe size in u-isparity as the new coorinate. It mitigates the quantization /15/$ IEEE 449

2 % Forwar Phase C U,1,1 M 1,1 for u=2:u max min C C U p p, u 1 p1: max arg min M C U p p, u 1 p1: max Figure 1. Principle of the conventional metho. noise cause by integer coorinates in u-isparity an enhances the accuracy of the etecte free space bounary to the subpixel level. II. CONVENTIONAL METHOD Conventional u-isparity-base free space bounary estimation [2] exploits two assumptions. First, as the camera is installe parallel to the groun surface an the sie surfaces of obstacles are perpicular to it, pixels in a column within an obstacle are suppose to have almost the same isparity value. That is, an obstacle will form a peak series in u-isparity, which represents the pixel number having a specific isparity value for each column. Secon, as an obstacle has a smooth bounary, isparity values of the peak series in consecutive columns are suppose to have a small change. The conventional metho exploits DP to etect the peak series in u-isparity. Positions in the input image or XZ plane (X: lateral irection, Z: longituinal irection) corresponing to the peak series is referre to as the free space bounary as it is the bounary between the rivable space an obstacles. DP is one of the representative methos which can eterministically solve a problem efine recursively [6]. To hanle the problem at han, it uses the u-isparity value multiplie by a minus sign as its ata term an the absolute change of isparity inexes (or corresponing Z-epths) as its smoothness term, respectively. The vertical axis is assume to be about the isparity inex instea of isparity because the isparity can be treate in the resolution less than 1 pixel. Fig. 1 shows the principle of the conventional metho. The vertical an horizontal otte lines epict the u coorinates an isparity inexes in u-isparity, respectively. u-isparity U,u enotes the number of pixels whose isparity inex is in the u-th column. As the larger value means a higher possibility that the position (,u) belongs to the peak series, ata term -U,u is ae to the cost of paths passing through the position. The absolute change of vertical coorinates between two positions, (p,u-1) in the previous column an (,u) in the current column, measures the iscontinuity of a path passing through the two positions. As the smaller value means the higher possibility, the smoothness term % Backwar Phase s u arg m in C m ax 1: for u=u max -1:-1:1 s M u su 1, u 1 m ax Figure 2. MATLAB-like pseuo coe of the conventional metho. Figure 3. Free space bounary example estimate by the conventional metho. p (1) is ae to the cost of the path passing through the two positions, (p,u-1) an (,u). Fig. 2 shows the MATLAB-like pseuo coe of the conventional metho. C,u an M,u enotes the optimal cost an the previous isparity inex corresponing to it when the position (,u) belongs to the peak series. During the forwar phase, as the optimal path up to the previous column is suppose to be known the optimal path of the current position (,u) can be set to the previous isparity inex p which minimizes the sum of the optimal cost up to the previous column C p,u-1 an weighte sum of ata term -U,u an smoothness term -p. Where, ω enotes a weighting factor controlling the contribution of the ata term an smoothness term. In the backwar phase, starting from the position having the minimum cost in the last column, isparity inex S u of free space bounary in each column can be etermine by recursively following the optimal previous isparity inex M,u. Fig. 3 shows an example of free space bounary estimate by the conventional metho. 450

3 Figure 5. Examples of moifie u-isparity. Figure 4. The effect of quantization noise in u-isparity: an oblique line with a graual slope is represente by a combination of horizontal an 45 -slope lines. III. PROPOSED METHOD The conventional metho has two rawbacks. First, if the smoothness term of DP is efine as the absolute change of isparity inexes, it oes not properly reflect the assumption that the bounary of an obstacle shoul be smooth. That is, if a vehicle is locate in the riving irection as shown in Fig. 4, its sie surface will appear as an oblique line in u-isparity an will increase cost by the absolute change of isparity inexes. In other wors, even when an obstacle bounary is a straight line, it will increase cost by a ifferent amount accoring to its slope in u-isparity. Secon, as isparity inex an u coorinate is represente by an integer, an oblique line with a graual slope is represente by a combination of horizontal an 45 -slope lines. It is because, as shown in Fig. 1, the smallest slope of a path between two positions in consecutive columns is 45 except 0. As even a small ifference in u-isparity makes a significant ifference in the XZ plane if the position is far away, such a ifference, or quantization error, can isturb accurate estimation of the rivable space. Aitionally, when stixel motion is estimate by matching stixels of the consecutive two images [7], although its effect on the magnitue of motion coul be ignorable, its effect on the irection of motion shoul be seriously consiere because a relatively stable obstacle might have unstable coorinate on the bounary of the rouning operation. Consequently, it coul egrae the stixel clustering-base stixel segmentation [8] as the irection of motion is an important feature in the clustering. To aress these problems, this paper proposes a preview DP exploiting slope change as the smoothness term an moifie u-isparity approximating subpixel-level coorinate. To mitigate the effect of the quantization error cause by representing a position in u-isparity with integer coorinates, this paper proposes to use moifie u coorinate u an moifie isparity inex, corresponing to the CG of a winow with a fixe size in u-isparity, instea of u coorinate an isparity inex as new coorinates. They are referre to as moifie u-isparity. u an is efine as in (2) an (3), respectively: average u an weighte by u-isparity value of all positions within a winow which is centere on a position (,u) in u-isparity an whose sie length is 2W+1. Where, the resultant value is restricte to the bounary of rouning operation. W u W u U j i W j u W u m in m ax, u 0.5, u 0.5 W u W U j i W j u W W u W U j i W j u W m in m ax, 0.5, 0.5 W u W U j i W j u W Fig. 5 shows an example of moifie u-isparity efine in this way. The horizontal an vertical axis is about the u coorinate an isparity inex, respectively, an the area epicte by a square correspons to a position in u-isparity. Notice that the figure enlarges a portion of u-isparity rather than the input image. The intensity epicts the u-isparity value: the arker the area the higher the accumulation. If a position of u-isparity (,u), corresponing to the center of the square, is represente by integer coorinates, it is har to properly represent an oblique peak series with a graual slope as shown in Fig. 5. The peak series will be epicte by a combination of horizontal an 45 -slope lines, which inevitably inclues significant quantization noise. The re ots in Fig. 5 epict positions whose horizontal an vertical coorinate is u an, respectively. In this case, W=2. Three ots esignate by arrows show that the moifie u-isparity can represent the path of the peak series more properly. The moifie u-isparity seems to approximate the true peak position as a general peak in u-isparity is istribute over a wie area of u-isparity an the CG is the best estimator. In (2) an (3), moifie u-isparity is restricte to the bounary of the rouning operation. It is because the resultant value is suppose to lose the meaning of the coorinate of a position in u-isparity when it crosses over the bounary. Moifie u-isparities locate above an below those esignate by arrows show examples of this case. We can recognize that the moifie u-isparities in the positions where they are going to enter properly represent the peak series. Therefore, new coorinates crossing over the rouning operation oes not seem to be require. This paper proposes to use the absolute change of slope as a smoothness term instea of the absolute change of isparity inex such that a straight line of the obstacle bounary will not (2) (3) 451

4 % Forwar Phase for n=1: max C U,1, n,1 M 1,1, n increase the smoothness term. As the conventional DP cannot implement such types of smoothness term, this paper proposes a variation of DP consiering also the next column, which will be referre to as the preview DP. Fig. 6 shows the principle of the propose metho. Like Fig. 1, the vertical an horizontal otte lines epict u coorinates an isparity inexes in u-isparity, respectively. u-isparity U,u enotes the u-isparity value of a position (,u) in u-isparity. In Fig. 1, only the previous position (p,u-1) is consiere, but in Fig. 6 the next position (n,u+1) is also consiere. The propose metho exploits the absolute change of slopes of two lines, one from a position of the previous column to the current position an the other from the current position to a position in the next column, n, u 1 u u u u p, u 1 n, u 1 p, u 1 as the smoothness term. Figure 6. Principle of the propose metho. Fig. 7 shows the MATLAB-like pseuo coe of the propose metho. C,u,n an M,u,n enotes the optimal cost an the previous isparity inex corresponing to it assuming that a position (,u) is connecte to the position (n,u+1) in the next column. Meanwhile, as the propose metho oes not consier only the previous column but also the next column, it cannot etermine the optimal path passing through the current position using only information until the current position. We notice that, if the next position is assume, the optimal previous position (p,u-1) passing through the current position an the assume next position can be etermine. That is, although the next position of the optimal path cannot be etermine on the current position, the optimal path for each next position is eterministic. On the current position, the optimal path for each next position passing through the current position is recore, an the ecision is postpone to the next column. Overall structures of the forwar an backwar phase are almost the same as those of the conventional metho. The major ifference is that C,u,n an M,u,n for each potential next position n are calculate in the main loop of the forwar phase. Where, notice the meaning of the term C p,u-1, : among all paths passing through a position (p,u-1) in the previous column, (4) for u=1:u max for n=1: max C C U m in n, u 1 p, u 1, n p, u 1, p1: u u u u max n, u 1 p, u 1 only the path assuming the current position (,u) as the next position is consiere. Fig. 8. shows an example of free space bounary estimate by the conventional an propose methos. The blue an re soli line epicts the results of the conventional an propose methos, respectively. The results of the propose metho is similar to those of the conventional metho in a large scale. However, when enlarging the lower right part (area marke by a otte rectangle) of the image, we can recognize that the propose metho reuces most parts of the quantization noise of the conventional metho. IV. EXPERIMENTS This paper qualitatively evaluates the propose metho by applying it to the Daimler Urban Scene Segmentation Benchmark Dataset [9]. The use ataset consists of 500 stereo image pairs with corresponing ense isparity maps. The isparity maps are obtaine by semi-global matching stereo [10] at a resolution of an ownloae from the ataset homepage [11]. To the best of our knowlege, n, u 1 M, n arg m in C U p, u 1, u u u u p1: max m in C C U,1 p, u 1, m ax m ax m ax p1: arg m in M C U m ax,1 p, u m ax 1, p1: % Backwar Phase s u arg m in C u s m ax 1: M u m ax 1 su, u m ax m ax,1 for u=u max -2:-1:1 s M u su1, u 1, su2, m ax,1 p, u 1 n, u 1 p, u 1 Figure 7. MATLAB-like pseuo coe of the propose metho. 452

5 (a) left input image Figure 8. Comparison of free space bounary estimate by the conventional an propose metho. (b) isparity map there has been no previous research estimating the free space bounary in subpixel level accuracy. Therefore, there is no preceent about how to input groun truth in subpixel level accuracy an what kin of measures are proper for quantitative evaluation. We hope we can evise a reasonable quantitative evaluation metho in the near future an share it. It is inclue in the further stuy in the conclusion. Fig. 9 shows the first example of the experimental results. Figs. 9(a) an (b) show the left input image an isparity map, respectively; (c) shows the u-isparity an free space bounary estimate by the propose metho; an () epicts the free space bounary estimate by the conventional an propose metho on the left input image. The blue an re soli line shows the results of the conventional an propose methos, respectively. Where, free space bounary estimate in u-isparity is projecte onto the left input image exploiting the vertical roa profile, which is estimate by our metho exploiting Hough transform an DP [12]. As mentione previously, the propose metho shows similar results to the conventional metho in a large scale. To verify the effect of the propose metho, the right sie of the secon preceing vehicle in Fig. 9 is enlarge. Fig. 10(a) is the result when the u-isparity of Fig. 9(c) is enlarge an Fig. 10(b) is the result when the input image of Fig. 9() is enlarge. Although quantization noise remains because of the resolution of the u-isparity, we can recognize that the propose metho can represent oblique lines more properly than the conventional metho. In particular, in the portion where the peak series is almost horizontal, we can recognize that the proper slope is estimate exploiting the surrouning information. Fig. 11 shows the secon example of the experimental results. Fig. 11(a) shows the free space bounary estimate by the conventional an propose methos. Figs. 11(b) an (c) enlarge the u-isparity an input image corresponing to the yellow otte rectangle in (a), respectively. We can recognize (c) u-isparity an free space bounary estimate by the propose metho () free space bounary estimate by the conventional (blue line) an propose metho(re line). Figure 9. Experimental result example 1. (a) enlarge u-isparity (b) enlarge input image Fig. 10. Enlarge view of the right sie of the secon preceing vehicle in Fig

6 (a) free space bounaries estimate by the conventional (blue) an propose metho (re) in the input image. evaluation is conucte. As the propose metho is just a potential iea, we think a lot of aitional research is require. Further stuy inclues the following topics: 1) the input metho of the groun truth of the free space bounary in subpixel level accuracy, 2) mathematical valiation of the moifie u-isparity, 3) quantitative evaluation of the free space bounary in the XZ plane exploiting range sensor ata, 4) quantitative evaluation of the improvement of stixel matching-base stixel motion estimation, an 5) algorithm improvement for the reuction of execution time or harware implementation using a graphics processing unit (GPU) or fiel programmable gate array (FPGA). (b) enlarge u-isparity corresponing to the yellow otte rectangle (c) enlarge input image corresponing to the yellow otte rectangle Figure 11. Experimental result example 2. that the portion represente by a combination of horizontal an 45 -slope lines is represente by lines with a graual slope. V. CONCLUSION This paper proposes moifie u-isparity an preview DP to estimate the free space bounary in the accuracy of subpixel level. By qualitative evaluations using an open ataset, it is confirme that the propose metho can significantly mitigate the problem of the conventional metho. We expect that such an effect will improve rivable space estimation in the XZ plane, motion estimation by stixel matching, an stixel segmentation using the motion. However, as there is no preceent as to how to input the groun truth of the free space bounary in subpixel level accuracy, the propose metho cannot be quantitatively evaluate. Instea, only qualitative REFERENCES [1] J. Ziegler, P. Ber, M. Schreiber, H. Lategahn, T. Strauss, C. Stiller, T. Dang, U. Franke, N. Appenrot, C. G. Keller, E. Kaus, R. G. Herrtwich, C. Rabe, D. Pfeiffer, F. Linner, F. Stein, F. Erbs, M. Enzweiler, C. Knoppel, J. Hipp, M. Haueis, M. Trepte, C. Brenk, A. Tamke, M. Ghanaat, M. Braun, A. Joos, H. Fritz, H. Mock, M. Hein an E. Zeeb, Making Bertha Drive? An Autonomous Journey on a Historic Route, IEEE Intelligent Transportation Systems Magazine, Vol.6, No.2, pp.8-20, summer [2] Davi Pfeiffer an Uwe Franke, Moeling Dynamic 3D Environments by Means of The Stixel Worl, IEEE Intelligent Transportation Systems Magazine, vol. 3, no. 3, pp , fall [3] Florin Oniga, an Sergiu Neevsch Processing Dense Stereo Data Using Elevation Maps: Roa Surface, Traffic Isle, an Obstacle Detection, IEEE Transactions on Vehicular Technology, Vol.59, No.3, pp , Mar [4] Thien-Nghia Nguyen, Bern Michaelis, Ayoub Al-Hama Michael Tornow an Marc-Michael Meinecke, Stereo-Camera-Base Urban Environment Perception Using Occupancy Gri an Object Tracking, IEEE Transactions on Intelligent Transportation Systems, Vol.13, No.1, pp , Mar [5] Davi Pfeiffer, Uwe Franke, Towars a Global Optimal Multi-Layer Stixel Representation of Dense 3D Data, Proceeings of the British Machine Vision Conference, University of Dunee, 29 Aug. 2 Sep. 2011, pp [6] C. H. Papaimitriou an U. V. Vaziran 6. Dynamic programming, Algorithms, July 18, 2006, pp [7] Bertan Günyel, Rorigo Benenson, Rau Timofte, an Luc Van Gool, Stixels Motion Estimation without Optical Flow Computation, 12th European Conference on Computer Vision, Firenze, Italy, 7-13 Oct. 2012, Lecture Notes in Computer Science, vol. 7577, pp [8] Frierich Erbs, Beate Schwarz, Uwe Franke, Stixmentation Probabilistic Stixel base Traffic Scene Labelling, Proceeings of the British Machine Vision Conference, University of Surrey, 3-7 Sep. 2012, pp [9] Timo Scharwächter, Markus Enzweiler, Uwe Franke, Stefan Roth, Efficient Multi-cue Scene Segmentation, 35th German Conference on Pattern Recognition (GCPR 2013), Saarbrücker, Germany, 3-6 Sep. 2013, Lecture Notes in Computer Science, vol. 8142, pp [10] Hirschmuller, H., Stereo Processing by Semi-global Matching an Mutual Information, IEEE Transactions on Pattern Analysis an Machine Intelligence, vol. 30, no. 2, pp , Feb [11] Daimler Urban Scene Segmentation Benchmark Dataset, available at accesse on 31 Dec [12] Jae kyu Suhr, Ho Gi Jung, Dense Stereo-base Robust Vertical Roa Profile Estimation Using Hough Transform an Dynamic Programming, IEEE Transactions on Intelligent Transportation Systems, online publishe on 4 Dec