A probabilistic distribution approach for the classification of urban roads in complex environments

Transcription

1 A probabilistic distribution approach for the classification of urban roads in complex environments Giovani Bernardes Vitor 1,2, Alessandro Corrêa Victorino 1, Janito Vaqueiro Ferreira 2 1 Automatique, Systèmes Embarqués, Robotique (ASER) - Heudiasyc Technology of Information and Systems Université de Technologie de Compiègne - UTC - France 2 Autonomous Mobility Laboratory (LMA) Department of Computational Mechanics Mechanical Engineering - UNICAMP - Brazil 1 / 23

2 Summary 1 Introduction / 23

3 Motivation Challenges Objective Motivation Detecting the road area ahead of a vehicle, as it appears in inner-city... is central to modern driver assistance systems; it could improve interlinked or dependent tasks as path planning; Road Following and Visual Servoing. 3 / 23

4 Motivation Challenges Objective Motivation Detecting the road area ahead of a vehicle, as it appears in inner-city... is central to modern driver assistance systems; it could improve interlinked or dependent tasks as path planning; Road Following and Visual Servoing. 3 / 23

5 Motivation Challenges Objective Motivation Detecting the road area ahead of a vehicle, as it appears in inner-city... is central to modern driver assistance systems; it could improve interlinked or dependent tasks as path planning; Road Following and Visual Servoing. 3 / 23

6 Motivation Challenges Objective Motivation Detecting the road area ahead of a vehicle, as it appears in inner-city... is central to modern driver assistance systems; it could improve interlinked or dependent tasks as path planning; Road Following and Visual Servoing. 3 / 23

7 Motivation Challenges Objective Motivation Detecting the road area ahead of a vehicle, as it appears in inner-city... is central to modern driver assistance systems; it could improve interlinked or dependent tasks as path planning; Road Following and Visual Servoing. 3 / 23

8 Motivation Challenges Objective Challenges Applications for road detection using camera sensors must deal with a set of problems such as: Continuously changing backgrounds in different environments (inner-city, highway, off-road); Different road types (shape and color); The presence of different objects (signs, vehicles, pedestrian); differences in imaging conditions (variation of illumination and weather conditions). 4 / 23

9 Motivation Challenges Objective Challenges Applications for road detection using camera sensors must deal with a set of problems such as: Continuously changing backgrounds in different environments (inner-city, highway, off-road); Different road types (shape and color); The presence of different objects (signs, vehicles, pedestrian); differences in imaging conditions (variation of illumination and weather conditions). 4 / 23

10 Motivation Challenges Objective Challenges Applications for road detection using camera sensors must deal with a set of problems such as: Continuously changing backgrounds in different environments (inner-city, highway, off-road); Different road types (shape and color); The presence of different objects (signs, vehicles, pedestrian); differences in imaging conditions (variation of illumination and weather conditions). 4 / 23

11 Motivation Challenges Objective Challenges Applications for road detection using camera sensors must deal with a set of problems such as: Continuously changing backgrounds in different environments (inner-city, highway, off-road); Different road types (shape and color); The presence of different objects (signs, vehicles, pedestrian); differences in imaging conditions (variation of illumination and weather conditions). 4 / 23

12 Motivation Challenges Objective Challenges Applications for road detection using camera sensors must deal with a set of problems such as: Continuously changing backgrounds in different environments (inner-city, highway, off-road); Different road types (shape and color); The presence of different objects (signs, vehicles, pedestrian); differences in imaging conditions (variation of illumination and weather conditions). 4 / 23

13 Motivation Challenges Objective Objective The main objective is the development of a robust system based on visual perception to achieve the detection of roads in challenging inner-city environments. 5 / 23

14 Segmentation Method Texton Maps Dispton Maps 1 Introduction / 23

15 Introduction Segmentation Method Texton Maps Dispton Maps A. Segmentation Watershed Transform Image Capture B. Texton Maps Filter-Bank Texton Color Texton HoG Texton LocationTexton Probabilistic Distribution using Joint Boosting Classifier C. Dispton Maps U-Dispton V-Dispton 7 / 23

16 Segmentation Method Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [1]: Conversion to grayscale ; Morphological Gradient Adjusted; Morpho. Reconstr. Area Close; Morpho. Reconstr. Hmin; Watershed Transform; Objective: The segmentation process works on grayscale image. [1] G. B. Vitor, D. A. Lima, A. C. Victorino, J. V. Ferreira, "A 2D/3D Vision Based Approach Applied to Road Detection in Urban Environments.", IV 2013, Gold Coast, Australia, / 23

17 Segmentation Method Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [1]: Conversion to grayscale ; Morphological Gradient Adjusted; Morpho. Reconstr. Area Close; Morpho. Reconstr. Hmin; Watershed Transform; Objective: This filter detects the intensity variations of pixel values in a given neighborhood, adjusting areas with low intensity. [1] G. B. Vitor, D. A. Lima, A. C. Victorino, J. V. Ferreira, "A 2D/3D Vision Based Approach Applied to Road Detection in Urban Environments.", IV 2013, Gold Coast, Australia, / 23

18 Segmentation Method Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [1]: Conversion to grayscale ; Morphological Gradient Adjusted; Morpho. Reconstr. Area Close; Morpho. Reconstr. Hmin; Watershed Transform; Objective: The conception of this filter is to remove from a binary image its connected components with area smaller than a parameter λ. [1] G. B. Vitor, D. A. Lima, A. C. Victorino, J. V. Ferreira, "A 2D/3D Vision Based Approach Applied to Road Detection in Urban Environments.", IV 2013, Gold Coast, Australia, / 23

19 Segmentation Method Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [1]: Conversion to grayscale ; Morphological Gradient Adjusted; Morpho. Reconstr. Area Close; Morpho. Reconstr. Hmin; Watershed Transform; Objective: This filter removes the connected local minima given a parameter h. [1] G. B. Vitor, D. A. Lima, A. C. Victorino, J. V. Ferreira, "A 2D/3D Vision Based Approach Applied to Road Detection in Urban Environments.", IV 2013, Gold Coast, Australia, / 23

20 Segmentation Method Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [1]: Conversion to grayscale ; Morphological Gradient Adjusted; Morpho. Reconstr. Area Close; Morpho. Reconstr. Hmin; Watershed Transform; Objective: It applies the Local Condition Watershed Transform to obtain the superpixel. [1] G. B. Vitor, D. A. Lima, A. C. Victorino, J. V. Ferreira, "A 2D/3D Vision Based Approach Applied to Road Detection in Urban Environments.", IV 2013, Gold Coast, Australia, / 23

21 Morphological Gradient Adjusted Segmentation Method Texton Maps Dispton Maps Result of the Morphological Gradient Adjusted Formulation: MG Adj = { c [(f g e) (f g i)] γ, if { x f (x) < ρ} (f g e) (f g i), otherwise c = max((f ge) (f g i )) max((f g e) (f g i )) γ Figure 1 : Enhancing the contrast of higher frequency in shadow areas. (a) Original image, (b) gradient image and (c) the gradient image with shadow area enhanced. 9 / 23

22 Morphological Reconstruction Area Close Segmentation Method Texton Maps Dispton Maps Result of the Morphological Reconstruction Area Close Formulation: γ a λ(x) = {x X Area(C x(x)) λ} Figure 2 : (a) Original image, (b) Morphological Reconstruction Area Close result. 10 / 23

23 Morphological Reconstruction Hmin Segmentation Method Texton Maps Dispton Maps Formulation: IR(p, q) (min(m(i, j)), M(i, j) NNG(p, q) M(i, j) < M(p, q)) I(p, q) Figure 3 : Morphological Reconstruction Hmin. 11 / 23

24 Watershed Transform Introduction Segmentation Method Texton Maps Dispton Maps Results of Watershed Transform changing λ and h. Figure 4 : Example of an influence surface for the parameters λ and h on the number of segments in an given image. In (a) λ 1 = 5, h 1 = 2 and 4090 segments; (b) λ 2 = 30, h 2 = 5 and 427 segments; (c) λ 3 = 80, h 3 = 15 and 73 segments. 12 / 23

25 Texton Maps Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [2]: Histogram of Gradient (HoG); Filter Bank; Pixel Location; LAB color; Textonization Process: Figure 5 : Input image example. [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, / 23

26 Texton Maps Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [2]: Histogram of Gradient (HoG); Filter Bank; Pixel Location; LAB color; Textonization Process: Figure 5 : Input image example. [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, / 23

27 Texton Maps Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [2]: Histogram of Gradient (HoG); Filter Bank; Pixel Location; LAB color; Textonization Process: Figure 5 : Input image example. [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, / 23

28 Texton Maps Introduction Segmentation Method Texton Maps Dispton Maps Method similar to [2]: Histogram of Gradient (HoG); Filter Bank; Pixel Location; LAB color; Textonization Process: Figure 5 : Input image example. [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, / 23

29 Dispton Maps Introduction Segmentation Method Texton Maps Dispton Maps Disptonization Process: Method: V-Dispton; U-Dispton; Figure 6 : Disparity input image example. 14 / 23

30 Dispton Maps Introduction Segmentation Method Texton Maps Dispton Maps Disptonization Process: Method: V-Dispton; U-Dispton; Figure 6 : Disparity input image example. 14 / 23

31 Introduction Segmentation Method Texton Maps Dispton Maps - It uses a adapted version of the Joint Boosting algorithm [3], inspired by [2]. Probabilistic Approach: Filter Bank { P(Xjb) j Db} Color { P(Xjc) j Dc} HoG V-Disparity U-Disparity Dispton Maps h(cl ) = ( aδ(d(wc, Sr ) > θ) + b κc l g, if {cl L}, otherwise where r Location Texton Maps Segmen. The weak-learner classifier is modeled by: Sr g { P(Xj ) j D } Joint Boosting Classifier d(wc, Sr ) = 1 [P(xrand ) P(Xr = x f )]2 j { P(Xjl) j Dl} The strong classifier: { P(Xju) j Du } { P(Xjv) j Dv } The Probability distribution under super-pixel (Sr ) is denoted by P(Xr ) and given by: n P o S f f 1 P(Xr ) = { f F ir P(xj ) j D } Z H(cl ) = PM h (c ) m=1 m l [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, [3] A. Torralba, et.al, "Sharing features: efficient boosting procedures for multiclass object detection," in Computer Vision and Pattern Recognition (CVPR), / 23

32 Introduction Segmentation Method Texton Maps Dispton Maps - It uses a adapted version of the Joint Boosting algorithm [3], inspired by [2]. Probabilistic Approach: Filter Bank { P(Xjb) j Db} Color { P(Xjc) j Dc} HoG V-Disparity U-Disparity Dispton Maps h(cl ) = ( aδ(d(wc, Sr ) > θ) + b κc l g, if {cl L}, otherwise where r Location Texton Maps Segmen. The weak-learner classifier is modeled by: Sr g { P(Xj ) j D } Joint Boosting Classifier d(wc, Sr ) = 1 [P(xrand ) P(Xr = x f )]2 j { P(Xjl) j Dl} The strong classifier: { P(Xju) j Du } { P(Xjv) j Dv } The Probability distribution under super-pixel (Sr ) is denoted by P(Xr ) and given by: n P o S f f 1 P(Xr ) = { f F ir P(xj ) j D } Z H(cl ) = PM h (c ) m=1 m l [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, [3] A. Torralba, et.al, "Sharing features: efficient boosting procedures for multiclass object detection," in Computer Vision and Pattern Recognition (CVPR), / 23

33 Introduction Segmentation Method Texton Maps Dispton Maps - It uses a adapted version of the Joint Boosting algorithm [3], inspired by [2]. Probabilistic Approach: Filter Bank { P(Xjb) j Db} Color { P(Xjc) j Dc} HoG V-Disparity U-Disparity Dispton Maps h(cl ) = ( aδ(d(wc, Sr ) > θ) + b κc l g, if {cl L}, otherwise where r Location Texton Maps Segmen. The weak-learner classifier is modeled by: Sr g { P(Xj ) j D } Joint Boosting Classifier d(wc, Sr ) = 1 [P(xrand ) P(Xr = x f )]2 j { P(Xjl) j Dl} The strong classifier: { P(Xju) j Du } { P(Xjv) j Dv } The Probability distribution under super-pixel (Sr ) is denoted by P(Xr ) and given by: n P o S f f 1 P(Xr ) = { f F ir P(xj ) j D } Z H(cl ) = PM h (c ) m=1 m l [2] J. Shotton, et.al., "Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context", International Journal of Computer Vision, [3] A. Torralba, et.al, "Sharing features: efficient boosting procedures for multiclass object detection," in Computer Vision and Pattern Recognition (CVPR), / 23

34 Road Benchmark Quantitative Analysis Qualitative Analysis 1 Introduction / 23

35 Road Benchmark Quantitative Analysis Qualitative Analysis Road Benchmark The experiments using real driving situations are based on the Urban Kitti-road dataset; The data are categorized in three sets having each one a subset of training and test images; The evaluation process is done on the metric space in order to capture the fact that vehicle control happens in the 2D environment. Table 1 : Dataset statistics of the KITTI-ROAD dataset[4]. Abbreviation train test description UU urban unmarked UM urban marked two-way road UMM urban marked multi-lane road URBAN all three urban subsets [4] J. Fritsch, T. Kuehnl, and A. Geiger, "A new performance measure and evaluation benchmark for road detection algorithms," in International Conference on Intelligent Transportation Systems (ITSC), / 23

36 Road Benchmark Quantitative Analysis Qualitative Analysis Quantitative Analysis Table 2 : Results [%] of pixel-based for the all categories on the metric space evaluation. Urban Unmarked (UU) F max AP Prec. Recall FPR FNR Baseline ANN Our Urban Marked (UM) F max AP Prec. Recall FPR FNR Baseline ANN Our Urban Marked Multi-Lane (UMM) F max AP Prec. Recall FPR FNR Baseline ANN Our / 23

37 Road Benchmark Quantitative Analysis Qualitative Analysis Quantitative Analysis Table 2 : Results [%] of pixel-based for complete URBAN ROAD area evaluation. F max AP Prec. Recall FPR FNR Baseline[4] ANN[1] Our [1] G. B. Vitor, D. A. Lima, A. C. Victorino, J. V. Ferreira, "A 2D/3D Vision Based Approach Applied to Road Detection in Urban Environments.", IV 2013, Gold Coast, Australia, [4] J. Fritsch, T. Kuehnl, and A. Geiger, "A new performance measure and evaluation benchmark for road detection algorithms," in International Conference on Intelligent Transportation Systems (ITSC), / 23

38 Introduction Road Benchmark Quantitative Analysis Qualitative Analysis Qualitative Analysis The result to UU dataset. Left: Perspective Image, Right: BEV image LEGEND: Red denotes false negatives, blue areas correspond to false positives and green represents true positives. 19 / 23

39 Introduction Road Benchmark Quantitative Analysis Qualitative Analysis Qualitative Analysis The result to UM dataset. Left: Perspective Image, Right: BEV image LEGEND: Red denotes false negatives, blue areas correspond to false positives and green represents true positives. 19 / 23

40 Introduction Road Benchmark Quantitative Analysis Qualitative Analysis Qualitative Analysis The result to UMM dataset. Left: Perspective Image, Right: BEV image LEGEND: Red denotes false negatives, blue areas correspond to false positives and green represents true positives. 19 / 23

41 Conclusions Future Works 1 Introduction / 23

42 Conclusions Future Works conclusions The probabilistic distribution based on Texton and Dispton maps to model weak classifiers used in the Joint Boosting classifier presents promising results; Quantitative evaluation of our algorithm presents 87.21% of correctness for challenging urban Kitti-road benchmark, despite the presence of shadows and other objects in the scene, inherent from the complexity of innercity environments; The result also provides the benefits of our approach over existing methods. 21 / 23

43 Conclusions Future Works conclusions The probabilistic distribution based on Texton and Dispton maps to model weak classifiers used in the Joint Boosting classifier presents promising results; Quantitative evaluation of our algorithm presents 87.21% of correctness for challenging urban Kitti-road benchmark, despite the presence of shadows and other objects in the scene, inherent from the complexity of innercity environments; The result also provides the benefits of our approach over existing methods. 21 / 23

44 Conclusions Future Works conclusions The probabilistic distribution based on Texton and Dispton maps to model weak classifiers used in the Joint Boosting classifier presents promising results; Quantitative evaluation of our algorithm presents 87.21% of correctness for challenging urban Kitti-road benchmark, despite the presence of shadows and other objects in the scene, inherent from the complexity of innercity environments; The result also provides the benefits of our approach over existing methods. 21 / 23

45 Conclusions Future Works Future Works It could be interesting work with the road pattern and extending the recognition for different classes such as vehicles, builds, sidewalks, etc. More Details The complete application will be embedded in a real carlike robot, sponsored by the project ROBOTEX, from the Heudiasyc laboratory, to perform autonomous driving in urban environments. More Details 22 / 23

46 Conclusions Future Works Future Works It could be interesting work with the road pattern and extending the recognition for different classes such as vehicles, builds, sidewalks, etc. More Details The complete application will be embedded in a real carlike robot, sponsored by the project ROBOTEX, from the Heudiasyc laboratory, to perform autonomous driving in urban environments. More Details 22 / 23

47 Conclusions Future Works Thank you for your attention! 23 / 23