Organized Segmenta.on Alex Trevor, Georgia Ins.tute of Technology PCL TUTORIAL @ICRA 13
Overview Mo.va.on Connected Component Algorithm Planar Segmenta.on & Refinement Euclidean Clustering Timing Results Applica.ons Demos (Time PermiRng) Q & A Alexander J. B. Trevor Organized Segmenta.on 2
Mo.va.on Segmenta.on is used for many applica.ons Object Detec.on & Recogni.on Seman.c Mapping Obstacle avoidance Goal is to make a fast, general approach focused on RGB- D data We provide an open- source implementa.on in PCL A.J.B. Trevor, S. Gedikli, R.B. Rusu, H. I. Christensen. Efficient Organized Point Cloud Segmentation with Connected Components. 3rd Workshop on Semantic Perception Mapping and Exploration (SPME). Karlsruhe, Germany. May, 2013. Alexander J. B. Trevor Organized Segmenta.on 3
Mo.va.on RANSAC takes a lot of.me, which is okay for scanning lasers, but not for RGB- D sensors Convex hulls poorly represent the shape of many surfaces Key idea: we can exploit the organized structure to solve both issues Alexander J. B. Trevor Organized Segmenta.on 4
Organized Point Clouds RGB- D sensors can produce organized point clouds by back- projec.ng RGB onto the depth image Result is an image like structure, but with addi.onal informa.on channels such as X, Y, Z coordinates Neighboring pixels can be accessed in constant.me We can augment this with other densely computed features such as surface normals Key advantage is that keeping NaN points enables us to maintain organiza.on Monochrome Image Point: p = {i} RGB Image Point: p = {r, g, b} RGBD Point: p = {r, g, b, x, y, z} RGBD+Normal Point: p = {r, g, b, x, y, z, n x,n y,n z } Alexander J. B. Trevor Organized Segmenta.on 5
Normal Es.ma.on Surface normals can be useful for many segmenta.on tasks We use Integral Image Normal Es.ma.on of Holzer et. al Enables real-.me surface normal computa.on S Holzer, RB Rusu, M Dixon, S Gedikli, and N Navab. Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 2684 2689. IEEE, 2012. Alexander J. B. Trevor Organized Segmenta.on 6
Connected Components Well known algorithm from computer vision Can be applied to organized point cloud segmentation Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision 0 Can be applied to organized point cloud segmentation Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision 0 0 Can be applied to organized point cloud segmentation Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision 0 0 0 Can be applied to organized point cloud segmentation Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision 0 0 0 0 Can be applied to organized point cloud segmentation Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision Can be applied to organized point cloud segmentation 0 0 0 0 0 Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision Can be applied to organized point cloud segmentation 0 0 0 0 0 1 Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision Can be applied to organized point cloud segmentation 0 0 0 0 0 1 1 Requires a comparison function to determine if pixels should be part of the same segment or not
Connected Components Well known algorithm from computer vision Can be applied to organized point cloud segmentation 0 0 0 0 0 1 1 0 Requires a comparison function to determine if pixels should be part of the same segment or not
Union Find 1 2 2 2 An additional pass is required to merge labels in some cases The union-find algorithm can be used to do this efficiently 3 3
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Boundary Detection Organized data makes it easy to find the boundary of segmented regions using standard contour tracing Provides the actual boundary shape, rather than a convex shape Example convex hull timing (80k pts): 0.012 sec Example boundary timing (80k pts): 0.000048 sec
Comparison Func.ons Different comparison func.ons can be used for different segmenta.on tasks Allows us to easily modify the approach for different applica.ons These comparison func.ons have the form: Alexander J. B. Trevor Organized Segmenta.on 18
Planar Segmenta.on Point Representation: Distance Metrics: Comparison Function: Alexander J. B. Trevor Organized Segmenta.on 19
Plane Coefficient Comparator Alexander J. B. Trevor Organized Segmenta.on 20
See PCL s Organized Segmenta.on Demo for more: pcl_organized_segmenta.on_demo Alexander J. B. Trevor Organized Segmenta.on 21
Refinement of planar regions Normals near surface edges can be noisy We can address this issue with an addi.onal two passes Instead of comparing neighboring pixels values, we compare to the planar model of neighboring pixels Alexander J. B. Trevor Organized Segmenta.on 22
Refinement of planar regions Distance Metric: Comparison Function: Refinement requires two addi.onal passes to extend regions Alexander J. B. Trevor Organized Segmenta.on 23
Code Example Alexander J. B. Trevor Organized Segmenta.on 24
Using Color Informa.on We can easily segment on color and geometric informa.on at liole addi.onal cost We can use the same comparison as we did for planes, plus an addi.onal constraint that neighboring pixels have similar color Alexander J. B. Trevor Organized Segmenta.on 25
Euclidean Cluster Extrac.on Use planar regions extracted in the first pass as a mask Simply compare euclidean distance between neighboring points Alexander J. B. Trevor Organized Segmenta.on 26
Code Example Alexander J. B. Trevor Organized Segmenta.on 27
Timing Results Timing results were computed on a 2.6 GHz Intel Core i7 CPU Point Clouds from the publicly available TUM RGBD Dataset were used: fr1_desk fr1_floor fr2_pioneer_slam Alexander J. B. Trevor Organized Segmenta.on 28
Timing Results Planar Segmenta.on: Planar Segmenta.on with refinement: Alexander J. B. Trevor Organized Segmenta.on 29
Mapping Applica.on Segmenta.on approach has been used in a mapping applica.on Planar segments from mul.ple images are merged in a map frame Alexander J. B. Trevor Organized Segmenta.on 30
Stereo Ground Segmenta.on Applica.on Technique also works on stereo data, as this is also organized Has been applied to ground segmenta.on on outdoor wide- baseline stereo Segmenta.on & Refinement steps are used Alexander J. B. Trevor Organized Segmenta.on 31
Organized Edge Extrac.on Similar organized extrac.on techniques can be used to extract edges Supports mul.ple edge types: Occluding Edges Occluded Edges High Curvature Edges pcl_openni_organized_edge_detec.on Alexander J. B. Trevor Organized Segmenta.on 32