Superpixel Segmentation using Depth Information Superpixel Segmentation using Depth Information David Stutz June 25th, 2014 David Stutz June 25th, 2014 01
Introduction - Table of Contents 1 Introduction Oversegmentation and Superpixels Motivation 2 SEEDS Hierarchy and Histograms Iterations Parameters 3 Results Qualitative 4 Depth Information David Stutz June 25th, 2014 2
Introduction - Oversegmentation and Superpixels Oversegmentation and Superpixels Segmentation and Oversegmentation are not clearly defined throughout the literature... Notion (Superpixel) The term superpixel is used to describe the oversegmentation of an image into groups of pixels using low-level features. Well, but what is the difference between usual segmentation and oversegmentation? David Stutz June 25th, 2014 3
Introduction - Oversegmentation and Superpixels Oversegmentation and Superpixels Figure: Original image. David Stutz June 25th, 2014 4
Introduction - Oversegmentation and Superpixels Oversegmentation and Superpixels Figure: Ground truth segmentation one segment per object. David Stutz June 25th, 2014 5
Introduction - Oversegmentation and Superpixels Oversegmentation and Superpixels Figure: Superpixel segmentation multiple superpixels per object. David Stutz June 25th, 2014 6
Introduction - Motivation Motivation There is a whole bunch of superpixel algorithms out there [4] [1] [13] [16] [15] [10] [3] [12]... Only few provide near-realtime [4] [1]... Some of them utilize depth information [1] [10] [15]... SEEDS [4] provides state of the art performance and near-realtime runtime on 2D data! Goal: integrate depth information into SEEDS. David Stutz June 25th, 2014 7
SEEDS - Table of Contents 1 Introduction Oversegmentation and Superpixels Motivation 2 SEEDS Hierarchy and Histograms Iterations Parameters 3 Results Qualitative 4 Depth Information David Stutz June 25th, 2014 8
SEEDS - Overview Overview Given an initial superpixel segmentation, SEEDS iteratively refines this segmentation by... exchanging blocks of pixels between neighboring superpixels (block updates), and exchanging pixels between neighboring superpixels (pixel updates). The initial superpixel segmentation will be a uniform grid. This grid is subdivided to define blocks of pixels. David Stutz June 25th, 2014 9
SEEDS - Overview Overview Figure: Initial superpixel segmentation. David Stutz June 25th, 2014 10
SEEDS - Hierarchy and Histograms Hierarchy and Histograms Step 1 Set up a hierarchy SEEDS is based on a simple hierarchy consisting of L levels... Group w min h min pixel to form one block on level l = 1; and group 4 blocks at level (l 1) to form one block at level l > 1. The blocks at level L represent the initial superpixel segmentation! Step 2 Compute color histograms for each block at level l > 0. David Stutz June 25th, 2014 11
SEEDS - Hierarchy and Histograms Hierarchy and Histograms Figure: Initial blocks at level three (with a total of four levels). David Stutz June 25th, 2014 12
SEEDS - Hierarchy and Histograms Hierarchy and Histograms Figure: Initial blocks at level two (with a total of four levels). David Stutz June 25th, 2014 13
SEEDS - Iterations Block Updates Step 3 Perform block updates For each level l = L 1,..., 1: For each block at level l: If a neighboring block belongs to a different superpixel (= label), consider changing the label. When to change the label? If the histogram intersection of the block histogram and the superpixel histogram is higher than before... David Stutz June 25th, 2014 14
SEEDS - Iterations Hierarchy and Histograms Figure: Block updates at level three. David Stutz June 25th, 2014 15
SEEDS - Iterations Hierarchy and Histograms Figure: Block updates at level two. David Stutz June 25th, 2014 16
SEEDS - Iterations Hierarchy and Histograms Figure: Block updates at level one. David Stutz June 25th, 2014 17
SEEDS - Iterations Pixel Updates Step 4 Perform pixel updates For each pixel: If a neighboring pixel belongs to a different superpixel (= label), consider changing the label. When to change the label? If the color of the pixel is represented well in the superpixel histogram (= intersection distance on pixel level)... and most pixels in a local neighborhood belong to the new superpixel (smoothing term, optional). David Stutz June 25th, 2014 18
SEEDS - Iterations Pixel Updates Figure: Pixel updates without smoothing term. David Stutz June 25th, 2014 19
SEEDS - Iterations Pixel Updates Figure: Pixel updates with smoothing term. David Stutz June 25th, 2014 20
SEEDS - Iterations Pixel Updates Figure: Pixel updates with smoothing term and based on color means. David Stutz June 25th, 2014 21
SEEDS - Parameters Parameters Step 5 Take some time to choose parameters... How many bins to use for the histograms? How many superpixels to obtain? How to define local neighborhood? David Stutz June 25th, 2014 22
Results - Table of Contents 1 Introduction Oversegmentation and Superpixels Motivation 2 SEEDS Hierarchy and Histograms Iterations Parameters 3 Results Qualitative 4 Depth Information David Stutz June 25th, 2014 23
Results - Qualitative Qualitative Results Well, but what does this actually look like? David Stutz June 25th, 2014 24
Results - Qualitative Easy Examples... Figure: Part of the BSDS500 validation set. David Stutz June 25th, 2014 25
Results - Qualitative Easy Examples... Figure: Superpixel segmentation with 400 superpixels, 6 bins, smoothing term and mean based pixel updates. David Stutz June 25th, 2014 26
Results - Qualitative Easy Examples... Figure: Reconstruction by coloring superpixels according to their mean color. David Stutz June 25th, 2014 27
Results - Qualitative Easy Examples... Figure: Original image. David Stutz June 25th, 2014 28
Results - Qualitative Easy Examples... Figure: Superpixel segmentation. David Stutz June 25th, 2014 29
Results - Qualitative Easy Examples... Figure: Reconstruction. David Stutz June 25th, 2014 30
Results - Qualitative Qualitative Results OK, but these were easy, what about some hard ones? David Stutz June 25th, 2014 31
Results - Qualitative The Hard Ones... Figure: Difficult original image. David Stutz June 25th, 2014 32
Results - Qualitative The Hard Ones... Figure: Superpixel segmentation. David Stutz June 25th, 2014 33
Results - Qualitative The Hard Ones... Figure: Reconstruction. David Stutz June 25th, 2014 34
Results - Qualitative The Hard Ones... Figure: Difficult original image. David Stutz June 25th, 2014 35
Results - Qualitative The Hard Ones... Figure: Superpixel segmentation. David Stutz June 25th, 2014 36
Results - Qualitative The Hard Ones... Figure: Reconstruction. David Stutz June 25th, 2014 37
Depth Information - Table of Contents 1 Introduction Oversegmentation and Superpixels Motivation 2 SEEDS Hierarchy and Histograms Iterations Parameters 3 Results Qualitative 4 Depth Information David Stutz June 25th, 2014 38
Depth Information - Demonstration Depth Information Figure: Image taken form the NYU Depth Dataset V2.. David Stutz June 25th, 2014 39
Depth Information - Demonstration Depth Information Figure: Some boundaries which are really hard to see... at least for the algorithm. David Stutz June 25th, 2014 40
Depth Information - Demonstration Depth Information Figure: Segmentation using SEEDS Revised purely color based... Could be better! David Stutz June 25th, 2014 41
Depth Information - Demonstration Depth Information Figure: Better! Pixel and block updates are based on depth information a penalty is added if the depth between blocks or pixels differs by more than 10cm. David Stutz June 25th, 2014 42
Depth Information - Conclusion Depth Information Can depth information lead to better oversegmentation? Yes... especially where color fails! But: Color alone already is very good! Well, but how to integrate depth information? Some ideas... David Stutz June 25th, 2014 43
Depth Information - Ideas Ideas Use depth information for mean updates (= integrate depth information into euclidean distance)... Done! Use 3D point coordinates for pixel updates (= integrate (x, y, z) into euclidean distance)... Already done! But SEEDS is based on histograms... Build histograms using... point normals; or use feature histograms for example Fast Point Feature Histograms (FPFH) [18]! David Stutz June 25th, 2014 44
The End - The End Thanks for your attention! Comments? Ideas? Questions? David Stutz June 25th, 2014 45
Appendix - Recall Recall Let G be a ground truth segmentation and S be a superpixel segmentation. Some definitions [7]... True Positives (TP): The number of boundary pixels in G for which there is a boundary pixel in S in range d. False Negatives (FN): The number of boundary pixels in G for which there is no boundary pixel in S in range d. Then Boundary Recall is defined as R(G, S) = T P T P + F N (1) David Stutz June 25th, 2014 46
Appendix - Undersegmentation Error Undersegmentation Error Let G be a ground truth segmentation, S be a superpixel segmentation and N be the total number of pixels. For a superpixel S S and a ground truth segment G G we define S in = S G; and S out = S S G. The Undersegmentation Error is defined as UE(G, S) = 1 N G G min(s in, S out ) (2) S G David Stutz June 25th, 2014 47
Appendix - Compactness Compactness Let G be a ground truth segmentation, S be a superpixel segmentation and N be the total number of pixels. The isoperimetric quotient Q(S) for a superpixel S S with area A S and perimeter L S is defined as Q(S) = 4πA S L 2 S (3) Then the compactness measure is given as CO(S) = 1 Q(S) S (4) N S S David Stutz June 25th, 2014 48
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, SLIC Superpixels, Technical report, EPFL, Lausanne, 2010. X. Ren, J. Malik, Learning a classification model for segmentation, ICCV, 2000. P. F. Felzenswalb, D. P. Huttenlocher, Efficient graph-based image segmentation, International Journal of Computer Vision, Volume 59, Number 2, 2004. M. Van den Bergh, X. Boix, G. Roig, B. de Capitani, L. van Gool, SEEDS: superpixels extracted via energy-driven sampling, ECCV, 2012. P. Arbelaez, M. Maire, C. Fowlkes, J. Malik, Contour Detection and Hierarchical Image Segmentation, PAMI, 2011. A. Schick, M. Fischer, R. Stiefelhagen, Measuring and evaluating the compactness of superpixels, ICPR, 2012. P. Neubert, P. Protzel, Superpixel benchmark and comparison, Forum Bildverarbeitung, Regensburg, 2012. David Stutz June 25th, 2014 48
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, SLIC superpixels compared to state-of-the-art superpixel methods, PAMI, 2012. A. P. Moore, J. D. Prince, J. Warrell, U. Mohammed, G. Jones, Superpixel lattices, CVPR, 2008. D. Weikersdorfer, D. Gossow, M. Beetz, Depth-adaptive superpixels, ICPR, 2012. C. Conrad, M. Mertz, R. Mester, Contour-relaxed superpixels, Lecture Notes in Computer Science, Volume 8081, 2013. M.-Y. Li, O. Tuzel, S. Ramalingam, R. Chellappa, Entropy rate superpixel segmentation, CVPR, 2011. A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson, K. Siddiqi, Turbopixels: fast superpixels using geometric flows, PAMI, 2009. Y. Zhang, R. Hartley, J. Mashford, S. Burn, Superpixels via pseudo-boolean optimization, ICCV, 2011. David Stutz June 25th, 2014 48
J. Papon, A. Abramov, M. Schoeler, F. Worgotter, Voxel cloud connectivity segmentation - supervoxels for point clouds, CVPR, 2013. A. Vedaldi, S. Soatto, Quick shift and kernel methods for mode seeking, ECCV, 2008. N. Silberman, D. Hoiem, P. Kohli, R. Fergus, Indoor segmentation and support inference from RGBD images, ECCV, 2012. R. B. Rusu, N. Blodow, M. Beetz, Fast point feature histograms (FPFH) for 3D registration, ICRA, 2009. David Stutz June 25th, 2014 48