Growing Depth Image Superpixels for Foliage Modeling

Transcription

1 z Growing Depth Image Superpixels for Foliage Modeling Daniel Morris, Saif Imran Dept. of Electrical and Computer Engineering, Michigan State University, East Lansing, MI 48824, USA Jin Chen, David M. Kramer MSU-DOE Plant Research Lab, Michigan State University, East Lansing, MI 48824, USA Abstract This paper presents a method for segmenting depth images into superpixels without requiring color images. Typically superpixel methods cluster pixels based on proximity in a multidimensional color space. However, building superpixels from time-of-flight depth images poses a number of new challenges including: pixels do not have color channels for similarity comparisons, the resolution of depth cameras is low compared to color, and there is significant depth noise. To address these we propose a superpixel method that approximates a depth image with set of planar facets. Facets are grown from seed points to cover the scene. Facet boundaries tend to coincide with high curvature regions and depth discontinuities, typically giving an over-segmentation of the scene. This work is motivated by automated foliage modeling, and the data we consider are of dense 3D foliage. Superpixel results are shown on foliage and are quantified using labeled data. Keywords-Superpixels; depth image; segmentation; energy minimization; RGB-D; time-of-flight camera I. INTRODUCTION Superpixels have significant utility in image analysis. With greater information content than individual pixels, they allow for richer low-level descriptors, and at the same time there are far fewer of them than raw pixels, which reduces the dimensionality of graphs and optimization problems built on them. As a result they have been used for numerous computer vision tasks including segmentation [1], [2], [3], object detection [4], object tracking [5], and stereo depth estimation [6]. The key property that makes them useful for these tasks is that they over-segment the image, and boundaries of objects are included in the boundaries between superpixels. This property means superpixels do not need to be split and can be treated as low-level building blocks for scene and object analysis. The recent availability of consumer depth cameras, such as time-of-flight sensors [7], has provided an important new modality for sensing the world. These depth images can be used for many of the same computer vision tasks as color imagery including segmentation, object detection, object tracking, etc., and similar problems are faced in achieving these tasks. In the application that motivates this paper, a time-of-flight sensor is placed over plants in a growth chamber and leaf growth over a number of days is recorded. A key need is to automatically segment the leaves and leaf fragments within dense foliage. While there are (c) Figure 1. Depth image of a plant generated by a time-of-flight camera. 0 Near-IR reflectivity measured-200 for each depth pixel. (c) Color image of100 the plant from adjacent camera. (d) Depth -100 image pixels 0 from 100 projected into 3D illustrating significant noise and edge artifacts. x methods for 3D modeling of leaves including [8], [9], [10], [11], automatic segmentation of them from depth imagery has not been solved, and these methods rely on color or intensity images for segmentation, in some cases leveraging superpixels of these [10]. Single-image color segmentation of leafs works well [12] with plane backgrounds, but can fail to segment uniformly-colored, overlapping leaves. While ideally a RGB-D superpixel method such as [13] could use both color and depth, in practice the parallax for closein plants can be a large fraction of leaf features making this impractical. The alternative is to do superpixel in both color and depth and combine the results afterwards. The missing component is a depth-based superpixel method, and achieving that is the goal of this paper. Future work will address combining superpixels from multiple modalities. In an ideal superpixel segmentation, the true object boundaries are included in the boundaries of the superpixels. Poor performance occurs when superpixels span the boundaries of (d) -100 y

2 Figure 2. An illustration of the failures in two of the best color-based superpixel methods when used on overlapping foliage in Fig. 1(c). A close-up of SLIC superpixels [15] with boundaries shown in orange. The manually identified outline of the leaves is shown in magenta. The SLIC pixel boundaries align well with the boundary between the leaves and the background. However the superpixel boundaries do not overlap the boundaries between overlapping leaves well. The graph cut superpixel method in [14] is used with a very low threshold to obtain this set of superpixels. More of the leaf boundaries are obtained, but nevertheless a number of true overlapping leaf boundaries (shown in magenta) are not obtained and these are merged within single superpixels. objects. Proximity in color space, as used by most superpixel methods [14], [15], [16], is a useful cue for segmentation since object boundaries frequently involve sharp changes in color space. Strategies for leveraging color proximity cues can lead to high scores in boundary recall [17]. On the other hand, proximity in depth can provide easy segmentation for well-separated objects, but is problematic when objects are closely spaced or are touching. Overlapping leaves are often closely spaced resulting in small depth separation cues between leaves. Thus, rather than use proximity in depth, this paper presents a depth image superpixel method based on the idea of superpixels being the 2D projection of a planar-patch approximation of the depth image. A patch model is different than a mesh model [18] as we do not require that the boundaries of facets connect. In part this is because superpixels have complicated shapes, but more importantly it is because edge discontinuities in a scene are very often not connected spatially, and connecting them with a mesh leads to blurring of boundaries. Sensor depth noise is an important issue as the depth separation of leaves can be close to the sensor noise level. Our method seeks a robust solution for approximating noisy depth surface patches while avoiding spanning boundaries between objects. II. RELATED WORK Superpixel methods can be broadly divided into graphbased methods [19], [14], [20], [21], [22] and hill-climbing methods [23], [24], [25], [15], [16]. Graph methods optimize an energy function over edge cuts that includes on pixel color similarity and shape and size priors. The more recent hill climbing methods such as SLIC [15] and SEEDS [16] start with a regular grid and refine these leading to fast performance with fairly evenly sized superpixels. Our method follows a similar approach of starting with a regular grid and refining it. However since depth pixels have no color or texture, our underlying local similarity model is quite different from all the color-based methods. With the availability of RGB-D datasets with a depth pixel for each color pixel, it is natural to do joint color and depthbased superpixel as in [13]. Both color and depth cues can be leveraged potentially achieving better object segmentation. With current sensors the baseline separating the depth and color cameras causes significant parallax and occlusion differences between color and depth cameras, particularly for complicated structures such as foliage. Obtaining a oneto-one mapping of color and depth pixels entails significant feature matching and object modeling often from multiple viewpoints as in [26]. In our case, constrained to viewing the scene from a single viewpoint, it is preferable to use the color and depth images separately, as in [27] where the color image is used to upsample the depth image with an enhanced bilateral filter. Unfortunately it is in regions of detailed structure and occlusions, as exhibited by overlapping leaves, that significant depth errors are reported [27]. In our work we focus on using superpixels to resolve depth discontinuities in the depth image alone, and leave it to future work to improve this with an adjacent color image. Superpixel can be used as a first step in 3D model building. An alternative is to collect data from a moving sensor, align and fuse it as done with Kinect Fusion using RGB- D images [28], [29], [26]. These methods involve building a voxel model of the scene and extracting an isosurface. However, in our application the camera remains stationary making these approaches not suitable, and that motivates this work to focus on the single-viewpoint segmentation problem. III. SENSOR DATA Our data was collected using a Creative Senz3D sensor. Like the Kinect 2, this is a time-of-flight sensor [7] but with a lower resolution of pixels. Despite the lower resolution, its smaller angular field of view along with closer minimum range of roughly 20cm, compared to 50cm for the Kinect 2, means that it can collect higher resolution depth images. This is important for modeling plants and foliage with fine structure. We used the raw depth values rather than the default smoothed values as these contain more details of the target and return values at greater depth. Sample data from this sensor is shown in Fig. 1. The depth camera returns both a dense pixel depth image and reflected intensity for each pixel although only the depth image was used here. In addition there is a color camera offset roughly 2.5cm from the depth camera. This offset creates in large parallax for close-in objects and significant occlusion differences.

3 IV. METHOD Unlike color images, depth-image pixels have no color or texture characteristic with which to group them. Instead we propose that depth-image superpixels should model the local geometry as a planar facets. Pixels on flat regions should be grouped together into segments. On the other hand, high curvature edges and depth discontinuities demark the boundaries of objects and so should lie on the boundaries of superpixels. A depth-image superpixel segmentation will thus be a faceted surface model. In building a superpixel representation we seek the following four characteristics. First, superpixel boundaries should conform to boundaries of 3D objects in the scene. That is, while the superpixels will subdivide objects, they should not span multiple objects. Second, superpixels should be contiguous in the image plane. Third, as argued in [15], it is desirable that superpixels be roughly uniform in size as they will contain similar information content for use in higher level algorithms for scene analysis. This constant size could be in 3D scene space as in [13], although the utility of this depends on the subsequent processing, and here we seek a constant image-space dimension for superpixels. Fourth, since depth-image superpixels provide a planar faceted approximation of the scene, the difference between these planar facets and the measured depths should be minimized. This section describes our approach to achieving these goals. A. Facet Estimation Our method starts using a regular grid to divide the depth image into equal-sized square cells. The cell size should be on the order of the smallest features that we wish to approximate, which in our application is the size of the smallest leaves. The significant challenge we face in building our faceted model is how to avoid crossing depth discontinuities when the depth change across them may be on the order of the sensor noise. Our approach is to let our facet model the largest contiguous planar region in each grid cell. If there are two surfaces spanned by a cell, then the facet should approximate just one. It is important that the facet is contiguous, because simply finding the best planar fit to the points in a cell has a high chance of finding a plane that crosses the discontinuity, as illustrated in Fig. 3. Hence we fit planes to pixel depths in a grid cell using a robust method that grows contiguous regions explained as follows. First we choose a set of seed points to start the plane fitting. For each seed point we want a depth and two slopes to define a plane. The slope will be noisy hence we sample these seed points from a smoothed depth image. Not all points on the smoothed depth image are likely to a plane. In particular the slopes at depth discontinuities are unlikely to result in a good planar fit. To avoid these cases we exclude pixels corresponding to peaks in the gradient images and Figure 3. A slice illustration of the challenge in fitting facets to data. The red data points (one per pixel) are the same in and. In a robust line is fit to the data, but the true surface is really a step function closer to. Our method seeks to grow facets that do not span these boundaries. also where the smoothed depth differs more than a threshold from the raw depth image. Next we need to estimate and evaluate the planar region defined by each seed point. There are two main considerations in evaluating this planar region. The first, H(s), for a set of pixels s, is a likelihood function of the plane given the data, which we seek to maximize in estimating the planar parameters. The second consideration is a prior on the shape or distribution of pixels belonging a planar region, which we refer to as a penalty term S(s). For instance this can capture a prior that planar regions should be compact. Finding a planar region involves hill climbing an energy function made from these two components: E(s) = H(s) αs(s). (1) In our implementation we chose the following instantiation for this equation. Our likelihood measure for a planar region was simply, N(s), the number of contiguous pixels whose depth points fall within a tight threshold around the plane. Our shape-based penalty term, S(s), was set equal to the number of boundary edges of the pixels on the plane. For a given number of pixels, a compact region will have fewer boundary edges than a region with narrow arms and so will be penalized less. Both measures are easily calculated. The weight, α, selects how compact the regions will be. What still needs to be described is how we obtain a contiguous set of pixels belonging to a seed point. The seed specifies the plane (with its depth and slopes), and so all points within a grid cell can be evaluated by their perpendicular distance to this plane. A mask is created for all pixels belonging to this plane. Then starting at the seed point a dilation operator is repeatedly used to expand the seed point, and at each step the dilated pixels are constrained to lie on the mask as well as not cross any gradient peaks above a threshold, t g, or to diverge by more than t p in depth from the depth pixels. After a fixed number of iterations the dilation stops and the set of pixels reached from within the mask are the points on the plane. The energy of this planar facet is evaluated with Eq. (1). By repeating this procedure for a randomly chosen set of good seed points, the pixels belonging to the highest scoring plane are chosen for that

4 grid cell, and plane is fit to them using least squares to represent the superpixel. B. Facet Expansion Up to now we have each grid cell modeled as a single facet covering a subset of the pixels. For instance when there is a discontinuity across a grid cell, the facet will ideally cover the region to one side of the discontinuity. We would like a way to explain the remainder of the pixels in each facet. Now assuming that the structures in the have dimension at least the size of the superpixel, then the planar region not covered is likely to extend out over adjacent superpixels. If this plane is modeled in one of these adjacent superpixels we can simply extend that plane to cover the uncovered pixels of the current superpixel. The expansion algorithm works as follows. We apply the contiguous planar facet fitting algorithm described in Section IV-A with the following modifications. We do not need to search for a seed as we start with the fitted plane for that superpixel. We increase the region in which we grow the plane from a single grid cell to a 3 3-cell region centered at that cell. Our mask is set to those pixels in the new cell region lying within an expanded distance threshold of 1.6t p from the surface, and that do not belong to any of the other facets. In this way unclaimed pixels from adjacent grid cells are added to a superpixel if they fall on the plane and are contiguous. This procedure is applied to all of the superpixels in a random order. The final result is a set of superpixels modeling the depth image as a set of planar facets. There remain pixels not claimed by any superpixel. These are primarily of two types neither of which is well approximated by a plane. At depth discontinuities the depth pixels may have mixed values between a surface and its occluding surface with high variance. Also pixels at long range or with low intensity have large noise and do not pass the thresholds we use for fitting planes. We could adjust the threshold based on the pixel noise, but for our application these pixels are not useful and excluding them from the superpixels does not impact the foliage surface estimation. V. RESULTS To test our method we labeled individual leaves on depth images collected from six plants, three of which are shown in Fig. 4 with their labeling. The leaves are densely packed presenting a challenge to segmentation and superpixel algorithms. A goal of superpixel models is that the superpixels boundaries include the leaf boundaries. We evaluate the results based on two stamdard evaluation criteria [16] for superpixel methods: the Corrected Undersegmented Error (CUE) and the Boundary Recall (BR) score. To calculate CUE we first assign each superpixel to one ground truth segment or the background based on its maximum overlap. The error for a superpixel is the sum of pixels falling outside its assigned segment. Then the CUE is the sum of the errors for all the superpixels divided by the sum of the ground truth pixels. The denominator intentionally excludes the background region since adding more background should not affect the CUE. Boundary Recall, on the other hand, means the fraction of boundary pixels that can be traced by the superpixels. Here it is the fraction of ground truth boundary pixels that are within a 1.5 pixel distance to an edge of a superpixel. We set the following parameters for all instances of our algorithm: α = 0.5 from Eq. (1), gradient peak threshold: t g = 5mm, and plane error threshold, t p = 5mm. Based on the ground-truth, the results of our algorithm are compared with two other methods: one baseline method and the other is SLIC [15]. The baseline method is based on splitting the image into regular grid where each grid is a superpixel. SLIC is modified by replacing the LAB color-space image with the depth image. Our method outperforms the other methods both in terms of CUE and BR scores in all the six canopies tested. For subjective evaluation, we plot the facet superpixel boundaries on the plant canopies as shown in Fig. 5. It shows the superpixel boundaries overlapping quite well the boundaries of the depth images. VI. CONCLUSION We propose a new superpixel approach that operates purely on depth images. It approximates depths images as collections of planar facets that correspond to superpixels. It is simple to implement, generates a mostly regular grid of superpixels that approximate the depth image as a set of planar facets, and achieves good fidelity to object boundaries. Our results show improvements over a baseline method for fidelity to ground truth. They also show practicality for using depth-image superpixels for foliage analysis. As far as we are aware this is the first depth-only method for superpixel segmentation. This opens the door to applications where color is not available or where color provides poor segmentation cues as in some of the foliage data. Since the depth superpixels are do not depend on color superpixels, they could be fused with a color-based approach; a future direction for our work ACKNOWLEDGMENT This research was supported by an MSU start-up grant, the U.S. Department of Energy, Office of Science, Basic Energy Sciences [award number DE-FG02-91ER20021], the National Science Foundation [award number ] and the MSU Center for Advanced Algal and Plant Phenotyping.

5 (c) (e) (f ) (d) Figure 5. Superpixel boundaries corresponding to Fig. 4 are plotted on depth images in, (c) and (e). For easier visual inspection we plot the same superpixels on the IR reflectance images in,(d) and (f ). (c) Figure 4. Ground-truth labeled segments for three of the six plants: Plant A, Plant B also shown in Fig. 1, and (c) Plant C. R EFERENCES [1] B. Fulkerson, A. Vedaldi, and S. Soatto, Class segmentation and object localization with superpixel neighborhoods, in Computer Vision, 2009 IEEE 12th International Conference on, Sept 2009, pp [2] A. P. Moore, S. J. D. Prince, J. Warrell, U. Mohammed, and G. Jones, Scene shape priors for superpixel segmentation, in Computer Vision, 2009 IEEE 12th International Conference on, Sept 2009, pp [5] F. Yang, H. Lu, and M.-H. Yang, Robust superpixel tracking, Image Processing, IEEE Transactions on, vol. 23, no. 4, pp , April [6] C. L. Zitnick and S. B. Kang, Stereo for image-based rendering using image over-segmentation, International Journal of Computer Vision, vol. 75, no. 1, pp , [7] M. Hansard, S. Lee, O. Choi, and R. Horaud, Time-of-Flight Cameras: Principles, Methods and Applications. New York, NY: Springer, [8] L. Quan, P. Tan, G. Zeng, L. Yuan, J. Wang, and S. B. Kang, Image-based plant modeling, in ACM SIGGRAPH 2006 Papers, ser. SIGGRAPH 06. New York, NY, USA: ACM, 2006, pp [3] J. Lu, R. Xu, and J. J. Corso, Human action segmentation with hierarchical supervoxel consistency, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, [9] G. Alenya, B. Dellen, and C. Torras, 3d modelling of leaves from color and tof data for robotized plant measuring, in Robotics and Automation (ICRA), 2011 IEEE International Conference on, May 2011, pp [4] Y. Yang, S. Hallman, D. Ramanan, and C. Fowlkes, Layered object models for image segmentation, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 34, no. 9, pp , Sept [10] G. Alenya, B. Dellen, S. Foix, and C. Torras, Robotized plant probing: Leaf segmentation utilizing time-of-flight data, Robotics Automation Magazine, IEEE, vol. 20, no. 3, pp , Sept 2013.

6 BR Scores CUE Metric Baseline Algorithm SLIC FacetSuperPixel [18] H. Hoppe, T. DeRose, T. Duchamp, M. Halstead, H. Jin, J. McDonald, J. Schweitzer, and W. Stuetzle, Piecewise smooth surface reconstruction, in Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, ser. SIGGRAPH 94. New York, NY, USA: ACM, 1994, pp [19] J. Shi and J. Malik, Normalized cuts and image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp , Aug Plant A Plant B Plant C Plant D Plant E Plant F Plant Canopy Baseline Algorithm SLIC FacetSuperpixel Plant A Plant B Plant C Plant D Plant E Plant F Plant Canopy Figure 6. CUE metric and BR score for six plants. Plants A, B and C are shown in Figs. 4 and 5. [11] C. Xia, J.-M. Lee, Y. Li, Y.-H. Song, B.-K. Chung, and T.- S. Chon, Plant leaf detection using modified active shape models, Biosystems Engineering, vol. 116, no. 1, pp , [12] N. Kumar, P. N. Belhumeur, A. Biswas, D. W. Jacobs, W. J. Kress, I. C. Lopez, and J. V. Soares, Leafsnap: A computer vision system for automatic plant species identification, in Computer Vision ECCV Springer, 2012, pp [13] D. Weikersdorfer, D. Gossow, and M. Beetz, Depth-adaptive superpixels, in Pattern Recognition (ICPR), st International Conference on, Nov 2012, pp [14] P. F. Felzenszwalb and D. P. Huttenlocher, Efficient graphbased image segmentation, International Journal of Computer Vision, vol. 59, no. 2, pp , [15] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk, Slic superpixels compared to state-of-the-art superpixel methods, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 34, no. 11, pp , Nov [16] M. Bergh, X. Boix, G. Roig, and L. Gool, Seeds: Superpixels extracted via energy-driven sampling, International Journal of Computer Vision, vol. 111, no. 3, pp , [17] D. Stutz, Superpixel segmentation: An evaluation, in Pattern Recognition, ser. Lecture Notes in Computer Science, J. Gall, P. Gehler, and B. Leibe, Eds. Springer International Publishing, 2015, vol. 9358, pp [20] A. P. Moore, S. J. D. Prince, J. Warrell, U. Mohammed, and G. Jones, Superpixel lattices, in Computer Vision and Pattern Recognition, CVPR IEEE Conference on, June 2008, pp [21] O. Veksler, Y. Boykov, and P. Mehrani, Computer Vision ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part V. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, ch. Superpixels and Supervoxels in an Energy Optimization Framework, pp [22] M. Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, Entropy rate superpixel segmentation, in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, June 2011, pp [23] D. Comaniciu and P. Meer, Mean shift: a robust approach toward feature space analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp , May [24] A. Vedaldi and S. Soatto, Computer Vision ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part IV. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, ch. Quick Shift and Kernel Methods for Mode Seeking, pp [25] A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson, and K. Siddiqi, Turbopixels: Fast superpixels using geometric flows, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 12, pp , Dec [26] P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, Rgbd mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments, The International Journal of Robotics Research, vol. 31, no. 5, pp , [27] J. Park, H. Kim, Y. W. Tai, M. S. Brown, and I. S. Kweon, High-quality depth map upsampling and completion for rgbd cameras, IEEE Transactions on Image Processing, vol. 23, no. 12, pp , Dec [28] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, and A. Fitzgibbon, Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera, in Proc. of ACM UIST, 2011, pp [29] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. W. Fitzgibbon, Kinectfusion: Real-time dense surface mapping and tracking, in Proc. of IEEE ISMAR, 2011, pp