Efficient Image Segmentation Using Pairwise Pixel Similarities

Transcription

1 Efficient Image Segmentation Using Pairwise Pixel Similarities Christopher Rohkohl 1 and Karin Engel 2 1 Friedrich-Alexander University Erlangen-Nuremberg 2 Dept. of Simulation and Graphics, Otto-von-Guericke-University Magdeburg Abstract. Image segmentation based on pairwise pixel similarities has been a very active field of research in recent years. The drawbacks common to these segmentation methods are the enormous space and processor requirements. The contribution of this paper is a general purpose two-stage preprocessing method that substantially reduces the involved costs. Initially, an oversegmentation into small coherent image patches - or superpixels - is obtained through an iterative process guided by pixel similarities. A suitable pairwise superpixel similarity measure is then defined which may be plugged into an arbitrary segmentation method based on pairwise pixel similarities. To illustrate our ideas we integrated the algorithm into a spectral graph-partitioning method using the Normalized Cut criterion. Our experiments show that the time and memory requirements are reduced drastically (> 99%), while segmentations of adequate quality are obtained. 1 Introduction The segmentation of images into meaningful regions is an important task of computer vision. One common approach is the definition of a similarity measure Φ(i, j) between two image pixels i, j based on e.g. color, texture, proximity or contour information. Let N in the following denote the number of image pixels. Usually from Φ a N N similarity matrix W is derived with W ij = Φ(i, j). W may be interpreted as an adjacency matrix for an undirected weighted graph G =(V,E), whose nodes V represent the image pixels which are connected by the affinity-weighted edges in E. In order to receive a partition of the image an arbitrary method may be applied to W. The most popular methods are spectral graph partitioning [1,2], deterministic annealing [3] and stochastic clustering [4]. Unfortunately, for most problems W easily becomes very large and unmanageable. For example, given a relatively small image (N = ) W contains 14.4 billion entries and requires about 53.6 GB of memory at single precision. Thus prevalently the number of connections per pixel is restricted. We present a two-step preprocessing method which substantially reduces the time and memory requirements such that even much larger problems may be addressed by any pairwise grouping algorithm. The remainder of this paper is organized as follows: In Sect. 2 we discuss related work. We present our algorithm in detail in Sect. 3. The Normalized Cut grouping method into which we F.A. Hamprecht, C. Schnörr, and B. Jähne (Eds.): DAGM 2007, LNCS 4713, pp , c Springer-Verlag Berlin Heidelberg 2007

2 Efficient Image Segmentation Using Pairwise Pixel Similarities 255 integrated our algorithm is reviewed in Sect. 4. In Sect. 5 we present the results of our method applied to the segmentation of synthetic and natural images. 2 Related Work A very common procedure to make pairwise grouping methods tractable is the creation of a sparse version of the similarity matrix by zeroing entries, i.e. by removing edges from the graph representation. Shi and Malik [1] proposed the approximation of W by setting a cutoff radius in the image plane such that each pixel is connected to only a few of its neighbors. Alternatives include the zeroing of randomly chosen entries and of, e.g. the smallest matrix entries. This approach was confirmed to be very effective by [5]. However, this requires all matrix elements to be calculated. Fowlkes et al. [5] utilize the Nyström method to approximate the similarity matrix. Initially, a set of randomly chosen sample pixels and their corresponding connections to all other pixels are extracted. This small portion of the similarity matrix is then used to estimate all remaining connections. The major reduction of the computational effort is achieved by calculating the row sums of the approximated similarity matrix without the need to estimate all matrix entries. Hence, this approach is attractive for methods that actually require the row sums of W. Keuchel and Schnörr [6] propose a singular value decomposition (SVD) approximation method for W based on probabilistic sampling. Sharon et al. apply a multiscale method to find an approximate solution to normalized cut measures [7]. Several pairwise segmentation techniques may take advantage of the mentioned approximations but not all can do so [8]. More general approaches [8, 9] generate a suitable oversegmentation of the image in terms of a preprocessing step. Ideally this oversegmentation does not miss any boundaries. It thus can be used to derive a new (much smaller) similarity matrix by means of a pairwise measure for the obtained image patches. This in turn is used to feed the pairwise grouping method. Keuchel et al. [8] created an irregular oversegmentation using the mean shift algorithm. The image patch affinities are described using a region feature, e.g. the mean color. Malik et al. [9] proposed to use a sparsified version of the complete similarity matrix in order to separate the image into patches. However, this procedure still does not reduce the costs, as a convenient approximation scheme for W is still required. In this paper, we propose a method that provides a suitable oversegmentation and concurrently exploits the availability of a pairwise pixel similarity measure. In our approach an oversegmentation into small coherent image patches is obtained in an iterative manner. The algorithm offers a clean interface and may be easily integrated into existing pairwise grouping methods. 3 Our Approach The high spatial resolution of modern images makes optimization on the level of pixels intractable [10]. However, image pixels are no natural entities, being

3 256 C. Rohkohl and K. Engel rather a consequence of the digital discretization process. For these reasons we define image elements as small coherent image patches which we refer to as superpixels [10]. The interpretation and preferences of superpixels can be compared to a jigsaw puzzle. The superpixels are the pieces that make up the segments which reveal the complete image. With increasing size of the tiles the jigsaw gets easier and can be completed significantly faster. This analogy reveals two important aspects our algorithm is based on: 1. The superpixels must respect the boundaries of the true segmentation. 2. The size of the superpixels limits the resolution of the segmentation. Any true segment smaller than the superpixel size cannot be correctly segmented. For the creation of superpixels our algorithm solely requires the pairwise pixel similarity function Φ. This allows for the use of clever implementation schemes, e.g. a caching mechanism. In our experiments we stored the information needed for the similarity function (e.g. color) and performed the computation on the fly. 3.1 Oversegmentation into Superpixels The superpixel segmentation is calculated in a two-step process. An initial tiling of the image plane is achieved which is then adapted to the local image structure. Figure 1 shows an example of the superpixel segmentation. To obtain the initial segmentation into k superpixel segments, Ren et al. apply the Normalized Cut algorithm which produces superpixels of similar size and shape [10]. This fact is exploited by our approach which uses a raw segmentation into hexagonal superpixels. We decided for a regular hexagonal grid (see Fig. 2a) as it offers beneficial properties over other tilings [11]. Choosing a proper value for k may require a training stage [10]. However, this shall not be addressed in the scope of this paper. We choose k given a predefined minimum size of the segments, i.e. the granularity of the superpixels is user supplied. Starting from this raw segmentation the segments then adapt iteratively to the local image structure characterized by the pairwise pixel similarities. The (a) (b) (c) (d) Fig. 1. An example of a superpixel segmentation using color information for k = 400 at a sampling rate of r sp =0.01 and m = 30 iterations (see text for explanations). (a) the original image from the Berkeley Segmentation Dataset [12]; (b) a reference segmentation; (c) the final superpixel segmentation; (d) a reconstruction of (b) from the superpixel segmentation: each superpixel is assigned to the segment with the maximum overlapping area [10].

4 Efficient Image Segmentation Using Pairwise Pixel Similarities (a) 0 iterations 257 (b) 5 iterations (c) 15 iterations (d) 30 iterations (e) Final result Fig. 2. Superpixel segmentation process. The background shows the superpixel segmentation of the current iteration. The dark outline constitutes the reconstruction of a reference segmentation [10]. (a) The original image (see Fig. 1) is partitioned into approximately k = 400 hexagons. (b) - (d) Given this raw segmentation the segments are iteratively adapted to the local image structure. (e) The final superpixel segmentation after region merging to obtain a maximum of k = 400 image patches. similarity between a pixel i and a segment s can be defined as the average of the 1 similarities between i and all the pixels in s, namely simp (i, s) = s Φ(i, j). j s i For a set S of concurrent image patches the most likely segment s S is the one having maximum similarity to pixel i, i.e. 1 si = arg max simp (i, s) = arg max Φ(i, j). (1) s S,s\{i} s S,s\{i} s j s The pixel i is ignored in any segment as it would bias the result. Since the computational effort depends on the size of the segments we introduce a function Rn ( ) which returns n randomly selected elements from a set of pixels. Modifying (1) an estimation for the most suitable segment s i for a pixel i is then s i = arg 1 s S,s\{i} n max Φ(i, j). (2) j Rn (s) The initial segments adapt to the local image conditions in the following way: 1. For each boundary pixel i of the current segmentation identify the set P of feasible superpixel segments (P will contain the neighboring and the current superpixels. Hence most similarities are ignored.). Select the new superpixel s i from P according to (2). 2. Repeat step 1 until a stopping criterion is fulfilled. 3. Merge the smallest superpixels until a maximum of k superpixel segments is left. The spreading of the pixels is again calculated using (2). For reasons of simplicity, in our experiments we used color and proximity information to characterize the pairwise pixel similarity, see Sect. 5.2 for details. The number n of random samples used in step 1 is set to a fraction rsp of the size of the current superpixel segment. In the following rsp will be referred to as superpixel sampling rate. A preset number of m iterations was chosen as stopping criterion. (Alternatives may include the supervision of the number of consecutive fails to assign boundary pixels of the current segmentation to a new superpixel, but this shall not be discussed here).

5 258 C. Rohkohl and K. Engel Figures 1 and 2 show the results for an image taken from the Berkeley Segmentation Dataset [12] at a sampling rate of 1% for k = 400 superpixels and m = 30 iterations. To obtain these results in overall 8 million calls have been made to Φ, which corresponds to 0.04% of all pairwise pixel similarities. 3.2 Definition of a Pairwise Superpixel Similarity By hypothesis the true segmentation can be computed from the k image patches obtained through the oversegmentation of an image according to Sect To make the superpixels viable for a pairwise grouping method, a pairwise superpixel similarity measure has to be defined on the basis of the pairwise pixel similarity measure. A straightforward expansion of the measure of similarity sim p (i, j) between a pixel i and a segment s given in (1), to the similarity of two image segments s 1 and s 2 is obtained using the average similarity of each pixel of s 1 to the segment s 2,namely: sim s (s 1,s 2 )= 1 1 sim p (i, s 2 )= Φ(i, j). (3) s 1 s i s 1 s 2 1 i s 1 j s 2 However, the calculation of all pairwise superpixel similarities requires to compute all pairwise pixel similarities, which is exactly what we wanted to avoid. By means of the function R n ( ) whichgivesn randomly selected elements for any set of pixels, (3) can be modified. An estimation ˆΦ(s, t) for the similarity between superpixel segments s and t is then given by: ˆΦ(s, t) = 1 Φ(i, j). (4) n 1 n 2 i R n1 (s) j R n2 (t) The numbers n 1,n 2 of random samples can be adapted for each segment using the superpixel sampling rate r sp introduced in the previous section. By providing a suitable pairwise pixel similarity function Φ along with the parameters for the superpixel segmentation, i.e. k (the number of superpixels), r sp (the superpixel sampling rate) and the number of iteration steps m, the according superpixel similarity measure ˆΦ can be computed. This may be easily plugged into an arbitrary pairwise grouping method, e.g. [1]. 4 The Normalized Cut Framework The Normalized Cut framework introduced by Shi and Malik [1] provides a pairwise grouping method inspired by spectral graph theory. Image segmentation is described in terms of a graph partitioning problem. Recall that W may be interpreted as an adjacency matrix for an undirected weighted graph G = (V,E). Let A, B be a partition of the graph, i.e. A B = V, A B =. TheNormalized Cut criterion allows to evaluate the quality of a partition by extracting the global impression of an image: Ncut(A, B) = cut(a, B) assoc(a) + cut(a, B) assoc(b), (5)

6 Efficient Image Segmentation Using Pairwise Pixel Similarities 259 where cut(a, B) = i A,j B W ij and assoc(a) = i A,j V W ij. The determination of the best partition is a NP-complete problem. However, fast approximation methods exist. Let D be a diagonal matrix with D ii = N j=1 W ij. The optimal partition can be computed as follows: y T (D W )y y =argminncut =argmin y y T, (6) Dy with y {a, b} N being a binary indicator specifying the group membership for each pixel in either A or B. Ify is relaxed to take on real values (6) can be optimized by solving the generalized eigenvalue system (D W )y = λdy. The eigenvector y with the second smallest corresponding eigenvalue can be used to recursively compute a bipartition of the image. We suppose to check l evenly spaced possible splitting points, such that the resulting partition has the best Ncut value. A complete image segmentation is obtained by further splitting the computed segments, starting e.g. with the largest of the current segments. Any two segments which differ in size for a maximum allowable ratio e are both split. The partition with minimum Ncut value is accepted. The segmentation process can be stopped, e.g. after t partitions, or when a partition exceeds the maximum Ncut value c max. Our algorithm is plugged into this framework by calculating an approximative similarity matrix Ŵst = ˆΦ(s, t) from the superpixel segmentation. 5 Experiments 5.1 Segmentation of a Test Case In the images I 0 displayed in Figs. 3a-3b, the dark and light segments I0 D and I0 B were created from the normal distribution N(μ =0,σ =0.1) and N(μ =1.2,σ =0.1), respectively. The image segmentation I 0 = I0 D IB 0 was taken as ground truth. The difficulty of the segmentation task was increased by minimizing the difference of the means of the probability distributions of the dark and light segments in I 0. We therefore formally introduce a problem difficulty p [0, 0.75]. Each image I p is derived from I 0 according to: Ip D = I0 D + p 2 and Ip B = I0 B p 2. The similarity of two pixels i, j is given by the Gaussian weighted Euclidean distance function Φ(i, j) = exp ( x i x j /2σ 2),wherex i is the intensity value of pixel i. The quality of the Ncut bipartition provided by the eigenvector with the second smallest eigenvalue is estimated using the Jaccard coefficient [13]: J(T,S)=n 11 (n 11 + n 10 + n 01 ) 1 [0, 1]. Here, T is the true solution and S the segmentation to evaluate; n 11,n 01 and n 10 denote the number of pairs of elements within the same segment in both S and T,onlyinS and only in T, respectively. For each problem difficulty the parameter σ was optimized by testing several evenly spaced values in the non-approximated segmentation. The number of iterations during the superpixel segmentation was fixed to m = 30. The results of this experiment are presented in Fig. 3. As expected, the number of superpixels used directly influences the quality of the segmentation as well

7 260 C. Rohkohl and K. Engel (a) (b) 0.6 Optimum σ Parameter vs. Problem Difficulty 1 Jaccard Score vs. Problem Difficulty: Superpixels Optimum σ Problem Difficulty Jaccard Score Problem Difficulty Complete W 450 Superpixels 208 Superpixels 90 Superpixels 40 Superpixels 20 Superpixels 10 Superpixels 6 Superpixels (c) (d) 0.84 Jaccard Score vs. Problem Difficulty: Sampling Rate 10 4 Time vs. Problem Difficulty Jaccard Score 0.78 Time [ms] Problem Difficulty Problem Difficulty Complete W (e) (f) Fig. 3. Top row: Benchmark images of size used for evaluating the quality of our approximation scheme. (a) The benchmark image I 0 of difficulty p =0.(b)The benchmark image I 0.75 of maximum difficulty p =0.75. Second and last row: A study of the segmentation quality depending on the problem difficulty for different configurations of the algorithm in 500 trials. The non-approximated solution to the similarity matrix is denoted Complete W. (c) The optimum value for σ that maximizes the Jaccard score. (d) Comparison of the segmentation quality for different numbers of superpixel segments at a sampling rate of r sp = 1. (e) Estimation of the segmentation quality at different sampling rates given a fixed number k = 20 of superpixels. (f) Empirical analysis of the running time for the segmentations of (e). as the computational effort of our algorithm. Interestingly, for a large number of sampling rates the results are of similar quality and compare to the segmentation quality of the Normalized Cut algorithm. The computation of the segmentation

8 Efficient Image Segmentation Using Pairwise Pixel Similarities 261 (a) r sp =0.1 (b) r sp =0.01 (c) k = 700 Fig. 4. Color image segmentations of our algorithm using color and proximity information. The images are taken from the Berkeley Segmentation Dataset [12] and are of a size of pixels. Column (a) shows the segmentation on the basis of k = 700 superpixels at a sampling rate of 10%, the average segmentation time was about 4 minutes. In column (b) the images were segmented using the same configuration but a lower sampling rate of 1%. The running time decreased to less than 1 minute. (c) shows the superpixel segmentations obtained during the segmentation of (a) after m =30 iterations. Results on the complete dataset are available at effpixsegment.oneder.de.

9 262 C. Rohkohl and K. Engel using the complete similarity matrix W took at average 4.5 seconds. In contrast, Ŵ could be computed in about seconds, using 20 superpixels and a sampling rate of 64%. Approximately 99% savings in calculation time and 97% savings in memory requirements could be achieved at a quality loss of just 15%. 5.2 Segmentation of Natural Images To present some segmentations of real world images, color images from the Berkeley Segmentation Dataset are used ( [12]. The pairwise pixel similarities are calculated according to: Φ(i, j) =exp C(i) C(j) 2 σ ci X(i) X(j) 2 σ sl ),where C( ) is the 3-dimensional vector in the nearly perceptual uniform L*a*b* color space and X( ) the spatial location of the image pixels i, j. The parameters are set to 15% of the range of the feature distance function, i.e. σ ci =0.15 d max, and σ sl =0.15 D, whered max denotes the maximum color distance between randomly chosen pairs of pixels and D is the length of the image diagonal. The Normalized Cut implementation was parametrized with: l = 50, e = 0.5, c max =0.5, t = 10 (see Sect. 4). A fixed number of m = 30 iterations of the superpixel segmentation was computed. Figure 4 summarizes some segmentation results (without any postprocessing) for two different configurations of the segmentation algorithm. The first group of segmentations was obtained at a sampling rate of r sp =0.1 andk = 700. The whole segmentation process for an image at this configuration took about 4 minutes in unoptimized Java code on a Pentium 4 with 3GHz. The second group was achieved at a sampling rate of r sp =0.01 and k = 700. The computation took at average less than one minute. A direct comparison to the non-approximated version is not possible as the complete similarity matrix does not fit into memory. However, to estimate the savings of our approximation scheme the number of all calls to Φ was compared with the number of all pairwise pixel similarities. For the two given configurations we calculated a fraction of 0.75% and 0.03% which resembles to reductions in the order of 99.25% and 99.97% compared to the real solution. The segmentation quality of course highly depends on Φ which in our case utilizes color and proximity information only. The integration of suitable proximity, texture and contour cues [9] would yield better results. 6 Conclusion We presented a general approach toward making grouping methods based on pairwise pixel similarities applicable to real world problems. The technique is very straightforward and may be plugged into existing systems as a preprocessing step. The pixels are replaced by small coherent image patches - the superpixels. In order to make the superpixels utilizable in a pairwise grouping method, in the second step a new pairwise similarity measure for the superpixels is defined. It relies on the basis of the pairwise pixel similarity function used in the iterative superpixel creation. We demonstrated the integration into the Normalized Cut framework. The speed and memory requirements for segmentation can be

10 Efficient Image Segmentation Using Pairwise Pixel Similarities 263 reduced drastically without loosing significantly in quality. The savings of our approximation scheme are in the order of > 99% of the original problem size. The reconstructed segmentations are good approximations of the original. Depending on the granularity of the superpixels, some less significant details in the human segmentation may be lost (see Fig. 1). However, we yield comparable results as other approximation and speed up techniques [8,7,10]. Segmentation results and the source code are available at effpixsegment.oneder.de. References 1. Shi, J., Malik, J.: Normalized cuts and image segmentation. In: Proc. IEEE CVPR, vol. 731 (1997) 2. Wang, S., Siskind, J.M.: Image segmentation with ratio cut. IEEE Trans. Pattern Anal. Mach. Intell. 25(6), (2003) 3. Hofmann, T., Buhmann, J.M.: Pairwise data clustering by deterministic annealing. IEEE Trans. Pattern Anal. Mach. Intell. 19(1), 1 14 (1997) 4. Gdalyahu, Y., Weinshall, D., Werman, M.: Stochastic image segmentation by typical cuts. In: Proc. IEEE CVPR, vol. 02, p (1999) 5. Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the Nyström method. IEEE Trans. Pattern Anal. Mach. Intell. 26(2), (2004) 6. Keuchel, J., Schnörr, C.: Efficient graph cuts for unsupervised image segmentation using probabilistic sampling and SVD-based approximation. In: 3rd Internat. Workshop on Statist. and Comput. Theories of Vision, France (2003) 7. Sharon, E., Brandt, A., Basri, R.: Fast multiscale image segmentation. In: IEEE CVPR, vol. 1, pp (2000) 8. Keuchel, J., Heiler, M., Schnörr, C.: Hierarchical image segmentation based on semidefinite programming. In: Rasmussen, C.E., Bülthoff, H.H., Schölkopf, B., Giese, M.A. (eds.) Pattern Recognition. LNCS, vol. 3175, pp Springer, Heidelberg (2004) 9. Malik, J., Belongie, S., Leung, T., Shi, J.: Contour and texture analysis for image segmentation. Int. J. Comput. Vision 43(1), 7 27 (2001) 10. Ren, X., Malik, J.: Learning a classification model for segmentation. In: IEEE ICCV, vol. 10 (2003) 11. Brimkov, V.E., Barneva, R.P.: Analytical honeycomb geometry for raster and volume graphics. Comput. J. 48(2), (2005) 12. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. IEEE ICCV vol. 2, pp (2001) 13. Halkidi, M., Batistakis, Y., Vazirgiannis, M.: Clustering algorithms and validity measures. In: Proc. 13th Int. Conf. Scient. Stat. Datab. Man., USA, pp IEEE Computer Society, Los Alamitos (2001)