A Hierarchical, Multiscale Texture Segmentation Algorithm for Real-World Scenes

Transcription

1 A Hierarchical, Multiscale Texture Segmentation Algorithm for Real-World Scenes V Lakshmanan, Advisor: Dr. V. E. DeBrunner, Advisory Committee: Dr. J. Cheung, Dr. J. Havlicek, Dr. T. Przebinda, Dr. R. Rabin Oct. 25, 2001

2 Summary Summary 1. Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. 2. We can incorporate hierarchical segmentation into a K-Means clustering technique by steadily relaxing inter-cluster distances. 3. Thus, it is possible to hierarchically segment images based solely on texture measurements. 4. This hierarchical, multiscale segmentation is useful in identifying and tracking weather images. 1

3 Terms Introduction Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. What is segmentation? How has it been done in the literature? 2

4 Segmentation Segmentation Segmentation: splitting an image into several components, by assigning one of these components to each pixel in the image Figure 1: The image on the right shows the image on the left segmented. Pixels are assigned to either skin or wound (the two components of the scene). 3

5 Segmentation methods Segmentation How is segmentation typically done? 4

6 Segmentation Amplitude thresholding, which was used in the previous image. This works when there is clear separation between the pixel values of the components. Figure 2: Amplitude Segmentation. The gray-level histogram of the pixels in the photograph of wound in Figure 1a. Figure based on that of [1]. 5

7 Segmentation Edge-based methods which try to find edges and then fill in the edges. Figure 3: Contour-based segmentation 6

8 Segmentation Templates If you know what the objects should look like, just take a template and move it around the image. Those pixels belonging to a close match to the template belong to the object. 7

9 Segmentation Region growing When an unlabeled pixel is encountered, the next possible label is assigned to that pixel. Every neighbor of that pixel (a 4-neighborhood here) which should belong to the same region as that pixel is also associated with the new label. This process is carried out recursively until there are no neighbors who should belong to the same region. 8

10 Multiscale segmentation Multiscale Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. What do we mean by multiscale outputs? 9

11 Multiscale Segmentation Multiscale Many image processing algorithms and techniques lend themselves to a concept of scale that the results of the analysis would be different if one were concerned with a different level of detail. Left image: detailed segmentation (windows, floors, entrance, sky) Middle image: less detailed (building, trees, sky) Right image: coarse (building, sky) 10

12 Multiscale Outputs Multiscale But the multiscale results shown in the previous slide are not the usual definition of multiscale. Typically, in image processing, when people talk about multiscale, they are thinking of multiscale inputs, not multiscale outputs. We are talking about multiscale outputs. 11

13 Multiscale Segmentation Multiscale Traditionally, multiscale segmentation is done in one of two ways: 1. Image pyramids where wavelets or filter banks are employed to obtain the image at different scales (with the original image as the most fine resolution available). Each of these images is then segmented. 2. Quad-tree decomposition where the entire image is assumed to be a single region, then split into smaller regions, on each of which the process is repeated. Similar regions are merged at each stage. 12

14 Relationship between regions Multiscale How are the regions obtained at each of the scales related? Typically, the relationship between the segmented regions at the different scales are of no interest. If they are, then components at different scales have to be associated in some, often heuristic, manner. For example, in the image pyramid case, you move links until a global cost function is minimized: In our case, the regions obtained at each scale are related in a very specific manner. 13

15 Nested Partitions Multiscale Instead of trying to associate regions obtained from different scales, our segmentation process will automatically put the regions into nested partitions. Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. 14

16 Hierarchical clustering Multiscale Such a partitioning method, where the partitions at each scale are arranged such that every partition at a particular scale wholly contains one or more regions in the next most detailed scale is called hierarchical clustering. You can do this either by merging smaller clusters (agglomeration) or dividing up large clusters. 15

17 Hierarchical segmentation Hierarchical Segmentation is the process by which we assign image pixels to various regions. Can we combine hierarchical clustering with segmentation? At the time we assign pixels to regions, can we assign the regions in a nested manner? Yes. 16

18 Watershed Segmentation Hierarchical Watershed segmentation [2] is a well known hierarchical segmentation method. Images are considered to be topographic reliefs, with the gray level at a pixel standing for the elevation of that point. Watershed segmentation is the process of finding watershed lines which separate catchment basins. Watershed segmentation can provide nested partitions because there is a way to test how salient a watershed line is, and thus to combine regions. 17

19 If we want nested partitions... Hierarchical So, why don t we simply use watershed segmentation? Because watershed segmentation can not deal with texture. What is texture? 18

20 Texture Texture Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. 19

21 Texture Texture Texture analysis is commonly used when there is significant variation between the intensities of adjacent pixels even in the absence of an edge between the pixels. In the presence of texture, many image processing algorithms fail. This is because image processing algorithms often rely on the idea that when there is a gradient in pixel value, there is an edge between different objects in the scene. 20

22 Texture Segmentation Texture The idea is to associate with each pixel, not just a single pixel value, but a vector of texture measurements. The underlying assumption in texture segmentation is that the different regions that are the result of segmentation possess different textural attributes. Approaches in the literature include: 21

23 Texture Gabor filters: a Gaussian filter that has been modulated by a sinusoid. By choosing the standard deviation of the Gaussian kernel whose orientations define the feature vector, the scale at which the regions are identified may be chosen. In our comparisons, we will use the Gabor filter method introduced in [3] by Ma and Manjunath. 22

24 Texture Markov Random Field Approach The most popular MRF models use cliques. A clique is a set consisting of the pixel in question and zero or more of its neighbors. For example, in a second order MRF model (which corresponds to a 3x3 neighborhood) has these cliques: Unsupervised methods proposed (for example, [4, 5]) involve one or more of these assumptions: 1. The image can be histogram quantified into a small number of gray levels. 2. The number of different textured regions in the image 23

25 is known. Texture 3. The image can be divided into mostly homogeneous blocks so that texture parameters can be estimated on the blocks. We will use test images from the original study [6] to compare. 24

26 Texture Kolmogorov-Smirnov test The Kolmogorov-Smirnov (KS) test can be used to test the hypothesis that an observed distribution function of independent random variables belongs to a specified distribution function at varying confidence levels. Assume that the segmentation has been initialized in some manner. Then, we can compute the distribution of the features in the pixels within each region. We can then test whether the local distribution of these features around a pixel is part of the global distribution of these features within a region. We will compare our method with the KS Test method introduced in [7]. 25

27 Texture K-Means clustering K-Means clustering is a clustering technique where the clustering proceeds by computing the affinity of each entity with each of K clusters that already exist. The entity is assigned to the cluster to which it is closest in the measurement space. The means are updated and the process is repeated for the next entity. The entities are cycled through until no entities are moved between clusters. We will compare against the contiguity-enhanced K-Means approach of [8]. 26

28 But how? Texture Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. Such a texture segmentation would be hierarchical. How can we do hierarchical texture segmentation? Why would we want to do hierarchical texture segmentation? 27

29 Motivation Texture Tracking storms is very useful for meteorological algorithms [9]. The notion of scale is natural in the storm tracking context. A multiscale tracking algorithm would be a significant improvement over current tracking schemes which concentrate either on small scales [9] or on large scales [10]. Any successful segmentation technique that is useful for identifying storms at small scales from infrared imagery would be a useful contribution to the applied meteorology community. A multiscale segmentation algorithm for weather imagery, whether radar reflectivity or satellite infrared, would be an advance over the current methods. 28

30 Method Method We can incorporate hierarchical segmentation into a K-Means clustering technique by steadily relaxing inter-cluster distances. 29

31 K-Means Clustering Method Steps in the iterative multiscale segmentation method proposed in this thesis: 1. Associate a vector of textural measurements with each pixel. 2. Requantize image into K levels using K-Means Clustering. (a) Initialize the k means somehow we simply divided up the measurement space into equal intervals. (b) Assign the closest mean to each pixel. (c) Start iterating on the clustering scheme by reassigning pixels based on the Markov assumption. (d) Iterate until stable 3. Parcel off into connected regions. 30

32 Method 4. If region is too small, choose the closest region (in textural measurement space) amongst its neighbors and reassign it. 5. Increase the allowed inter-cluster difference and repeat to obtain regions at different scales. 31

33 Texture measurements Method There are many choices that can be made for texture measurements. There is no optimal texture vector. That depends on the image in question. We used the pixel value, mean, variance, kurtosis, contrast, etc. in a neighborhood. We assumed the texture was statistical in nature. Our method works equally well with other ways of computing texture. For example, using wavelet coefficients as the elements of the texture vector. 32

34 Method 33

35 Neighborhood Method When computing statistical texture, we compute the statistics in the neighborhood of a pixel. How big a neighborhood? We would expect that with a tight neighborhood, we would get a large number of small regions, and that with a large neighborhood, we would get a fewer number of larger regions. That is what happens. 34

36 Method Figure 4: Segmenting a satellite image using statistics computed in neighborhoods of varying sizes (3,5 and 7) 35

37 K-Means Method Step 1: compute texture vector at each pixel. Step 2: Requantize image into K levels using K-Means Clustering. 1. Initialize the k means. 2. Assign the closest mean to each pixel. 3. Start iterating on the clustering scheme by reassigning pixels based on the Markov assumption. 4. Iterate until stable 36

38 What is K? Method K is not the number of regions in the final segmented output. It is the number of significant gray levels in the image into which we requantize the pixel values. We initialize the K values by uniformly quantizing the measurement space and assigning to each pixel the cluster with the closest mean. 37

39 Method The effect of varying K when segmenting the building image: a b c d Figure 5: The effect of varying K on the most detailed segmentation. (a) Original image, from [11]. Segmentation result (most detail) with (b) K=2 (c) K=4 (d) K=8 As the number K increases, the clusters cover a smaller range in the texture space. 38

40 Method In case the number of regions is not known a priori, a very high value of K may be chosen. The most detailed segmentation may have too many regions, but coarser levels might yield the desired result. This is one advantage of using a hierarchical technique. 39

41 K-Means contd. Method 1. Take as candidates all the labels in the 8-neighborhood of the pixel. 2. Compute the contiguity distance d c (k) that would result if the label were changed to k. 3. Compute the distance between the mean of the kth cluster and the pixel s texture vector, d m (k). 4. Assign to the pixel the label for which E(k) is minimum. If this causes a change in the label of the pixel, another pass through the image is required. 40

42 Method We iteratively move pixels minimizing E(k) = λd m (k) + (1 λ)d c (k) 0 λ 1 (1) where the distance in the measurement space is: d m (k) = µ n k T xy (2) and the discontiguity measure is:: d c (k) = (1 δ(sij n k)) (3) ijɛn xy 41

43 What is lambda? Method The weight, λ is the weight given to the texture component of the global energy, d m (k), relative to the contiguity measure, d c (k). 42

44 Method a b c d e f Figure 6: (a) λ = 0.2 (b) λ = 0.4 (c) λ = 0.6 (d) λ = 0.8 (e) λ = 0 (f) λ = 1 43

45 Method Any one of the mid-range λ s does just as well as any of the others. We can control the smoothness of the resulting segmentation using the value of λ, but not by much. 44

46 Hierarchical Segmentation Method A region growing algorithm is employed to build a set of connected regions, where each region consists of 8-connected pixels that belong to the same K-Means cluster. If a connected region is too small, then its cluster mean (the mean of the texture vectors at each pixel in the region) is compared to the cluster means of the adjoining regions and the small region is merged with the closest mean. The result of the K-Means segmentation, region growing and region merge steps is the most detailed segmentation of the image. 45

47 Method The inter-cluster distances of all adjacent clusters (or regions) in the image are computed. A threshold is set such that half the pairs fall below this threshold. If a pair of clusters differ by less than this threshold, they are merged and cluster means updated. Continue until no two adjacent regions are closer in cluster space than the threshold. When this process is complete, we have the next coarser scale of the segmentation. Repeat this process until no changes happen. 46

48 Comparison with other multiscale approaches What s new The image pyramid approach employs images at different scales. The proposed method does not decompose the original image, only the regions identified in the original image. Merging of regions in the quad-tree approach is based on local characteristics and a hard threshold. In the proposed approach, the choice of which regions to merge is made globally. In the image pyramid approaches (in [12] for example), the texture boundaries are not reliable at coarse scales. In our approach, boundaries remain reliable even at the most coarse segmentation. The scale in the image pyramid approach, especially, is tied to the image space. In the proposed method, the scale in question refers to the components themselves. 47

49 What s new In a multiscale tracking problem, the matching of segmented components will have to match not only across frames but across scale as well. By ensuring that there is no leeway in the association of components across scale, we reduce the dimensionality of the association problem in multiscale tracking. 48

50 Comparison with Ahuja What s new In [13], the author states as a goal, one similar to ours:... the objective is to derive multiscale segmentation of the image and represent it through a hierarchical, tree structure in which the different image segments, their parameters, and their spatial interrelationships are made explicit. The methodology followed by [13], is however, different from the one proposed in this thesis. In [13], the image is transformed using a transformation that partitions the image such that each cell of the partition has a characteristic property. This is done by computing an attraction-force field over the image with the force at each point denoting the affinity to the rest of the 49

51 What s new image. Thus, spatial extent is carried through the entire process, albeit in a different form. 50

52 Comparison with Kim What s new In [14], a lot of the segmentation work is carried out in the region domain rather than in the image domain. The motivation of the work is, however, different it is to simply use a single set of constraints in both the segmentation and interpretation problems. The two normally independent stages of the typical computer vision problem cooperate to minimize errors. In fact, the interaction happens via region clusters [14], which are the atomic unit in our multiscale approach as well. However, [14] does not deal either with texture or with multiscale segmentation. 51

53 Comparison with other K-Means Methods What s new In our variation of K-means clustering, K is better thought of as similar to the number of gray-level values we wish to requantize the measurement space into. This idea itself is not novel a similar approach was taken in [4] in the context of MRF-based texture segmentation. A histogrambased technique, not K-means clustering, was used in [4]. We follow the clustering and region growing stage with steps that use texture-vector distances between statistically unsound regions and their statistically sound neighbors to yield a robust segmentation at the finest level of detail. We show how the use of inter-cluster distances leads naturally to a hierarchically arranged tree of identified regions. 52

54 Results Comparison: other texture segmentation algorithms 53

55 Results a b c d e f Figure 7: (b) detailed (c) coarser (d) Using the method of [6] (a Gabor method, from which this image was taken) assuming that there are four clusters. (e) MRF-based approach of [15]. (f) Gabor method of [3] 54

56 Comparison: medical image Results 55

57 Results a b c d e f Figure 8: Segmenting a photograph of a leg wound. (c) Most detailed segmentation (d) Coarser (e) MRF-based approach of [15]. (f) Gabor filter texture segmentation method of [3] 56

58 Results These results are possible, even in the absence of any customization to the problem of interest, because the ulcer image satisfies the requirements of what we call real-world scenes : the texture is statistical, rather than regular and periodic, and the different objects in the scene (skin and lesion, in this case) differ significantly in texture. The MRF-segmented image does not hew to the actual boundaries of the wound. The Gabor filter captures all the edges, not just the salient ones. 57

59 Segmentation speed Results Method CPU time (µsec) Time sec Memory kb A 256x178 image of a leg ulcer wound Gabor [3] MRF [15] Method of this dissertation A 307x307 image of radar reflectivity Gabor [3] MRF [15] Method of this dissertation Table 1: The computer resources required by each of the texture segmentation methods compared in this section. 58

60 Segmentation Accuracy Measure Results For synthetic images where we know the actual object boundaries: a = j S i 1 j N i1 jɛe 1 S i2 j N i2... jɛe k 1 S ik j N ik (4) The larger the value, the more accurate the segmentation. 59

61 Brodatz textures different values of K Results Left: images segmented Right: K=4, 8. (second most detailed in both cases) a b c d 60

62 Brodatz textures results Results Test/Technique KS Test K-Means [8] Hierarchical K-Means (Parameter) [15, 16] K=2or3 K=4 K=2 K=4 K=8 K=16 D12 and D D24, D38, D D84, D94, D D112 and D D15, D112,D D38, D µ σ/µ

63 Brodatz textures different methods Results (a) Image comprised of Brodatz textures D24, D38 and D68. The result of segmentation by (b) the method of [15]. (c) the method of [8] with K=3 (there are 3 regions). (d) the method of this dissertation with K=2. (e) the method of this dissertation with K=8 (the best performance). (f) the method of this dissertation with K=16. This is the second test referred to in the previous table. 62

64 Results a b c d 63

65 Brodatz textures nested partitions Results (a) Image comprised of Brodatz textures D112 and D19. The result of segmentation using the method of this dissertation with K=16 at various scales. (b) most detailed (accuracy=0.04) (c) second most detailed this is what is used in the accuracy computation in the table. accuracy=0.53. (d) coarse this is actually the most accurate (accuracy=0.76). This is the fourth test referred to in the table. 64

66 Results a b c d 65

67 MPEG-4 methods Results Techniques reported in the literature usually utilize one of the major segmentation approaches described and compared. For example, the MPEG-4 methods of [17] and [18] both use clustering. The method of [17] uses statistical texture vectors while the method of [18] uses density gradients. We did not develop our segmentation technique to fit into the MPEG- 4 framework, or to follow the MPEG specifications. However, it is possible to compare the results of segmenting images using the method described in this dissertation with MPEG-4 methods that provide access to this intermediate output, as reported in the literature. 66

68 MPEG comparison Results 67

69 Results a b c d Figure 9: (a) The image from [17] was segmented using MPEG-4 segmentation techniques and compared against the method of this thesis. (b) When segmented using the MPEG-4 method of [17] (c) When segmented using the MPEG-4 method of [18] (d) When segmented using the method of this dissertation 68

70 Weather radar imagery Results 69

71 Results 70

72 Weather satellite images Results 71

73 Results a b c d e f (b) Markov Random Field (MRF) approach of [15]. (c) Using the method 72

74 Results of this dissertation (the most detailed scale). (d) The next higher scale of segmentation. (e) 1K regions. (f) Using the watershed segmentation approach of [2]. 73

75 Conclusions Results 1. Texture segmentation can lead to multiscale outputs in which the partitions at successive scales are nested. 2. How: We can incorporate hierarchical segmentation into a K-Means clustering technique by steadily relaxing inter-cluster distances after the clusters have been identified on the basis of texture measurements and morphology. 3. Thus, it is possible to hierarchically segment images based solely on texture measurements. 4. This hierarchical, multiscale segmentation is useful in identifying and tracking weather images. 74

76 Resources Resources These slides, and papers written on various components of this dissertation are online at: lakshman/papers/ You can reach me by at: Seminar on application of this technique to weather images: Nov. 2 at 10.30am: NSSL Main Conference Room 75

77 Resources References [1] T. Jones, Improving the Precision of Leg Ulcer Area Measurement with Active Contour Models. PhD thesis, U. Glamorgan, May [2] L. Najman and M. Schmitt, Geodesic saliency of watershed contours and hierarchical segmentation, IEEE Trans. Patt. Anal. and Mach. Intell., vol. 18, pp , [3] W. Ma and B. Manjunath, Edge flow: a framework of boundary detection and image segmentation, in Proc. IEEE International Conference on Computer Vision and Pattern Recognition, (San Juan, Puerto Rico), pp , June [4] P. Andrey and P. Tarroux, Unsupervised segmentation of Markov Random Field modeled textured images using selectionist relaxation, IEEE Trans. on Pattern Anal. and Mach. Intell., vol. 20, no. 3, pp , [5] B. Manjunath and R. Chellappa, Unsupervised texture segmentation using markov random field models, IEEE Trans. on Pattern Anal. and Mach. Intell., vol. 13, no. 5, pp , [6] T. Hofmann, J. Puzicha, and J. Buhmann, A deterministic annealing framework for unsupervised texture segmentation, Tech. Rep. IAI-TR-96-2, Institut fr Informatik III, U. Bonn, [7] C. Kervrann and F. Heitz, A markov random field model-based approach to unsupervised texture segmentation using local and global spatial statistics, IEEE Trans. on Img. Proc., vol. 4, pp , Jun [8] J. Theiler and G. Gisler, A contiguity-enhanced K-Means clustering algorithm for unsupervised multispectral image segmentation, in Proc. SPIE, vol. 3159, pp , [9] J. Johnson, P. Mackeen, A. Witt, E. Mitchell, G. Stumpf, M. Eilts, and K. Thomas, The storm cell identification and tracking algorithm: An enhanced WSR-88D algorithm, Weather and Forecasting, vol. 13, pp , June [10] M. Wolfson, B. Forman, R. Hallowell, and M. Moore, The growth and decay storm tracker, in 8th Conference on Aviation, (Dallas, TX), pp , Amer. Meteor. Soc.,

78 Resources [11] J. van Hateren and A. van der Schaar, Independent component filters of natural images compared with simple cells in primary visual cortex, in Proc. R. Soc. Lond., vol. B 265, pp , [12] H. Choi and R. Baranuik, Multiscale texture segmentation using wavelet-domain hidden markov models, in Proc. 32nd Asilomar Conf., Nov [13] N. Ahuja, A transform for multiscale image segmentation by integrated edge and region detection, IEEE Trans. on Pattern Anal. and Mach. Intell., vol. 18, no. 12, pp , [14] I. Kim and H. Yang, An integrated scheme for image segmentation and labeling based on markov random field model, IEEE Trans. on Pattern Anal. and Mach. Intell., vol. 18, no. 1, pp , [15] J. Blum and J. Rosenblat, Probability and Statistics. W.B. Saunders Company, [16] V. Lakshmanan, V. DeBrunner, and R. Rabin, Texture-based segmentation of satellite weather imagery, in Int l Conference on Image Processing, (Vancouver), pp , Sept [17] J. Kim and T. Chen, A VLSI architecture for image sequence segmentation using edge fusion, in International Workshop on Computer Architectures for Machine Perce, (Padova, Italy), Sep [18] D. Comaniciu and P. Meer, Robust analysis of feature spaces: Color image segmentation, in IEEE Conf. on CVPR 97, (San Juan, Puerto Rico),