Improved watershed segmentation using water diffusion and local shape priors

Transcription

1 Improved watershed segmentation using water diffusion and local shape priors Hieu T. Nguyen, Qiang Ji Department of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute, USA {nguyeh2, Abstract The watershed algorithm has many nice properties in terms of robustness to image noise, topology, and effective handling non-rigid deformations. It also has drawbacks including oversegmentation and lack of regularization. We present new methods to overcome these drawbacks. We propose a novel region merging algorithm based on the water diffusion principle. Starting with a large number of markers, lakes formed around the markers are merged in the order they meet during the immersion. The merging is not performed immediately but delayed until the amount of water diffusion between two lakes is significant enough to overwhelm the small lake. The delay makes the merging result robust to leaks in object boundaries, when weak edges could trigger the merging of object into the background as in traditional methods. Regularization is achieved by imposing priors of local shape configurations. Local shape features are extracted from Gaussian derivatives of the object indicator function. The ensemble of shape features at multiple scales increases representation power. These features are used to incorporate smoothness and domain knowledge into the evolution of region boundaries in the watershed algorithm. The method has been successfully applied to segmentation of worms. 1 Introduction In image segmentation, achieving robustness to image noise and background clutter, flexibility for incorporation of domain knowledge, and insensitivity to initialization is always a challenging task. There have been two main approaches to segmentation: edge-based or region-based. The edge-based methods aim to find discontinuities of some image features at object boundaries [10, 3]. The approach inherently has problem with getting stuck at irrelevant edges before reaching the object border, since it cannot determine the edge strength that is sufficient for stopping the contour movement. As a consequence, the initial contour should be placed close to the object border. The region-based approach, on the other hand, finds regions that match an object model. Many region-based methods rely on a region uniformity model in intensity, color or texture [14, 6, 24]. The uniformity model, however, is often violated due to various types of image noise. Region-based methods also have problems with initialization of region-seeds. In recent years, progresses have been achieved in incorporation of prior knowledge into segmentation. Regularization is clearly an important component of any segmentation method, since optimization relying solely on low level image data will have many local optima representing irregular segmentations. The early form of regularization is smoothness [10, 3] which is too general and not sufficient to discriminate many types of objects. Other models include the active shape model [7], or recently proposed level set models [5, 19, 17, 4] minimizes a distance measure to one or a set of referenced contours. The method in [11] incorporates priors about curvature profile. These models are suitable for segmenting rigid objects, but not for objects under non-rigid deformations since the geometrical transformation of the object needs be included in the model and be estimated along with segmentation. This paper strives toward a segmentation method that is suitable for objects under non-rigid deformations like worms and snakes, and at the same time, robust to noise and initialization, and flexible for incorporation of prior knowledge. We base our method on the watershed algorithm from mathematical morphology [13, 21]. The algorithm is a nice balance of the edge-based and region-based approaches. While being a region growing method, it does not require a region model and grows regions using edge information. Furthermore, the algorithm converges to the strongest edges between regions, overcoming the local minima problem encountered by many other edge-based methods. Unfortunately, the watershed algorithm also has limitations including over-segmentation and lack of prior knowledge. Although these issues have been addressed in existing literature, drawbacks remain. The presented paper proposes new methods to tackle the two aforementioned limitations

2 of the watershed algorithm, making it an ideal tool for segmentation in the condition of image noise and non-rigid deformations. The paper is structured as follows. Section 2 discusses the related work. An overview of the method is given in Section 3. Section 4 presents a new watershed algorithm which merges regions during immersion based on a water diffusion model. Section 5 presents a new approach to incorporation of prior knowledge into the watershed algorithm, including smoothness and domain knowledge. Section 6 demonstrates the segmentation results of images of nematode worms. 2 Related work Recall that the standard watershed algorithm [13, 21] is a region growing algorithm that takes as input a set of region seeds and a gradient surface f. The region seeds, also called markers, are usually defined from regional minima of the surface or are imposed. The surface is then immersed into water with holes at the markers. Water will flood the surface through the holes and form a lake (catchment basin) around each marker. A dam is built at every point where two growing lakes meet to prevent the mixing of water. These dams will form the boundary at the final segmentation. In the original algorithm, markers are selected from local regional minima of the surface, which usually are in a large number and lead to an over-segmentation. In the existing literature, there are two main groups of methods for reducing overs-segmentation. The first group uses the waterfall model [1, 2]. This approach reconstructs the surface by replicating a rain fall in which catchment basins representing sub-local minima will be filled up and merged into adjacent lower and more significant minima. Re-applying the watershed algorithm to this surface removes those watershed contours that are completely surrounded by higher ones. While the idea is nicely demonstrated in one dimensional case, in the two dimensional case, it is sensitive to leaks on object border, which are border segments of weak edge strength. This problem can be seen the example of Figure 1b, where the edge leak on the border of region 2 leads to the merging of region 2 into region 1, although one would like to merge region 2 into region 3 since the overall edge strength along the border of region 2 is stronger than the edges around region 3. Other approaches to oversegmentation include pre-processing smoothing [23] and merging of watershed regions in a post-processing phase, using a similarity measure [21, 8]. This measure can be based on intensity, color, the length and edge strength along region border. It is difficult however to derive a robust similarity measure that could unify all these factors without a careful tuning of parameters. In addition, separation of merging and immersion compromises the accuracy of the watershed algorithm in detecting strongest edges. a) Figure 1. a) Illustration of surface modification by the waterfall algorithm [1], b) An example where the water-fall and other alike algorithms could fail due to the overflow between R 1 and R 2 through the leak on the boundary of R 2. The second limitation of the watershed is lack of a framework allowing for regularization of segmentation results. There is a still small number of work in this topic. The approaches impose smoothness of watershed lines only [15, 16, 12]. The method in [9] incorporates prior information of intensity of each region into the topographical distance which is then used to determine the watershed line. In [15], an energy-based formulation of the algorithm is proposed. Nevertheless, the method remains imposing smoothness only. Incorporation of any shape knowledge that need be learned a priori remains an open question. To overcome the drawbacks of the previous approaches, this work concentrates on fixing the edge-leaking problem as well as developing a more powerful regularization model for the watershed segmentation. 3 Overview of the proposed method We consider the segmentation of one object from an given image region. As illustrated in Figure 2, the algorithm has two stages, each is an improved version of the watershed algorithm. In the first stage, the watershed algorithm is initialized with a large number of markers which are subsequently merged into two main regions corresponding to the object and the background respectively. Both the regions are then eroded, leaving a band of uncertain pixels between them. In the second stage, the two contours of these regions advance toward each other with a speed function controlled by image data, smoothness and domain knowledge. The final contour is determined as the collision points of these contours. R1 R2 R3 b)

3 crater lake R1 image diffusion based watershed erosion incorporation of prior knowledge R3 R2 R3 R1 R2 a) b) Figure 2. Overview of the proposed method. 4 Diffusion based watershed To make the watershed algorithm insensitive to marker selection, we start with a large number of markers R 1,...R n, and then merge the corresponding lakes during the immersion process. For lake merging, it is desirable that lakes are merged in the order that they meet. Such a merging preserves the advantage of the original watershed in terms of converging to strongest edges. In fact, if the gradient is high over the entire object border, all lakes inside the object will merge together before they meet any lake in the background. The same thing holds for lakes in the background. The only reason that could make the merging go wrong is again the leaks on object border, which could lead to early overflow between the object and background. An example is illustrated in Figure 1b, where the classical water-fall model also fails. Our suggestion is to delay the merging of two lakes until a later point in time when there is enough evidence that no significant edge separates these lakes. This delay is achieved by allowing diffusion of water between lakes. Specifically, when two lakes meet, instead of building a dam we place a gate at each border point that allows water from the lakes to enter each other. More gates are built as more border points are determined. We then measure the amount of water diffused across the gates cumulatively over time, using a flux rate ν. The lakes are merged when the amount of diffusion exceeds the volume of one of the lakes, in which case the small lake is considered to be overwhelmed by water from the large lake. The merging is illustrated in Figure 3. The proposed merging principle encourages the merging of small and shallow lakes into the bigger ones as well as the merging of lakes sharing a long but low watershed line. The region intensity properties can also be taken into account by setting the flux parameter to a region similarity measure in intensity. We use: ν ij = γ Īi Īj (1) where ν ij is the flux rate between regions R i and R j, Īi and Ī j are the mean of intensities in R i and R j respectively, and Figure 3. a) The illustration of the diffusion based watershed algorithm. b) In this example, although R 2 meets R 1 first, R 2 will merge with R 3 first due to the stronger diffusion and small size of R 3. γ is a predefined constant. In our implementation, time is measured in the number of pixels added to growing regions. About the stopping criteria, one can require the merging be stopped when the entire surface is flooded, and then tune the flux parameter to control the final number of regions. In this work, we merge regions until a required number of regions K is reached. In several applications, this number is known in advance, especially in a local segmentation problem like the focus of this work, where the task is to segment only one object from a specified image patch. Small lakes at high places can cause problems for the proposed merging however. Although small, these crater lakes will not be merged until the end of flooding, see Figure 3a. Since the final regions that correspond to such lakes are small as well, the problem is resolved by running the algorithm twice. In the first run, the number of regions is set to a high value. Markers in small regions are then removed. The remaining regions are then eroded and used as markers for the second run of the algorithm in which the number of final regions is set to the actual required value. 5 Incorporation of prior knowledge This section presents a new approach for incorporation of prior knowledge into the watershed algorithm for the case of two-layer object/background segmentation. For this purpose, we view the watershed line as the locus of points of collision between the expanding contour C in and the shrinking contour C out, see Figure 2. For the original watershed algorithm, the speed of these contours along their 1 normal can be expressed as f provided f is reconstructed so that the markers become its only regional minima [20]. Prior knowledge is incorporated into the contours, and hence, the final segmentation, by adding appropriate regu-

4 larization terms to the speed functions as follows: ( ) C in 1 = H t f V n regularization,in, ( C out = H 1 ) t f V n regularization,out (2) where n denotes the contour normal, V regularization,in and V regularization,out denote the shape regularization terms for C in and C out respectively. H is the Heavy side function: { 1, v 0 H(v) = (3) 0, v < 0 which ensures that the curves monotonically advance and therefore, the evolution can be efficiently implemented by the fast-marching algorithm [18]. 5.1 Regularization using multiscale local shape features The regularization terms V regularization,in and V regularization,out are defined for individual pixels on the contour. We derive these terms from a set of features that characterize the local shape of the contour at different resolution scales. B= 1 B=+1 Figure 4. Multiscale Gaussian derivatives of the object indicator function B can be used to represent different local shape structures. σ For each contour, define the object indicator function: { 1 if x is inside the contour Cin B in (x) = (4) 1 otherwise The function B out (x) is defined in a similar way. To avoid unnecessary duplication of presentation, we will use the notation B. and C. where the dot. denotes either in or out. For each contour pixel x C. consider the Gaussian derivatives of B.: B (σ) pq,.(x) = g (σ) pq (r; σ)b.(x + r)dr (5) where pq denotes the pq-order derivative, g pq is the derivative of the Gaussian function, and σ denotes the Gaussian scale. Since C. is a zero level curve of B., various local geometrical invariants of C. can be calculated from the derivatives of B.. In this work, we use the two following types of invariants: 1. the gradient norm: 2. and the curvature: B. (σ) = κ (σ). = B(σ) 20,. B(σ)2 B (σ)2 10,. + B(σ)2 01,., (6) 01,. 2B(σ) 11,. B(σ) 10,. B(σ) 01,. + B(σ) 02,. B(σ)2 10,. (B 10,. (σ)2 +. B(σ)2 01,. )3/2 (7) At a single scale, these two features do not tell much of the local contour structure. However, the ensemble of the features at multiple scales brings enough information for the detection of complicated contour structures. In case of worm-like objects, measurements of curvature at multiple scales can tell, for example, an abnormal protrusion that is a segmentation error from a sharp end of the object, see Figure 4. As will be shown in the next section, these measurements also allow for more accurate estimation of contour curvature. Measurements of gradient norm at multiple scales can provide information about the width of the object if the size of the Gaussian kernel is set large enough to reach the other side of the object, see Figure 4. Thus, the local shape vector is defined as: B (σ1). (x)..., s.(x) = B. (σm) (x) (8) κ (σ1). (x)... κ (σm). (x) where σ 1,...,σ m are the given set of scales. This vector is used to define the regularization terms each as the sum of two subterms: V regularization,.(x) = 1+V smooth,.(s.(x)) + V prior,.(s.(x)) (9) where V smooth,. is the smoothness term, and the term V prior,. is for incorporation of domain knowledge. These subterms will be elaborated in the next subsections. Note that the Gaussian derivatives can be efficiently computed and updated during contour evolution. Specifically, for the computation of B pq,.(x), (σ) the scanning over the neighborhood of x need be done only once when x is assigned to one of the markers. When more pixels in the

5 neighborhood of x are added to the marker, B (σ) pq,.(x) may change and need be recomputed. However, any changes of this type can be calculated efficiently without re-scanning the neighborhood of x. 5.2 Multiscale smoothness Although smoothness has been incorporated into watershed segmentation in previous work [15, 16, 12], the multiscale approach offers better performance. Smoothness of an evolving contour is achieved by adding a curvature term to the speed function. However, using a curvature estimation at a single scale usually does not bring satisfactory results. While low scale curvatures may not produce enough smoothness, large scale curvatures have no effect for the contour jaggedness. In addition, the estimate in eq. (7) is subject to dicretization error when used in a digital grid. To overcome the scale dependence and the discrepancy between the continuous and discrete domains, we propose a method to estimate the radius of the osculating circle, i.e. the maximal digital circle that resides inside or outside the contour and passes through the contour pixel in consideration. The reciprocal of its radius is the curvature. Denote the radius of the maximal circles inside and outside the contour as R + and R respectively. One of those radii is infinity. We assume R has the negative sign. R+ Figure 5. The maximal circles that are used for definition of scale independent curvature in a digital grid. A set of digital disks of radius ranging in [ R max, +R max ] are generated in advance. For a negative radius, the exterior of the disk is viewed as the interior. For every disk, the curvature in eq. (7) is calculated for all boundary pixels and for different values of σ. For each pair of radius R and scale σ, the minimal and maximal curvature values are then determined: κ (σ) min (R) = min x R κ(σ) (x), R κ (σ) max(r) =max x R κ(σ) (x) (10) where R denotes the border of the disk of radius R. Using the above limits, we can estimate the radius of the maximal disks using the multiple curvature estimates κ (σ1) (x),...κ (σm) (x) as follows: R + (x) = max{r σ : κ (σ) (x) <κ (σ) max(r)} (11) R (x) = min{r σ : κ (σ) (x) >κ (σ) min (R)} (12) The scale-independent curvature is defined as: κ(x) = 1 R + (x) if R + (x) <R max 1 R (x) if R (x) > R max 0 otherwise (13) The advantage of this curvature is that it is defined for the discrete domain and independent on scale. The smoothness term is defined as: V smooth,.(x) =ακ (σ). (x) (14) where α is the weighting coefficient. 5.3 Domain knowledge V prior is meant to prevent the contour from developing configurations considered as illegal for the domain of interest. We distinguish two types of illegal configurations: interior and exterior. Illegal interior configurations are defined for pixels of C out that advance too deep into the interior of the object. On the other hand, illegal exterior configurations indicate the case when C in move too far to the exterior of the object. Examples of illegal interior and exterior configurations are shown in Figure 9a and b respectively. In case of illegal interior configurations we want to stop the motion of C out. Likewise we want to stop C in for exterior configurations. To detect illegal configurations we train a priori two classification functions φ in and φ out such that: { > 0, s Sin φ out (s) = < 0, s S out S a { (15) > 0, s Sout φ in (s) = < 0, s S in S a where S in and S out denote the sets of illegal interior and exterior configurations respectively, and S a is the set of admissible configurations. In the current system, we have manually collected the training examples for S in, S out, and S a. For the training, the kernel-based Support Vector Machines classifier [22] is used. The prior terms are then defined as: V prior,.(s) =βh(φ.(s)) (16) As such, these terms turn effective only when illegal configurations are detected.

6 a b c d e f g h Figure 6. Example of the result of the proposed diffusion based watershed algorithm: a) A C-elegan, b)the set of markers used in the first merging, c-d) the result of the first merging, e) the set of markers in the second run of merging, f-h) the result of the second merging. a b c d Figure 7. The merging result of the waterfall algorithm [1] for the hierarchy levels from 0 to 3. 6 Experiments We have applied the algorithm for segmenting images of Caenorhabditis elegans (C-elegans), the nematode worms used in biology for investigation of neural system functions. The segmentation takes as input an initial rectangle enclosing the worm of interest. The exterior of the rectangle is used as an exterior marker for the watershed algorithm. A set of other markers are obtained by generating a uniform grid of points over the interior of the rectangle. For the merging stage, the diffusion parameter in eq.(1) is set as γ = In the first merging round, the number of regions is set to k =32. For the chosen grid size and the size of worms used in our experiments, this value of k is enough to detect all crater lakes. After the first merging, all regions are eroded by a disk of radius 5 pixels. This erosion also removes small regions. The remaining regions are then used as markers for the second merging which reduces the number of regions down to two. An example of the result of the first and second merging is shown in Figure 6. For comparison, in Figure 7 we show the results of the well-known waterfall algorithm [1] for different levels of the merging hierarchy. The zero level is the result of the original watershed using the same marker set in Figure 6b. The gradient surface is then reconstructed by first raising the surface value to 255 almost everywhere except for the watershed line. The raised surface is then eroded recursively with the condition that it should not be lower than the surface before the reconstruction. Running the watershed algorithm on this surface produces the first level merging in Figure 7b. As observed in Figure 7c, the second merging level

7 Figure 8. The segmentation results for nematode worms (C-elegans). The first and third rows show the results of the original watershed algorithm with no prior knowledge. The second and fourth rows show the corresponding results with prior knowledge incorporated by the proposed method. gives the best result, but still leaves a few of isolated region in the background. Moreover, there is no control of the desired number of regions. At the third level of the waterfall algorithm, both foreground and background are merged into one region, see Figure 7d. In the stage of prior incorporation, local shape at a contour pixel is characterized by a 8-dimensional vector that is composed of the values of B (σ) and κ (σ) in four scales: σ 1 =1,σ 2 =2,σ 3 =5,σ 4 =15. The standard kernelbased SVM algorithm [22] with a kernel size of 10 is used to train two classifiers that detect interior/exterior illegal local shape configurations respectively. Examples of training configurations used are shown in Figure 9. For the weighting coefficients in eq.(14),(16) we use α =1,β =2. Figure 8 shows the segmentation results with and without incorporation of prior knowledge. Since the worm motion is highly non-rigid, the regularization methods that minimize the distance to a reference shape will not work in these images. Although the worms have in general lower brightness than the background, there are much of variations of intensities due to light reflection and also worm tracks. The worm contour does not always coincide with strongest edges. As a consequence, the original watershed lines are not only noisy but also inaccurate. However, the results of the proposed algorithm are much better in terms of both the improved smoothness and the elimination of irregular shapes. 7 Conclusion We have proposed novel techniques to improve the watershed algorithm. To handle the over segmentation, a large number of

8 a b c Figure 9. a) training examples of illegal interior contour configurations, b) illegal exterior configurations, c) admissible configurations. markers are selected, which are then fused during the immersion process using a water diffusion model. Since flooding and merging take place simultaneously, the advantage of the classical watershed algorithm in terms of converging to strongest edges is preserved. Since the decision on merging two lakes is delayed until a significant amount of diffusion is reached, the algorithm can effectively handle the problem of edge leaking at region boundaries as well as naturally take into account many region properties including size, edge strength and border length. Prior knowledge can be incorporated by regularizing the speed function of region contours. For segmenting objects under non-rigid deformations, the regularization can be achieved effectively and flexibly using local shape features. The Gaussian derivatives of the object indicator function appear a good choice since they are invariant to translation and rotation and can be efficiently computed during the evolution of object contour. Furthermore, combination of the measurements of these features at multiple scales increase the ability of the algorithm in discriminating different local shape configurations. We have successfully used these features to impose smoothness and domain knowledge for worm-like objects. References [1] S. Beucher. Watershed, hierarchical segmentation and waterfall algorithm. In Math. Morphology and Its Appl. to Image Processing, pages 69 76, [2] A. Bleau and L.J. Leon. Watershed-based segmentation and region merging. Computer Vision and Image Understanding, 77(3): , [3] V. Caselles, R. Kimmel, and G. Sapiro. Geodesic active contours. Int. J. Computer Vision, 22(1):61 79, [4] T. Chan and W. Zhu. Level set based shape prior segmentation. In CVPR, pages II: , [5] Y. Chen, S. Thiruvenkadam, H.D. Tagare, F. Huang, and D. Wilson. On the incorporation of shape priors into geometric active contours. In IEEE Workshop in Variational and Level Set Methods, pages , [6] P.B. Chou and C.M. Brown. The theory and practice of Bayesian image labeling. IJCV, 4(3): , [7] T.F. Cootes and C.J. Taylor. Active shape models: Smart snakes. In BMVC, pages , [8] L. Garrido, P. Salembier, and D. Garcia. Extensive operators in partition lattices for image sequence analysis. Signal Processing, 66(2): , [9] V. Grau, A.U.J. Mewes, M. Alcaniz, R. Kikinis, and S.K. Warfield. Improved watershed transform for medical image segmentation using prior information. IEEE Trans. on Medical Imaging, 23(4): , [10] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. IJCV, 1(4): , [11] M. Leventon, O. Faugeras, E. Grimson, and W. Wells. Level set based segmentation with intensity and curvature priors. In Math. Methods in Biomed. Image Analysis, (MMBIA), [12] B. Marcotegui and F. Meyer. Bottom up segmentation of image sequences for coding. Annals of telecommunications, 52(7-8): , [13] F. Meyer and S. Beucher. Morphological segmentation. Journal of Visual Comm. and Image Rep., 1(1):21 46, [14] D. Mumford and J. Shah. Optimal approximation by piecewise smooth functions and associated variational problems. Comm. Pure and Applied Math., 42: , [15] H.T. Nguyen, M. Worring, and R. van den Boomgaard. Watersnakes: Energy-driven watershed segmentation. IEEE Trans. on PAMI, 25(3): , [16] M. Pardas and P. Salembier. Time-recursive segmentation of image sequences. In EUSIPCO94, pages 18 21, [17] M. Rousson and N. Paragios. Shape priors for level set representations. In ECCV, page II: 78 ff., [18] J. A. Sethian. A fast marching level set method for monotonically advancing fronts. In Proc. Nat. Acad. Sci., volume 93, pages , [19] A. Tsai, A.J. Yezzi, Jr., W.M. Wells, III, C. Tempany, D. Tucker, A. Fan, W.E.L. Grimson, and A.S. Willsky. Model-based curve evolution technique for image segmentation. In CVPR, pages I: , [20] L. Vincent. Morphological gray scale reconstruction in image analysis: Applications and efficient algorithms. IEEE Trans. on Image Proc., 2: , [21] L. Vincent and P.Soille. Watershed in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans. on PAMI, 13(6): , [22] A. Webb. Statistical Pattern Recognition. Wiley, [23] J. Weickert. A review of nonlinear diffusion filtering. In ScaleSpace, pages 3 28, [24] S. C. Zhu and A. Yuille. Region competition: unifying snakes, region growing, and Bayes/MDL for multiband image segmentation. IEEE Trans. on PAMI, 18(9): , 1996.