216 F.MAES ET AL. tomation, semi-automatic methods that improve interaction by providing intelligent assistance are often superior to fully automatic

Transcription

1 AUTOMATIC IMAGE PARTITIONING FOR GENERIC OBJECT SEGMENTATION IN MEDICAL IMAGES F.MAES, D.VANDERMEULEN, P.SUETENS AND G.MARCHAL Laboratory for Medical Imaging Research y, ESAT-Radiology K. U. Leuven, Kardinaal Mercierlaan 94, 3001 Heverlee, and U. Z. Gasthuisberg, Radiology, Herestraat 49, 3000 Leuven, Belgium. Abstract. In this paper a generic strategy is presented to facilitate the segmentation of anatomical objects in medical images by providing powerful and intelligent assistance to the clinical user. Partitioning of the image in segmentation primitives, such that all pixels within the same primitive are likely to belong to the same object, reduces the segmentation problem to selecting those primitive regions that belong to the object of interest. This dramatically increases segmentation eciency, because no individual pixels need to be considered. The power of this approach is that such a partitioning can be computed independently of the object to be segmented, such that this strategy is very general and can be applied for the segmentation of a variety of anatomical shapes in dierent imaging modalities. This is illustrated for MR images of the brain, for CT images of the abdomen and for angiographic images of the coronary arteries. 1. Introduction Segmentation of the anatomical objects of interest in medical images is a necessary requirement for their quantication and three-dimensional (3D) visualization. Manual delineation of objects in 3D images is very timeconsuming and fast and accurate computer-aided segmentation methods are needed to improve the physician's productivity in clinical practice [9]. Because clinical users are in many ways superior to any form of current auy F. Maes is Aspirant of the Belgian National Fund of Scientic Research (NFWO).

2 216 F.MAES ET AL. tomation, semi-automatic methods that improve interaction by providing intelligent assistance are often superior to fully automatic methods, which may be computationally expensive and may require a lot of manual correction. Because of the growing number of applications and the increasing requirement for accurate representation of a diversity of complex shaped objects, the segmentation algorithms need to be adaptable in a exible way to a variety of anatomical (and articial) shapes and should be robust with respect to varying imaging conditions. Such a multi-purpose approach should make it possible to oer the clinical user a single tool that enables him to quickly segment objects with dierent appearance in the image, without limiting restrictions regarding image contrast or imaging modality. In this paper we propose such a generic strategy, which aims at facilitating the segmentation of anatomical objects in a variety of dierent applications by providing powerful and intelligent assistance to the clinical user. This intelligence is based on the idea that partitioning of the image in segmentation primitives such that all pixels within the same primitive are likely to belong to the same object, reduces the segmentation problem to selecting those primitive regions that belong to the object of interest. Because all pixels that belong to the same primitive region can be treated as one single entity, the number of segmentation decisions to be made is drastically reduced, which increases segmentation robustness and eciency. The power of this approach is that such a partitioning is independent of the object of interest to be segmented and can be computed in advance as a pre-processing step without the need for user interaction to specify the object of interest in the image. Moreover, the method presented here to partition the image in meaningful segmentation primitives is independent of image specic parameters and can be applied without adaptation for the segmentation of a variety of objects in very dierent images obtained with dierent imaging modalities. Because of these features, the approach presented here is very well suited to be used in a generic segmentation algorithm with large applicability. Section 2 of this paper describes how relevant segmentation primitives, whose boundaries correspond to image edges, can be computed automatically from the image data using a fast pseudo-watershed procedure, resulting in an oversegmentation with many, relatively small regions. Section 3 describes how the Minimum Description Length (MDL) criterion is used to reduce the oversegmentation by selectively merging neighbouring watershed regions, such that the total boundary length of the partitioning is reduced while the relevant object boundaries are maintained. Segmentation of the object of interest from the image partitioning is discussed in section 4. The power and the generality of the approach is illustrated in section 5 for MR images of the brain and CT images of the abdomen.

3 IMAGE PARTITIONING FOR GENERIC SEGMENTATION Construction of segmentation primitives We want to partition the image in primitive regions such that all pixels within one region belong to the same object. Boundaries between dierent objects in a medical image are often characterised by a signicant change in image intensity across the boundary, resulting in an image edge. Thus, for all pixels within one region to be likely to belong to the same object, there should be no evidence of edges within that region and all edges in the image should coincide with one of the boundaries of the primitive regions. However, edge detection is very sensitive to noise and image contrast and usually requires additional processing in order to obtain closed boundaries. We use the watershed of the gradient magnitude image (see [4, 15] and the references therein) to partition the image in relevant segmentation primitives. Because the boundaries of the watershed regions coincide with the ridges of the gradient magnitude surface, pixels that are separated by an image edge that corresponds to a gradient magnitude ridge will be assigned dierent labels by the watershed transformation. When compared with classical edge detection techniques, such as the Canny edge detector [3], the watershed procedure has no problems detecting junctions and, because of the region-based approach, the boundaries found are always closed, independent of image contrast or noise level. A true watershed transformation, dened by an immersion process analogy, is a complex and computationally expensive procedure [15]. Instead, we use a simple and fast pseudo-watershed algorithm [13], which involves growing downhill Maximum Gradient Paths (MGP, [11]), starting from each pixel in the image, by selecting in a 3 3 (in 2D, or in 3D) neighbourhood of each pixel the pixel with the smallest gradient magnitude. In order to reduce the inuence of image noise on the gradient computation, the image is smoothed rst with a Gaussian kernel before computing the gradient magnitude image using the Sobel [6] operator. Each MGP terminates at a local minimum of the gradient magnitude and each minimum an MGP terminates in is given a distinct label. Watershed regions are obtained by assigning to each pixel in the image the label of the minimum to which its downhill MGP leads (gure 1). Except for the smoothing parameter used to calculate the gradient magnitude image, the watershed procedure is completely parameterless and does not require any a priori knowledge regarding the objects to be segmented. First results show that this method is sensitive to very small intensity changes in the image, as long as these correspond to a ridge in the gradient magnitude image. This might not be the case for the boundaries of thin line-like (or plate-like) structures, such as for instance the small blood vessels in gure 1(c), which will therefore probably be less well segmented.

4 218 F.MAES ET AL. a b c d Figure 1. (a) Part of one slice of a coronal MR image of the brain, showing a brain tumor ( ). (b) Result of the pseudo-watershed partitioning procedure applied on the image of gure a (1913 regions). (c) Angiographic image of the coronary arteries ( ). (d) Result of the pseudo-watershed partitioning procedure applied on the image of gure c (4434 regions). In both examples, Gaussian smoothing with = 2 was applied before gradient computation. 3. Merging regions using the MDL criterion The partitioning obtained with the watershed procedure consists of many, relatively small primitive regions. In order to make this partitioning more useful for segmentation purposes, we reduce the oversegmentation by selectively merging those primitive regions that are likely to belong to the same object, without merging regions that belong to dierent objects. Selecting the neighbouring regions to be merged is based on the observation that the image intensity is often rather homogeneous within one object, but diers signicantly between objects, such that the image intensity distributions of two neighbouring primitive regions that belong to the same object are likely to be more similar than the intensity distributions of two neighbouring regions that belong to dierent objects. Therefore, merging regions with similar intensity distributions only will most likely preserve the relevant region boundaries in the partitioning. This similarity can be measured by the entropy of the distributions, as described below. Apart from inten-

5 IMAGE PARTITIONING FOR GENERIC SEGMENTATION 219 sity similarity, merging small regions that share a large common boundary segment compared to their size reduces the oversegmentation more than merging large regions that only share a small common boundary segment and should therefore be preferred. These criteria can be formulated mathematically as a global optimisation problem using the Minimum Description Length (MDL) principle: the optimal image partitioning is the one for which the total number of bits required to completely describe the image data in terms of a specied descriptive language is minimal [12, 14]. In our case, such a description consists of the number of bits needed to code the image intensity ofeach pixel in each region and the number of bits needed to code the region boundary information. The image intensity data are coded using separate code alphabets for each region R i by treating the original image intensity x of all pixels in R i as a stochastic variable with distribution p Ri (x). According to information theory [5], the number of bits B I (R i ) required to code the image intensity of all pixels in R i using an optimal coding scheme is given by B I (R i ) = n(r i )H(R i ) =,n(r i ) P x p R i (x)log 2 (p Ri (x)), with n(r i ) being the number of pixels in R i and H(R i ) the entropy of p Ri (x). We estimate p Ri (x) from the histogram h Ri (x) of the original image in the region R i as the convolution of h Ri (x) with a Gaussian window, assuming that the number of pixels in each region is sucient to obtain reliable image statistics over the range of image intensity values in that region. The boundary of a 2D region can be coded by specifying one starting point and a chain code of boundary line elements. The overall number of bits needed to code the region boundary information in the image partitioning R using this scheme is given by B B (R) =N r (R)b 1 +N b (R)b 2, with N r (R) being the number of regions in R, N b (R) the total boundary length of the partitioning, b 1 the number of bits required to code the starting point and b 2 the number of bits required to code each element of the boundary chain code. We take b 1 =28 = 16 for a 2D image and b 2 = log 2 (4) = 2, as the alphabet size of the chain code elements is 4. The surface of a 3D binary object can be represented as a directed graph [1, 10], linking each boundary surface element with two neighbouring boundary elements. Given one boundary element as a starting point and using this graph representation, the surface is completely dened by specifying for each boundary element the orientation of 2 neighbouring boundary elements, for which only 4 dierent values are possible for each neighbour. Therefore, the number of bits needed to code the boundaries of 3D regions is given by the same expression as for the 2D case, using b 1 =3 8 = 24 for coding the starting point and b 2 =2 log 2 (4) = 4 for each boundary surface element. The total number of bits to code the image data from the image par-

6 220 F.MAES ET AL. titioning R using this coding scheme for both 2D and 3D partitionings is given by B(R) =B I (R)+B B (R) = P i B I(R i )+B B (R). Merging two neighbouring regions R i and R j of R results in a new partitioning R 0, containing the region R k = R i +R j. While more bits will be required to code the image information of R k than of R i and R j separately, the total description length might decrease because no bits are needed to code the common boundary segment ofr i and R j. The description length reduction B(R i ;R j ) (either positive or negative) of merging R i and R j is B(R i ;R j ) = B(R), B(R 0 ) = [n(r i ) H(R i )+n(r j ) H(R j )+b 1 + n b (R i ;R j ) b 2 ],(n(r i )+n(r j )) H(R i + R j ) with n b (R i ;R j ) being the number of common boundary elements of R i and R j. If B(R i ;R j ) < 0, the entropy increase of merging R i and R j is not completely compensated by the description length reduction of eliminating the boundary between these regions and these regions should not be merged. If B(R i ;R j ) > 0, the image intensity distributions of R i and R j are suciently similar to consider these regions to belong to the same object and R i and R j should be merged into one region R k in order to minimise B(R). This changes the neighbour relations, requiring an update of the description length reductions of merging this new region R k with all of its neighbours. Therefore, the order in which the regions are merged inuences the outcome of the merging process, making global minimisation of the objective function B(R) a computationally expensive procedure. Instead, we use an iterative procedure in which regions are merged in a deterministic order. First, the histogram is constructed for each region, the neighbour relations of all regions are extracted from the original image partitioning and the common boundary line length and the description length reduction is computed for every pair of neighbouring regions. At each step the pair of neighbouring regions with the largest positive description length reduction is merged, the neighbour relations are updated and all values B(R i ;R j ) that are aected are recalculated. This is repeated as long as regions can be merged with positive description length reduction. Merging thus terminates when a partitioning is obtained that corresponds to a local minimum of the description length function B(R). Figure 2 shows the result of applying this procedure on the image partitionings of gure 1. While the oversegmentation has been drastically reduced, most region boundaries corresponding to relevant object boundaries were preserved. Generally, the merging procedure does not completely resolve the oversegmentation problem and additional processing is required in order to segment the object of interest, which is discussed in the next section.

7 IMAGE PARTITIONING FOR GENERIC SEGMENTATION 221 a b c d Figure 2. (a) Result of the MDL merging procedure applied on the partitioning of gure 1b (279 regions). (b) Overlay of the partitioning of gure a on the original image. (c) Result of the MDL merging procedure applied on the partitioning of gure 1d (387 regions). (d) Overlay of the partitioning of gure c on the original image. 4. Generic object segmentation The power of partitioning the image in segmentation primitives is that the segmentation problem is reduced to identifying all primitives in the partitioning that belong to the object of interest in the image. Instead of having to consider all pixels individually, the segmentation algorithm can treat all pixels that belong to the same primitive region as a single entity. This dramatically reduces the number of decisions that needs to be made by the segmentation algorithm, thus increasing segmentation eciency and robustness. Because of the generality of the partitioning approach, several dierent segmentation strategies, which have been designed for specic applications, can benet from the approach and use the information of the image partitioning to facilitate the segmentation task. One way to extract the object to be segmented from the image is to do this manually. The user can interactively select all regions in the image partitioning that make up the object of interest by dragging the mouse pointer over the object in the original image, while the partitioning itself

8 222 F.MAES ET AL. a b Figure 3. (a) Segmentation of the brain tumor in the image of gure 1a using the partitioning of gure 2a. (b) Segmentation of the coronary arteries in the image of gure 1c using the partitioning of gure 2c. In both examples, the segmentation was done interactively using the intelligent paintbrush tool. remains hidden to the user. The mouse pointer hereby behaves like an intelligent paintbrush [13, 2] and colors or erases each primitive region it hits, reducing the interaction to selecting one point of each region that belongs to the object to be segmented. Because of the generic properties of the image partitioning, the same tool can be applied successfully to a variety of dierent applications (gure 3). The paintbrush technique is very ecient and accurate when compared to manual delineation of object contours. When compared to most (semi)automatic segmentation methods, this (semi)interactive method eliminates the need for object and image dependent models and parameters completely, thus avoiding problems with robustness and repeatability. The accuracy of the method depends on the overlap of the boundaries in the image partitioning with the true object boundaries in the original image. Manual editing might be needed to correct for segmentation errors. More automated and application-specic methods to extract the object of interest from the image partitioning can be considered, in order to further speed up the segmentation process. Because the primitive regions are computed such as to contain only pixels that are likely to belong to the same object, image statistics calculated over all pixels within one region are directly correlated with the image properties of the object to which the region belongs and are therefore more reliable and more signicant than statistics calculated rather heuristically within a pixel neighbourhood or within a user specied region. Region growing methods could benet from this directly, by extending these algorithms to merging primitive regions instead of pixels. The procedure described in section 3 to merge the regions obtained by the watershed procedure is an example of such a strategy. This

9 IMAGE PARTITIONING FOR GENERIC SEGMENTATION 223 method can be transformed into a true region growing method by imposing the restriction that regions can only be merged with an initially specied region, which belongs to the object to be segmented. Additional a priori knowledge regarding the object of interest can be incorporated in the formulation of the MDL criterion, for instance by using image intensity models when coding the image data [12]. Another feature that might be useful to extract the object of interest from the image partitioning is the shape of the primitive regions. In the image of gure 2(c) for instance, the regions that belong to the blood vessels have a rectangular shape and the long axis of this rectangle is indicative for the direction of the blood vessel through that region. Therefore, it is to be expected that vessel tracking algorithms exploiting this property will perform better than pixel based algorithms, for which the most likely direction of a vessel through a pixel is usually estimated by local neighbourhood evaluations. 5. Results The image of gure 4(a) is one slice of a CT image of the abdomen, showing the liver in an advanced state of metastasis. Figure 4(b) shows the boundaries of the segmentation primitives after merging of the original watershed regions. The total computation time on an IBM RS/ workstation was about 10 seconds. Although there is no clear boundary between the metastatic tissue and the background, the partitioning procedure succeeded in discriminating between them and large parts of the slightly darker metastatic tissue were identied to form one single region. Figure 5 illustrates the segmentation of 3D objects using the image partitioning approach. This image is one slice of a transversal 3D MR image of the brain with a voxel size of 1 mm 1mm 1:2 mm. Slice-by-slice 2D image partitionings were computed for all 33 image slices in which a brain tumor was visible, using Gaussian smoothing with = 2 before calculating the gradient magnitude image. This takes less than 10 seconds per slice on an IBM RS/ workstation, including gradient computation, watershed transformation and region merging. It takes the user about 2 minutes to completely segment the tumor on all slices when using the intelligent paintbrush tool. A 3D watershed partitioning resulted in D regions, which was reduced to only 111 after merging. Computation time was about 10 minutes. In this case, the tumor object corresponded precisely to exactly one region in the image partitioning, such that the tumor could be extracted entirely from the original image by selecting one point of it. Tumor volume calculated from the number of pixels in this region turned out to be 73:4 cm 3.

10 224 F.MAES ET AL. a b c d Figure 4. (a) One slice of a CT image of the abdomen (512512). (b) Image partitioning of the image of gure a (595 regions). (c) Overlay of the partitioning on the original image. (d) Segmentation of metastatic liver tissue using the partitioning of gure b. 6. Discussion and conclusion We use an automatic watershed-like transformation of the gradient magnitude image to compute a partitioning of the image in segmentation primitives. This method is somewhat similar to the method described by Grif- n [11], but has the advantage that no continuous representation of the image intensity function needs to be constructed and that we don't need to rst detect the saddle points of the gradient magnitude image before tracking the maximum gradient paths. Other image partitionings can be obtained that are based on another denition of ridges and crest lines. For instance, Eberly [8] uses a ridge ow model based on the height ridge definition [7], in which ridge points are computed from the image intensity itself, rather than from the gradient magnitude image. It is not clear to us how the results obtained with these methods compare to ours.

11 IMAGE PARTITIONING FOR GENERIC SEGMENTATION 225 a b c d e f Figure 5. (a) One slice of an MR image of the brain ( ). (b) Tumor segmented automatically from the 3D MR image by the image partitioning procedure (33 slices). (c,d,e,f) Overlay of the contours of the segmented tumor on some selected slices. In order to reduce the oversegmentation of the watershed partitioning, we use the Minimum Description Length criterion to selectively merge watershed regions based on their image intensity distribution similarity. The outcome of the merging process depends on the scheme that is used to code the image data and the region boundary information. For instance, region merging is favored if a larger bin size is used when constructing the region histograms, because all histograms will be more compact and more similar. The merging procedure implicitly constructs a hierarchy of image partitionings of dierent granularity, such that the boundaries of each partitioning at any level form a subset of the boundaries of the partitionings at all lower levels. Instead of only using the nal partitioning obtained by merging for the segmentation of the objects of interest in the image, one or more of the intermediate partitionings in the hierarchy may be used too, for instance one with large granularity to quickly but roughly segment the object of interest, and another with smaller granularity for more detailed editing. As illustrated by the examples shown in this paper, facilitating the segmentation of generic objects by constructing a partitioning of segmentation primitives is a powerful technique. The clinical usefulness of the method is currently being evaluated on some applications from clinical practice. Fur-

12 226 F.MAES ET AL. ther work will focus on strategies to automate the extraction of the object of interest from the image partitioning. The segmentation of dierent object types will most likely require dierent strategies, resulting in dierent specialised object-nders for each type [9]. Acknowledgements This research is supported by the Belgian National Fund for Scientic Research (NFWO). The images were provided by the Department of Radiology, UZ Gasthuisberg, Leuven, Belgium. The algorithms were developed using the Application Visualization System (AVS) from Advanced Visual Systems Inc. on an IBM RS/6000 workstation. D. Delaere and E. Bellon contributed to the implementation of the paintbrush tool. References 1. E. Artzy, G. Frieder, G. T. Herman, \The theory, design, implementation, and evaluation of a three-dimensional surface detection algorithm", Computer Graphics and Image Processing, Vol. 15, pp. 1-24, D. V. Beard, D. Eberly, B. Hemminger, S. Pizer, R. Faith, C. Kurak, M. Livingston, \Interacting with image hierarchies for fast and accurate object segmentation", Proc. SPIE Medical Imaging VIII, Newport Beach, CA, Vol. 2167, pp , J. Canny, \A computational approach to edge detection", IEEE Trans. on PAMI, Vol. 11, no. 2, pp , A. C. F. Colchester, \Network representation of 2D and 3D images", in 3D Imaging in Medicine, K. H. Hohne, H. Fuchs and S. Pizer, pp , Springer-Verlag, T. M. Cover, J. A. Thomas, \Elements of information theory", J. Wiley, R. O. Duda, P. E. Hart, \Pattern classication and scene analysis", J. Wiley, D. Eberly, \A dierential geometric approach to anisotropic diusion", in Geometry driven diusion in computer vision, B. M. ter Haar Romeny, Kluwer Academic Publishers, D. Eberly, S. M. Pizer, \Ridge ow models for image segmentation", Proc. SPIE Medical Imaging VIII, Newport Beach, CA, Vol. 2167, pp , F. A. Gerritsen et al., \Some requirements for and experience with Covira algorithms for registration and segmentation", Technical Report, XLB , CDS Advanced Development, Philips Medical Systems Nederland, D. Gordon, J. K. Udupa, \Fast surface tracking in three-dimensional binary images", Technical Report, No. MIPG117, Medical Image Processing Group, University of Pennsylvania, L. D. Grin, A. C. F. Colchester, G. P. Robinson, \Scale and segmentation of greylevel images using maximum gradient paths", Image and Vision Computing, Vol. 10, no. 6, pp , Y. G. Leclerc, \Constructing simple stable descriptions for image partitioning", Intern. J. Computer Vision, 3(1), pp , F. Maes, D. Vandermeulen, P. Suetens, G. Marchal, \Computer-aided interactive object delineation using an intelligent paintbrush technique", presented at CVRMed'95, Nice, April 3-5, J. Rissanen, \Minimum-description-length principle", In Encyclopedia of Statistical Sciences, Vol.5, pp , Wiley: New York, L. Vincent, P. Soille, \Watersheds in digital spaces: an ecient algorithm based on immersion simulations", IEEE Trans. on PAMI, Vol. 13, no. 6, pp , 1991.