Computer-aided Interactive Object Delineation Using an Intelligent Paintbrush Technique Frederik Maes?, Dirk Vandermeulen, Paul Suetens, Guy Marchal L

Transcription

1

2 Computer-aided Interactive Object Delineation Using an Intelligent Paintbrush Technique Frederik Maes?, Dirk Vandermeulen, Paul Suetens, Guy Marchal Laboratory for Medical Imaging Research?? Katholieke Universiteit Leuven, Belgium. Department of Electrical Engineering, ESAT, Kardinaal Mercierlaan 94, B-3001 Heverlee, Belgium. Department of Radiology, University Hospital Gasthuisberg, Herestraat 49, B-3000 Leuven, Belgium. Abstract. A method for fast generic object segmentation is presented that allows the user to quickly paint the object of interest in the image using an intelligent paintbrush. This intelligence is based on a partitioning of the image in segmentation primitives, which are computed automatically by merging watershed regions with similar image intensity distribution using the Minimum Description Length principle. We show results for Magnetic Resonance images of the heart and of the brain and for Computerized Tomography images of the abdomen. 1 Introduction Segmentation of medical images is a necessary requirement for three-dimensional (3D) visualization and quantication of the anatomical structures of interest. There are numerous clinical applications, for instance in surgery and radiotherapy treatment planning and in quantitative diagnosis. Because completely manual object delineation on large 3D datasets is very tedious and time-consuming, computer-assistance is needed to facilitate the segmentation task in clinical routine. Fully automatic methods, relying on a priori knowledge such as shape and intensity models, are precluded by the complexity of the images, the diversity of the objects of interest and the occurrence of pathological deformations. In this paper, we present a semi-automatic segmentation method, whereby the user, assisted by the computer, interactively segments the object of interest by colouring it in the image using an intelligent paintbrush [1]. This approach is based on a partitioning of the image in primitive regions and assumes that the object of interest is the union of one or more of these regions. In that case, only one point of each region that belongs to the object needs to be selected to completely segment it, which dramatically speeds up the interactive segmentation process.? Frederik Maes is Aspirant of the Belgian National Fund of Scientic Research (NFWO). Corresponding author. frederik@uz.kuleuven.ac.be?? Directors: Andre Oosterlinck & Albert L. Baert

3 78 Section 2 describes how a relevant partitioning is computed automatically using a fast watershed-like procedure. In order to reduce the need for user interaction in the paintbrush tool, regions that are likely to belong to the same object need to be merged selectively while maintaing the relevant object boundatries. Section 3 describes how this is achieved using the Minimum Description Length (MDL) criterion [9]. Section 4 illustrates the power of the method by showing results for Magnetic Resonance (MR) images of the heart and of the brain and for Computerized Tomography (CT) images of the abdomen. 2 Construction of primitive image regions Boundaries between dierent objects in an image are often characterised by a signicant change in image intensity across the boundary. Such boundaries correspond to ridges in the gradient magnitude image and the contours of the objects coincide with the crest lines of these ridges. Therefore, a partitioning of the image in meaningful segmentation entities can be obtained by detecting these crest lines using the watershed of the gradient magnitude image. A true watershed transformation, however, is intractable, because it is a computationally expensive operation [10]. Instead, we construct pseudo-watershed regions by tracking downhill Maximum Gradient Paths (MGP) [6], starting from each point in the image and terminating in a local gradient minimum. The gradient of the image intensity is computed using the Sobel [4] operator, after Gaussian smoothing of the original image in order to reduce the inuence of noise on the gradient computation. The downhill MGP of each pixel p is tracked by recursively selecting the pixel q in the 8-connected (in 2D, or 26-connected in 3D) neighbourhood of p for which the gradient magnitude is minimal. If more than one such pixel q exists, we systematically select the rst one found. The MGP terminates in the pixel m if there is no pixel in its neighbourhood with strictly smaller gradient magnitude. Each such pixel m is marked as a local minimum of the gradient magnitude and given a distinct label. If a minimum is reached in whose neighbourhood another minimum with the same gradient magnitude value has already been detected, the new minimum is disregarded by giving it the same label as the neighbouring one in order to reduce the number of regions. Each pixel in the image is assigned the label of the minimum its downhill MGP terminates in. A pixel p is a boundary pixel of the region R(p) it belongs to if there is at least one pixel q in the 4-connected (in 2D, or 6-connected in 3D) neighbourhood of p that does not belong to R(p). This pixel q is a boundary pixel of its region R(q) and the common pixel boundary of p and q is a boundary line element (in 2D, or surface element in 3D) of R(p) and R(q). The boundary between two regions is the union of all their common boundary elements. We now argue that the sodened downhill region boundaries coincide with the crest lines of the gradient magnitude. Indeed, if the pixel p is located on the crest line of a gradient ridge, the gradient of the image gradient magnitude points in opposite directions on both sides of the ridge, ipping direction in p. Therefore, the neighbourhood of

4 p will contain at least two pixels that are mapped on two dierent local minima on opposite sides of the crest and p is a boundary pixel of its downhill region. The result of this algorithm is an oversegmentation of the image, as illustrated in Fig. 1. The correspondence of the watershed boundaries to object contours can be compared with the result of an edge detector, such as the Canny edge detector [2] (Fig. 1c). A typical problem of Canny edge detection is that the edges found are not closed and that additional processing for explicit detection of junctions and linking of edge segments is usually required. The watershed method on the contrary, has no problem detecting junctions and, because of the region based approach, all boundaries are closed implicitly, regardless of noise or low intensity contrast. 79 [t] a b c Fig. 1. (a) Short-axis MR cross-section of the heart at diastole (200 times 200). (b) Watershed regions of the gradient magnitude of the smoothed image ( = 2). (c) Canny edges thresholded at 10% of the maximum edge strength (sigma = 2). 3 Selectively merging primitive regions Because the image intensity is often rather homogeneous within one object and signicantly dierent between objects, the oversegmentation of the watershed can be reduced while preserving the relevant object boundaries by selectively merging regions with similar intensity distributions. Apart from intensity similarity, merging small regions that have a large common boundary compared to their size should be preferred, because this reduces the oversegmentation more than merging regions that only share a small common boundary. These criteria can be formulated mathematically as an optimization problem using the Minimum Description Length (MDL) principle: the optimal image partitioning is the one for which the total number of bits required to describe the image data in terms of a specied descriptive language is minimal [8, 9]. In our case, such a description consists of the number of bits needed to code the

5 80 image intensity of each pixel in each region and the number of bits needed to code the region boundaries. In this section, we only discuss the 2D case, but extension to 3D regions is straightforward. We code the image data using separate code alphabets for each region R i by treating the original image intensity x of all pixels in R i as a stochastic variable with distribution p Ri (x). According to information theory [3], the number of bits B I (R i ) required to code the image intensity of all pixels in R i using an optimal coding scheme is given by B I (R i )=n(r i )H(R i ), with n(r i ) being the number of pixels in R i and H(R i ) the entropyofp Ri (x). The image intensity distribution p Ri (x) is estimated from the histogram of the original image in the region R i as the convolution of with a Gaussian Parzen-window [4]. The boundary line of each region is coded by specifying one starting point andachain code of boundary line elements, such that the total number of bits needed to code the region boundary information of the partitioning R is B B (R) = N R (R) b 1 + N b (R) b 2, with N R (R) being the number of regions in R, N b (R) the total boundary length, b 1 the number of bits required to code the starting point and b 2 the number of bits required to code each element of the boundary chain code. We take b 1 =28 = 16 for a 2D image and b 2 = log 2 4=2, as each 2D boundary element corresponds to one of 4 possible directions. The total number of bits to code the image data from the image partitioning R using this coding scheme is B(R) =B I (R)+B B (R) = P i B I(R i )+B B (R). If two neighbouring regions R 1 and R 2 are merged into a new region R 3, more bits will be required to code the image intensity information of R 3 that of R 1 and R 2 separately. However, the total description length might decrease because no bits are needed to code the common boundary segment ofr 1 and R 2. The description length gain B(R 1 ;R 2 ) of merging R 1 and R 2 into R 3 is given by B(R 1 ;R 2 )=n(r 1 )H(R 1 )+n(r 2 )H(R 2 ),n(r 3 )H(R 3 )+b 1 + n b (R 1 ;R 2 )b 2, with n b (R 1 ;R 2 ) being the length of the common boundary of R 1 and R 2. The merging process proceeds iteratively by selecting at each step the couple of neighbouring regions with the largest positive description length gain, merging these into a new region and updating the neighbour relations. This procedure is repeated as long as regions can be merged with positive description length gain and merging terminates at a local minimum of the description length function. Figure 2 shows the result of this procedure on the partitioning of Fig. 1b. This partitioning is used as input to the intelligent paintbrush tool. 4 Results obtained with the intelligent paintbrush tool The original image and the region label image of the primitive region partitioning are used as input to the intelligent paintbrush tool. Only the original image is presented to the user, while the region label image remains hidden. When the user selects a point on the screen by clicking the left mouse button, the region the selected point belongs to is extracted from the region label image and shown in colour overlay on the screen. The user can erase coloured regions by clicking

6 81 [t] a b c Fig. 2. (a) Merged region boundaries for the image of gure 1(b). (b) Overlay of (a) on the original image. (c) The myocardium segmented with the intelligent paintbrush tool. the right mouse button. Dragging the mouse with the left or right mouse button pressed, colours or erases all regions the mouse pointer hits. In the current implementation of this tool, the colour overlay is opaque and hides the underlying image, which might make it dicult for the user to assess the result of the segmentation. For clinical use, the colourings should be transparent and the user should be oered the choice of displaying the lled coloured regions or only the boundary of their union. Manual editing of the contour of the delineated region might also be needed to correct segmentation errors. [t] a b c Fig. 3. (a) Original MR image of the brain ( ). (b) Merged watershed partitioning. (c) Tumor segmented with the intelligent paintbrush. Figure 2c shows the segmentation of the myocardium in the image of Fig. 1, using the partitioning of Fig. 2a. This partitioning was computed automatically from the original image in less than 5 seconds on an IBM RS/6000 workstation. It

7 82 takes the user about 5 seconds to complete the segmentation with the intelligent paintbrush tool. Figure 3 shows the segmentation of a tumor in one slice of a transversal 3D MR image of the brain. Slice by slice segmentation of the tumor on all 33 slices takes the user less than 3 minutes. Figure 4 shows one slice of a 3D CT image of the abdomen and the segmentation of the prostate in this image. [t] a b c Fig. 4. (a) Original CT image of the abdomen ( ). (b) Merged watershed partitioning. (c) Prostate segmentation with the intelligent paintbrush. 5 Discussion The accuracy of the intelligent paintbrush segmentation depends on the overlap of boundaries in the image partitioning with the contours of the object of interest. First results show that the pseudo-watershed procedure is able to detect small intensity dierences in the image, as long as these correspond to a ridge in the gradient magnitude image. This might not be the case for the boundaries of thin line-like (or plate-like) structures, which are likely to be blurred with the background due to the smoothing needed for the gradient computation. Other image partitionings can be obtained that are based on another ridge denition. Grin [6] describes a method to partition the image in districts by growing MGP's both uphill and downhill from the saddle points of the gradient magnitude image, computed at dierent scales from a continuous representation of the orginal image. Eberly [5] uses a ridge ow model, whereby ridges, corresponding to medial structures, are computed from the image intensity itself. The MDL criterion requires estimation of the image intensity distribution in each region. For small regions, the number of samples might be too small to obtain reliable statistics. However, the image intensity is likely to be rather homogeneous within each region, such that the average number of samples per histogram bin might still be signicant. Region merging is favored by histogram binning,

8 because all region intensity distibutions will be more compact and therefore more similar. Other optimization strategies to minimize the description length function, such as simulated annealing [], might result in more optimal partitionings with lower description length, but are computationally more expensive than the procedure used here. Instead of using one single image partitioning as input to the paintbrush tool, several partitionings of the same original image could be used with dierent granularity. A hierarchy of dierent image partitionings is constructed implicitly during merging. One partitioning with larger granularity could be used for fast, but rough, colouring of the object of interest, while another with smaller granularity could be used for erasing and more detailed editing Conclusion Partitioning of the image in segmentation primitives is a powerful technique. It allows fast and robust computer-aided interactive segmentation of the object of interest using the intelligent paintbrush tool. Further work includes extension of the merging procedure to 3D with the purpose of true 3D object segmentation, and evaluation of the accuracy of the method on clinical applications by medical experts. Acknowledgements This research is supported by the Belgian National Fund for Scientic Research (NFWO). The algorithms were developed with AVS (Application Visualization System) on an IBM RS/6000 workstation. We thank Dr. J. Bogaert and Dr. F. Demaerel for providing the images and D. Delaere and E. Bellon for their contribution to the implementation of the intelligent paintbrush tool. References 1. Beard, D. V., Eberly, D., Hemminger, B., Pizer, S., Faith, R., Kurak, C., Livingston, M.: Interacting with image hierarchies for fast and accurate object segmentation. Proc. SPIE Newport Beach, CA, 2167 (1994) 10{ Canny, J.: A computational approach to edge detection. IEEE Trans. PAMI 11 (1986) 679{ Cover, T.M., Thomas, J.A.: Elements of information theory. John Wiley & Sons (1991). 4. Duda, R.O., Hart, P.E.: Pattern classication and scene analysis. John Wiley & Sons (1973). 5. Eberly. D., Pizer, S.M.: Ridge ow models for image segmentation. Proc. SPIE Newport Beach, CA, 2167 (1994) 54{ Grin, L.D., Colchester, A.C.F., Robinson, G.P.: Scale and segmentation of greylevel images using maximum gradient paths. Image and Vision Computing 10 (1992) 389{402.

9 84 7. Kirkpatrick, S., Gelatt, C.D, Vecchi, M.P.: Optimization by simulated annealing. Science 220 (1982) 671{ Leclerc, Y.G.: Constructing simple stable descriptions for image partitioning. Intern. J. Computer Vision 3 (1987) 73{ Rissanen, J.: Minimum-description-length principle. Encyclopedia of Statistical Sciences. Wiley, New York 5 (1987) 425{ Vincent, L., Soille,P.: Watersheds in digital spaces: an ecient algorithm based on immersion simulations. IEEE Trans. PAMI 13 (1991) 583{598. This article was processed using the LATEX macro package with LLNCS style