Segmentation Using Deformable Models With. Department of Computer and Information Science, University of Pennsylvania,

Transcription

1 Segmentation Using Deformable Models With Anity-Based Localization? Timothy N. Jones Dimitris N. Metaxas Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA USA Abstract. We have developed an algorithm for segmenting objects with simple closed curves, such as the heart and the lungs, that is independent of the imaging modality used (e.g., MRI, CT, echocardiography). Our method is automatic and requires as initialization a single pixel within the boundaries of the object. Existing segmentation techniques either require much more information during initialization, such as an approximation to the object's boundary, or are not robust to the types of noisy data encountered in the medical domain. By integrating region-based and physics-based modeling techniques we have devised a hybrid design that overcomes these limitations. In our experiments we demonstrate that this integration automates and signicantly improves the object boundary detection results, independent of the imaging modality used. 1 Introduction Most existing segmentation techniques can be classied as either model-based or region-based. Model-based techniques such as snakes [5] and Fourier parameterized models [11] start with a deformable boundary and attempt to align this boundary with the object boundary, typically using gradient features. The solution to these systems generally involves minimizing an energy functional which quanties the shape of the model and image information near the boundary of the model. To avoid becoming stuck in local minima, most model-based techniques require that the model be initialized near the solution or supervised by an interactive interface or some higher-level reasoning entity. In [6], a modied snake model is presented which reparameterizes the model depending on local topology, potentially avoiding some of the problems in previous designs. Region-based techniques such as region growing [1] assign membership to objects based on homogeneity statistics. The advantage here is that image information inside the object is considered as well as on the boundaries. However, there is no provision in the region-based framework for including the shape of the region in the decision making process, which can lead to noisy boundaries and holes in the interior of the object.? Published in the Proceedings of CVRMed/MRCAS '97, March 1997 Grenoble, France. Available as Lecture Notes in Computer Science vol. 1205, Springer.

2 We have designed an integrated algorithm which combines the model-based and region-based techniques into a hybrid framework which helps to overcome the limitations mentioned above. Several other researchers ([2],[3], [9], [12]) have also recently described hybrid algorithms. In [2], the model-based and regionbased techniques are set in a game-theoretic setting where each represents a \player." The segmentation process proceeds in an iterative fashion, where at each iteration each player updates their strategy based on the results from both players in the previous iteration. The idea is that each player will pull the other away from noise and local minima, while pushing it toward the solution. In [3], a framework is presented in which the solution is in the form of a maximum matching of region, gradient, and curvature terms. The high-level matching functional is a product of individual feature matches, which maximizes the correlation of the individual features. In [9], the probabilistic deformable model energy functional is augmented such that in certain locations within the interior of objects, where an image (gradient) energy can be computed, a gradient-based energy is used such that the model boundary is driven in the direction of the boundary of the object in the priors. In [12], a unifying framework is derived which generalizes the deformable model, region growing, and prior matching approaches and presents a new energy functional which represents them. By combining the model-based and region-based techniques, these approaches oer greater robustness than either technique alone. However, most still require signicant initialization to avoid local minima. Furthermore, most of the above approaches use prior matching for their region-based statistics, which we would rather avoid to increase usefulness in situations where a comprehensive set of priors may not be available. Our integrated approach works as follows. Starting from a single pixel within the boundary of an object, we make an initial estimate of the object's boundary using the techniques of fuzzy anity [10] and clustering. A deformable surface model is then tted to the extracted boundary data to ll in the missing boundary data and to override the spurious boundary data due to image noise. This is achieved by generalizing the denition of our deformable models [7] to incorporate simple domain-specic knowledge. The elastic properties of the model help it to override small patches of noise, while the domain knowledge attenuates the rest. Once the deformable model achieves an equilibrium, we rene the extracted boundary by using forces computed from the intensity gradient. The above two step boundary extraction process is recursively applied, by using the result of the deformable model t to rene and improve the anity statistics, which are then used to better initialize the deformable model. 2 Boundary Estimation Using Fuzzy Anity We describe the region-based process by which we initially estimate the object boundary which is then used by the deformable model tting process, described in a later section, to further improve the accuracy of the boundary detection. Fuzzy anity is the assignment of a probability (real number between 0 and 1) to two pixels in an image belonging to the same object. The value of the probability denotes the certainty of the coherency. Pairs of pixels which have an anity of 0 are said to be denitely disconnected (belonging to dierent objects), 2

3 while an anity of 1 indicates that the pixels are denitely connected (belonging to the same object). This probability is a function which is based on the distance between the pixels and the image features in the neighborhoods of the pixels. Typical features used are the intensities of the pixels, and the gradient of the pixel intensities. In [10], the following general model of fuzzy anity is proposed: u k (c; d) = h(u a (c; d); f(c); f(d); c; d); (1) where c and d are the locations of the two pixels in the image, u a is an adjacency function based on the distance between the two pixels, and f(i) is the intensity of pixel i. A simplied, shift-invariant version which was used in [10] is the following: u k (c; d) = ua (c; d)(w i h i (f(c); f(d)) + w g h g (f(c); f(d))) if c 6= d 1 otherwise; (2) which is a linear combination of fuzzy intensity, h i, and fuzzy gradient, h g, anities, with weights w i and w g whose sum is unity. The adjacency function, u a, is a 4-neighbor test: u a (c; d) = 1 if p (cx? d x ) 2 + (c y? d y ) otherwise; (3) where h i and h g are functions which rate the anities between the two pixels based on intensity and intensity gradient magnitude, respectively: h i (f(c); f(d)) = e? 1 2 [ 1 2 (f(c)+f(d))?m i s i ] 2 ; h g (f(c); f(d)) = e? 1 2 [ jf(c)?f(d)j?mg ] sg 2 : (4) Here h i assigns a higher anity to two pixels whose average intensity is similar to the intensity of other pixels believed to be in the object. From these other pixels, the intensity mean and standard deviation, m i and s i, are computed. h g assigns a lower anity to two pixels whose dierence in intensity is similar to that of other pixels believed to be in the object. From these pixels, the gradient magnitude mean and standard deviation, m g and s g, are computed. Previous approaches [10] for estimating the other pixels believed to be in the object used for computing m i, s i, m g, and s g include thresholding or userselection. Our approach uses an iterative feedback mechanism, where the pixels used for these statistics are updated based on the shape of the deformable model. Initially, we apply this method only to the pixels in the local neighborhood of the user-provided starting pixel, which gives an initial approximation to pixels which are known to be within the object boundaries. The optimal distribution for the intensity and gradient weights w i and w g, respectively (see (2)), depends on the properties of the object being segmented as well as the imaging modality. We initially use an equal distribution (w i = w g = 0:5) and propose methods for automatically tuning the distribution based on a specic imaging situation, where the default may fail because of excessive 3

4 or minimal gradient information or intensity homogeneity. One simple method for automatically selecting the weight distribution would be to match the image against a catalog of previously segmented objects (priors). The weights for the object with the best match would be used. Another, adaptive, method is one in which the weights are initially 0 for the intensity and 1 for gradient. The gradient is used as the initial feature as it is often the most striking in image segmentation. Once we estimate the initial boundary by clustering the pixels based on their respective anities computed above (see Sect. 2.2), we initialize a deformable model to further rene the estimated boundary and to update the boundary pixel anities. At each iteration of the model tting process, the quality of the gradient information at pixels near the nodes is evaluated, and if it is poor (i.e., gradient information is missing at many pixels), the gradient weight is lowered and the intensity weight is raised. Placing this in functional form we have w i = 1? w g ; w g = ny i=1 rf(m(i)); (5) where is a normalizing coecient, n is the number of nodes in the model, m(i) is the pixel nearest the model node i, and rf is the gradient magnitude at a pixel. 2.1 Fuzzy Connectedness Based on this anity function and thresholding, [10] describes a segmentation algorithm designed for an interactive setting as follows. A dynamic programming technique is used to generate the connectedness of all pixels in the image to an initial pixel by considering all pairwise anities starting with the immediate neighbors of the initial pixel and recursively fanning out until the entire image has been examined. The connectedness of a pixel to the initial pixel is the strength of the strongest path from the initial pixel to the pixel in question. The strength of a path is the minimum anity between any two pixels along the path. Once this connectedness map is generated, the pixels can be segmented into object/non-object regions based on thresholding (i.e. strongly connected pixels are considered to be inside the object, and pixels with low connectedness are outside). (a) (b) (c) Fig. 1. (a) MRI slice of left ventricle, (b) gradient magnitude, and (c) fuzzy connectedness map demonstrating leaking The problem with this approach in our experiments has been that noisy images with deteriorated gradients can cause the connectedness to \leak" into 4

5 non-object pixels such that boundary information is lost. Consider anity parameters such that the anity between white pixels and black pixels is 0, and the anity between all other pairs of pixels is close to 1. Given an image where white and black pixels are separated by grey pixels, when the dynamic programming algorithm visits the white pixels adjacent to the grey pixels, it will assign a strong anity to the grey pixels. When the algorithm visits the grey pixels, it will then assign a strong anity to the black pixels. A strong anity is thereafter assigned to the rest of the black pixels, and the boundary information is completely lost. Figure 1 demonstrates this problem on an MRI slice of a human heart. The left ventricle in the center of the image is the object of interest. The algorithm was initialized with statistics corresponding to a circular region of approximately 80 pixels at the center of the object, and anity function weights w i = w g = 0:5. The gradient of the LV deteriorates in the upper right portion, causing the dynamic programming algorithm to assign relatively high anities to non-object pixels. While careful initialization could likely overcome this problem on an image-by-image basis, our experiments suggest that the general approach is not well suited for automation. 2.2 Anity Clustering Our aim is to automatically determine, based on the theory of fuzzy anity, a reliable estimate of the object boundary that will be used to initialize the deformable model. To overcome the leaking problem in the fuzzy connectedness approach, we do not use pixels with high connectedness ratings in the model tting stage. Instead of trying to identify pixels inside the object, we identify the pixels that lie outside the object by considering those pixels with low connectedness ratings. This is based on the observation that the anity between pixels along the exterior of the boundary of objects tends to be very low. Of course, we lose information at points where the boundary deteriorates due to noise, but we consider this a far better situation than that of gaining misleading information in noisy areas. We therefore estimate the boundaries of objects as those areas where there are pixels with low anities. To lter noise from the anity computations, we extract the connected components of pixels with low anity, discard those with small area as likely noise, and retain the rest as potential boundary regions. Each region is assigned a unique identier which is used later by the knowledgebased noise suppression rules. As shown in Fig. 3(c), spurious boundaries are computed that based on our model-based approach will be ignored and the correct boundary will be determined. 3 Deformable Models We summarize our recently developed physics-based framework for deformable models [7] and describe the integration of anity data into the framework. Our model is a 2D superquadric ellipse with local deformations. The position of a point on the model is given by x = c + Rp where c and R are the translation and rotation of the model coordinate system with respect to the world coordinate system and p is the position of the point with respect to the model coordinate system. p is decomposed into p = s+d, where s represents the model's global 5

6 shape and d is the local displacement at the point. We collect all the model degrees of freedom (translation, rotation, and local displacements) into the vector q. The nite element method is used to compute the local displacement, d, by dividing the model into small regions (elements). Between the nodal points within the model, the displacement is approximated using a small number of interpolating polynomials known as shape functions. Points on the model move under the inuence of externally applied forces. The Lagrangian dynamics provides us with a set of equations describing this motion while considering mass, damping, and a deformation strain energy. The simplied Lagrange equations we use in estimation problems [7] are: _q + Kq = f q ; (6) where K is the stiness matrix and f q are the generalized external forces. In addition, we experimentally select the material parameters which compose the stiness matrix, K. However, this process can also be automated by by integrating our recent work in elastically adaptive models [8] which can be done in a straightforward manner. 3.1 External Force Calculation In the model tting stage we apply forces f q to the nodes of the model such that the model boundary becomes aligned with the object boundary. One approach for deforming models to image data is to project the nodes of the model onto the plane of the image, and apply forces to the nodes based on the intensity gradient at the point of projection. A potential function is dened P (x; y) = kr(g I)k; (7) where G is a Gaussian smoothing lter of width and I is the original image. Based on this potential we dene a force f = rp ( Y x); (8) where controls the strength of the force and Q is a projection of nodal points to the image plane. One problem with this approach is that of localizing the nodes close enough to the real solution so that they neither get stuck in local minima nor become inuenced by gradients at boundaries other than at the object of interest. A partial solution to this problem is the so called \balloon force" [4] which constantly pushes nodes away from the center of the model. f = n(x)? rp krp k ; (9) where n(x) is the surface normal at point x, controls the strength of the balloon force and controls the strength of the opposing potential force. When using a balloon force, the nodes are initialized inside the object so that the balloon force is guaranteed to push them toward the real solution (the object's boundary). Local minima can still be a problem in noisy images where there is too much gradient information inside the object. 6

7 3.2 Integration of Anity Information into the Deformable Model Framework Our solution to the model tting problem is to localize the nodes with a balloon force which uses an anity-based potential rather than the gradient-based potential of (7). We use the estimated boundary regions derived in Sect. 2.2 as the basis of our opposing potential force P in (9). When a node is pushed into an estimated boundary region, the region \pushes back" on the node with an opposing force. The goal is to have the nodes come to a stop at the object boundary. Fig. 2. Depiction of forces applied to model nodes Once the model has reached equilibrium, the entire process is repeated in a feedback loop where we use the pixels covered by the model to generate more accurate statistics for the anity function. This yields an even better boundary potential which we use to achieve a better model t. During each iteration the t of the model to the object boundary is improved, and the statistics used to generate the anity function are optimized. Our experiments show that 2-3 iterations of the two phases results in a very convincing boundary. For particularly noisy images, or objects that contain \holes" such as tumors, not all of the estimated boundary regions lie on the true object boundary. If there are enough false regions inside the object arranged such that they aect nodes of the model on all sides, the nodes may get stuck in local minima (see Fig. 3). It is here that we use the domain-specic knowledge to overcome the noise. The goal of the rules is to assign a condence to all regions, and ignore those regions with a low condence. One simple yet eective rule is to assign condence to a region proportional to the number of model nodes near it. When the condence of a few regions becomes strong enough, we discard the other regions that have a low condence. 4 Gradient-Based Fine Tuning Once the balloon forces and anity forces have reached equilibrium and the nodes of the model have come to rest, the model is considered localized to the estimated boundaries of the object. Because we took a conservative approach to selecting the boundary regions to avoid noise in the interior of the object, there are likely to be parts of the object boundary for which we have no anity data. Consequently, at the end of the localization phase, the positions of the 7

8 (a) (b) (c) (d) Fig. 3. (a) CT slice of lung, (b) estimated boundaries, (c) model stuck on false boundary regions, and (d) model localized after skipping false regions model nodes along these areas are interpolated from areas where there is anity data. Unless the true shape of the object happens to match the nite element shape functions we use for the interpolation, there is likely to be some minor misalignment at this point. To resolve this error, we switch to a gradient-based potential to push the nodes toward the true object boundary. Because many of the nodes will have been localized at the true object boundary (assuming reasonable anity data), the problems of local minima should not be a problem. The nodes which were correctly localized will not move far, and the \oating" nodes along parts of the boundary where there was no anity data will be constrained by the deformation strain forces of the model from the well-localized nodes. 5 Results We have tested our combined, automated approach to object segmentation using region-based and physics-based modeling techniques on several datasets of internal organs from MRI, CT, and echocardiography. The results have been very promising, yielding a convincing estimation of organ boundary while requiring no interaction or detailed initialization from the user. Spatial limitations constrain our presentation of results to a few cases from MRI, CT, and echocardiography, but similar results have been veried on many others. (a) (b) (c) (d) Fig. 4. (a) Potential boundaries of left ventricle, (b) initialized model, (c) localized model, and (d) nal t to gradient data Figure 1(a) shows an MRI slice of a human heart. Figure 4(a) shows the estimated boundaries of the left ventricle produced in the initial iteration of our algorithm using anity data. A deformable model was initialized on the image in Fig. 4(b). Forces were applied to the nodes of the model as described in Sect. 3 until it reached equilibrium as shown in Fig. 4(c). Finally, the gradient forces 8

9 were applied as described in Sect. 4 and the nal results after two iterations are shown in Fig. 4(d). Figure 3(a) shows a CT slice of a human lung. Figure 3(b) shows the estimated boundaries, which contain some false boundary regions. Once a model is initialized to the boundary data and becomes stuck on the false boundaries as show in Fig. 3(c), we apply the noise suppression rules described in Sect The nal t after one iteration is shown in Fig. 3(d). (a) (b) (c) Fig. 5. (a) CT slice of left ventricle, (b) boundary potential, and (c) tted model with gradient data Our algorithm is demonstrated on a CT slice from a human heart in Fig. 5. Figure 5(a) shows the original data. The estimated boundaries produced from the anity data is shown in Fig. 5(b). The nal t of the model after two iterations is shown in Fig. 5(c). (a) (b) (c) Fig. 6. (a) Echo slice of left ventricle, (b) boundary potential, and (c) tted model with gradient data An echocardiography image from a human heart is shown in Fig. 6. The original data is in Fig. 6(a). The white contour near the corners of the image is an artifact of the imaging hardware and did not aect the results of the algorithm. The estimated boundary data is shown in Fig. 6(b). The nal t after three iterations is shown in Fig. 6(c). Note the high quality of the estimated boundary data as compared to the gradient data shown in the nal t. 6 Conclusions In this paper we have shown how the boundary-based and region-based segmentation techniques can be combined to yield good results with very little initialization or interactive supervision. The fuzzy anity concept is reviewed and used to estimate possible boundary regions of an object. A two-phase segmentation method is presented which consists of localizing a deformable model 9

10 to the estimated boundary regions and ne-tuning the model t by adding gradient forces once the model is localized. Results of the algorithm on images of human hearts and lungs from MRI, CT, and echocardiography are presented. The algorithm is undergoing further verication on other organs and using data from other modalities, and we are in the process of extending it to 3D. Acknowledgments The authors are grateful for the contribution of data from Dr. Eric Homan of the University of Iowa and Drs. Leon Axel and Ivan Salgo of the University of Pennsylvania. Discussions with Dr. Jayaram Udupa of the University of Pennsylvania are also gratefully acknowledged. This work has been partially supported by NSF Career Award NSF References 1. D.H. Ballard and C.M. Brown. Computer Vision. Prentice Hall, A. Chakraborty and J.S. Duncan. Integration of boundary nding and regionbased segmentation using game theory. In Y. Bizais et al., editor, Information Processing in Medical Imaging, pages 189{201. Kluwer, A. Chakraborty, M. Worring, and J.S. Duncan. On multi-feature integration for deformable boundary nding. In Proc. Intl. Conf. on Computer Vision, pages 846{851, L.D. Cohen. On active contour models and balloons. CVGIP: Image Understanding, 53(2):211{218, M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. Intl. J. of Computer Vision, 1(4):321{331, T. McInerney and D. Terzopoulos. Topologically adaptable snakes. In Proc. Intl. Conf. on Computer Vision, D.N. Metaxas. Physics-Based Modeling of Nonrigid Objects for Vision and Graphics. PhD thesis, University of Toronto, D.N. Metaxas and I.A. Kakadiaris. Elastically adaptive deformable models. In Proc. European Conf. on Computer Vision, R. Ronfard. Region-based strategies for active contour models. Intl. J. of Computer Vision, 13(2):229{251, J.K. Udupa and S. Samarasekera. Fuzzy connectedness and object denition. In Medical Imaging, M. Worring, A.W.M. Smeulders, L.H. Staib, and J.S. Duncan. Parameterized feasible boundaries in gradient vector elds. Computer Vision and Image Understanding, 63(1):135{144, S.C. Zhu, T.S. Lee, and A.L. Yuille. Region competition: Unifying snakes, region growing, and bayes/mdl for multi-band image segmentation. In Proc. Intl. Conf. on Computer Vision, pages 416{423, This article was processed using the LATEX macro package with LLNCS style 10