Simon K. Warfield, Michael Kaus, Ferenc A. Jolesz and Ron Kikinis

Transcription

1 Medical Image Analysis (2000) volume 4, number 1, pp c Elsevier Science BV Adaptive, Template Moderated, Spatially Varying Statistical Classification Simon K. Warfield, Michael Kaus, Ferenc A. Jolesz and Ron Kikinis Surgical Planning Laboratory, Brigham and Women s Hospital and Harvard Medical School, Dept. of Radiology, 75 Francis St., Boston, MA, 02115, Abstract A novel image segmentation algorithm was developed to allow the automatic segmentation of both normal and abnormal anatomy from medical images. The new algorithm is a form of spatially varying statistical classification, in which an explicit anatomical template is used to moderate the segmentation obtained by statistical classification. The algorithm consists of an iterated sequence of spatially varying classification and nonlinear registration, which forms an adaptive, template moderated (ATM), spatially varying statistical classification (SVC). Classification methods and nonlinear registration methods are often complementary, both in the tasks where they succeed and in the tasks where they fail. By integrating these approaches the new algorithm avoids many of the disadvantages of each approach alone while exploiting the combination. The ATM SVC algorithm was applied to several segmentation problems, involving different image contrast mechanisms and different locations in the body. Segmentation and validation experiments were carried out for problems involving the quantification of normal anatomy (MRI of brains of neonates) and pathology of various types (MRI of patients with multiple sclerosis, MRI of patients with brain tumors, MRI of patients with damaged knee cartilage). In each case, the ATM SVC algorithm provided a better segmentation than statistical classification or elastic matching alone. Keywords: segmentation, elastic matching, nonlinear registration, nearest neighbor classification, multiple sclerosis, knee cartilage, neonate, brain, tumor. 1. INTRODUCTION The segmentation of structures or types of tissue from medical images is still a difficult problem. In particular, the segmentation of pathology often requires intensive manual interaction to achieve good, or even acceptable, segmentations. Our goal was to develop a generally applicable segmentation algorithm that could aid in the automation of medical image analysis tasks by successfully segmenting both normal anatomy and common types of pathology. A strategy of feature detection and classification (spectral segmentation) has been widely used for the identification of tissue types. When spatial information (such as shape and spatial relationships between structures) is significant, a variety of deformable models have been proposed. Some segmentation problems can be solved simply by feature identification and classification. In this case, there is no need to make use of an anatomical template, and the segmentation is a straightforward process. Similarly, some segmentation problems can be solved by matching deformable Corresponding author ( warfield@bwh.harvard.edu) models directly to the image data (Collins, 1994; Kumar et al., 1996; Solloway et al., 1997). Here we deal with those segmentation problems for which feature identification and classification, or matching deformable models alone, are insufficient. These problems are typified by spectral overlap and/or structures to be segmented that are not modelled in, or cannot be matched by, a template of normal anatomy. Statistical classification and nonlinear registration are often complementary segmentation strategies. For example, classification is often successfully applied for the global segmentation of major tissue types (Vannier et al., 1985; Cline et al., 1990; Wells et al., 1996; Warfield, 1996). Deformable models have been successfully applied to the localization of particular anatomical structures. Tissue classification is unsuccessful when different structures to be identified have the same or overlapping spectral properties. Deformable models often need accurate initialization and are usually optimized for particular structures (sometimes with a single closed boundary), and can fail in the presence of abnormal anatomical variability or even in the presence of normal but highly variable structures, such as the cortex Collins (1994); Warfield et al. (1995). Alignment optimiza-

2 44 S. K. Warfield et al. tion strategies have usually been driven by local gradient information which is often insufficient to distinguish particular structures of interest. Explicit anatomical templates have been successfully used to identify anatomical structures through nonlinear registration. Tumors or lesions are not modelled in an anatomical template of normal anatomy, and so cannot be directly segmented in this way. We developed a new algorithm, ATM SVC, which overcomes the above limitations by embedding a traditional multiple feature k-nn classification into a higher dimensionality problem space. The additional dimensionality is derived from an anatomical template, and acts to moderate the statistical classification. In the special case of the classification problem involving separable classes with nonoverlapping anatomy accurately described by a matched anatomical template, the ATM SVC solution is the same as the original k-nn classification problem. In the common case involving some spectral overlap, ATM SVC resolves the ambiguity of the feature space with anatomical context, as well as allowing the classification of structures not present in the anatomical template. 2. METHOD Our algorithm iterates between a classification step to identify tissues and an elastic matching step to align a template of normal anatomy with the classified tissues. The template of anatomy is used to modify the classification to produce a spatially varying, rather than global, classification. The steps are iterated until the matched anatomy and the classification agree. Figure 1 illustrates the ATM SVC algorithm. Initialization for the algorithm consists of image acquisition, tissue class prototype selection and initial registration of a template of normal anatomy to the image data. The anatomical template is converted into a set of features describing anatomical localization with a distance transform. A segmentation based upon these features is computed with k-nn classification. The segmentation is then put through a feedback path that is used to refine the segmentation. A fast elastic matching algorithm is used to refine the alignment of the template of normal anatomy to the classified patient data. The anatomical localization is recomputed with the refined atlas, and the process is iterated Prior to Segmentation Image acquisition It is usually necessary to optimize the image acquisition protocol for automated image analysis, beyond that necessary for inspection by radiologists. Our acquisitions are designed to optimize the tradeoff of resolution, contrast to noise and time in the scanning device for the patient. The goal is to maximize the contrast of the structures we wish to analyze. The ultimate goal of image acquisition optimization is to render segmentation a trivial operation. Usually this is not possible, but good acquisitions make image analysis significantly simpler. We have developed several databases of scans of patients exhibiting particular types of pathology. The automated segmentation of such databases can aid in understanding the nature of these pathologies, through improved quantification and visualization. These databases allow for the assessment of the robustness and validity of new segmentation algorithms Template generation We choose to represent normal anatomy with a 3D volumetric digital atlas. The atlas consists of a 3D volume with each voxel labelled according to the anatomical structure present. A hierarchical representation of the anatomy can be imposed upon the atlas by describing groups of labels according to higher level anatomical constructs. For example, the cerebrum consists of left and right hemispheres, each of which consists of grey matter and white matter, each of which is divided into cortical gyri and subcortical structures and so on. We construct templates of normal anatomy by selecting a scan of a normal volunteer and segmenting the scan by statistical classification followed by manual editting (Kikinis et al., 1996). We have generated atlases for the brain, the knee joint, the abdomen and the chest The ATM SVC algorithm When we come to segment a particular patient, several steps are involved in the initialization of data for the algorithm. Initialization involves the application of feature detection or image enhancement algorithms to construct images that have improved contrasts for the structures of interest (e.g. nonlinear diffusion for noise smoothing, local structure enhancement), the execution of an initial alignment strategy to align the template to the patient scan, and the selection of prototype voxels for tissue classes of interest to allow the construction of a statistical model for the distribution of features for each tissue class. The initial alignment depends upon whether the anatomy to be aligned is rigid or piecewise rigid, and examples are presented in Section 3. For brain MR, the usual rigid registration techniques are suitable, and in the past we have used surface matching (Warfield et al., 1995), whilst more recently we have used a faster, more robust, method based on comparison of dense labelled data (Warfield et al., 1998). For the knee joint, we use a piecewise rigid registration by manual alignment of each of the bones. The atlas and patient bones are displayed in windows showing three orthogonal slices with linked cursors. When a cursor is moved in any window, or a new slice is selected, cursors in each of the linked windows also move, maintaining correspondence of spatial location between the two sets of three orthogonal slices. The user inputs modifications to the transform parameters (three rotation, three translation and three scale parameters) until the bones are aligned. We do not yet have extensive experience with nonrigid areas of the body, but we expect a spline based nonlinear registration

3 Adaptive, Template Moderated, Spatially Varying Statistical Classification 45 Volumetric image data Feature identification knn classification Segmentation Tissue class prototypes Anatomical template Elastic matching Figure 1. Schema for Adaptive Template Moderated Spatially Varying Classification. Initialization consists of image acquisition, tissue class prototype selection and initial registration of a template of normal anatomy to the image data. The anatomical template is converted into a set of features describing anatomical localization with a distance transform. A segmentation based upon these features is computed with k-nn classification. The segmentation is then put through a feedback path that is used to refine the segmentation. A fast elastic matching algorithm is used to refine the alignment of the template of normal anatomy to the classified patient data. The anatomical localization is recomputed with the refined atlas, and the process is iterated. could be successful for initial alignment of a template to a patient scan Supervised Statistical Classification The k-nearest Neighbor (k-nn) classification rule is a technique for nonparametric supervised pattern classification. An excellent description of k-nn classification and its properties is provided in (Duda and Hart, 1973). Given a training data set, P, consisting of N prototype patterns (vectors) of dimension D and the corresponding correct classification of each prototype into one of C classes, a pattern v of unknown class, is classified as class w i if most of the k closest prototype patterns are from class w i (Cover and Hart, 1967). Distance is measured with a distance metric appropriate to the problem domain. The main characteristic of the ATM SVC algorithm is to use Euclidean distance in a modified feature space, as described in Section Feature Detection A wide range of enhancement and filtering techniques tailored to specific applications are known. When an enhancement technique is identified to improve the contrast of a particular structure for a particular segmentation problem it can be used as a feature (also called channel) in a statistical classifier. Local filtering strategies for feature enhancement could be motivated by the anatomical template. We routinely use nonlinear diffusion (Gerig et al., 1992) as an enhancement strategy. It was used for the enhancement of the data used for the examples in Section 3, which include brain scans of patients with multiple sclerosis, brain scans of patients with brain tumors, brain scans of neonates and scans of the knee joint. The matched template is the source of new features that provide anatomical localization, as described in Section These act to improve the segmentation by increasing the separation between different classes in the feature space. Classifier Training Prototypes are voxels selected to act as examples of particular tissue classes. These prototypes are used to model the probability density function of the features of particular tissue classes. The prototypes are selected by an experienced operator, usually a trained technician familiar with the anatomy of the subject to be segmented, and the operation of the classification process. This is similar to the prototype selection that has been used in earlier statistical classification methods (Kikinis et al., 1992; Cline et al., 1990). The prototype selection process records both the spectral information of the prototype (the value of the prototype in each of the features ) and its spatial location. Its spatial location is not directly used in the classification process, but it is recorded so that it can be used to recompute the spectral information if any feature values are changed (such as occurs during the adaptation of the anatomical localization information). Typically prototypes are selected for each class to be segmented. This usually takes less than five minutes of user interaction. For brain MR examples of Section 3 prototypes for background, white matter, grey matter and cerebrospinal fluid were collected. Depending on the nature of the brain MR, we also selected, white matter lesion, tumor, edema, and unmyelinated white matter. In the case of knee MR, we used prototypes for background, skin, fat, ligament, meniscus, muscle, patella cartilage, femoral cartilage, tibial cartilage, fibula cartilage, patella, femur, tibia and fibula Spatially Varying Classification The alignment of the template is used to modify the classification so as to improve the overall segmentation. In our applications, the aligned template either does not model all of the structures to be segmented (such as tumors or lesions), or the nonlinear registration alone does not capture the relevant local shape differences (the scan to be aligned is too variable or does not contain sufficient correspondences or has different topology). k-nn Classification with Anatomical Localization We make use of the approximately aligned template to modify the classification. A particular voxel v of unknown class, is classified by assigning the class that appears most frequently in the set of k nearest prototypes. Determining

4 46 S. K. Warfield et al. the k nearest prototypes involves a distance computation of the form: d 2 D v f f 1 where d is distance, v f is the value of voxel v in feature f, p f is the value of prototype p in feature f, and D is the number of features. If we have no spatial model for the expected anatomy, then this classification scheme is optimal in that it minimizes the risk of making a wrong classification. If we have a perfectly accurate model for the expected anatomy, we could reject from consideration any prototypes lying outside the expected anatomical region. Since we do not have a precisely localized model, instead we modify the computation of the distance to prototypes in a way that reflects our confidence of its spatial location. Consider a segmentation problem where we have anatomical localization of only one structure and that a mechanism exists for transforming the aligned template into a map of certainty of anatomical localization. In the presence of perfect certainty of anatomical localization, the desired effect could be achieved by modifying the distance computation in the following way: d 2 a D v f f 1 where a 0 if p and v are located within the anatomical structure, and a if either p or v are located outside the anatomical structure. When p and v are inside the anatomical structure, d is unmodified from that of the original classification problem, and when either p or v are outside the anatomical structure d becomes larger. We can consider a range of confidence in our anatomical localization, ranging from the regions we have reason to suspect are far away from the anatomical structure, to those regions we suspect are very likely to be the anatomical structure. The nature of the misalignment of the nonlinear registration makes our confidence in the anatomical localization weakest at the boundary of the model. We can also generalize the anatomical localization to allow for more than one anatomical region. A possible modification of the distance computation then becomes: d 2 M v a a 1 p a 2 p f 2 p f 2 D v f f 1 where we have added M features of spatial information, and v a p a 2 represents the difference in anatomical localization of the voxel to be classified and the prototype. Qualitatively, when v and p are from the same anatomical region, v a and p a will vary together, leading to a small addition (ideally zero) to the overall distance, and so the distance will be mainly determined by the features. When v and p are from different anatomical regions, v a and p a p f 2 will vary differently, leading to a large change to the overall distance, causing the distance to be large irrespective of the values in the features. Combining the calculation for all of the features, the distance function is simply d 2 D v f f 1 where D D M. An important advantage of using k-nn classification in this way is that it is straightforward to set up a balance between the influence of the anatomical localization and the other feature channels. When the existing feature channels have insufficient contrast to identify a structure, the anatomical localization can provide weighting for the prototypes from which the classification for a voxel is computed. As the anatomical localization weighting increases, the contrast in feature space between the different structures increases. An incorrect template alignment will have no effect if the anatomical localization weight is less than the contrast in the other feature channels. Estimation of Anatomical Localization We generate anatomical localization feature channels by converting each of the relevant structures of an aligned template into a distance map. We model the uncertainty of anatomical localization, which depends upon the size of the potential error in elastic matching. A straightforward model of error is to use a penalty of 0 where labels for the matched structure is present, and increase the penalty linearly or quadratically with distance from the anatomical template until saturating it. When better error bounds on the anatomical localization are known, they can be directly incorporated by modifying the penalty in the relevant regions. This model is efficient to compute: a volume the size of the data is initialized with large values, the anatomical structure is copied from the aligned template into the volume with value 0, and a distance transform of the volume is computed. The largest penalty is limited by clamping the distance function. This limit is applied since we do not wish very distant structures to dominate the classification distance computation Nonlinear Registration We achieve fast nonlinear registration with an elastic matching algorithm based upon that of Dengler and Schmidt (1988), which is described below. The precise formulation of the nonlinear registration algorithm is not intrinsic to the ATM SVC algorithm, and it is possible to use other nonlinear registration algorithms (Bajcsy and Kova ci c, 1989; Christensen et al., 1994; Thirion, 1998). We have used the method of Dengler and Schmidt (1988) because it is sufficiently fast to be used routinely. The goal of the elastic matching algorithm is, given a source data set g 1 x and a target data set g 2 x, to find a deformation vector field u x such that the function g 1 x u x is as similar to the function g 2 x as possible. p f 2

5 Adaptive, Template Moderated, Spatially Varying Statistical Classification 47 We require u to be slowly varying, so that we can make the following Taylor series approximation. This is reasonable because we only expect to capture local shape differences, and the initial alignment process accounts for global shape differences. The basic method of computing u is via template matching: for a fixed value of x, consider the problem of finding the value of u that minimizes E x u w x x g 2 x g 1 x u 2 dx The resulting value of u is taken as the value of u x. Here, w is a window function whose width determines the size of the region used to compute u x. We make the approximation g 1 x u g 1 x u T g 1 x leading to the expression E x u w x w x x x g 2 x g 1 x u T g 1 x 2 dx g 2 g 1 2 g 2 g 1 u T g 1 dx 2 u T g 1 The value of u minimizing this expression is given by the solution of Qu F where and Q w x x g 1 x g T 1 x dx F w x x g 1 x 2 g 2 x g 1 x dx Our implementation calculates the deformation field with a multiresolution approach which computes u incrementally so that the approximation of small u which is needed to derive the equation Qu F can be satisfied at each resolution. At each resolution level, the deformation field is computed with window functions of several widths and the best window function width is determined adaptively (?). The coarsest level is determined by the additional requirement that for each axis, the width of the window function w is no more than 2 1 of the width of the image. At the top level, the deformation field u is estimated. This is then interpolated down to the next lower level. This procedure is repeated until the bottom level is reached Iteration We compute a classification, and use the classification to update the template alignment. The updated template is then used to generate a new anatomical localization and used to compute a new classification. To recap, the segmentation at iteration number j, s j, is computed with a k-nn classification applied in the feature space derived from the original image acquisition and the spatial localization information derived from anatomical template t j. Following the computation of the segmentation, an updated template, t j 1, is computed by elastic matching of the template t j to the segmentation just derived, s j. The spatial localization is then recomputed from the template t j 1, which is then used to compute the segmentation s j 1 and so on. The iteration is initialized by selecting as t 0 the initial registration of the atlas to the patient. For the segmentation problems described in Section 3, the algorithm converges to a satisfactory segmentation around five or fewer iterations of this process. The current implementation does not make use of a test of convergence (such as a low rate of change of the classification from iteration to iteration), but rather is iterated a fixed number of times, the precise number of which depends upon the particular application because of the different levels of contrast and anatomical variability. The conditions under which k-nn classification is optimal are well understood (Section 2.2.1) and its use here is designed to meet those requirements as well as a practical implementation can. The elastic matching algorithm has been designed to capture slowly varying local nonlinear deformations, and has been successful in practice, but is not guaranteed to converge under all circumstances. In particular, shape differences due to cutting or motion which would require a discontinuity in the deformation field are not captured. A more general nonlinear registration algorithm would be better suited to such applications and could be incorporated in the ATM SVC scheme. 3. RESULTS In this section the application of ATM SVC to segmentation problems involving both normal anatomy and pathology is presented. For each of these problems, segmentation with a global (non-spatially varying) classification was also carried out and a visual comparison was made. In each case the spatially varying classification generated by ATM SVC better reflects the underlying anatomy than global classification techniques Illustrative Example: Segmentation of Brain MR of Patients with Multiple Sclerosis White matter lesions appear in the brain of patients with multiple sclerosis and evolve over time (Guttmann et al., 1995). During the lifetime of a lesion some parts of the lesion develop signal intensity characteristics that overlap those of normal grey matter on conventional spin echo MR images (Warfield et al., 1995). Figure 2 shows a conventional spin echo MR image from a scan of a patient with multiple sclerosis. Figure 3 illustrates the difficulty of classification of lesion and grey matter classes caused by the overlap of the MR intensity range of these classes. Figure 4 shows the grey matter and white matter derived from a matched atlas, and the anatomical localization channels for white matter and grey matter. For the

6 48 S. K. Warfield et al. segmentation of scans of patients with multiple sclerosis we have developed a grey matter localization algorithm, based upon region growing, that improves upon the localization of cortical grey matter that is possible with elastic matching alone (Warfield et al., 1995). We applied that improved grey matter localization algorithm here, as an extra step following elastic matching, in order to improve the localization of grey matter and white matter prior to computing spatially varying penalties for these classes. This improved localization allows a larger penalty to be applied for white matter and grey matter classification outside the area indicated by the refined anatomical model. Figure 5 illustrates that the ATM SVC improves the segmentation of normal white matter, lesion and grey matter, over that of statistical classification alone. The segmentation is derived from a four channel classification incorporating proton density weighted images, T2 weighted images, grey matter localization and white matter localization. The anatomical localization channels act to improve the discrimination between the lesion and grey matter classes. Algorithmic component time (min. x # itns) k-nn classification 0.9 x 5 affine registration 5.9 x 5 distance transform 3.5 x 5 elastic matching 2.5 x 5 erosion/dilation/region growing 1.6 x 5 operator interaction 4.0 x 1 Table 1. Wallclock computation time (minutes x number of iterations) for a typical brain tumor case segmentation. Total time for ATM SVC was 76 minutes (with 4 minutes of operator interaction for initialization) whereas manual segmentation required 180 minutes. Observer ATM SVC C.V. (%) Table 2. Coefficient of variation of volume of manual and ATM SVC femoral cartilage segmentation Classification of Cortical and Subcortical Grey Matter from MR of Neonates In the developing brain, rapid changes in the amount of cortical grey matter occur. These changes can be studied with MR imaging (Hüppi et al., 1997, 1998). Since the brain is still developing, the signal intensity characteristics of different parts of the brain can be quite similar, making it difficult to apply conventional classification techniques. We applied the ATM SVC algorithm to this problem with the goal of improving the segmentation of cortical grey matter and subcortical grey matter. These have similar intensity ranges, but different spatial locations, and accurate measurement of cortical grey matter is a clinically interesting problem. Figure 6 illustrates the improvement in the segmentation that is achieved with ATM SVC. The segmentation was carried out with MR intensity features from SPGR (Spoiled Gradient Recalled acquisition in the steady state), and conventional spin echo imaging (PDw, T2w). Anatomical localization was generated for the basal ganglia and brain stem structures, and a channel representing everything that is not these structures Brain Tumor Segmentation Figure 7 illustrates the segmentation of an MRI of a patient with a brain tumor (a meningioma). The MRI is an SPGR, covering the entire brain with a voxel resolution of x0.9375x1.5 mm 3. A sagittal image from this acquisition is shown in Figure 7(a). For preoperative surgical planning, a segmentation of the skin, brain, ventricles and tumor is desired. Figure 7(b) shows the segmentation of these structures obtained by manual outlining. Figure 7(c) shows statistical classification (k-nn) based on the MR intensity is unable to identify these structures because they have overlapping MR intensity ranges. Elastic matching of a template of normal anatomy can project the boundaries of structures present in the atlas, as shown in Figure 7(d), but does not segment the tumor. Figure 7(e) shows the segmentation obtained by ATM SVC. The anatomical localization provided by the atlas is used to model the skin (fat and bone structures) and the brain. A spatial localization model for the tumor is derived from the k-nn classification and morphological processing (erosion and dilation). Together these three anatomical localization channels and the SPGR channel are combined to form the segmentation of Figure 7(e). Table 1 indicates the wallclock computation time of each of the algorithmic components for the segmentation of a typical brain tumor case. The times were measured on a Sun Microsystems Ultra HPC 5000 server with 8 UltraSPARC-I (167 MHz) CPUs and 2 GB of RAM Segmentation of Knee Cartilage from MRI Figure 8(a) is a single image from an MR of a patient with damaged knee cartilage. A defect in the bright cartilage along the femur/femoral cartilage boundary can be seen. Figure 8(b) is an image showing the ATM SVC segmentation of the cartilage of the patella, femur and tibia. We compared the ATM SVC algorithm with manual outlining for the segmentation of the femoral cartilage. Four experts segmented the patient image 8(a) with a single focal defect, 5 to 10 times on separate occasions over a period of one month. One of the experts also carried out repeated initializations of the automatic segmentation, so the variability induced in the segmentation by differences in initialization could be determined. The volume of the femoral cartilage was recorded, and the coefficient of variation of the volume for each of the experts and the automatic segmentation is shown in Table 2. The use of the ATM SVC was found to greatly reduce the variability of the segmentation.

7 Adaptive, Template Moderated, Spatially Varying Statistical Classification 49 (a) Early echo. (b) Late echo. Figure 2. Conventional spin echo MRI (proton density weighted and T2 weighted) from a patient with multiple sclerosis (lesions appear bright in both images) Grey Matter Lesion Number of voxels Proton Density Intensity (a) Histogram of lesion and grey matter from the proton density weighted MR image from one patient shown in Figure 2. (b) k-nn classification of tissue classes background, ventricles, grey matter, lesion and white matter (in order of increasing intensity). Figure 3. Figure 3(a) illustrates the overlapping range of proton density weighted intensity for the lesion and grey matter tissue classes, derived from the MRI scan of the patient shown in Figure 2. Figure 3(b) shows how a classification algorithm (k-nn) will misclassify lesion and grey matter as a result of this overlap.

8 50 S. K. Warfield et al. (a) Simplified anatomical template showing grey matter, white matter and background. (b) Anatomical localization of white matter. (c) Anatomical localization of grey matter. Figure 4. An example of anatomical localization channels for grey matter and white matter from the brain MR of a patient with multiple sclerosis (Figure 2). Figure 4(a) is a simplified atlas (showing white matter and grey matter) from which the spatially varying classification penalty is computed. Figure 4(b) and Figure 4(c) illustrate the spatially varying penalty applied for white matter and grey matter respectively white regions represent a low penalty and the penalty increases as the image intensity becomes darker. 250 Number of voxels WM Distance PD Intensity (a) Feature space of white matter localization and proton density intensity showing the distribution of grey matter (large peak) and lesion (small peak). (b) ATM SVC segmentation. Figure 5. Figure 5(a) shows a normalized two channel histogram of MR proton density weighted intensity and white matter anatomical localization. Since multiple sclerosis lesions primarily appear in the white matter, a cluster appears for lesion at a distance zero from the white matter. The white matter localization channel ensures that the grey matter cluster appears at a distance from white matter. The amount of the separation can be controlled by adjusting the white matter localization distance function. In the new feature space the lesion and grey matter classes no longer overlap, and so these classes can be accurately segmented, as shown in Figure 5(b).

9 Adaptive, Template Moderated, Spatially Varying Statistical Classification 51 (a) Single slice of neonate brain SPGR MR. (b) Single slice of neonate T2 weighted MR. (c) k-nn classification. (d) ATM SVC segmentation. Figure 6. MRI of neonate, k-nn classification and ATM SVC segmentation. The segmentation shows, in order from darkest to lightest, background, ventricles, unmyelinated white matter, myelinated white matter (present here primarily in the region of the brainstem), cortical grey matter and subcortical grey matter. The quantification of cortical grey matter is a clinically important problem, but it is difficult to distinguish from subcortical grey matter because of the similarity of MR intensity. The ATM SVC algorithm allows the spatial distribution of grey matter to be modelled, and as shown, generates an improved segmentation of cortical and subcortical grey matter over that of k-nn classification alone.

10 52 S. K. Warfield et al. (a) SPGR. (b) Manual segmentation. (c) k-nn classification. (d) Matched atlas. (e) ATM SVC segmentation. (f) Surface rendering of skin, tumor and ventricles. Figure 7. Illustration of the segmentation of a brain tumor from MRI. 4. DISCUSSION AND CONCLUSION We have developed a new algorithm which is an adaptive, template moderated, spatially varying statistical classification. The examples presented in Section 3 demonstrate that ATM SVC can achieve better segmentations than either statistical classification or nonlinear registration alone for the illustrated problems. These problems involve both normal anatomy and also certain types of pathology. The ATM SVC algorithm may be unable to segment structures which have similar characteristics in all feature channels and are also not spatially distinct. When the structures to be segmented have similar characteristics in all feature channels the classification is strongly dependent upon the spatial localization that can be derived from the nonlinear registration. The classification is then strongly dependent upon the boundaries identified by elastic matching. If the structures to be segmented have a boundary that cannot be discerned by elastic matching, the segmentation in the region of the boundary can be wrong, up to the size of the error of the spatial localization. Related work has examined different types of adaptation in the classification problem. The segmentation of tissue classes in the presence of intensity inhomogeneity was the motivation for the development of an adaptive tissue classification algorithm by Wells et al. (1996). This method, based upon the Expectation-Maximization (EM) algorithm, iteratively adapts a model of intensity inhomogeneity to drive a tissue classification algorithm. This has the advantage of making it possible to classify patient scans without per-patient prototype selection. For ATM SVC we carry out protype selection on a per-patient basis and iterate the classification with respect to an adaptive anatomical model. For the examples of Section 3, we have found it unnecessary to explicitly include estimation of intensity inhomogeneity, although it would be straightforward to incorporate. An alternative modification to the usual statistical classification model is to use a spatially varying a priori probability field. Van Leemput et al. (1998) incorporated such a field into the adaptive segmentation algorithm of Wells et al. (1996). This improved the performance of the algorithm by providing improved a priori information about the spatial distribution of tissue classes (white matter, grey matter and cerebrospinal fluid). Kamber et al. (1992) built

11 Adaptive, Template Moderated, Spatially Varying Statistical Classification 53 (a) Single slice of knee MR showing bones (patella, femur, tibia and fibula), cartilage (bright signal), with a defect in the cartilage of the femur. (b) ATM SVC segmentation of the cartilage of the patella, femur and tibia (in order from left to right across the image). (c) Surface rendering showing the femur, the tibia, and cartilage of the femur, tibia and patella. (d) A second surface rendering showing the segmentation of Figure 8(c) from another angle. Figure 8. MRI of a patient knee (Figure 8(a)), the ATM SVC segmentation of the cartilage of Figure 8(a) and surface renderings of the ATM SVC segmentation of the entire volume, showing the automated segmentation of bone and cartilage.

12 54 S. K. Warfield et al. a probabilistic spatially varying a priori model which was successful at reducing the number of false positives in a lesion classification problem. However, neither of these algorithms used nonlinear registration to adapt the spatially varying model to the particular patient being segmented. A probabilistic atlas of this type provides useful information about the population from which the atlas is derived. The ATM SVC approach makes use of an individualized anatomical template which is adapted to each subject. Future extensions of the ATM SVC approach could include the incorporation of Expectation-Maximization based intensity inhomogeneity correction and the use of a probabilistic atlas to incorporate a spatially varying prior probability field for the spatial distribution of tissue classes. Acknowledgements This investigation was supported (in part) by a grant from the National Multiple Sclerosis Society (SKW) and by NIH grants P41 RR , R01 RR A and P01 CA REFERENCES Bajcsy, R. and Kova ci c, S., (1989). Multiresolution Elastic Matching. Computer Vision, Graphics, and Image Processing, 46, Christensen, G. E., Rabbitt, R. D. and Miller, M. I., (1994). 3D brain mapping using a deformable neuroanatomy. Phys. Med. Biol., 39, Clarke, L. P., Velthuizen, R. P., Phuphanich, S., Schellenberg, J. D., Arrington, J. A. and Silbiger, M., (1993). MRI: Stability Of Three Supervised Segmentation Techniques. Magnetic Resonance Imaging, 11, Cline, H. E., Lorensen, W. E., Kikinis, R. and Jolesz, F., (1990). Three-Dimensional Segmentation of MR Images of the Head Using Probability and Connectivity. Journal of Computer Assisted Tomography, 14, 6, Collins, D. L., (1994). 3D Model-based segmentation of individual brain structures from magnetic resonance imaging data. Ph.D. thesis, McGill University. Cover, T. M., (1968). Estimation by the Nearest Neighbor Rule. IEEE Transactions On Information Theory, IT-14, 1, Cover, T. M. and Hart, P. E., (1967). Nearest Neighbor Pattern Classification. IEEE Transactions On Information Theory, IT-13, 1, Dengler, J. and Schmidt, M., (1988). The Dynamic Pyramid A Model for Motion Analysis with Controlled Continuity. International Journal of Pattern Recognition and Artificial Intelligence, 2, 2, Duda, R. O. and Hart, P. E., (1973). Pattern Classification and Scene Analysis. John Wiley & Sons, Inc. Friedman, J. H., Baskett, F. and Shustek, L. J., (1975). An Algorithm for Finding Nearest Neighbors. IEEE Transactions On Computers, C-24, 10, Gerig, G., Kübler, O., Kikinis, R. and Jolesz, F. A., (1992). Nonlinear Anisotropic Filtering of MRI Data. IEEE Transactions On Medical Imaging, 11, 2, Guttmann, C. R. G., Ahn, S. S., Hsu, L., Kikinis, R. and Jolesz, F. A., (1995). The Evolution of Multiple Sclerosis Lesions on Serial MR. AJNR, 16, Hüppi, P., Warfield, S., Kikinis, R., Barnes, P. D., Zientara, G. P., Jolesz, F. A., Tsuji, M. K. and Volpe, J. J., (1998). Quantitative Magnetic Resonance Imaging of Brain Development in Premature and Mature Newborns. Ann. Neurol., 43, 2, Hüppi, P., Warfield, S., Taranto, R., Ringer, S., Jolesz, F. and Volpe, J., (1997). Cortical brain development in preterm and fullterm infants: surface and volume changes measured by 3D-MRI. In The Society for Pediatric Research 1997 Annual Meeting, vol. 41, p Kamber, M., Collins, D. L., Shinghal, R., Francis, G. S. and Evans, A. C., (1992). Model-based 3D segmentation of multiple sclerosis lesions in dual-echo MRI data. In SPIE Vol. 1808, Visualization in Biomedical Computing, pp Kikinis, R., Shenton, M. E., Gerig, G., Martin, J., Anderson, M., Metcalf, D., Guttmann, C. R. G., McCarley, R. W., Lorenson, W. E., Cline, H. and Jolesz, F., (1992). Routine Quantitative Analysis of Brain and Cerebrospinal Fluid Spaces with MR Imaging. Journal of Magnetic Resonance Imaging, 2, Kikinis, R., Shenton, M. E., Iosifescu, D. V., McCarley, R. W., Saiviroonporn, P., Hokama, H. H., Robatino, A., Metcalf, D., Wible, C. G., Portas, C. M., Donnino, R. M. and Jolesz, F. A., (1996). A Digital Brain Atlas for Surgical Planning, Model Driven Segmentation, and Teaching. IEEE Transactions on Visualization and Computer Graphics, 2, 3, Kumar, A., Tannenbaum, A. R. and Balas, G. J., (1996). Optical Flow: A Curve Evolution Approach. IEEE Transactions On Image Processing, 5, 4, Solloway, S., Hutchinson, C., Waterton, J. and Taylor, C., (1997). The use of active shape models for making thickness measurements of articular cartilage from MR images. Magn Reson Med, 37, 6, Thirion, J., (1998). Image matching as a diffusion process: an analogy with maxwell s demons. Medical Image Analysis, 2, 3, Van Leemput, K., Maes, F., Vandermeulen, D. and Suetens, P., (1998). Automatic Segmentation of Brain Tissues and

13 Adaptive, Template Moderated, Spatially Varying Statistical Classification 55 MR Bias Field Correction Using a Digital Brain Atlas. In MICCAI 98: First International Conference on Medical Image Computing and Computer-Assisted Intervention, eds. Wells, W., Colchester, A. and Delp, S., vol of Lecture Notes in Computer Science, pp Boston, USA. Vannier, M. W., Butterfield, R. L., Jordan, D., Murphy, W. A., Levitt, R. G. and Gado, M., (1985). Multispectral Analysis of Magnetic Resonance Images. Radiology, 154, Warfield, S., (1996). Fast k-nn Classification for Multichannel Image Data. Pattern Recognition Letters, 17, 7, Warfield, S., Dengler, J., Zaers, J., Guttmann, C. R., Wells III, W. M., Ettinger, G. J., Hiller, J. and Kikinis, R., (1995). Automatic identification of Grey Matter Structures from MRI to Improve the Segmentation of White Matter Lesions. Journal of Image Guided Surgery, 1, 6, Warfield, S. K., Jolesz, F. and Kikinis, R., (1998). A High Performance Computing Approach to the Registration of Medical Imaging Data. Parallel Computing, 24, 9 10, Wells, W. M., Kikinis, R., Grimson, W. E. L. and Jolesz, F., (1996). Adaptive segmentation of MRI data. IEEE Transactions On Medical Imaging, 15, APPENDIX In this appendix, the K Nearest Neighbor classification algorithm and its properties are described in more detail. Each prototype voxel has associated with it a class, and the most frequently occurring class is determined by counting occurrences of classes of prototypes in N v, where N v is the subset of prototypes, drawn from the set of training prototypes P, with distance less than or equal to the distance to the kth nearest prototype, d k : N v p P : v p d k!#" It has been shown that, with mild conditions, in the infinite training data case the conditional risk of the k-nn rule is 1 1 $ k R&, where R& is the Bayes risk (minimum possible) (Cover, 1968). Each pattern is a D-dimensional vector constructed from the value of the voxel in each of the D features of the image. The k-nn rule has higher accuracy and stability for MRI data than other common statistical classifiers, but straightforward implementations can be slow (Clarke et al., 1993). For low dimensionality problems it is efficient to compute a lookup table for every possible pattern that could occur in the image data, and then classify the volume by finding the location of the value of each voxel in the lookup table (Warfield, 1996). For arbitrary dimensionality problems, such as those our ATM SVC algorithm is designed to solve, an efficient k-nn algorithm is that of Friedman et al. (1975). The algorithm described by Friedman et al. (1975) reduces the number of distance calculations required to find the k-nn. First the prototypes are ordered on the values of each feature axis, generating D ordered sets. In order to classify a test pattern v, an estimate of the local sparsity in the region of v for each feature axis is made, and the ordering corresponding to the largest sparsity is chosen. The location of the test pattern in the ordered prototypes is determined and the search for the k-nn is carried out by considering the prototypes nearest the test pattern using the selected order. The search for the k-nn can be terminated when the distance to the next pattern along the feature axis exceeds that of the k th pattern already found. Statistical analysis (Friedman et al., 1975) indicates the expected number of distance calculations is O(Fk 1' D N 1( 1' D ), where F is the number of patterns to classify, k is the number of nearest neighbors to find, N is the number of training prototypes, and D is the dimensionality of the feature space. That is, if the set of prototypes N v consists of k prototypes, k wi of which have label w i, then the joint probability p v" w i can be estimated by p v" w i k wi $ N V where V is the volume enclosed by k samples, and so the a posteriori probabilities P w i % v can be estimated by: P w i % v k wi k and the minimum risk classification of the voxel v is the class w i.