Nonlinear Registration and Template Driven. Segmentation

Transcription

1 Nonlinear Registration and Template Driven Segmentation Simon Warfield Andre Robatino Joachim Dengler Ferenc Jolesz Ron Kikinis I Introduction Registration is the process of the alignment of medical image data. Nonlinear registration is the set of techniques that allow the alignment of data sets that are mismatched in a nonlinear or nonuniform manner. Such misalignment can be caused by a physical deformation process, or can be due to intrinsic shape differences. For example, deformation of the brain can occur during neurosurgery when the skull is opened and CSF drained, and when a tumour is removed. Nonlinear deformation is a characteristic of the motion of the nonrigid organs of the abdomen. Also, shape differences arise when a comparison is made between the brains of different people. Although the broad structure of the brain is similar throughout different people (and even different species), both normal anatomical variability and various disease states lead to local nonlinear shape differences. Harvard Medical School and Brigham and Women s Hospital, Department of Radiology, 75 Francis St., Boston, MA 02115, Vision Systems, Neckargemuend, Germany This appeared as Chapter 4 of Brain Warping, editted by Arthur W. Toga

2 Applications of nonlinear registration have included the alignment of scans of different brains for the purpose of characterizing both normal and abnormal anatomical variability (brain mapping), and the alignment of a model of anatomy with particular data for the purpose of segmentation (segmentation by registration). In the future, with the development of rapid and robust algorithms, nonlinear registration will most likely become more widely used, with potential applications ranging from surgery to the enhanced analysis of functional images. The term matching is used to refer to any process that determines correspondences between data sets, and warping is used to refer to those matching techniques that involve the computation of a deformation field between points of correspondence. Here elastic matching is used to refer to a subset of the available nonlinear registration techniques which operate by computing a deformation field through the minimization of a functional consisting of a term measuring local similarity and a regularization constraint based on a physical model of an elastic material. Loosely, we also refer to techniques with any physically-based regularization model as elastic matching, even when the model does not correspond precisely to the properties of an elastic body. Some implementation issues have a serious impact on the potential to transfer nonlinear registration techniques from research to clinical applications. The computation of high order nonlinear transformations can require multiple 3D (floating point) data sets be held in working memory whilst a lengthy nonlinear optimization process is performed. Long running times on standard workstations, or even supercomputers, ultimately limits the number, and frequency, of such computations. This does not present problems whilst such techniques are in the research phase, or while small scale studies are carried out. However, in order for nonlinear registration to become a widely used clinical tool, techniques need to be available that can execute in a clinically compatible timeframe on widely available workstations. In this work, we have concentrated on the development of a nonlinear registration algo- 2

3 rithm as a clinically compatible tool, seeking techniques that can allow rapid and robust elastic matching. Our approach is a tradeoff between accuracy of the physically-based regularization model and speed. Typically the formulation of an elastic regularization constraint makes use of assumptions of consistent topology and of homogeneous and isotropic material properties. Examination of these assumptions indicates there is scope to determine methods by which it is acceptable to use fast approximations to these physical models. A physical model of elastic material may apply when a particular patient is undergoing a physical deformation, for example, when pressure is applied (or released) to a patient s brain during neurosurgery. However, when an atlas is being matched to a patient scan or certain kinds of intra-patient time series matching, the regularization model is being applied to the anatomical variability. It is not clear that a model of an elastic material is sufficient to capture all kinds of anatomical variability. Hence, we use the regularization term as a convenient facility for building into the matching process a tradeoff between faithfulness to the model of normal anatomy and faithfulness to the data, and not for its precise representation of the properties of physical bodies. This chapter is organized in the following manner. Section II is an overview of nonlinear registration techniques. Section III discusses our template-driven segmentation approach and our elastic matching algorithm (which is based on the Dynamic Pyramid developed by Dengler (Dengler and Schmidt, 1988)), and Section IV presents some applications of the technique. II Nonlinear Registration Techniques Several methods of nonlinear registration, with varying degrees of generalizability, have been proposed. Each is suitable for certain types of applications. Matching schemes can encode anatomical information as an implicit or explicit set of rules. For example, Brummer (Brummer, 1991) has employed a priori anatomic knowledge in his 3

4 method for detection of the longitudinal fissure of the brain using the Hough transformation. Rule-based (for example if.. then.. statements) identification of anatomic regions and expert systems technology have been developed for the purpose of brain segmentation (Dellepiane et al., 1987; Serpico et al., 1987; Vernazza et al., 1987). The chief drawbacks to these approaches are that the correction of low-level segmentation errors (misclassification of voxels into the wrong tissue class, or mislabeling of boundary segments) is problematic, and rule-based systems tend to become very complex due to the inherent arbitrariness of anatomic definitions. A different approach to segmenting anatomic structures is to use a reference data set and to compare it with the patient data by finding a transformation that aligns the reference data set with the patient data. The goal of automated registration in the case of cerebral imaging, as defined by Evans and coworkers (Collins et al., 1992; Collins et al., 1994; Collins, 1994), is an automatic, reliable, repeatable, robust identification of sub-structures of the brain. There are two components to the problem of automatic nonlinear registration: 1. establishing a relationship between the model (atlas or canonical data set) and the patient specific data, and 2. the actual warping procedure used to establish the transform between atlas and patient. Matching techniques can be categorized by the form assumed for the computed transformation. A 3D geometric transform maps an image from the coordinate system defined by (u,v,w) into a new image defined by the coordinate system (x,y,z). where the mapping functions are! 4

5 Often a fixed model is chosen, and parameters of that model are then estimated. Examples include linear registration, where rotation, scaling, and translation parameters are usually estimated, as well as nonlinear models such as cubic transforms. The nonlinear registration techniques compared here aim to compute high order mapping functions using local information, and constrain the mapping functions based on a physical model of elastic materials. The class of techniques that aim to nonlinearly deform a template (model anatomy), restricting the deformation with a smoothness constraint (a regularization or elasticity constraint), is generally referred to as elastic matching. It involves the construction of a high-order nonlinear transform describing how the template must be deformed in order to maximize the match to the patient scan. Goals associated with this process include segmentation, determination of normal anatomy and anatomical variability through atlas construction, and change detection through the analysis of time series. Elastic matching aims to match a template, describing the anatomy expected to be present, to a particular patient scan so that the information associated with the template can be projected directly onto the patient scan on a voxel to voxel basis. The template can be an atlas of normal anatomy (deterministic or probabilistic), or it can be a scan from a different modality, or it can be a scan from the same modality. The template can contain information typically found in anatomical textbooks, but unlike normal textbooks, can be linked to any form of relevant digital information. Usually matching is a two step process with the first step being to determine a linear registration to globally align the model with the patient scan. The linear registration establishes corresponding regions for both data sets and accounts for translation, rotation and scale differences. A nonlinear transform is then sought that maximizes the similarity of these regions. Most of the elastic matching techniques can be cast in terms of the following framework for a template data set (such as an atlas) and a subject data set (such as an MRI scan) 5

6 defined on a regular rectangular lattice a deformation field which matches data set to data set is estimated. A vector describing the deformation at location is. The goal of an elastic matching algorithm is to estimate a deformation field,, such that the sum of the local similarities between the subject,, and the template,, under the deformation field is maximized, similarity subject to some constraint upon the deformation field the regularization (or elasticity ) constraint, Different approaches to elastic matching may differ in 1. the local similarity measure 2. the features from which the local similarity is computed and the density of features in the template 3. the regularization constraint 4. the method of the solution of the system of equations arising from the chosen model II.A Review of Elastic Matching Techniques and Applications Existing elastic matching techniques are mathematically unified under the umbrella of regularization theory (Poggio et al., 1985). We characterize the available techniques as volumetric or non-volumetric, according to whether or not local volumes are used in the computation of the similarity metric. Some typical applications have been to 6

7 segment normal anatomical structures by matching with an atlas of normal anatomy (Bajcsy and Kova ci c, 1989; Gee et al., 1994; Collins et al., 1992), develop a probabilistic atlas of gross morphometric variability of the human brain (Collins et al., 1994), improve the estimation of metabolism from PET images (Christensen et al., 1994), segment anatomical structures in the presence of white matter abnormalities (Warfield et al., 1995), characterize sulcal variability (Thompson et al., 1996) and abnormal brain structure (Thompson et al., 1997), characterize the hippocampus (Haller et al., 1997; Joshi et al., 1997), characterize subcortical grey matter differences in schizophrenia (Iosifescu et al., 1997). II.A.1 Non-Volumetric Elastic Matching Models Linear registration algorithms that compute a global transform matching two data sets can be used with the same goal as elastic matching. However, it has been found that a single global transformation is unable to account for the normal anatomical variability found between patients and normal volunteers (Collins, 1994). Some investigators have used operator-guided determination of the relationship between reference and patient data. Bookstein, for example, used interactive identification of landmarks to guide matching with a thin-plate spline regularization (Bookstein and Green, 1992). Davis et al. developed an elastic body spline for matching medical images (Davis et al., 1997). The spline is based on a physical model of an elastic body. 7

8 Spline warping and local linear transformations can be used to modify a linear registration to attempt to account for local deformations. However, these systems often require intensive expert operator intervention and guidance in order to select control points or regions to warp. Meyer et al. describes the use of a mutual information metric to refine the position of control points in a thin-plate spline based registration (Meyer et al., 1996). Some attempts have been made to use only the surfaces of objects to discover the transformation that aligns the objects. It is possible to formulate an elastic transformation that deforms the surface of an object to match the surface of a target object. One recent scheme used an iterative refinement approach to compute the elastic transformation (Moshfeghi et al., 1994). First contours of the surface of each object were extracted from each slice of the 3D volume using a semi-automatic active contour model. A surface was formed by triangulating between the contours of neighbouring slices. A distance function was used to describe how well local patches of the two surfaces match. This distance function was a combination of Euclidean distance and similarity of surface orientation. The surfaces of the objects were matched by computing a displacement vector for each triangle vertex on the basis of this distance function. In order to capture the interaction between neighbouring vertices, a deformation field is computed by Gaussian smoothing of the displacement vectors. This procedure is iterated with a decreasing stiffness of the surfaces. The elastic transformation is computed by interpolation of the deformation field over the entire volume. In order to reduce the computation time the contours were subsampled and each was limited to 100 points. This imposes a limit to the fidelity of the surface representation. Nonlinear surface matching has been used to transform the cortical surface segmented from MRI scans to a common parametric representation (a sphere) (MacDonald et al., 1994). The surface is represented with a polygonal model. The surface is matched to the image data by minimizing a cost function,, consisting of the weighted sum of a term measuring the 8

9 difference between the surface and the image data,, and a term imposing a form for the recovered surface, where denotes the surface and denotes the image data. The difference between the surface and the image data is where is a point on the surface, and is a threshold representing an edge in the image data. The model constrains the possible surface deformation by imposing a local constraint, and causing every vertex to be attracted to the centroid of the neighbouring vertices where is the centroid of the neighbours. Thompson and Toga (Thompson and Toga, 1996) have developed a surface-based technique specialized for matching brain images. The scheme involves the determination of several model surfaces, a warp between these surfaces and the construction of a volumetric warp from the surface warp. First, a Chen surface (loosely, a superquadric modulated by spherical harmonics (Thompson and Toga, 1996)) is estimated from a set of manually chosen landmarks for each of the lateral ventricles and the external cortex. The Chen surface is then used to initialize an active contour minimization process where the surface is attracted towards an estimate of the desired surface formed by thresholding an edge map. This process generates estimates of the cortical and lateral ventricle surfaces. In a separate step, major sulci are manually traced and the set of points obtained is converted into a parametric mesh representation of the sulcal surface. The surface representations were used to form surface warps by mapping a parametric grid of fixed size onto each surface. Points on two surfaces were called corresponding points if they 9

10 had the same grid location within their respective surfaces. Shape differences were represented by computing a 3D displacement vector between each such corresponding nodal points. The set of surface warps were interpolated through space to form a volumetric warp able to match a patient scan to an atlas data set. A qualitative comparison of warping in both 2D and 3D, with synthetic images, MRI to MRI matching, and cryosectioned images to MRI matching indicate excellent performance. Analysis of the sulcal surface representations has been applied to characterizing sulcal variability (Thompson et al., 1996) and to the characterization of abnormal brain structures (Thompson et al., 1997). II.A.2 Volumetric Elastic Matching Models In order to improve the anatomical localization of functional information, atlases of normal anatomy have been introduced for the analysis of PET. One approach made use of a stereotactic PET atlas, constructed from a scan of a normal volunteer (Minoshima et al., 1994). Major cortical and subcortical brain structures were identified. Patient scans were matched to the atlas by first computing a linear registration to the stereotactic coordinates, and then resolving small regional differences by nonlinear warping. 3D display was used to visualize the matched data. The stereotactic atlas was found to aid the visual interpretation of PET images and to enhance the accuracy of structural identification. They report difficulties in dealing with normal anatomical variability, and warn of limitations when dealing with subjects who do not have normal brain structures (such as in dementia patients). Similarly, a Volume-Of-Interest (VOI) atlas has been used for quantitative interpretation of functional images (Evans et al., 1991), where it is desirable to provide accurate and reproducible measures of regional cerebral function. The VOI atlas was constructed by hand-contouring structures from MRI slices. MRI and PET images were registered using a semi-automatic method for minimizing the RMS (root mean square) distance between paired points. Paired 10

11 points were automatically determined using a system of fiducial markers. The VOI atlas was matched to the MRI scan in three steps. First some common structures were selected and a global affine transform computed in order to establish a rough match between the VOI atlas and the MRI scan. Local transforms for each structure were then computed to refine the match. The final step involved manual editing of the VOI contours to maximize the match accuracy. The contour representation was chosen to allow for easy manual manipulation, in order to readily account for abnormalities such as tumours, abscesses, strokes and hemorrhages. Several groups have proposed automatic elastic matching schemes for volumetric anatomical models (Dengler and Schmidt, 1988; Nagel, 1983; Collins et al., 1992; Bajcsy and Kova ci c, 1989; Gee et al., 1992; Miller et al., 1993; Christensen et al., 1994). These matching techniques will be reviewed in detail below. Most of these techniques have been successfully applied to the matching of structures of normal brains. An important problem is the development of automatic methods capable of matching successfully in the presence of a range of abnormalities. For widespread use of a matching algorithm to be practical in a clinical environment, it is necessary that the technique have both reasonable computation and memory requirements. Optical Flow Dengler et al. (Dengler and Schmidt, 1988) developed the Dynamic Pyramid, a matching method originally for the computation of velocity fields for 2D image sequences. The similarity measure is based on an optical flow model (Nagel, 1983). where is the deformation field being estimated, measures the local similarity of the images and is a regularization constraint. The model is a 2Dvector field generalization of a controlled continuity functional where is continuous almost everywhere. Discontinuities are allowed where there are occluding edges. In this model 11

12 the deformation field is like an elastic membrane attached to one of the images, warping it to fit another image. The membrane is allowed to tear in some places. This is important since it allows the possibility to relax the regularization constraint in the presence of anatomical abnormalities. A multiresolution Laplacian pyramid is constructed from the image data. Matching is carried out on the sign of the pixels in the Laplacian pyramid. The sign of the Laplacian maintains important indicators of the structure of the images (edges) whilst being less sensitive to noise and sampling effects than the grey scale values. The cross-correlation,, between the sign images over a local neighbourhood is This is approximated with a bivariate quadratic polynomial: where and, the Hessian matrix of the local curvatures, and, are determined on a 3x3 neighbourhood. The regularization constraint is where the first term represents isotropic stretch and the second represents transverse contraction in a 2D elastic membrane. The first term is modified to become a continuity control stabilizer, allowing tears in the membrane at regions of high tension, and the second is set to zero. 12

13 The deformation field is found at each pixel by solving the Euler-Lagrange equations corresponding to the functional. A finite element method discretization is applied and an efficient nested multigrid algorithm with conjugate gradient relaxation is used to solve the resulting system of equations (Schmidt and Dengler, 1989). Gradient of Intensity An elastic matching method using the gradient of the pixel intensity of subject and template MRI scans as the main feature to drive the deformation has been reported (Collins et al., 1992; Collins et al., 1994). The local similarity measure used here is normalized cross-correlation. Let be the normalized cross-correlation between the local neighbourhood of in and the corresponding neighbourhood in : where the summation is carried out over the spherical local neighbourhood of, denotes the interpolated feature value from the volume at voxel position., and The features used for matching are image intensity and gradient magnitude, calculated from a multiresolution pyramid of the data. The deformation vector field is estimated only at voxels with a significant gradient magnitude. lattice The similarity measure between volume and volume for a deformation over the is defined by The deformation field at voxel is calculated by maximizing. is the set of points at which the deformation is estimated, and is constructed in a multiresolution scheme by decimating the highest resolution data set with distance between samples along each axis. Hence is the parameter controlling the size of the local region over which 13

14 similarity is measured. Since the estimate of a deformation at one node affects the deformation of surrounding nodes, this procedure is iterated until the deformation estimates have converged. In early work by Collins no constraint was imposed upon the estimated deformations. This was found to allow unrealistic deformations, so a limit to the maximum deformation at each node was imposed. The neighbourhood is obtained after forming a multiresolution Gaussian pyramid from the volumes and. A four layer pyramid is constructed from the MRI scans, consisting of a volume, constructed by cubic interpolation of the original MRI scan, to having a voxel resolution of 1mm and 3 Gaussian blurred volumes (obtained with Gaussian bandwidth mm). At a given resolution level, the neighbourhood consists of those voxels within a distance of from. Validation experiments involving the recovery of a deformation field from a simple 3D model of a deformed rectangular prism were reported (Collins, 1994). Experiments with brain phantom data (constructed from the MRI scan of a normal volunteer) included recovery of imposed deformations with and without noise. When matching was carried out on one arbitrarily selected MRI volume, the increase in normalized correlation measure, as compared to that obtained with linear registration alone, for the whole brain was reported as 11.3% and the improvement in the residual of matching landmark points was reported as 15%. The template used for matching,, was a brain model obtained by averaging registered MRI scans and the grey scale values of an MRI scan were used directly for. This method was applied to construct an average anatomical variability map, and in order to perform modelbased segmentation of MRI. Bayesian Formulation One of the first reports of elastic matching schemes being used for medical imaging data described an efficient implementation using a multiresolution pyramid (Bajcsy and Kova ci c, 1989). 14

15 The model is based on the description of elastic properties of a material: where defines the deformation field, is the force driving the deformation (measured by local similarity of data and template) and defines the elastic properties of the material. As with the approach of (Collins et al., 1994) described above, the force is applied in regions where the local similarity function exhibits a substantial value. Here, the similarity function is a normalized cross-correlation computed from the magnitude of intensity and first partial derivatives of CT scans. Bajcsy (Bajcsy and Kova ci c, 1989) developed the matching scheme in order to match a brain atlas to CT scans. The features used for matching were the edges of the brain and the ventricles. The template data set,, was a brain atlas constructed by sectioning a human brain and tracing and digitizing the contours of 45 brain structures. The atlas was artificially coloured to have a grey level similar to that of the subject brain,. The model was solved using a finite difference approximation and Jacobi iteration. A finite difference approximation was used to model the similarity function and the force term was estimated over a neighbourhood 3x3x3 voxels. An atlas of 109 structures was obtained by manually outlining 135 sections from a young normal brain. This was used to evaluate the quality of the elastic match (Gee et al., 1993). Deformations were applied to the atlas after artificially colouring the sections, and the elastic matching algorithm applied to recover the deformation. The mean absolute volumetric percent difference for all structures when brain surface, ventricles and grey/white matter boundaries were used to drive the match was 6%, and the mean overlap error was 22%. 15

16 A similar experiment was carried out to investigate the significance of varying the elasticity parameter. It was found that a more rigid atlas gives better immunity to noise but requires more iterations of matching to converge. No significant difference in the quality of the match was found for a wide range of the elasticity parameter. Recently the same group has investigated a probabilistic formulation of the elastic matching problem (Gee et al., 1994; Gee et al., 1995). The probabilistic formulation for finding the deformation that maps each point to a corresponding point in. This formulation allows for the mapping between the reference intensity and the to be determined at the same time as the deformation, by modelling it as a conditional probability density. The elastic matching problem was formulated as solving for and to minimize: where, and the global affine map, true correspondence is assumed to have a distribution is the correspondence due to the deformation is a set of user-defined corresponding points, where the vector and is the strain vector arising from the elastic material model. and is the stress This is equivalent to the formulation "! $ # "! where represent feature vectors derived from the template and subject anatomies and is a normalized cross-correlation. The last term represents the deformation energy penalty, which imposes a smoothness constraint on the deformation. An iterative finite element method was used to solve the model in order to determine the 16

17 deformation field. The implementation used allows for the global affine map to be optimised, and for the elastic material parameters to be varied, as the iteration proceeds. An investigation of the use of other features (apart from boundaries) was carried out to determine the effectiveness of different features for elastic matching (Gee et al., 1994). Curvature, edges and classified data were used as the feature in separate matching experiments. It was found that the different features did not generate significantly different deformation fields, and that there was little dependence upon the precise value of the elasticity parameters. Bayesian Formulation with Stochastic PDE A stochastic partial differential equation (PDE) elastic matching formulation for projecting a model of normal anatomy onto PET, CT and MRI scans was developed (Miller et al., 1993) as an evolution of work with simple deformable template models for biological shape recognition. A physical model of visco-elastic solids was used to formulate the matching as a regularization problem: with where is an atlas, is the subject anatomy, is the deformation field mapping from. Local similarity is measured by the sum of the squares of the difference in intensity between the atlas and the subject over a region, and the regularization term comes from a physical model of visco-elastic material. The atlas anatomy was constructed by hand segmenting slices of MRI scans to match the appearance indicated by a textbook of neuroanatomy. The 2D template anatomy was then 17

18 projected onto one slice from each of 3 patients. The template anatomy was observed to overlap the patient anatomy with near perfect alignment. In (Christensen et al., 1994; Christensen, 1994) another Bayesian formulation of the matching problem was formulated. This made use of both an elastic and a viscous fluid model as regularization terms. A parametric model of the deformation field was presented, defining the deformation vector field as a sum of basis vectors and allowing a coarse to fine solution method in which more terms are added to the deformation field as the resolution is increased. A model for the likelihood, where is 3D image data and are the parameters defining the vector field, was assumed to have a Gibb s form where and are the anatomical atlas and subject acquired with the same imaging modality. The constraint on the deformation vector field was imposed by assuming the prior obeys properties of elastic solids and has a Gibb s form with potential energy with being elasticity constants and a Lagrange multiplier. It was observed that such an elasticity constraint prevents the atlas from deforming without limit, since a penalty is imposed for being increasingly different from the atlas anatomy. A method was described to allow the restoring forces of the elastic material to decline over time the viscous fluid spatial transformation (VFST). The VFST satisfies the equation, 18

19 where is the instantaneous velocity of the deformation field and is the body force per unit volume. Samples of the posterior are generated using a stochastic differential equation where is a vector-valued Wiener process and is the Gibb s potential of the posterior. The sample mean converges to the conditional mean of the posterior. The VFST PDE is discretized and solved using successive overrelaxation (Christensen, 1994). This is a Bayesian formulation and a special case of this stochastic differential equation formulation, when there is no Wiener process, generates MAP estimates. In (Gee et al., 1994) a Bayesian formulation with MAP estimation of the deformation field is presented. In a validation experiment, an atlas anatomy slice and a slice from an MRI scan of a patient were matched. First an approximation to the deformation field was found by solving the elastic solid model with a multiresolution approach, involving adding more and more basis terms to the deformation field approximation as the iteration proceeds. The estimated deformation field was then used as an initial condition for the solution of the VFST. The quality of the elastic match was measured by recording the ratio of the area of the intersection of particular segmented structures to the area of the union of the structures. Typical results were 0.81 for the thalamus and 0.63 for the ventricles. A 3D deformation was carried out to match a patient volume to an atlas volume, but in order to achieve reasonable computation times, the patient volume was subsampled from its original resolution of 256x256x21 voxels to 64x64x21. Qualitative comparison of matched atlas and patient scan indicated a close match. Haller et al. (Haller et al., 1997) have recently investigated its application to the segmentation of the hippocampus and their work demonstrates that these techniques are quite close to a 19

20 robust, automatic segmentation of cortical structures. The shape of the surface of the hippocampus, and its variability, has been investigated by studying the variation of the deformation field (Joshi et al., 1997). This technique is motivated by the attempt to find a formulation that does not require a small deformation assumption. Successful matching of large deformations using this technique has been achieved (Christensen, 1994). A consequence of the use of this technique is that in projecting the model anatomy across long distances, the model anatomy becomes less significant in defining valid matches (the energy term representing a penalty for deformation away from the model is relaxed over time). The match is then driven solely by intensity differences, which may not always be sufficient to define anatomically meaningful boundaries. The implementation of this approach through solving a stochastic PDE required a long time (almost 10 hours for a 128x128x148 voxel 3D match) to compute matches with this model, even on a MASPAR massively parallel supercomputer (Christensen, 1994). Bro-Nielsen has demonstrated a method for fast viscous fluid registration (Bro-Nielsen and Gramkow, 1996) that achieves similar timing on a workstation. The approach was to use a linear filter to allow solution of the linear elasticity PDE faster than is possible using successive over-relaxation. II.A.3 Summary Table 1 summarizes techniques for volumetric elastic matching techniques described in the previous section. Our review is not intended to be comprehensive, and should be seen as an overview of the family of techniques for volumetric elastic matching. The technique that we have developed for fast volumetric elastic matching, based on the Dynamic Pyramid algorithm of Dengler (Dengler and Schmidt, 1988), is described in more detail in Section III.D. The major areas of similarity of these techniques are the assumptions that regions to be 20

21 Authors Bajcsy et al. (Bajcsy and Kova ci c, 1989) Local similarity measure normalized cross-correlation Features for matching intensity and edges features Constraint elasticity Method of solution multiresolution Jacobi iterations Authors Gee et al. (Gee et al., 1994) Local similarity measure normalized cross-correlation Features for matching intensity and edges, curvature, tissue class Constraint elasticity Method of solution iterative solver for finite element formulation Authors Dengler and Schmidt (Dengler and Schmidt, 1988) Local similarity measure optical flow model Features for matching sign of Laplacian pyramid Constraint elasticity Method of solution multiresolution pyramid with FEM discretization solved by nested multigrid with conjugate gradient relaxation Authors Christensen et al. (Miller et al., 1993; Christensen et al., 1994) Local similarity measure sum of squares of intensity differences Features for matching pixel intensity Constraint elasticity and viscosity Method of solution stochastic PDE evaluated Authors Collins (Collins et al., 1992; Collins, 1994) Local similarity measure normalized cross-correlation Features for matching edges Constraint maximum deformation limited Method of solution iterative maximization of normalized cross-correlation Table 1: Overview of automatic volumetric elastic matching techniques. matched have similar intensities, and that smooth deformations are desirable. Differences between these techniques are primarily related to the precise choice of regularization constraint and method of solution. The requirement to have similar intensities can be removed by first classifying the data, and then matching on the segmentation (we currently compute matches between classified data (Iosifescu et al., 1997)). As an alternative approach, we speculate that a similarity metric based on mutual information, similar to that used successfully for rigid registration (Wells et al., 21

22 1996b), may allow direct multimodality matching. Intra-operative nonlinear registration during surgery may require the computation of deformation fields incorporating discontinuities. Experiments with 2D sequence matching has shown such deformation fields can be computed with a controlled continuity deformation field (Dengler and Schmidt, 1988). Further investigation will be necessary to make this computation tractable for 3D data. Our elastic matching scheme assumes only small scale deformations. It is possible to compute matches when larger deformations are present by iterated small scale deformation matching, or through the algorithm developed by Christensen (Christensen, 1994). III Template Driven Segmentation A major component of our work has been to use nonlinear registration as an integrated component of a group of image analysis techniques. Using an image analysis scheme based on cooperative processes, a more robust behaviour can be obtained than when using any single process alone. This strategy allows following processes to use the results of earlier processing and also to correct for errors that earlier processes make. Our concept is to not require perfect accuracy from the elastic matching scheme since it can form a part of a pipeline of cooperative processes. In our work, we aim at designing a set of general purpose, robust, tuneable image analysis modules. Then, systems specialized for particular tasks are constructed through the design of pipelines using these modules and feedback mechanisms. Our integrated systems approach for segmentation is illustrated in Figure 1. The modules are feature detection, classification, and registration components. Our work on template-driven segmentation has concentrated on the following components: 1. generation of a digital brain atlas that is used as the template, 22

23 Images Feature Identification Classification Segmentation Model Alignment Figure 1: General configuration for template-driven segmentation. 2. development of linear registration algorithms for initial alignment of atlas and subject data set, 3. implementation of a fast elastic warping algorithm for nonlinear registration, and 4. development of a system that integrates all of these modules A key characteristic of the systems approach illustrated by Figure 1 is the use of feedback to refine the operation of each module. Often image analysis problems are approached with a single algorithm, or sequential application of a group of algorithms. We wish to solve problems where such techniques have so far been unsuccessful, and our approach is to build iterative systems where the solution is refined at each iteration. Robustness is increased by using the results of a previous step to improve results in next step. For example, when registration of an atlas to a patient is desired, initial alignment and nonlinear registration matching is based on linear registration. Following iterations of matching can be based on better nonlinear registration. Feature detection can use local structure analysis with feature enhancement techniques related to the structures expected to be present from the model. Classification can be spatially varying, for example, through the use of a spatially varying a priori probability for certain structures, propagated through the matched atlas. The segmentation of certain types of pathology is difficult to achieve directly from the 23

24 Figure 2: Surface rendering of volumetric digital atlas, showing some anatomical structures and visualization controls. matching of a model of normal anatomy. When an atlas of normal anatomy is used, the atlas does not contain examples of pathology such as lesions. Lesion detection can be built into a classification process in a way that takes advantage of the spatial information from the model. An example of such a process is given in Section IV.B.3. III.A 3D Digital Atlas of the Brain A high-resolution T1 weighted MR data set of x x 1.5 mm resolution, obtained from a prospectively studied normal volunteer, has been converted into a high-quality digital anatomic atlas. This was achieved by first identifying the brain and then using a semiautomatic procedure to segment gray and white matter. Two postdoctoral fellows have manually edited the data set by subdividing gray matter and white matter into over 120 label classes. Figure 2 shows a 3D surface rendering from this atlas (Kikinis et al., 1996). 24

25 III.B Feature Detection and Classification Robustness, automation, and speed are important characteristics of successful classification strategies. Classification of medical images is often successful when applied directly to the data, without any feature detection process. We have developed a number of classification strategies that have been successfully used in automated image analysis pipelines (Cline et al., 1990; Wells et al., 1996a; Warfield, 1996). Feature detection strategies can help with the analysis of newer types of imaging data, such as diffusion MR, and with the enhancement of images where structures of a particular shape are expected to be found. For example, in the analysis of blood vessels from magnetic resonance angiography (MRA), or the segmentation of cartilage from MRI of the knee. III.C Linear Registration We want to restrict our nonlinear registration to capture variability that is not significantly different from the normal template, so that it is clear that aligning to the template is reasonable. In order to reduce the differences between patient and template we first carry out linear registration. We have identified some applications where particular linear registration techniques are successful. Intra-patient registration For intra-patient and multiple modality registration we have developed a very successful rigid registration algorithm based on the maximization of mutual information (MI) (Wells et al., 1996b; Hata et al., 1996). Inter-patient registration For inter-patient registration we have investigated the use of surfacematching methods. We have used surface matching successfully in a number of intra- and interpatient registration applications including MRI to skin surface alignment for image guided 25

26 surgery (Grimson et al., 1996), intra-patient registration for MS lesion change detection (Ettinger et al., 1994) and atlas to patient registration as a precursor to nonlinear registration (Iosifescu et al., 1997). III.D Nonlinear Registration For elastic matching, we are using an approach that is similar in concept to the work reported by Bajcsy and Kova ci c (Bajcsy and Kova ci c, 1989), and Collins and Evans (Collins, 1994). However, our implementation uses several different algorithmic improvements to speed up the processing including a multiresolution approach with fast local similarity measurement, and a simplified regularization model for the elastic membrane (Dengler and Schmidt, 1988). Our matcher, which is implemented in C, is based on essentially the same algorithm as that implemented by Dengler in APL, with a few improvements and modifications. III.D.1 The algorithm The goal of the matcher is that given a source and target data set and, respectively (where ), one would like to find a vector field such that the function is as similar to the function as possible. Ideally should satisfy the condition that the map is one-to-one, but due to computational difficulties and the limited benefit that such a condition could provide to matching two data sets which contain qualitative differences (for example, those coming from different patients) and so for which a reasonable exact one-to-one match is unachievable, it is convenient to relax this condition. Our basic method of computing is via template matching : for a fixed value of, consider the problem of finding the value of (here considered as a variable, not a function) that minimizes 26

27 The resulting value of is taken as the value of. Here, is a window function whose width determines the size of the region used to compute. In order for this procedure to be valid, must be essentially constant over this region. An initial linear registration of the source and target data sets, together with an iterative multiresolution approach to computing, will ensure that this expression need only be evaluated for small. In this case we can make the approximation leading to the expression The value of minimizing this expression is given by the solution of where and If this is used to compute for each, then the width of determines how fast can vary. The obvious problem is that choosing a fixed function for computing everywhere is unreasonable. On the one hand, a wide would be needed to allow to be influenced by values of and at distant values of. On the other hand, a narrow is needed to allow 27

28 to vary rapidly where this is appropriate. A solution is to allow the function to depend on. This is done by first computing matrices and each of which corresponds to one of some small number of successively wider window functions, where for some. In our implementation, these have the property that is roughly twice as wide as for each, and the maximum width is comparable to the minimum axis length for any given resolution level. Thus is smaller for lower resolution levels. One then uses some set of weights (where each ) to write and as a linear combination of the s and s, respectively: For reasons given below, the weight function used is (in fact, turns out to be for each, so this makes sense if for at least one ). The resulting matrices and are used to compute via the formula above at the point in question. Note that this procedure is equivalent to computing using the values of and computed with a window function which depends on through the. Before computing, the value of is computed to see if it exceeds some positive threshold. If not, one instead just sets. The threshold used in our implementation is, where and are the mean and standard deviation, respectively, of over the entire data set. The need to compute and and hence the threshold is why it s necessary to normalize so that : the solution of the equation 28

29 at any given point where is not affected since the normalization constant multiplies both sides of the equation. For those values of where for all, one avoids dividing by zero by simply setting, resulting in and hence. To explain the use of as a weighting function, Dengler noted that this nonnegative number is a good measure of the topological uniqueness of the patch of involved in the computation of - in other words, the extent to which the patch would be altered by translations. One way to see this is to note that the expression for forms what is known as a Gramm determinant (Gel fand, 1961). Let be the vector whose components are the values of at the points where is nonzero, for. These vectors live in a vector space, where is the number of voxels in the patch in question. The vectors generate a subspace of dimension, and the square of the (signed) volume generated by these vectors in this subspace can be shown to be equal to. Clearly, this determinant tends to be large when the patch of in question contains large gradients in a variety of directions, which is precisely what is required. III.D.2 Implementation Our implementation uses a multiresolution approach which computes incrementally so that the approximation of small which is needed to derive the equation can be satisfied at each resolution. A match function implements the computation of at a given resolution level as described above, except that in order to keep as small as possible, what it actually computes is a vector field which minimizes the overall difference between and, instead of and. What is actually needed as output of the matcher is still a vector field which does the latter. Note that for uniformly small the two conditions are 29

30 approximately the same, so a perturbation approach is appropriate for reconciling them. Since this is required anyway in the multiresolution computation of, nothing is lost by implementing the function this way. When calculating the displacement field, a multiresolution pyramid is used. The maximum number of levels can be selected by the user at runtime. The top level is determined by the additional requirement that for each axis, the width of any of the window functions used at that level is no more than of the width of the image. At the top level, the deformation field is estimated. This is then interpolated down to the next lower level. This procedure is repeated until the bottom level is reached. At each level, the window functions used have the property that for each, along each axis, is twice as wide as, and the maximum value of is determined by the requirement that the width of along each axis may be no more than of the width of the image. IV Results The basic operation of our nonlinear registration algorithm is illustrated with some examples of it s performance on synthetic data. These examples can help to understand and reason about the performance of the matcher on clinical data. Some of the applications where the technique has been successfully applied are then briefly described. All timing results reported in this section were computed on a single 167Mhz UltraSPARC CPU workstation with 256MB of RAM from Sun Microsystems. Reported times are the average of ten repetitions of each experiment. 30

31 IV.A Examples With Synthetic Data Figure 3 illustrates the matching of a target square with a copy of the target shifted by ten columns. Each square is 107x107 pixels in size with an intensity of 255, in an image of 256x256 pixels, with a background of intensity 0. A window function of size 13x13 pixels was used to define the local search region. The time required to compute the match was 4.0 seconds, and the time to transform the shifted square according to this transform was 0.2 seconds. The transformed square perfectly matches the target. Figure 4 shows an example of recovering a nonlinear deformation. A deformation was applied to an image of 256x256 pixels of four intensity levels as shown (63, 127, 191, 255 from inside to outside). The deformed image was then matched to the original image, using a window function size of 9x9 pixels. The matching operation was iterated three times, and the iterations are shown in Figure 4. The total time required (for the 3 matching and 3 transform operations) was 9.5 seconds. The matched atlas shown in Figure 4(e) almost perfectly matches the target. The boundaries of the matched atlas are slightly noisy because sampling artifacts perturb the initial atlas boundaries and because the window function width used to estimate the deformation is much larger than the deformation noise at the boundaries. IV.B IV.B.1 Examples With Medical Data Illustration of Brain Matching In order to match our atlas to a patient, we first compute a linear registration (3 rotation, 3 scale and 3 translation parameters), and resample the atlas into the patient coordinate system. We then compute an elastic match between the linearly registered atlas and patient scan, and then resample the atlas to match the patient. A typical match result is illustrated in 2D in Figure 5. The images are 256x256 pixels, and the time to compute the matched atlas was 3.4 seconds. It can be seen that before the match the atlas and patient are most different in the cortex, ventricles, 31

32 (a) Target square (b) Atlas, target square shifted left ten columns (c) Magnitude of atlas minus target (d) Atlas matched to target (e) Magnitude of matched atlas minus target Figure 3: Matching a displaced square to a square required 4.2 seconds. The overlap of the matched square to the target square is 100%. 32

33 (a) Target (b) Atlas (c) Atlas matched to target, iteration 1 (d) Atlas matched to target, iteration 2 (e) Atlas matched to target, iteration 3 Figure 4: Example of matching a nonlinearly deformed square to a square. The image (e) is the result of three iterations of matching the atlas image (b) to the target image (a), which required 9.5 seconds to compute. This demonstrates the accuracy with which longer range deformations can be captured. 33

34 cerebellum and brain stem. After matching the remaining differences are minor differences at the boundaries of structures. It is particularly striking how well the shape differences of the cortex, ventricles, cerebellum and brain stem have been captured. IV.B.2 Comparison of Subcortical Structures of Normal and Schizophrenic Volunteers from MRI We have applied a template-driven segmentation approach to analyze the volumes of subcortical structures from MRI. In particular, the use of manual outlining has demonstrated that there is a difference between the size of particular subcortical structures in normal volunteers and in schizophrenic patients. Since manual outlining is very time consuming, we were interested to determine if the same result could be obtained through the used of template-driven segmentation. We found that our automatic segmentation technique predicted the same differences between normal and schizophrenic volunteers as time consuming manual outlining by experts (Iosifescu et al., 1997). We applied an automated analysis to a series of 28 MR scans, from 14 schizophrenia patients and 14 normal controls. First, we used an automated classification program to differentiate between white matter, cortical and subcortical grey matter, and cerebrospinal fluid (Wells et al., 1996a). Next, we nonlinearly registered our volumetric digital atlas to fit each subject s brain using the method described in Section III.D, by matching the white matter and subcortical grey matter segmentations of the atlas to the classified patient scans. We segmented the patient scan by projecting the position of the subcortical structures in each matched atlas onto the patient scan. To assess the accuracy of these measurements, we compared 11 brain structures segmented by elastic matching, with the same structures outlined manually. The similarity between the measurements (the relative difference between the manual and 34

35 (a) Patient (b) Atlas (c) Magnitude of atlas minus patient (d) Atlas matched to patient (e) Magnitude of matched atlas minus patient Figure 5: Illustration of typical elastic match performance, which required 3.4 seconds to compute. After matching the remaining differences are minor differences at the boundaries of structures. It is particularly striking how well the shape differences of the cortex, ventricles, cerebellum and brain stem have been captured. 35

36 CSE MRI Classification Model Alignment Re-classification Segmentation Output Figure 6: Template-driven segmentation of white matter lesions from conventional spin echo (CSE) MRI the automated volume) was 97% for whole white matter, 92% for whole gray matter, and on average 89% for subcortical structures. The relative spatial overlap between the manual and the automated volumes was 97% for whole white matter, 92% for whole gray matter, and on average 75% for subcortical structures. In the schizophrenia group, compared to the controls, we found a 16.7% increase in MRI volume for the basal ganglia (i.e. caudate nucleus, putamen and globus pallidus), but no difference in total gray/white matter volume, or in thalamic MR volume. This finding reproduces previously reported results, obtained in the same patient population with manually drawn structures, and suggests the utility and efficacy of our automated template-driven segmentation algorithm over more labor intensive manual outlining. IV.B.3 Segmentation of MR Scans of Patients with Multiple Sclerosis In conventional spin echo imaging white matter lesions, such as occur in multiple sclerosis, appear as regions of increased signal intensity. However, the segmentation of these scans is difficult because the MR intensity range of white matter lesions overlaps that of normal tissue (particularly, of grey matter). Intensity based statistical classification techniques misclassify some lesions as grey matter and some grey matter as lesion. We developed a template-based segmentation technique to segment the grey matter structures of the brain, and to identify a mask of the white matter region. Healthy white matter and white matter lesions can then be segmented from the region of white matter without interference from grey matter (Warfield et al., 1995). The approach, illustrated in Figure 6, is essentially a two iteration version of the general scheme indicated in Figure 1. 36

37 A single slice from a 3D segmentation is shown in Figure 7. This patient scan consists of 256x256x54 voxels and nonlinear registration of the atlas to a patient at the resolution of 128x128x128 required 347 seconds. Figure 7(d) shows a region of the slice after the volume was classified with the Expectation-Maximization segmentation algorithm (Wells et al., 1996a). The classification shows, in order of increasing intensity, background, CSF, grey matter, lesion and white matter classes. This algorithm corrects for intensity inhomogeneity variation in the MRI at the same time as classifying the voxels. Since the MR intensity of lesions overlaps that of grey matter, some lesion is misclassified as grey matter and some grey matter is misclassified as lesion. The matched atlas, shown in Figure 7(e), has been used to identify the region of both healthy and diseased white matter. Figure 7(f) shows the improved segmentation of diseased white matter, healthy white matter and grey matter, after the white matter region was reclassified as either healthy or diseased white matter (Warfield et al., 1995). IV.B.4 Brain Tumour Segmentation In Figure 8, the results from two different segmentations of a data set from a patient with a large glioblastoma multiforme are shown. The left image shows the result obtained using a combination of classification (Section III.B) and operator controlled segmentation. Approximately one hour of work was required for this result. The right image in Figure 8 shows the result obtained from our template driven segmentation algorithm, which required about 5 minutes of operator interaction. Nonlinear registration was used to project a model of normal anatomy onto the patient scan, and then region growing limited to the region indicated by the matched atlas, was used to extract the tumour. The results of the segmentations are very similar. On a voxel to voxel comparison the intersection of the segmentations is over 85%. Most of the difference in the segmentations is due to over-smoothing by the operator in the manual segmentation. 37

38 (a) Early echo (b) Late echo (c) Template-driven Segmentation (d) Region of EM Segmentation (e) Region of matched atlas labelled to show only white matter, grey matter and ventricles (f) Region of template driven segmentation Figure 7: Template driven segmentation of CSE MRI of a patient with multiple sclerosis. Image (d) shows the segmentation result of classification without anatomical context from the matched atlas. Image (c) and (f) show the improved segmentation of diseased white matter, healthy white matter and grey matter using template driven segmentation. 38

39 (a) Manual (b) Automatic Figure 8: Manual and automatic extraction of a tumour from an SPGR brain scan. On a voxel to voxel comparison the segmentations overlap by over 85%. V Discussion Nonlinear registration is becoming an increasingly important tool for the analysis of medical imaging data. Powerful nonlinear registration techniques that capture and describe shape differences have become useful as parts of a template driven segmentation strategy. Future developments will be to improve nonlinear registration techniques to cope with a variety of more complex shape differences. We are investigating the application of our automatic volumetric elastic matching method to the parcellation of neocortical grey matter regions. The normal anatomical variability of the neocortex, and the lack of a signal intensity difference in conventional MR between different functionally significant structures makes this a challenging problem. Current methods rely on significant sulcal landmarks. We believe nonlinear registration will become a particularly useful for image-guided surgery. Most nonlinear registration schemes are built on the assumption that it is desirable to find a de- 39