CLASSIC: Consistent Longitudinal Alignment and Segmentation for Serial Image Computing

Transcription

1 NeuroImage 30 (2006) CLASSIC: Consistent Longitudinal Alignment and Segmentation for Serial Image Computing Zhong Xue,* Dinggang Shen, and Christos Davatzikos Section of Biomedical Image Analysis, Department of Radiology, University of Pennsylvania, PA 19104, USA Received 7 February 2005; revised 22 September 2005; accepted 30 September 2005 Available online 4 November 2005 This paper proposes a temporally consistent and spatially adaptive longitudinal MR brain image segmentation algorithm, referred to as CLASSIC, which aims at obtaining accurate measurements of rates of change of regional and global brain volumes from serial MR images. The algorithm incorporates image-adaptive clustering, spatiotemporal smoothness constraints, and image warping to jointly segment a series of 3-D MR brain images of the same subject that might be undergoing changes due to development, aging, or disease. Morphological changes, such as growth or atrophy, are also estimated as part of the algorithm. Experimental results on simulated and real longitudinal MR brain images show both segmentation accuracy and longitudinal consistency. D 2005 Elsevier Inc. All rights reserved. Keywords: Longitudinal brain image analysis; Image segmentation; Fuzzy clustering; Brain atrophy; Brain growth; Serial scans; Volumetry Introduction * Corresponding author. addresses: zhong.xue@uphs.upenn.edu (Z. Xue), dinggang.shen@uphs.upenn.edu (D. Shen), christos.davatzikos@uphs.upenn.edu (C. Davatzikos). Available online on ScienceDirect ( MR brain image segmentation is a key processing step in many brain image analysis applications, such as morphometry, automatic tissue labeling, tissue volume quantification, image registration, and computer-integrated surgery (Bezdek et al., 1993; Pappas, 1992; Udupa and Samarasekera, 1996; Brandt et al., 1994; Lim and Prefferbaum, 1989; Pham and Prince, 1999; Chen and Giger, 2004; Rezaee et al., 2000). Analysis of a series of 3-D data of the same subject captured at different times or of a 4-D image is important in many neuroimaging studies that concentrate on normal development and aging, as well as on evolution of pathology (Resnick et al., 2000; Tang et al., 2001; Freeborough and Fox, 1997). In these studies, longitudinal stability is critical when measuring subject changes with time since true signal is overwhelmed by measurement error. However, existing 3-D segmentation algorithms may not be able to provide adequate longitudinal stability since they process each 3-D image separately. Herein, we propose a method that overcomes this limitation and significantly improves longitudinal stability or temporal consistency of segmentation by formulating the segmentation problem in 4-D. Consistent segmentation of serial MR brain images is particularly important in the literature of aging and Alzheimer s disease (AD) since subtle brain changes that might be indicative of early stages of underlying pathology must be estimated from image series. It is also a challenge when the presence of vascular or other pathologies changes signal characteristics, such as tissue contrast, thereby rendering tissue segmentation unreliable. In 3-D segmentation, fuzzy algorithms (Bezdek et al., 1993; Pappas, 1992; Udupa and Samarasekera, 1996; Brandt et al., 1994; Lim and Prefferbaum, 1989; Bezdek et al., 1984) have been proven to be more suitable for MR images than other hard segmentation algorithms since the intensity of each voxel of an MR image may represent a combination of different tissues. Fuzzy C-Means (FCM) algorithms have been used in many segmentation applications, often accounting for intensity inhomogeneity (Pham and Prince, 1999; Chen and Giger, 2004; Guillemaud and Brady, 1998; Ahmed et al., 2002) and incorporating spatial information among voxels (Rezaee et al., 2000; Liew et al., 2000; Pham, 2001). The intensity of inhomogeneity can be well modeled by the product of the original image and a gain field (Ahmed et al., 2002) or by the summation of them (Pham and Prince, 1999). It is also desirable that the clustering algorithm be spatially adaptive to relatively local image intensity variations in order to smoothly and adaptively segment tissues of structures. Different algorithms have been proposed to incorporate the spatial image context information. They include the methods using smoothness constraints of the spatially varying centroid function (Rezaee et al., 2000), using intensity dissimilarities of neighboring voxels (Liew et al., 2000) and using smoothing operations of the fuzzy membership functions (called Robust FCM algorithm (RFCM)) (Pham, 2001). By combining Pham and Prince (1999) and Pham (2001), Pham and Prince proposed a Fuzzy And Noise Tolerant Adaptive Segmentation Method (FANTASM), which is robust to the effects of both intensity inhomogeneities and noise while providing a soft /$ - see front matter D 2005 Elsevier Inc. All rights reserved. doi: /j.neuroimage YNIMG-03529; No. of pages: 12; 4C:

2 Z. Xue et al. / NeuroImage 30 (2006) segmentation [ Compared to other methods, the advantage of FANTASM is that it uses the intermediate information of the segmentation results while compensating the gain field and performing smoothness on the membership functions. However, in FANTASM, the spatial smoothness constraints of fuzzy membership functions are the same for all the locations in an image, which can result in an overmoothing effect of the fuzzy membership functions across tissue boundaries. More importantly, FANTASM and other segmentation methods are designed for application on a 3-D image at a time, and they might yield inconsistent results when applied to a series of scans of the same subject, thereby rendering estimations of rates of brain atrophy and growth noisy. In this paper, we propose a novel algorithm for longitudinal MR brain image segmentation based on FANTASM, which we refer to as CLASSIC (Consistent Longitudinal Alignment and Segmentation for Serial Image Computing). CLASSIC not only jointly segments a series of longitudinal 3-D MR brain images of the same individual, but also estimates the longitudinal deformations in the image series, e.g. tissue atrophy. It iteratively performs two steps: (i) it jointly segments a series of 3-D images using a 4-D image-adaptive clustering algorithm based on the current estimate of the longitudinal deformations in the image series, and (ii) it then refines these longitudinal deformations using a 4-D elastic warping algorithm (Shen and Davatzikos, 2002, 2004). In this way, we obtain both a longitudinally consistent segmentation and an estimate of longitudinal deformation of anatomy (e.g. atrophy) in a series of 3-D images. The 4-D image-adaptive clustering algorithm used in CLASSIC extends the RFCM and FANTASM algorithms in three aspects. First, a new temporal consistency constraint term on the fuzzy membership functions is used in order to obtain temporally consistent segmentation results. Second, the spatial and temporal constraints of the fuzzy membership functions are made adaptive to the smoothness of the image, that is, they are stronger in the regions that have more uniform image intensities and vice versa. Thus, the fuzzy membership functions are not necessarily overly smooth across tissue boundaries. Third, the clustering centers at each voxel location are adaptive to relatively local image intensity variations. In this way, the proposed 4-D clustering algorithm not only provides temporally consistent segmentation results, but also adapts to local image intensity variations. Experiments are performed to segment-simulated and real longitudinal MR brain images. Both quantitative and visual comparisons are performed to compare the performance of CLASSIC and FANTASM. The longitudinal 3-D T1-SPGR MR images of healthy older adults from the Baltimore Longitudinal Study of Aging (BLSA) (Resnick et al., 2000) are used in the experiments, which provide a very challenging data set from older adults that displays both brain atrophy and changes of tissue contrast due to vascular and possibly other pathologies. In all the experiments, we focused on evaluating the performance of the algorithms in terms of obtaining temporally consistent segmentation, capturing global and local intensity/contrast changes, as well as estimating longitudinal deformations. The results demonstrate that CLASSIC gives consistent segmentation results across different years and adapts to image intensity variations. The remainder of the paper is organized as follows. In Methods, after briefly introducing the FCM, RFCM, and FANTASM algorithms, upon which CLASSIC builds, we introduce the framework of CLASSIC and the 4-D image-adaptive clustering algorithm in detail. Results gives the experimental results, and Summary and discussion provides a summary and discussion from this study. Methods Previous algorithms In this section, the previous algorithms including FCM, RFCM, and FANTASM are briefly introduced, which act as the foundation of the proposed 4-D image-adaptive clustering algorithm used in CLASSIC. The standard FCM (Bezdek et al., 1984) is formulated as finding the fuzzy membership functions l and the centroids c by minimizing the objective function, E FCM ðl; cþ ¼ K i a X k ¼ 1 l q i;kð x i c k Þ 2 : ð1þ Here, X is the set of voxel locations of a 3-D image, i is the index of a voxel in X, x i is the intensity of voxel i, c k is the centroid of class k, and K is the number of classes. l i,k is the fuzzy membership function, reflecting the degree of voxel i belonging to the kth class. q is a weighting exponent greater than 1. It is clear that Eq. (1) does not reflect any spatial relationships among image voxels. Thus, the standard FCM is generally not able to yield smooth segmentation results for MR brain images. The RFCM algorithm (Pham, 2001) uses an additional spatial constraint term of the fuzzy membership functions in order to yield spatially smooth segmentation results. Its objective function is E RFCM ðl; cþ ¼ K i a X k ¼ 1 K i a X k ¼ 1 l q i;kð x i c k Þ 2 þ a l q i;k 5; ð2þ l q j;m j a N ðþv s m a M k i where N i V represents a spatial neighborhood of voxel i (not include voxel i), and M k ={m Z {1,2,...,K}, m m k. The second term is minimized when the membership function for a particular class at location i is large and the membership functions for the other classes of the neighboring voxels are small and vice versa. It is a penalty term for smoothing the membership functions, thus the RFCM yields spatially smoother segmentation of an input image. However, in RFCM, the spatial smoothness constraints of the fuzzy membership function are the same for all the voxel locations, which might smooth fuzzy membership functions across tissue boundaries. By combining the smoothness constraints of fuzzy membership functions of Eq. (2) with the spatially adaptive gain field estimation proposed in the Adaptive FCM algorithm (Pham and Prince, 1999), a Fuzzy And Noise Tolerant Adaptive Segmentation Method (FANTASM) was proposed by Pham and Prince. FANTASM is robust to the effects of both intensity inhomogeneities and noise while providing a soft segmentation [ fantasm/]. However, this 3-D segmentation algorithm may not be able to provide adequate longitudinal stability, when applied to a series of MR images from the same individual. Furthermore, the ability of FANTASM to adapt to intensity variations aims at removing an MR gain field, but it is not optimal for local image intensity variations caused by anatomical boundaries. Based on RFCM and FANTASM, this paper proposes a 4-D image-adaptive clustering algorithm, which is then used under the 3

3 390 Z. Xue et al. / NeuroImage 30 (2006) CLASSIC framework. The 4-D clustering algorithm extends the previous methods in the following three aspects. First, a new temporal constraint term on the fuzzy membership functions is used in order to obtain temporally consistent segmentation results. Second, the spatial and temporal constraints of the fuzzy membership functions are made adaptive to the smoothness of the image. Third, the clustering centers at each voxel location are adaptive to local image intensity variations. In this way, the 4-D image clustering algorithm not only provides temporally consistent segmentation results, but also fully adapts to local image intensity variations. The framework of CLASSIC A 4-D MR brain image in the context of this work is a series of 3-D MR brain images obtained from the same subject at different time. Because the brain changes that we might be interested in can be extremely small, for example in early stages of Alzheimer s disease (AD), temporally inconsistent segmentation can significantly reduce the statistical power of longitudinal neuroimaging studies aiming at determining early structural changes as markers of AD. In this paper, we focus on jointly segmenting a 4-D image and follow the underlying temporal changes in anatomical structures in order to provide more stable and consistent tissue segmentation across different years. The idea behind CLASSIC is to classify the tissue of each voxel according to the intensities around that voxel plus incorporating image-adaptive spatiotemporal constraints at that location. For this purpose, point correspondences across the temporal domain have to be obtained in order to apply temporal constraints. Therefore, a 4-D image segmentation framework, referred to as CLASSIC, is proposed, which iteratively performs the following two steps: (i) given a current estimate of the longitudinal deformations necessary to align 3-D images, the first step is to jointly segment the image series using a 4-D imageadaptive clustering algorithm, and (ii) the second step is to refine the longitudinal deformations using a 4-D high-dimensional image warping algorithm (Shen and Davatzikos, 2004). The framework of CLASSIC can be summarized in Fig. 1. The pre-processing of the input image series includes: correct global intensity inhomogeneity (Pham and Prince, 1999) and globally normalize the intensities of each image according to the histogram of the first image (Nyul et al., 2000); transfer the subsequent images onto the space of the first image using rigid transformations. After pre-processing, CLASSIC is applied to consistently segment the rigidly aligned image series I t, t Z T ={t 1, t 2,...,t Y }, with initial longitudinal deformations from the first image I t1 to other images I t as F t1yt, t = t 2, t 3,...,t Y, and Y being the total number of the serial images. If no initial longitudinal deformations among the images are available, F t1yt (i) = 0, where i refers to a voxel location in the first image I t1, that is, initially, there is no deformation at all. The standard FCM algorithm is also performed on each image I t to give initial values of the clustering centroids. Then, CLASSIC iteratively performs the following two steps: (1) Apply the proposed 4-D image-adaptive clustering algorithm to the image series I t, t Z T, based on the current estimate of the longitudinal deformations F t1yt, t = t 2, t 3,...,t Y, and obtain the segmented images I t (seg), t Z T (this clustering algorithm will be described in The 4-D image-adaptive clustering algorithm in detail). (2) Use the 4-D volumetric registration algorithm, 4-D HAMMER (Shen and Davatzikos, 2004), to register the Fig. 1. The framework of CLASSIC. segmented images I (seg) t, t ={t 2, t 3,...,t Y } onto the reference image series formed by repeating the first segmented image (seg) I t1 for Y 1 times, i.e. I (seg) t1, I (seg) t1,...,i (seg) t1. After performing 4-D HAMMER, the longitudinal deformations F t1yt, t = t 2, t 3,...,t Y are refined according to the 4-D segmentation results of step (1). Go to step (1) if the amount of deformation changes between two iterations is larger than a prescribed threshold; otherwise, terminate with I (seg) t and F t1yt as the final segmentation results and the final estimate of the longitudinal deformations, respectively. Instead of registering images individually, in the 4-D registration algorithm (Shen and Davatzikos, 2004), a series of segmented images are registered to the first image simultaneously. Therefore, longitudinal stability of deformations among serial images is achieved using temporal smoothness constraints in 4-D warping algorithm. These two steps are performed iteratively so that consistent segmentation results can be obtained, and, in practice, a few iterations are enough to obtain stable segmentation results. It is worth noting that CLASSIC uses this iterative procedure because the longitudinal deformations in 3-D image series are unknown, otherwise, the proposed 4-D image-adaptive clustering algorithm can be applied directly to segment the images. Starting from an initial estimate of the longitudinal deformations if it were available, the iterative procedure of CLASSIC is used in order to refine the longitudinal deformations based on 4-D segmentation results. That is, the 4-D clustering algorithm yields spatially adaptive and temporally consistent segmentation results based on the current longitudinal deformations; the 4-D HAMMER then takes the segmented images as inputs and refines the temporally smooth deformations among those images. Therefore, two parts of the longitudinal changes are captured by CLASSIC. The longitudinal deformations reflect mainly volume/shape changes of structures (e.g. atrophy), while the adaptive clustering parameters reflect spatially and longitudinally intensity variations. From the experimental results, we found that this iterative procedure improves the

4 Z. Xue et al. / NeuroImage 30 (2006) segmentation results and also captures the longitudinal deformation well. In The 4-D image-adaptive clustering algorithm, we describe the 4-D image-adaptive clustering algorithm used in step (1) of CLASSIC in detail. The 4-D image-adaptive clustering algorithm Algorithm formulation Given image series I t, t Z T and the longitudinal deformations F t1yt, t = t 2,...,t Y, the purpose of the 4-D segmentation is to calculate the segmented images I t (seg), t Z T. Since F t1yt is the deformation from I t1 to I t, the corresponding point of voxel i of image I t1 will be F t1yt (i) in image I t. For simplicity, we denote point F t1yt (i) in image I t as (t,i), and x (t,i) as its intensity. According to the 4-D image-adaptive clustering algorithm, x (t,i) (t Z T, i Z X) is classified into different tissue types by finding c (t,i),k, the kth clustering center at location (t,i), and l (t,i),k, the fuzzy membership function of x (t,i) belonging to class k and by minimizing the objective function in Eq. (3), which includes three terms. The first term is the weighted squared error among the intensities around each voxel and the clustering centroids. It extends the first term of Eq. (2) to 4-D and uses different clustering centroids for different voxels in order to adapt to local image intensity variations; the second term in the following equation stands for the spatially adaptive smoothness constraints, and it is similar to the corresponding second term of Eq. (2); the third term represents the temporally adaptive smoothness constraints defined in the same manner. Eðl; cþ ¼ t a T i a X þ a 2 ( 1 K t a T i a X þ b 2 where l ðþ s ðt;iþ;k ¼ 1 N 1 ¼ 1 N 2 SN t;i t a T i a X l q ðt;iþ;k x ðs;j k ¼ 1 ðs;jþa N ðt;iþ q ðþ s ðt;iþ k k ¼ 1 h q ðþ t ðt;iþ K k ¼ 1 l q ðt;jþ;m ðt;jþa N ðþv s m a M k ðt;iþ ðs;i l q ðs;iþ;m : Þa N ðþv t m a M k ðt;iþ l q ðt;i h l q ðt;i h i ) 2 Þ c ðt;iþ;k Þ;k l ðþ s ðt;iþ;k ; and l t Þ;k l ðþ t ðt;iþ;k ðþ ðt;iþ;k i i ; The fuzzy membership functions are subject to K l ðt;iþ;k ¼ 1; for all iax; t at: ð4þ k ¼ 1 In Eq. (3), N (t,i) is the spatiotemporal neighborhood of point (t,i), which is a combination of its spatial neighborhood N (t,i), and its temporal neighborhood N (t) (t,i) ={(s,i): s t T N }, thus N (t,i) = N (t) (t,i) N (t,i). S(N (t,i) ) represents the number of voxels within N (t,i). In the first term of Eq. (3), the centroids c (t,i),k are adaptively changed at different image locations based on local image intensity variations within the spatiotemporal neighborhood N (t,i) of each location. The second term of Eq. (3) reflects the spatial constraints of the fuzzy membership functions, which is analogous to the second term of Eq. (2). The difference is that, in Eq. (3), an additional factor q (t,i) is used as an image-adaptive weighting coefficient. It will be detailed in the next paragraph. According to this term, stronger smoothness constraints are applied to the fuzzy membership functions in the ð3þ image regions that have more uniform intensities and vice versa. The third term reflects the temporal consistency constraints. Similar to q (t) (t,i),q (t,i) is a weighting coefficient that reflects the temporal (t) q smoothness of the image. l (t,i),k and l (t,i),k are the means of l (s,l),k in the spatial and temporal neighborhoods N V (t)v (t,i) and N (t,i) of the V (t)v current position (t,i), respectively. Notice that N (t,i) and N (t,i) do not include the point (t,i), and their sizes are different from those of N (t,i) and N (t) (t,i). a and b are the weighting coefficients, and N 1 and N 2 are the numbers of addends for normalization. q (t,i) is the spatial smoothness factor of image I t at voxel (t,i).the value of q (t,i) is close to 1 when the image around voxel (t,i) is spatially smooth and close to 0 when the image around voxel (t,i) is not spatially smooth. Using a spatial difference operator D r along each of the three spatial axes r, q (t,i) can be defined as ( q s t;i ¼ exp X ) h i 2 ðd r TI t Þðt;iÞ =2r2 s ; ð5þ r where (D r * I t ) (t,i) refers to first calculating the spatial convolution (D r * I t ) and then taking its value at voxel location (t,i). (t) q (t,i) is the temporal smoothness factor at location (t,i). The (t) value of q (t,i) is close to 1 when the image around voxel (t,i) is temporally smooth and close to 0 when the image around voxel (t,i)is not temporally smooth. n q t o 2 t;i ¼ exp D t Tx ðt;iþ t =2r2 t ; ð6þ where D t refers to the temporal difference operator, and (D t * x (t,i) )t refers to first calculating the temporal convolution (D t * x (t,i) ) and then taking its value at time t (location (t,i)). The size of the spatiotemporal neighborhood N (t,i) at each location (t,i) can also be adjusted adaptively. Therefore, different neighborhood sizes of different spatiotemporal locations are used. A small size will make the algorithm much adaptive to local image intensity variations, while a large size has to be used to capture adequate intensity information of different tissues around a voxel in order to label that voxel correctly. Note that no spatiotemporal smoothness constraints on c were used in Eq. (3). This is because we have found that if the sizes of the spatial neighborhoods N (t,i) are large enough to capture adequate intensity information of different tissues around voxels and these neighborhood sizes change smoothly across different voxel locations, the spatial and temporal constraints of l in Eq. (3) are adequate for yielding smoothly varying centroids. Finding the solutions of the 4-D clustering algorithm Using Lagrange multipliers to enforce the constraint in Eq. (4), the new objective function is written as follows J ¼ Eðl; cþþ t a T i a X k ðt;iþ 1 K k ¼ 1 l ðt;iþ;k : ð7þ Setting the partial derivative of Eq. (7) with respect to l (t,i),k to zero and using Eq. (4), we get the equation to update the fuzzy membership functions l ðt;iþk ¼ ð x ðt;iþ Þ C ðs;jþ;k Þ R 2 ðs;jþa N ðt;iþ S N ðs;jþ K k ¼ 1 R ðs;jþa N t;i ð ð Þ Þ C ðs;jþ;k Þ 2 x ðt;iþ ð Þ S N ðs;jþ ð Þ þ aq ðþ s ðt;i Þ l ðþ s ðt;iþ;k þ bq ðþ t ðt;i Þ l t ð ðþ t;iþ;k þ aq ðþ s ðt;i Þ l ðþ s ðt;iþ;k þ bq ðþ t ðt;i Þ l t ð 1 q 1 ðþ t;iþ;k 1 q 1 : ð8þ

5 392 Z. Xue et al. / NeuroImage 30 (2006) Since different spatiotemporal neighborhood sizes are used for different image locations, in Eq. (8), N (t,i) ={(s, j): (t,i) Z N (s,j) }. Setting the partial derivative of Eq. (7) with respect to c (t,i),k to zero, the equation to update the centroids can be acquired, c ðt;i Þ;k ¼ ðs;j Þa N ðt;i Þ l q ðs;jþ;k x ðs;jþ ðs;jþa Nðt;iÞ l q ðs;jþ;k : ð9þ Given a series of 3-D images and the longitudinal deformations among them, the 4-D image-adaptive clustering algorithm then jointly segments them by iteratively calculating the fuzzy membership functions using Eq. (8) and the centroids using Eq. (9) until convergence. In order to determine adaptively the size of each neighborhood N (t,i), in our algorithm, we first initialize an identical neighborhood size for all the locations and then adaptively adjust these sizes in every iteration: we segment the images using the current fuzzy membership functions and then calculate the Fractional Anisotropy (FA) (Zhu et al., 2004) of point (t,i) within the current neighborhood N (t,i), denoted as a (t,i). Since FA describes difference proportions of three tissue classes, the size of neighborhood N (t,i) is increased if its a (t,i) is greater than a prescribed threshold a high,or is decreased if a (t,i) is smaller than a threshold a low, or remains unchanged if a (t,i) is between the two thresholds. Finally, the neighborhood sizes are smoothed across the image series. The 4-D image-adaptive clustering algorithm can be summarized as follows: (1) Set the parameters a, b, r s, r t, a high, and a low and sizes of neighborhoods N (t,i), N V (t)v (t,i), and N (t,i). (2) Compute fuzzy membership functions using Eq. (8). (3) Compute centroids using Eq. (9). In order to accelerate the calculation speed, we only calculate the centroids on downsampled grid points and linearly interpolate the values at other locations. (4) Classify the input images using the fuzzy membership functions. If the algorithm were converged (the difference of the values of the objective function between two iterations is smaller than a prescribed threshold), then output of the segmentation results, otherwise, update the size of each spatiotemporal neighborhood N (t,i) and back to step (2). Results In this section, two sets of experiments were performed to evaluate the proposed 4-D image segmentation algorithm by segmenting simulated and real longitudinal MR brain images. Fig. 2. An example of the segmentation results for simulated 4-D images with global intensity/contrast decrease (only the images at time t 1, t 5, and t 9 are shown). Top: simulated serial images, middle: FANTASM results, bottom: CLASSIC results.

6 Z. Xue et al. / NeuroImage 30 (2006) Fig. 3. An example of the segmentation results for simulated 4-D images with local (see the white circle) longitudinal intensity/contrast decrease. Top: simulated serial images, middle: FANTASM results, bottom: CLASSIC results. It can be seen that, for FANTASM, because of the intensity and contrast decrease in the region, the overall centroid for WM becomes lower, which results in over segmentation of WM. On the other hand, since the centroids of CLASSIC are adaptively changed according to local image properties and also satisfy the temporal consistency measure, CLASSIC obtains better results. Quantitative measures on goodness of segmentation were also used to compare the performance of CLASSIC and FANTASM. Segmentation of simulated MR brain images without longitudinal deformation To generate simulated image series with no longitudinal deformations, starting from a 3-D segmented template image, we set the means of cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM) to prescribed intensity values and inserted random spatially correlated noise to the image. We also simulated partial volume effects due to finite voxel size as well as a bias field, as described in Goldszal et al. (1998). The initial values of the means of CSF, GM, and WM of the first image I t1 were set to 25, 85, and 105, respectively. Two sets of longitudinal data were simulated. The first set (see Fig. 2) generated longitudinal images with global intensity/contrast decrease and the second set (see Fig. 3) with local intensity/contrast decrease. Each set includes nine simulated 3-D images, in which the means of GM and WM (m g and m w ) were gradually decreased with time t to simulate an intensity/contrast decrease often seen in older individuals, due to vascular or other pathologies. The rate of decrease of GM intensity is defined as r g =(m g (t Y ) m g (t 1 )) / (m g (t 1 )( Y 1)) and that of WM is defined as r w =(m w (t Y ) m w (t 1 )) / (m w (t 1 )( Y 1)), where Y is the number of longitudinal images simulated. Different combinations of r g and r w yield different simulation results. Given r g, r w, m g (t 1 ), and m w (t 1 ), the decrease rate of contrast can also be determined: r c =(m w (t Y ) m g (t Y ) m w (t 1 )+m g (t 1 )) / ((m w (t 1 ) m g (t 1 ))( Y 1)). Local intensity/contrast decrease was achieved by setting the rates of decrease of GM and WM intensities r g and r w to some prescribed values within a spherical region and setting them to zeros outside that region. Gaussian function was used to smooth the rates of intensity decrease across the boundary of the spherical region in order to obtain smooth simulated images. CLASSIC and FANTASM were then used to segment these simulated images. In all the experiments, the parameters of CLASSIC were set as follows, a = 150, b = 200, r s = 25, r t = 35, a high = 0.3, a low = 0.1, the initial size of spatial neighborhood N (t,i) was set to 35 (radius) and was adaptively adjusted in the (t) V (t)v clustering procedure, and neighborhoods N (t,i), N (t,i) and N (t,i) were set as the immediate spatial or temporal neighborhoods of (t,i) and were fixed in all the calculation. Fig. 2 gives an example Fig. 4. Comparison of CCR for segmenting simulated images (in Fig. 2) with global intensity/contrast decrease. Fig. 5. Comparison of CCR for segmenting simulated images (in Fig. 3) with local intensity/contrast decrease.

7 394 Z. Xue et al. / NeuroImage 30 (2006) Fig. 6. An example of segmenting the simulated longitudinal data with local atrophy and intensity/contrast decrease. Top: the simulated serial images, in which the white circle indicates the spherical area within which atrophy and intensity decrease are simulated, middle: segmentation results of FANTASM, bottom: segmentation results of CLASSIC. of the simulated images with global intensity/contrast decrease, as well as the segmentation results using FANTASM and CLASSIC were r g =0,r w = 0.013, r c = 0.066, and the standard deviation of the noise was set to 4. In this case, the noise is relatively strong, and the contrast between WM and GM is very small at time t 9.It can be seen from the figure that FANTASM failed to segment the image especially at time t 9, while CLASSIC obtains good results for all time points. Fig. 3 illustrates an example of the simulated images with local intensity/contrast decrease and the segmentation results (r g, r w, and r c are the same with Fig. 2, and the standard deviation of the noise was set to 2). A quantitative measure of the Correct Classification Rate (CCR) was used to evaluate the similarity between the segmented images and the ground truth. CCR is defined as the percentage of the number of brain voxels that have been correctly labeled according to the ground truth with respect to the total voxel number of the entire brain. The results of CCR of these two examples are reported in Figs. 4 and 5, respectively. It can be seen that, although the contrast between GM and WM decreased temporally, the CCR of CLASSIC remains consistently high, while the CCR of FANTASM drops rapidly with time. Segmentation of simulated MR brain images with longitudinal deformations In the previous experiments, there are no longitudinal deformations involved in the simulated image series. In order to evaluate the ability of CLASSIC to segment simulated image series with deformations that might occur with atrophy or other pathologies, i.e. to estimate accurately longitudinal deformations while yielding consistent segmentation results, the image series with longitudinal deformations are simulated by combining simulated atrophy and intensity/contrast decrease of GM and WM within a spherical area. The atrophy is simulated by matching the Jacobian of the simulated Fig. 7. Comparison of CCR for segmenting simulated images (in Fig. 6) with local atrophy and GM/WM contrast decrease. Fig. 8. Accuracy to measure volumes of the simulated atrophy from CLASSIC segmentation.

8 Z. Xue et al. / NeuroImage 30 (2006) Fig. 9. Temporal consistency of GM and WM of different subjects. Small values indicate relatively less temporally consistent segmentation. Note that 100% consistency can be ideally obtained only if the brain has not changed at all during this 9-year period, which is not the case in these older adults. deformation to the desired volumetric changes subject to smoothness and topology preserving constraints (Karacali and Davatzikos, submitted for publication). The amount of atrophy can be described by the shrinkage rate, 0 < r s < = 1. For example, r = 0.9 implies a 10% atrophy within the spherical area. The definitions of the rates of decrease of GM and WM, r g, and r w are the same as in the previous experiments. Fig. 6 shows an example of the simulated images and the segmentation results using FANTASM and CLASSIC respectively. The circle reflects the spherical area within which the atrophy and intensity decrease were simulated. Fig. 7 shows the CCR of the whole brain region of the segmentation results in Fig. 6. The shrinkage rate was r s = 0.8, the rates of intensity decrease were r g = 0.006, r w = 0.013, r c = 0.035, and the standard deviation of noise is 2. By comparing the segmentation results of FANTASM and CLASSIC, it can be seen that CLASSIC yields temporally consistent segmentation results, while capturing the longitudinal deformations at the same time. Furthermore, by comparing Fig. 3 and Fig. 6 or Fig. 5 and Fig. 7, we find that the results of FANTASM in Fig. 6 are better than those of Fig. 3. This is because the rates of contrast decrease are different. It can also be seen that CLASSIC adapts to local intensity variations very well. For FANTASM, some larger local intensity variations, e.g. contrast decrease, may affect the segmentation results at other image locations. The reason is that FANTASM models the intensity changes through a very smooth gain field, whereas the 4-D clustering algorithm of CLASSIC is fully adaptive to local image intensity variations. Finally, in order to evaluate the accuracy of CLASSIC to follow longitudinal volume changes of the simulated atrophy, the volumes within the spherical region of atrophy are measured and compared with the ground truth. The results are shown in Fig. 8. It can be seen that the volumes measured from CLASSIC segmentation results follow the ground truth very well (the average volume differences are 1.6% (b = 200) and 2.2% (b = 300) respectively). It is worth noting that, although CLASSIC assumes smoothness in longitudinal deformations (and thus tissue volumes) between subsequent time points, experimental results reveal that the algorithm can tolerate longitudinal changes of anatomical structures, such as normal development and aging, and reasonable levels of atrophy well. For example, it can be seen from Figs. 6 and 7 that CLASSIC is able to capture atrophy, as well as obtain longitudinally consistent segmentation results. Segmentation of real longitudinal MR brain images In this experiment, we used CLASSIC to segment 18 sets of longitudinal MR brain images from the BLSA data (Resnick et al., 2000), which are 3-D T1-SPGR MR images of healthy older adults. The nine serial scans of each subject were obtained during a period of nine consecutive years. In order to quantitatively analyze Fig. 10. Comparison of a typical segmentation result of BLSA serial scans using CLASSIC and FANTASM. The top row shows the original serial scans after rigid alignment, the middle row indicates the segmentation results using FANTASM, and the bottom row gives the results of the CLASSIC. The two images in the right column show the number of label changes L i. It can be seen that CLASSIC gives not only spatially smooth, but also temporally consistent segmentation results. It is worth noting that some atrophy is present between these serial scans, thereby contributing to the label changes on the right.

9 396 Z. Xue et al. / NeuroImage 30 (2006) Fig. 11. An example of segmentation results for a series of three MR brain images. The images are overlays of MRI and segmentation boundaries of the GM and WM. The top row shows the results of FANTASM, and the bottom row shows the results of CLASSIC. the segmentation results, we used a temporal consistency (TC) factor to reflect the temporal consistency of the segmentation results. Suppose x seg (t,i) is the segmentation result (label) of x (t,i), according to the final estimate of the longitudinal deformations F t1yt, the segmentation results of voxel i across different times can be denoted as x seg (t 1, i), x seg (t 2, i),...,x seg (t Y, i). Denote L i as the number of label changes of corresponding voxels across time, then the segmentation of the corresponding voxels is consistent if L i is small, and vice versa. Therefore, the TC of segmentation results is measured by TC ¼ 1=SðXVÞ ð1 L i = ðy 1ÞÞ; ð10þ i a XV where XV is the voxel set of the region of interest for evaluating the results, and S(XV) is the number of voxels in XV. Fig. 9 gives the TCs of GM and WM of the entire brains calculated from the segmentation results of CLASSIC and FANTASM on the 18 image series, respectively. It can be seen that TCs of CLASSIC are much higher than those of FANTASM, which indicates that CLASSIC gives more temporally consistent results. Fig. 10 shows a typical segmentation result of CLASSIC and FANTASM respectively. For comparative purposes, the images shown are aligned images using rigid transformations. The two images on the right column illustrate the number of label changes L i of corresponding voxels projected onto the first image space, where white indicates many label changes and black means no changes across time. Fig. 11 shows another example of the segmentation results, where the number of serial images is three. In order to demonstrate the results clearly, we show the overlay image with both of MRI and the segmentation boundaries of GM and WM. By comparing the segmentation results of CLASSIC and those of FANTASM, we can see that the former yields longitudinally consistent segmentation results while adapting to the longitudinal changes of anatomical structures. Fig. 12. GM and WM volumes of the entire brains. Top: results of FANTASM, bottom: results of CLASSIC.

10 Z. Xue et al. / NeuroImage 30 (2006) Fig. 13. GM and WM volumes in a local sphere placed near the supertemporal lobe. Top: results of FANTASM, bottom: results of CLASSIC. Finally, it is worth noting that, although CLASSIC incorporates temporal smoothness constraints of the fuzzy membership functions, it still maintains longitudinal change information. Moreover, CLASSIC captures these changes in a more stable and smooth way by means of the longitudinal deformations among the serial images and the parameters of the clustering algorithm. For example, Fig. 12 shows the GM and WM volumes of the entire brains calculated from the segmented images of 13 subjects using CLASSIC and FANTASM respectively. Fig. 13 gives similar plots of the volumes calculated within a local spherical region (see the region in first image of Fig. 10). We can see from the figures that the curves are quite smooth. Moreover, the GM and WM volumes calculated from the results of CLASSIC steadily decrease with time, which suggests tissue loss with aging. We also used the boundary shift integral (BSI) (Freeborough and Fox, 1997) to measure the GM volume changes of the BLSA data. BSI provides an efficient way to measure volume changes of different structures derived from registered MRI scans. Since proper registration has to be performed to align the subsequent scans with the baseline scan and the structures of interest have to Fig. 14. Global and local GM volume changes calculated using BSI. Top: FAN-TASM + BSI (application of BSI to FANTASM segmentation results), bottom: CLASSIC + BSI (application of BSI to CLASSIC segmentation results).

11 398 Z. Xue et al. / NeuroImage 30 (2006) be segmented prior to using BSI, in our experiments, for simplicity, only the global and local GM volume changes are measured using BSI, and the volume changes calculated by applying BSI on FANTASM segmentation results and on CLASSIC results are compared. Fig. 14 shows the average and SD of volume changes calculated at different time points for 13 BLSA subjects. It can be seen that the volume changes obtained from CLASSIC + BSI are more stable (with much smaller std values) than those using FANTASM + BSI. Although longitudinal analysis of MR brain images is much more complex than this example, the experiments indicate that CLASSIC is a promising tool for longitudinally consistent segmentation without compromising measurement of longitudinal atrophy. Effect of temporal consistency In CLASSIC, since temporal deformations (e.g. atrophy) are mainly captured by 4-D registration, temporal smoothness of segmentation is assumed for longitudinally corresponding anatomical structures. Notice that the value of longitudinal smoothness constraints factor b in Eq. (3) may also affect the sensitivity of the algorithm to atrophy. However, from experiments, we found out that the performance of the algorithm is very stable with respect to b, hence the result is not sensitive to the temporal consistency coefficient b. For example, we simulated an image series with the same local atrophy as shown in Fig. 6 (shrinkage rate r s = 0.8), but with no longitudinal intensity and contrast changes (r g =0,r w =0,r c = 0), where the standard deviation of the noise was set to 4. By fixing a (a = 150) and setting different values for b, the average CCRs of the segmentation results of the simulated serial images were calculated and shown in Fig. 15. It can be seen from the figure that we obtained similar CCRs for a relatively large range of b. Notice that CLASSIC is not applicable when changes between different time points are large (e.g. a tumor appears in a series of images from some time point). However, we are interested in how CLASSIC tolerates smaller longitudinal changes. In order to quantitatively evaluate this performance, we simulated different levels of atrophy within a spherical region located on the supertemporal lobe area. The center of the spherical region is selected manually, and the radius is set to 27 mm. Each simulation generates a series of three images with different levels of atrophy between two time points, represented by a percentage of the volume change between two time points. Then, CLASSIC is applied to these image series, and the CCR for the second image of each image series is calculated. Fig. 16 shows the results. It can be seen that, although the segmentation accuracy drops along with the increasing of longitudinal atrophy levels between two time points, CLASSIC did quite a good job for smaller longitudinal changes. Fig. 15. The effect of temporal consistency coefficient b. Fig. 16. Sensitivity of CLASSIC to longitudinal changes: evaluation using simulated local atrophy. These results indicate that CLASSIC can be effectively applied to the applications that require to consistently follow smaller longitudinal changes, for example, longitudinal studies and analysis of Alzheimer s disease. Summary and discussion Temporally consistent segmentation of serial MR brain images is particularly important in the literature of aging and Alzheimer s disease (AD) since subtle brain changes that might be indicative of early stages of underlying pathology must be estimated from MR images. However, existing 3-D segmentation algorithms may not be able to provide adequate longitudinal stability since they process each 3-D image separately. Therefore, the purpose of this study was to develop a 4-D image segmentation algorithm for serial MR brain image processing and to improve the longitudinal measurement stability. In the proposed CLASSIC algorithm, the conventional 3-D fuzzy clustering algorithm, FANTASM, was improved by incorporating the temporal consistency constraints and making the clustering adaptive to local image intensity variations. In some way, temporal consistency constraints improve image SNR since noise across different times is uncorrelated. Thus, by jointly segmenting a series of scans, we effectively look for consistent anatomical patterns and remove random noise. Moreover, the algorithm becomes more resistant to errors that might be introduced by changes in tissue contrast. The parameters of CLASSIC were selected according to experience. The radius of the initial spatial neighborhood N (t,i) is set to 35 mm, about 10% of the brain volume, so that there is sufficient information of different tissues within the neighborhood. r s is chosen so that the regions with uniform intensity values yield q (t,i) values close to 1, while the boundary points yield smaller q (t,i) values (i.e. 0.6 or smaller). The selection of r t is similar to that of d t. Other important parameters include a and b, the weighting coefficients of the spatial and temporal constraints in Eq. (3). Since the first term of Eq. (3) acts as the major rule of the segmentation, a and b are chosen so that the summation of the second and the third terms is not greater than the first term, if the value of the objective function was evaluated for a given/known segmentation result. In the experiments, we tried different a and b values and obtained slightly higher temporal consistency (TC) values for the BLSA data with a = 150 and b = 200 over other values and thus fixed them in all the experiments. In fact, we found that CLASSIC tolerates the values of a and b well, and if using a = 50 and b = 50,

12 Z. Xue et al. / NeuroImage 30 (2006) the average TC drop of the WM tissues over the 18 subjects is not more than 1%. An important component of CLASSIC is that it jointly estimates segmentation and longitudinal deformation. Without the latter, it is inappropriate to impose longitudinal constraints since the anatomy, and therefore the tissue labeling, might be changing with time. In the present paper, we have focused on tissue segmentation. However, the fuzzy nature of our algorithm allows us not only to look at volumetric changes, but also to examine shifts in the cluster centers, due to underlying pathology. Ultimately, the interplay of volumetric and signal changes is very important for understanding disease progression. In conclusion, we proposed an algorithm for segmentation of serial MR brain images, which yields spatially adaptive and temporally consistent segmentation results. Moreover, the longitudinal deformations among the image series that reflect the underlying structural changes across time are also estimated simultaneously. Experiments with simulated and real longitudinal MR brain images have confirmed the advantages of CLASSIC over more conventional 3-D segmentation analogous formulation. Acknowledgments This work was supported in part by grants R01AG14971, N01- AG We thank Dr. Dzung Pham and Dr. Jerry Prince from Johns Hopkins University for providing the software of the FANTASM algorithm and Dr. Susan Resnick from NIH for access to the BLSA data. References Ahmed, M., Yamany, S., Mohamed, N., Farag, A., Moriarty, T., A modified fuzzy c-means algorithm for bias field estimation and segmentation of MRI data. IEEE Trans. Med. Imag. 21 (3), Bezdek, J., Ehrlich, R., Full, W., FCM: the fuzzy c-means clustering algorithm. Comput. Geosci. 10, Bezdek, J., Hall, L., Clarke, L., Review of MR image segmentation techniques using pattern recognition. Med. Phys. 20 (4), Brandt, M., Bohan, T., Kranmer, L., Fletcher, J., Estimation of CSF, white and gray matter volumes in hydrocephalic children using fuzzy clustering of MR images. Comput. Med. Imaging Graph. 18, Chen, W., Giger, M., A fuzzy c-mean (FCM) based algorithm for intensity inhomogeneity correction and segmentation of MR images. IEEE International Symposium on Biomedical Imaging (ISBI 2004). IEEE, Arlington, VA, pp Freeborough, P.A., Fox, N.C., The boundary shift integral: an accurate and robust measure of cerebral volume changes from registered repeat MRI. IEEE Trans. Med. Imag. 16 (5), Goldszal, A., Davatzikos, C., Pham, D., Yan, M., Bryan, R., Resnick, S., An image processing system for qualitative and quantitative volumetric analysis of brain images. J. Comput. Assist. Tomogr. 22 (5), Guillemaud, R., Brady, M., Estimating the bias field of MR images. IEEE Trans. Med. Imag. 20 (1), Karacali, B., Davatzikos, C., submitted for publication. Simulation of tissue atrophy using a topology preserving transformation model. IEEE Trans. Med. Imag. Liew, A., Leung, S., Lau, W., Fuzzy image clustering incorporating spatial continuity. IEE Proc.,Vis. Image Signal Process. 147 (2), Lim, K., Prefferbaum, A., Segmentation of MR brain images into cerebrospinal fluid, white and gray matter. J. Comput. Assist. Tomogr. 13, Nyul, G., Udupa, J., Zhang, X., New variants of a method of MRI scale standardization. IEEE Trans. Med. Imag. 19 (2), Pappas, T., An adaptive clustering algorithm for image segmentation. IEEE Trans. Signal Process. 40 (4), Pham, D., Spatial model for fuzzy clustering. Comput. Vis. Image Underst. 84 (2), Pham, D., Prince, J., Adaptive fuzzy segmentation of magnetic resonance images. IEEE Trans. Med. Imag. 18 (9), Resnick, S., Goldszal, A., Davatzikos, C., Golski, S., Kraut, M., Metter, E., Bryan, R., Zonderman, A., One-year age changes in MRI brain volumes in older adults. Cereb. Cortex 10, Rezaee, M., van der Zwet, P., Lelieveldt, B., van der Geest, R., Reiber, J., A multiresolution image segmentation technique based on pyramidal segmentation and fuzzy clustering. IEEE Trans. Image Process. 9 (7), Shen, D., Davatzikos, C., HAMMER: hierarchical attribute matching mechanism for elastic registration. IEEE Trans. Med. Imag. 21 (11), Shen, D., Davatzikos, C., Measuring temporal morphological changes robustly in brain MR images via 4-D template warping. NeuroImage 21 (4), Tang, Y., Whitman, G., Lopez, I., Baloh, R., Brain volume changes on longitudinal magnetic resonance imaging in normal older people. J. Neuroimaging 11 (4), Udupa, J., Samarasekera, S., Fuzzy connectedness and object definition: theory, algorithms and applications in image segmentation. Graph. Models Image Process. 58 (3), Zhu, C., Liu, F., Zhu, L., Jiang, T., Anatomy dependent multicontext fuzzy clustering for separation of brain tissues in mr images. 2nd International Workshop on Medial Imaging and Augmented Reality. China. pp R Aug.