Fusing Markov Random Fields with Anatomical Knowledge and Shape Based Analysis to Segment Multiple Sclerosis White Matter Lesions in Magnetic Resonance Images of the Brain Stephan Al-Zubi a, c, Klaus Toennies a, c, Nils Bodammer b, d, Hermann Hinrichs b, e a Institute for Simulation and Graphics, b Department of Neurology II Otto-von-Guericke University of Magdeburg c {stephan, klaus}@isg.cs.uni-magdeburg.de, d bodammer@neuro.med.uni-magdeburg.de e hermann.hinrichs@medizin.uni-magdeburg.de ABSTRACT This paper proposes an image analysis system to segment multiple sclerosis lesions of magnetic resonance (MR) brain volumes consisting of 3 mm thick slices using three channels (images showing T-, T- and PD -weighted contrast). The method uses the statistical model of Markov Random Fields (MRF) both at low and high levels. The neighborhood system used in this MRF is defined in three types: () Voxel to voxel: a low-level heterogeneous neighborhood system is used to restore noisy images. () Voxel to segment: a fuzzy atlas, which indicates the probability distribution of each tissue type in the brain, is registered elastically with the MRF. It is used by the MRF as a-priori knowledge to correct miss-classified voxels. (3) Segment to segment: Remaining lesion candidates are processed by a feature based classifier that looks at unary and neighborhood information to eliminate more false positives. An expert s manual segmentation was compared with the algorithm. Keywords: Markov Random Field, Registration, Multiple Sclerosis, Classification, Shape based analysis.. INTRODUCTION Multiple Sclerosis is a disease of the central nervous system. It appears as the myelin sheathes protecting nerve axons break down causing plaques. As a result 90% - 95% of lesions occur within white matter tissue. Certain parts of the nervous system are more affected by these MS-plaques: The anterior angles of the lateral ventricles, the corpus callosum and periventriclar areas. Lesion sizes vary considerably in size from a few millimeters to lesions involving the entire Centrum semiovale. MSplaques in white matter are ellipsoidal in shape. Their intensity is characterized by a bright center with decreasing intensity as we move away from the center. Figure. Left: example of a slice showing PD contrast. Middle: lesion mask obtained by intensity based classification alone and the actual lesion mask. Right: Manual segmentation. As seen much grey matter is falsely classified as lesions. Segmentation based on intensity alone will not succeed because the intensity histograms of grey matter and lesions overlap in MRI images (See Figure ). This means that any classifier based on voxel intensity alone will classify some gray matter as lesions and lesions as gray matter. This effect is evident where partial volume effect exists (e.g. where cerebrospinal fluid (CSF) and grey matter are
averaged). This makes it necessary to employ higher level knowledge about distribution of grey/white matter in the brain and lesion features. There are a number of different approaches that employ different models for segmentation. Some researchers proposed matching a brain volume with an anatomical atlas to create grey/white mask constraints used to label lesions correctly. Warfield proposes segmenting the cortex by region growing and constraining its boundary by an elastically registered anatomical atlas. This is used to make the white matter mask containing lesions and white matter that can be separated by a simple intensity based classifier. In Warfield 4 a statistical k-nn classifier for gray levels is used to classify tissues based on a set of manually selected prototypes. A spatial mask obtained by elastic matching adds spatial features to the k-nn classifier resolving the ambiguity of overlapping histograms. The elastic model is re-aligned with the volume using the improved segmentation. The algorithm iteratively applies k-nn with elastic matching until convergence. Similar work can be found for segmenting tumors in Kaus et al. 3 and Kamber et al. 5. They propose the use of a brain tissue probability model. This model gives the probability distribution of each tissue type within a standardized Talairach space. A set of segmented brain volumes were registered into Talairach space and averaged together to construct the model. This model was used as a mask to confine search within white matter and as a way to provide geometric features used to classify lesions (e.g. probability of ventricular CSF is useful in detecting periventriclar lesions). Another way to classify lesions is using a feature space collected from possible candidate lesions to sort out false positives. Ardizzone 6 uses the fuzzy c-means algorithm by first obtaining a set of oversegmented regions followed by a re-clustering phase. The re-clustering uses shape and intensity features to label or split unknown clusters. After this phase three masks corresponding to WM, WM+GM and WM+GM+CSF are built. The holes in these masks correspond to candidate lesions. A feature vector for each candidate is passed through a neural classifier to decide whether it s a lesion or not. Those features include: Contact with WM and GM and CSF, mean intensity, shape measures like compactness and elongation and position of the lesions like the distance from the ventricular area. Fuzzy connectedness 7 is used to detect connected fuzzy objects representing white, grey and CSF objects. The holes remaining represent locations of potential lesions. False positives are eliminated by operator assistance. Fuzzy connectedness is defined by a measure that computes a connected component such that the strength of connection between any two pixels in the segment is above a certain threshold. The strength of connection between pixels is determined by the distance and intensity homogeneity. Johnson 8 applies Iterated Conditional Mode (ICM) algorithm used to segment noisy images with morphological operators. This method consists of initially segmenting the volume by ICM using all labels. The segmentation is post processed to get the white matter + lesion mask. Partial volume results in classifying the areas around lesions as grey matter. These gaps in the mask are closed by binary morphology. The resulting mask is processed again with ICM but this time classifying only two tissue types. Small lesions are eliminated from the final output. Kirshnan 9 uses the ratio image of PD/T to eliminate magnetic field inhomogeneity. He then applies a series of thresholds to extract the WM, CSF mask. The lesions are identified by intensity thresholding as the brightest spots and then applying those masks to remove false lesions outside white matter. To summarize, many authors aim to find the white matter mask containing the lesions and then segmenting the lesions afterwards. Some apply registration with an anatomical model to determine where grey matter is. Others apply feature based classifiers to eliminate false positive. The method represented in this paper combines alignment with an anatomical model, feature based classification and image restoration using Markov Random Fields. This paper is structured as follows: section () introduces the Markov Random Fields which will be used to restore noisy data at a voxel level and later to eliminate false positives for candidate lesion segments. Section (3) will describe the process pipeline to classify MS lesions and surveys some literature pertaining to each step. Section (4) shows the experimental results followed by discussion of future directions.
. MARKOV RANDOM FIELDS (MRF) Let Y= {y i,k,m i N x, j N y, k N z, m M} be a multi-modal input image of dimension N x N y N z and M channels. Let X= {x i,k,m : i N x, j N y, k N z } be the corresponding segmentation of Y with each x i,k,m taking a value form a set L={, l} of labels. Let x Ω be a specific segmentation of X. Ω is the space of all segmentations. Let x be an instance of X and similarly y an instance of Y. The objective is to maximize the probability of P(X Y). Given Bayes Theorem P(X Y) P(Y X)P(X) we get x * = arg max P( y x) P( x) () x Ω The term P(y x) in () is modeled as a multivariate Gaussian additive noise which is independent of location. Let µ p, p L be the vector representing the mean intensity values for a label (p) and Σ p be the corresponding covariance matrix. Then we can define P(y x) as l ( ) ( ) Σ x exp y i k x x ( i k x ) i j k, j, µ Σ y, i, j, k i, j, k, j, i, j, k P( x) = µ, i, k y π () The advantage of using the multi-variate normal () is that it can better compensate for correlated tissue variations due to partial volume effect 6. P(x) in () is the regularization term in which each voxel interacts with its neighbors to restore noisy images by inferring the correct value from neighbors. The homogeneous multi-level logistic (MLL) distribution will be used to define P(x). Specifically, for each voxel a first order neighborhood system consisting of its six adjacent voxels is defined: { x, x x } η (3) x =,,,,, i, k i± j k i j± k i, j, k± A potential function is defined on cliques of one voxel and cliques of adjacent voxel pairs, specifically, given two adjacent sites u, u : β u = u V ( u, u ) = (4) β otherwise And for each label type (k) we define an a-priori expectation α k and so that a single site potential is Using (4) and (5), P(x) is defined as V ( x i, k ) α x i, k Z is the normalizing constant called the partition function. = (5) exp V ( u, u ) + V ( u) adjacent u u u x (, ) P(x ) = (6) Z The local optimization method of iterated conditional modes ICM by Besag 0 works by iteratively updating one site at a time in raster scanning fashion until label changes per image iteration falls below a threshold. In the case of optimizing a single site (x i,k =u) we only have to maximize () for a single site which simplifies as the minimization of: U ( ) = y i, k µ u Σu ( y i, k µ u ) + ln( Σu ) V ( u, u ) α u (7) ( u) u η u
3. METHOD The algorithm pipeline goes through the following steps (See Figure ): () Initial gray level segmentation. () Heterogeneous clique computation for the MRF. (3) Magnetic field inhomogeneity correction. (4) Image restoration by the ICM algorithm. (5) Registration with a tissue probability distribution atlas. (6) Using the aligned atlas to correct misclassified voxels. (7) Eliminating more false positive lesions by applying an MRF classifier on lesion segments. It looks at shape features of those segments and their neighborhood. The following are the individual steps in detail: initial T, T, PD data () initial gray level segmentation () compute heterogeneous clique weights (3) magnetic field correction (4) ICM (6) apply atlas on areas marked as lesions and regions that surround them using ICM (5) register atlas with segmented data (7) apply shape based ICM to eliminate false positives (8) add resulting segmentation as an instance to the atlas 3.. Initial Grey Level Segmentation Figure. Algorithm pipeline The first step in the pipeline is to perform use a multivariate Gaussian grey level classifier where the sample mean and standard deviation are acquired by manual segmentations of some brain slices (see Figure ). 3.. Compute Heterogeneous Clique Weights The interaction between neighbors can be further improved if we include edge information into P(x). The voxel-voxel weight β in V (see eq. 4) is redefined as a local field {β u,v }between voxels u and v to better adjust to likely edges by a function similar to eq. (3): β β u, v = (8) + ( u ) / λ Where u is the direction between u and v. and y σ is the gradient of a Gaussian smoothed image. y This method represents a low cost approximation to the heterogeneous field which enables a better behavior for the ICM algorithm (See Figure 3). Many authors include edge information into the Markov Random Field, 3, 4. Equation (8) is an adaptation of anisotropic diffusion filters found in Weickert 5. 3.3. Intensity Inhomogeneity Correction The image Y is intensity corrected for magnetic field inhomogeneities by using the white matter mask found in step () and modeling the inhomogeneity as a multiplicative field (y corrected (i,k)=y i,k *m i,k ). A grid of 3D points is defined where at each point a local estimate of the mean gray value of white matter is calculated from a minimum neighborhood size of Q pixels. The multiplicative field is defined as the ratio between the local and the global average (m i,k =µ global / µ local (i,k)). Bilinear interpolation provides the estimation for the remaining voxels. This approach is similar to the one used in Rajapakse et al.. σ
Figure 3. Heterogeneous MRF neighborhood weights in X, Y, Z directions 3.4. Applying the ICM Algorithm to Restore Miss-classified Voxels The image is restored using the method described in section (). 3.5. Registering Atlas with Segmented Data X In this paper the atlas P atlas ={ p i,k (l) } is a probability distribution of segments where p i,k (l) represents the probability that the voxel i,k has a label l. The transformation (T =T affine +T B-Spline ) is used to warp the atlas to the input image y or its segmentation x where: T affine: Represents the global extrinsic transform of 9 parameters defined by the affine space. T B-Spline : B-Splines were used in to model elastic intrinsic deformation because of their: () built in smoothness () locality of control that enables fast registration (3) a small parameter space that includes only the xyz distances between control points (for more details see Rueckert et al. 7 and Studholme et al. 9 ). The normalized entropy measure 8 {H(A,B)=(H(A)+H(B))/H(A,B)} enables the registration between x and y directly because of its ability to compare images of different modalities. A better similarity measure between P atlas and x is defined by S( P atlas, x) = pi, k ( xi, k ) (9) n i, k It is possible to build an atlas by averaging several brain volumes by registration to build a probability distribution model for example see Chen et al. 0 and Studhome et al. 9. In this paper an initial atlas was obtained by segmenting a simulated volume in brainweb then blurring the distribution by a Gaussian kernel. The blurring improves registration by minimizing chances of local minima in the fitting process. As more segmented images are obtained we can refine the atlas by registering the segmentations with P atlas and adding the new instance to form a new distribution. n+ npatlas + x P atlas = (0) n + The atlas has the ability to include the a-priori probability distributions of lesions in the brain where pi,k(l lesion ) represents the expectancy of a lesion at location i,k. Initially we set p to some threshold (αpi,k(l white-matter )) which gets refined as more segmentation instances are added. 3.6. Lesion Detection Using the registered atlas in section 3.5, we can differentiate between true and false lesions. This is done by reapplying ICM at segments labeled as lesions and their neighboring voxels with an additional term in equation (7) using P atlas. n
Figure 4. Improvement after applying the anatomical atlas. Left: the lesions of Figure. Middle: lesions reclassified by applying the anatomical atlas. Right: the expert s manual segmentation. As we see the atlas was able to eliminate a great deal of false positives. The remaining false positives are handled by shape based MRF. ( ) U ( u) = y i, k µ u Σu ( y i, k µ u ) + ln( Σu ) V ( u, u ) u + ln( pi, k ( u)) α γ () This has the effect of adding special information to each voxel resulting in a better classification (see Figure 4). Registration errors may still misclassify certain voxels therefore feature based classification is needed to improve results. 3.7. Applying Shape Based MRF False positives resulting from section 3.6 are further eliminated by defining individual candidate lesions as shape units in a Markov Random Field. Each shape unit can be reassigned a state {MS lesion, grey matter} based on shape and neighborhood features. An iterative process similar to ICM is used to re-label those units until convergence. The following is the description of the steps in detail:. The watershed transform is used to isolate most individual plaques in each XY slice using the property that generally the intensity is bright at the center and decreases towards the edges. The connected components thus formed within white matter generally have an elliptical shape (unless they contact other structures like the ventricular system). Per slice connected components are used instead of 3D connected components because the Z aspect ratio of (3:) makes calculation of good shape features in 3D difficult (i.e. Lesions generally do not persist more than -4 slices and partial volume makes them stick together in the Z direction).. The shape units calculated in step () are considered random variables (sites) which may be relabeled as {lesion, grey matter}: S= {s s k }, L S : S {l grey_matter, l lesion } 3. Unary features are computed for each shape unit that measure () compactness () elongation (3) distance from the center of ventricular area (φ unary : S v unary ). 4. A neighborhood system (η S z : S S ) is defined for each shape unit where η S z (s i ) are all other shape units that have contact with s i in the Z direction. Similarly we define η S xy as the neighborhood that specifies contact in the XY direction. Binary features are computed for each shape unit that includes: () the area of contact with gray matter () the area of contact with other shape units in the neighborhood system {η S z, η S xy } (φ binary : S v binary ). 5. A binary classifier C: S η S z η S xy (P lesion, P grey_matter ) is defined. It assigns higher probability that a shape unit is a lesion whenever it is more oval or contacts more other lesions or has a certain location. It assigns it a higher probability that it is grey matter when it contacts more grey matter or has a certain location. u η u
Figure 5. Left: Lesion mask after applying the anatomical atlas. Middle: Lesion mask after applying shape based MRF. Right: Experts manual segmentation 6. Using the neighborhood system and shape features, an iterated conditional mode algorithm is defined where each site is assigned a state of lesion or gray matter based the classifier C. The Algorithm is iterated a few times until convergence (see Figure 5). The algorithm uses contact information between shape units to mimic what the expert does. For example, when the expert finds an isolated oval region in white matter and looking at a the slice before he discovers that this structure is connected to cortical gray matter so it must be a continuation of cortical matter, however, if it contacts another lesion then it is likely to be a continuation of that lesion structure. Another example are elongated structures embedded in gray matter are less likely to be lesions but classification errors. 4. RESULTS Experiments were conducted on three brain volumes with T, T, and PD weighted contrast. Each modality is of dimension 56 56 48 where the voxel size is 0.97 0.97 3.0 mm. Manual segmentations of the multiple-sclerosis lesions for the three volumes were obtained by a medical expert. Two measures were used to compare the manual and automatic segmentation:. Similarity index 8 : The similarity index between two segments A, A is a number between [0, ] defined by S = A I A A + A (). Automatic and Manual segmentation volumes which show the quantity of false positives eliminated after each stage of the algorithm. Figure 6 shows similarity index for a typical MS patient after each stage of the algorithm. The ICM algorithm restored the image and yielded a 5% improvement in similarity index for the whole volume. Using the atlas yields a 3% improvement in similarity. Using the shape based ICM yields a 3% improvement in similarity especially after slice -6 as shown in the figure. Figure 7 shows automatic versus the manual segmentation volumes for the same patient. Initially the overall correlation between manual and intensity based segmentation is 0.3. The correlation became 0.87 after applying the atlas because most false positives were eliminated at this stage. After applying the shape based ICM the correlation became 0.95 as can be seen between the closely matching curves. Experiments with the other two patients yielded a per-slice-correlation of 0.93 and 0.9 between the volumes of manual and automatic segmentation.
Intensity based Segmentation ICM segmentation Atlas segmentation Shape ICM 0.9 0.8 0.7 0.6 Similarity Index 0.5 0.4 0.3 0. 0. 0-0 -9-8 -7-6 -5-4 -3 - - -0-9 -8-7 -6-5 -4-3 - - 0 3 4 Slice Number Figure 6. Similarity Index for after each stage of the algorithm pipeline for one patient volume 3500 Manual Segmentation Intensity based Segmentation ICM segementation Atlas segmentation Shape ICM 3000 500 Volume on MS lesions 000 500 000 500 0-0 -9-8 -7-6 -5-4 -3 - - -0-9 -8-7 -6-5 -4-3 - - 0 3 4 Slice Number Figure 7. Manual segmentation volume vs. Automatic segmentation volume for each stage of the algorithm. 5. CONCLUSION AND FUTURE WORK A method to segment multiple sclerosis was presented. It employs a three stage algorithm which first segments and restores the image at low level and then an anatomical atlas is used to disambiguate between lesions and gray matter. A shape based MRF in which shape units consisting of lesion slices and a neighborhood system representing contact with other lesions and gray matter is used to eliminate false positives. Future work will concentrate on refining the shape based classifier. For example: () defining a spatial probability map for contact information. () Improving the feature space to include more lesion properties (3) Improving of classification rules to be more close to that of the expert.
6. REFERENCES. C. Poster, An Atlas of Multiple Sclerosis, The Parthenon Publishing Group, 988.. S. Warfield, J. Dengler, J. Zaers, R. Guttman, W. Wells, G. Ettinger, J. Hiller, R. Kikinis, Automatic identification of grey matter structures from MRI to improve the segmentation of white matter lesions. J Image Guid Surg, pp. 36-338, 996. 3. M. Kaus, S. Warfield, F. Jolesz, R. Kikinis, Adaptive Template Moderated Brain Tumor Segmentation in MRI. Bildverbeitung für die Medizin, 0-06, 999. 4. S. Warfield, M. Kaus, F. Jolesz, R. Kikinis, Adaptive, Template Mpderated, Spatially varying Statistical Classification, Medical Image Analysis 4(), pp. 43-55, 000. 5. M. Kamber, R. Shinghal, L. Collins, G. Francis, A. Evans, Model-Based 3D Segmentation of Multiple Sclerosis lesions in Magnetic Resonance Brain Images, IEEE Trans. On medical Images 4(3), pp. 44-453, 995. 6. E. Ardizzone, R. Pirrone, An Architecture for the Recognition and Classification of Multiple Sclerosis Lesions in MR Images, IDAMAP, 999. 7. J. Udupa, L. Wei, S. Samarasekera, Y. Miki, M. Vanbuchem, R. Grossman, Multiple sclerosis lesion quantification using fuzzy- connectedness principles, IEEE Trans. on medical imaging 6(), pp. 598-609, 996. 8. B. Johnston, M. Atkins, B. Mackiewich, M. Anderson, Segmentation of Multiple Sclerosis Lesions in Intensity Corrected Multispectral MRI, IEEE Trans. On Medical Imaging 5(), 996. 9. K. Kirshnan, M. Atkins, Segmenttion of Multiple Sclerosis in MRI- An Image Analysis Approach, SPIE Conf. On Image Processing, pp. 06-6, San Diego, 998. 0. J. Besag, On the statistical analysis of dirty pictures, J. Roy. Statist. Soc. B 48(3), pp. 59-30, 986.. S. Lakshmanan, D. Haluk, Simultaneous Parameter Estimation and Segmentation of Gibbs Random Fields Using Simulated Annealing, IEEE Trans. On Pattern Analysis and Machine Intelligence (8), pp 799-83, 989.. S. Geman, D. Geman, Stochastic Relaxtation, Gibbs Distributions, and the Bayesian Restoration of Images, IEEE Trans. On Pattern Analysis and Machine Intelligence 6(6), 984. 3. S. Nadabar, A. Jain, Parameter Estimation in Markov Random Field Contextual Models Using Geometric Models of Objects, IEEE Trans. On Pattern Analysis and machine Intelligence 8(3), 996. 4. R. Aykroyd, Bayesian Estimation for Homogeneous and Inhomogeneous Gaussian Random Fields, IEEE Trans. On Pattern Analysis and machine Intelligence 0(5), 998. 5. J. Weickert, Anisotropic Diffusion in Image Processing, B.G Teubner Stuttgart, 998. 6. M. Desco, J. Gispert, S. Reig, A. Santos, J. Pascau, N. Malpica, P. Garcia-Barreno, Statistical Segmentation of multidimensional brain datasets, SPIE Medical Imaging, pp 84-93, 00. 7. D. Rueckert, L. Sonoda, C. Hayes, D. Hill, M. Leach, D. Hawks, Nonrigid Registration Using Free-Form Deformations: Application to Breast MR Images, IEEE Trans. on Medical Imaging 8(8), pp 7-7, 999. 8. C. Studholme, D. Hill, D. Hawks, An overlap invariant entropy measure of 3D medical image alignment, Pattern Recognition 3, pp 7-86, 999. 9. C. Studholme, V. Cardenas, M. Weiner, Multi scale image and multi scale deformation of brain anatomy for building average brain atlases, SPIE medical Imaging, pp 557-568, 00. 0. M. Chen, T. Kanade, D. Pomerleau, J. Schneider, Probabilistic Registration of 3-D Medical Images, tech. report CMU-RI-TR-99-6, Robotics Institute, Carnegie Mellon University, 999.. J. Rajapakse, J. giedd, J. Rapoport, Statistical Approach to Segmentation of Single-Channel Cerebral MR Images, IEEE Trans. on medical imaging 6( ), pp. 76-86, 997.. Brainweb http://www.bic.mni.mcgill.ca/brainweb/ 3. C.A. Cocosco, V. Kollokian, R.K.-S. Kwan, A.C. Evans : "BrainWeb: Online Interface to a 3D MRI Simulated Brain Database", NeuroImage 5(4), part /4, S45, 997 -- Proceedings of 3-rd International Conference on Functional Mapping of the Human Brain, Copenhagen, 997.