Norbert Schuff Medical Center and UCSF Norbert.schuff@ucsf.edu Medical Imaging Informatics N.Schuff Course # 170.03 Slide 1/67
Objective Learn the principle segmentation techniques Understand the role of segmentation in medical imaging
Overview Definitions Segmentation methods Deterministic/probabilistic Comparisons New trends Role of Segmentation Summary & Conclusion Literature
The Concept Of Segmentation Identify classes (features) that characterize this image! Intensity: Bright - dark Shape: Squares, spheres, triangles Texture: homogeneous speckled Connectivity: Isolated - connected Topology: Closed - open
More On The Concept Of Segmentation: Can you still identify classes in each image?
Segmentation Based On Intensity & Topology Gray matter segmentation topology Intensity
Segmentation Based On Texture & Structure Texture Tensors
Definitions Segmentation is the partitioning of an image into regions that are homogeneous with respect to some characteristic. In medical context: Segmentation is the delineation of anatomical structures and other features of interest, i.e. lesions, tumors, functional units.
Formal Definition If the domain of an image is, then the segmentation problem is to determine classes Z k, whose union represent the entire domain N k 1 Classes are connected by intersections : Z k k Z k Z j ; j;0
More Definitions When the constraint of connectivity is removed, then the determination of classes Z k is termed pixel classification. Determining the total number of classes K can be a challenging problem. In medical imaging, the number of classes is often based on a-priori knowledge of anatomy, e.g. K=4 (gray, white, CSF, other) for brain imaging.
Labeling Labeling is the process of assigning a meaningful designation to each image region Freesurfer Brain Labeling Template This process is often performed separately from segmentation. Generally, computer-automated labeling is desirable Labeling and classes may not necessarily share a one-to-one correspondence
Dimensionality Dimensionality refers to whether the segmentation operates in a 2D or 3D domain. Generally, 2D methods are applied to 2D images and 3D methods to 3D images. In some instances, 2D methods can be applied sequentially to 3D images.
Characteristic and Membership Functions A characteristic function is an indicator whether a pixel at location j belongs to a particular class Z k k j 1 if. element. of. class 0 otherwise This can be generalized to a membership function, which does not have to be binary valued. k 1 0 j 1, for. all. pixels.&. classes N k k j 1, for. all. pixels The characteristic function describes a deterministic segmentation process whereas the membership function describes a probabilistic one.
y Class Y 0.0 0.2 0.4 0.6 0.8 1.0 y 0.0 0.2 0.4 0.6 0.8 1.0 y 0.0 0.2 0.4 0.6 0.8 1.0 Simple Membership Functions Binary Probabilistic b=50 b=25 b=12 0.0 0.2 0.4 0.6 0.8 1.0 Feature x (i.e. intensity x1 0.0 0.2 0.4 0.6 0.8 1.0 x1 0.0 0.2 0.4 0.6 0.8 1.0 x1 sigmoid. function 1 y 1 exp a x * b
Segmentation Has An Important Role Computational diagnostic Surgical planning Database storage/retrieval SEGM Image registration Atlases informatics Image transmission Quantification Partial volume correction Super resolution
Segmentation Methods
0 50 100 150 200 Threshold Method Angiogram showing a right MCA aneurysm Histogram (fictitious) Threshold (T A ) (T B ± ) Dr. Chris Ekong; www.medi-fax.com/atlas/brainaneurysms/case15.htm 0 5 10 15 20 25 30 a.0
Threshold Method Original Segmented
Threshold Method Applied To Brain MRI White matter segmentation Results in major anatomical errors
Threshold: Principle Limitations Works only for segmentation based on intensities/contrasts Robust only for images with global uniformity and high contrast to noise ratios Local variability causes distortions The intrinsic assumption is that the probability of features is uniformly distributed
Region Growing - Edge Detection Seed point Region growing groups pixels or subregions into larger regions. A simple procedure is pixel aggregation, It starts with a seed point and progresses to neighboring pixels that have similar properties. Region growing is better than edge detection in noisy images. Guided by e.g. energy potentials with a penalty on high energy Similarity: Ii I j V i, j z score i, j Edges: V i, r I I erf ( r) I i r r
Region Growing: Principle Limitations Segmentation results dependent on seed selection Local variability dominates the growth process Global features are ignored Generalizations: Unsupervised segmentation (i.e. insensitive to selection of seeds) Exploitation of both local and global variability The intrinsic assumption still is that the probability of features is uniformly distributed
Clustering Can be applied to pre-defined features, e.g. intensities, texture Usually requires transformation of image to a feature space Clustering seeks patterns in the feature space Two commonly used clustering algorithms K-mean Fuzzy C-mean
Definitions: Clustering Clustering is a process for classifying patterns in such a way that the samples within a class Z k are more similar to one another than samples belonging to the other classes Z m, m k; m = 1 K. The number of clusters is usually pre-defined
K-Means The k-means algorithm attempts to cluster n patterns based on attributes (e.g. intensity) into k classes k < n. The objective is to minimize total intra-cluster variance in the least-square sense: K k 1 jsk 2 j k µ k is the mean point (centroid) of cluster k
d2 K- means Hypothetical: Two features - three classes 70 30-10 -50-10 0 10 20 30 40 50 d1
d2 K - means Four classes 70 30-10 -50-10 0 10 20 30 40 50 d1
d2 K - means Original (3 classes) 140 90 40-10 0 20 40 60 80 d1
d2 K - means 2 clusters Two classes 140 90 40-10 0 20 40 60 80 d1
d2 K means TRAPPED! Three classes Three classes 140 90 40-10 0 20 40 60 80 d1
Fuzzy Clustering The fuzzy C-means algorithm is a generalization of K- means. Rather than assigning a pattern to only one class, the fuzzy C-means assigns the pattern a number m, with 0 <= m <= 1.
Component 2-20 0 20 40 Fuzzy C - means Four classes -40-20 0 20 40 60 Component 1 These two components explain 100 % of the point variability.
Fuzzy C- Means Segmentation I Two classes Original Class I Class 2
Fuzzy Segmentation II Four classes Class I Class 2 Original Class 3 Class 4
Brain Segmentation With Fuzzy C- Means Bias field inhomogeneity contributes to the problem of poor segmentation
Clustering: Principle Limitations Convergence to the optimal configuration is not guaranteed. Outcome depends on the number of clusters chosen. No easy control over balancing global and local variability Intrinsic assumption of a uniform feature probability is still being made Generalization needed: Relax requirement to predetermine number of classes Balance influence of global and local variability Possibility to including a-priori information, such as non-uniform distribution of features.
Segmentation As Probabilistic Problem Treat both intensities Y and classes Z as random distributions The segmentation problems is to find the classes that maximize the likelihood to represent the image
Segmentation As A Probabilistic Problem Prior pz ( Y) p( Y Z) * p( Z) Likelihood Z is the segmented image Y is the observed image p(z): p(y Z): P(Z Y): prior probability likelihood posterior distribution that best represents the data Posterior
Probability In Spatial Context Model classes Z as Markov Random Fields (MRF) MRF Rules: Classes exist: p(z) > 0 for all z in Z Probability of z at a location depends on the spatial neighborhood Observed intensities in the neighborhood are a random process
MRF Based Segmentation 1 st and 2 nd order MRFs (p,q) (p,q+1) (p+1,q) (p=1,q+1)
MRF Based Segmentation 1 st and 2 nd order MRFs Step I: Define prior class distribution energy, i.e. using a random process (p,q) (p,q+1) zz, prior 1, z z 1 z z prior prior (p+1,q) (p=1,q+1)
MRF Based Segmentation 1 st and 2 nd order MRFs (p,q) (p,q+1) Step I: Define prior class distribution energy: zz, true 1, z z 1 z z true true (p+1,q) (p=1,q+1) Step II: Select distribution of conditional observation probability, e.g gaussian: E y 2 p, q p, q. yz 2 2 z y p,q is the pixel value at location (p,q) z and z are the mean value and variance of the class z z
MRF Based Segmentation 1 st and 2 nd order MRFs (p,q) (p+1,q) (p,q+1) (p=1,q+1) Step III: Solve (iteratively) for the minimal distribution energy arg min E p, q z, z y 2 p. q y z s true 2 sn 2 z s z assignment similarity energy
probability Iterative Segmentation Process 3 2 Update MRF classify Global Distribution 1 intensity Distribution model
MRF Based Segmentations 4T MRI, SPM2, priors for GM, WM based on 60 subjects
Generalization: Mixed Gaussian Distributions y 2 arg min E p, q z, z. p. q z y s t 2 tn 2 z s White matter z q z arg min E p,, z z y s t tn s Find solution iteratively using Expectation Maximization (EM) White matter White matter lesion y y p. q w w p. q wml wml
probability EM Based Segmentation UpdateMRF classify Global Distribution intensity Update distribution model
EM Segmentation Standard 1.5 MRI, SPM2, tissue classes: GM, WM, CSF, WM Lesions
Limitations Of Energy Minimization Case where energy minimization might not work: green: wrong segmentation red: correct segmentation Because the separation energy depends on mean and variance of the classes, separation can go wrong if red variance is much smaller than green variance, even when red mean is larger than green mean.
EM segmentation problems EMS EMS gray matter segmentation areas where EMS has problems.
Potential Solution: Gradient Based Algorithms The separation penalty is defined based on magnitude and direction of image gradient: E p, q I 1 p I q Small gradient magnitude: Large separation penalty (no change in class) in gradient direction Small separation penalty (class change) in other directions small Large gradient magnitude Small separation penalty in grad directions Large penalty in other directions. Large penalty
Local Structure Gradients (LSG) Case where energy minimization did not work: green: wrong segmentation red: correct segmentation Using LSG leads to: higher separation energy along grad direction Lower separation energy in orthogonal directions leads to a correct red segmentation
Energy versus Gradient Based Segmentations EMS gray matter Geo-Cuts gray matter
Other Segmentation Methods Edge detection Split and merge Level sets Graph partitioning Fractal (multiscale)
Edge Detection Original Edge filtered Image edges may be defined by: Zero-crossings of the Laplacian (differential operator) Maxima of gradient fields Maxima of coefficients of basis functions (e.g. fourier, wavelets, curvelets others (huge related literature)
Split and Merge Principle Example: lower spine Source: http://abdeslam.mokrani.free.fr/
Level Sets Principle: A level set of a function, e.g. representing shape, is the set were the function takes on a constant value c, e.g. zero at the plane intersection. Example: White Matter Segmentation Source: http://graphics.stanford.edu/~mlrobert/
Graph Partitioning Principle: Example: Liver Tumor on CT Source: http://www.cis.upenn.edu/~jshi/graphtutorial/tutorial-imagesegmentationgraph-cut1-shi.pdf
Pixel order MultiScale: Co-Occurrance Matrix (CM) Definition: The CM is a tabulation of how often different combinations of pixel brightness values (gray levels) occur in an image. kernel CM Image Sharp Blurred image Pixel that stores calculated CM value More Blurred Pixel order
Evaluation of CM CM MRI with spatially varying intensity bias MRI with spatially homogenous intensity bias Medical Imaging Informatics 2011, N.Schuff Course # 170.03 Slide 60/67
Fractal Dimensions Intuitive Idea: Many natural objects have structures that are repeated regardless of scale. Repetitive structures can be quantified by fractal dimensions (FD). Sierpinski Triangle Medical Imaging Informatics 2011, N.Schuff Course # 170.03 Slide 61/67
Fractal Dimensions Definition: Box counting Sierpinski Triangle Medical Imaging Informatics 2011, N.Schuff Course # 170.03 Slide 62/67
Fractal Dimensions MEDICAL IMAGE SEGMENTATION USING MULTIFRACTAL ANALYSIS Soundararajan Ezekiel; www.cosc.iup.edu/sezekiel/publications/medical Medical Imaging Informatics 2011, N.Schuff Course # 170.03 Slide 63/67
Summary Automated Threshold MRF EMS Graph Partitioning Fuzzy C- Means Level Sets Split & Merge Initializing Semiautomated Region growing K-means Cooccurrence Edge Detection Manual Tracing Fractal Dimensions Medical Imaging Informatics 2011, N.Schuff Course # 170.03 Slide 64/67
Literature 1. Segmentation Methods I and II; in Handbook of Biomedical Imaging; Ed. J. S. Suri; Kluwer Academic 2005. 2. WIKI-Books 1. SPM: http://en.wikibooks.org/wiki/spm-vbm FSL-FAST: http://www.fmrib.ox.ac.uk/fsl/fast4/index.html Medical Imaging Informatics 2011, N.Schuff Course # 170.03 Slide 65/67