Medical Image Analysis Active Shape Models Mauricio Reyes, Ph.D. mauricio.reyes@istb.unibe.ch ISTB - Institute for Surgical Technology and Biomechanics University of Bern
Lecture Overview! Statistical Shape Modeling! Quick summary/review! Further considerations! Reference selection! Modeling scaling! Outlier detection! Modeling Physiology! Statistical Shape Modeling for Segmentation: Active Shape Models 2
Segmentation Overview Big Picture Simple Methods! > Thresholding! > Region-Growing! >! Classification! + Clustering! > knn! > SVM! >! Deformable Models!! > Snakes! > Level-Sets! >! Active Models! > ASM > AAM 3
Statistical distribution of shape How?!
Summary Statistical Shape Modelling! Model: Mean Shape Statistical Analysis via Principal Component Analysis (PCA) Eigen-decomposition of matrix provides eigenvectors/eigenvalues (λ,φ) Sφ k = φ k λ k With k, the kth eigen-vector/value
Summary Statistical Shape Modelling! -Common reference space - Find anatomical corresponding points amongst shapes
Summary Statistical Shape Modelling! - Find anatomical corresponding points amongst shapes Principal modes of deformation - Model variability
Further considerations > What if scaling is included (e.g. bone length variability is considered)? Non-rigid transf. Affine transf. Direct morphing of these two might cause problems! What we look for Affine transf. Non-rigid transf. 8
Further Considerations! - Modelling Schemes! Mesh to Mesh Registration Reference mesh Isosurface Image volume
Further Considerations! - Modelling Schemes! Mesh to Volume Registration Reference mesh Image volume
Further Considerations! - Modelling Schemes! Volume to Volume Registration Reference mesh Reference volume Image volume
Further Considerations! - Modelling Schemes! * = Masking out soft tissues and other non-desired structures: Improved registration->improved modeling!
Further Considerations! - Modelling Schemes! Parameterization to Parameterization Registration Common base domain Surface and parameterization Image volume Common reference space: No bias! J
Bias in reference selection and natural coordinates reference! Not the same! L Rueckert et al. Miccai 2001
Bias in reference selection and natural coordinates reference! Computing mean image: Guimond et al. CVIU 2000
Bias in reference selection and groupwise modelling! Computing mean image: Guimond et al. Miccai 2000
Bias in reference selection and groupwise modelling! Computing mean image: Guimond et al. Miccai 2000 Reference 1 Reference 2
Bias in reference selection and groupwise modelling! Computing mean image: Guimond et al. Miccai 2000 Average model using ref 1 Average Intensity using ref 2
Further considerations! - Detecting Outliers through SSM! Sometimes the input data for modeling is not as expected.. Coronal slide from a 4D CT scan
Further considerations! - Detecting Outliers through SSM! Contribution measure (Cr): Cr(i,k) = 1 n (b i k ) 2 λ k bb: modal amplitude for subject i on mode k. Lamba_k: kth eigenvalue m1 m2 m3
Further considerations! - Detecting Outliers through SSM! Example: Image artefact, segmentation problem, deformity, etc.. Synthetic deformation added to instance number 13.
Further considerations! - Detecting Outliers through SSM! Example
Statistical Modelling of Physiology!
Statistical Modelling of Physiology!
Statistical Modelling of Physiology! First mode of variation Mean transformations
Statistical Modelling of Physiology! Second mode of variation Mean transformations
SSM for Segmentation: Active Shape Models
Active Shape Models ASM > General idea Start with SSM and image Initially place SSM in image Sample along SSM surface normals, find largest gradient (= edge) Deform pose (scale, rotation, translation) and shape (PCA modes) of model to best fit to extracted image features (largest gradient) Cootes et al. 95 28
Initialization A Crucial Step > Approximately align SSM with structure of interest > Many different methods, choose the one for your application User interaction Centroid alignment Chamfer matching Atlas registration Global search on entire image (evolutionary algorithms) 29
Initialization A Crucial Step Importance of Initialization Initial 2 Iterations 20 Iterations 30 Image Source: Cootes et. al. SPIE Medical Imaging, 2001
Feature Extraction > Edge detection methods.. 31
Normals > SSM points + connections give mesh So we know the neighbors of each point of two vectors yields orthogonal vector This is the surface normal 32
Sampling along profiles > Given: Point on SSM Normal at that point > Sample along normal profile > Parameters Sampling range Sampling step size Yields 2k+1 samples > Choose point with largest feature value (gradient) > Model point should be transformed to that point 33
Pose Estimation > Goal: Rough alignment of both shapes! > Here: similarity transform in 2D! > Optimize registration process according to certain cost function (e.g. mean Euclidean distance)! X = T X t ICP, s, Θ( x) T X t, s, Θ x x = y y t t Θ + s cos ssin Θ ssin Θ x s cosθ y minimize Y X = Y T, s, Θ( x) X t 34
Shape Estimation > PCA-based deformation > Keep only k most important PCA modes (e.g. so that k modes account for 90% of variance) > Threshold b to allow for plausible shapes only, e.g. b i < 3 σ i (3 std.dev. capture more than 95% of variation in Gaussian distribution) b b x x + = = P P T T Pb ( x x) ( Y x) 35
Optimization Schemes > Optimize pose and shape part separately or jointly > Optimization Choose optimizer for pose estimation (e.g. gradient descent) Closed form solution for shape > Iterative method: Compute profile along normals Adjust pose Adjust shape Iterate from beginning until convergence b = P X = TX t, s, Θ( x + Pb) Y = TX t, s, Θ( x + Pb) T ( T X t 1, s, Θ ( Y) x) 36
Summary The Complete ASM Algorithm 37
Advanced Methods > Multiresolution pyramid > Handling noise by averaging along normal points > Intensity profiles 38
Advanced Methods > Intensity profiles: Modeling local structure 1. Samples along profile for ith image 2. Avoid intensity changes-> use gradient values 3. Normalize 4. Repeat for each data point 5. Assume Gaussian distribution: (g,s g ) 6. Quality of fit for new sample s: Mahalanobis distance 7. Pick lowest values (higher probability) 39
Examples ASM 2D Initial 2 Iterations 6 Iterations 18 Iterations 40 Image Source: Cootes et. al. SPIE Medical Imaging, 2001
Examples ASM 2D Clinical example Initial 1 Iteration 14 Iterations 41 Image Source: Cootes et. al. SPIE Medical Imaging, 2001
Examples ASM 3D [Fripp 2007] [Bauer 2012] 42
Example in CMF 43
Example in CMF ASM enhanced with muscle information Model Patient 44
Example in CMF Model enhanced with: Facial muscles Muscle fibers (anisotropy of tissue properties) Predefined surgical approaches Data preprocessing (e.g. cropping) Simulation-specific (e.g. sliding contact areas) 45
Statismo > Statismo provides a high level API. > Users works with shapes, not vectors. > Core functionality implemented only once. > Exchanging models becomes easy. Application 1 Sampling High level API Probabilities Representer (model semantic) Portable File (HDF5) Statismo Application n Conditional distributions 46
Image-guided soft tissue deformation for CMF surgery Statismo is Open Source Get it at http://statismo.github.com Statismo - A framework for PCA based statistical models Lüthi M., Blanc R., Albrecht T., Gass T., Goksel O., Büchler P., Kistler M., Bousleiman H., Reyes M., Cattin P., Vetter T. Insight Journal, http://hdl.handle.net/10380/3371 47
Building texture models > Warp each training sample to mean shape to obtain meanfree patch > Sample intensities from shape-normalized image to form a texture vector > Normalize intensity to eliminate *lighting* variations > Start from the beginning, same as SSM PCA on normalized texture data g g + P g b g > We want to obtain a combined appearance model, which contains both shape and texture variations 48
Combined models of appearance > Shape and texture summarized by parameter vectors b > Handle correlations between shape and texture parameters b s and b g > Apply PCA on combined b to get eigenvectors P c and eigenvalues c > c is vector of appearance parameters controlling both shape and texture b Wsb = bg b = s P c c Weighting to account for unit differences 49
Combined Models of Appearance > Linear model > We can express shape and texture directly as function of c > How to generate a new image: Generate shape-free intensity image from g Warp it using the control points described by x Q s x = x + g = g + = PW s Q c Q 1 s Q = P g g s g P P c cs cg 50
Model Fitting > Minimize difference between new image and the synthetic AAM image > Key step: estimate parameter update from current image sample > Difference vector δi = I I i m 51
Model Fitting More Detailed 1 > Estimate update step using Jacobian > Residual difference r(p) > p is parameter vector p(c,t,u) > Minimize error E(p) > Taylor expansion of r(p+δp) > Choose δp to minimize E(p+ δp) > Need estimate of Jacobian J > Estimate J by small displacements on training set r( p) = g s g m E ( p) = r( p) r( p + δp) = r( p) + 2 r δp p E( p + δ p) = r( p + δp) δp = Rr( p) r R = p T r p 1 r p 2 T r J = p 52
Model Fitting More Detailed 2 > Iterative algorithm: Measure residual error r(p) Predict correction δp=-rr(p) p=p+k δp Repeat to convergence r( p) = g s g m δp = Rr( p) 53
Summary The Complete AAM Algorithm 9. Repeat until convergence 54
Examples AAM 55
Blanz &Vetter et al. 99 Example: AAM in face synthesis > Parameters: Pose, shape, illumination
Examples AAM [Cootes 2004] [Dornaika 2003] 57
Comparison ASM AAM Differences! > ASM searches around current position along normals, AAM searches at current position! > ASM minimizes distance between model and corresponding points, AAM minimizes difference between synthetic and target image!! Evaluation > ASM usually has larger capture range > ASM is faster > ASM usually has more accurate feature point location > AAM gives better match to image texture 58
References > [1] Cootes T.F., Taylor C.J., Cooper D.H., Graham J., Active Shape Models Their training and application Computer Vision and Image Understanding, 1995 > [2] Cootes T., Taylor C.J., Statistical Models of Appearance for Computer Vision, TechReport University of Manchester, 2004 > [3] Heimann T., Meinzer H., Statistical Shape Models for 3D Medical Image Segmentation: A Review, Medical Image Analysis, 2009 59
Evaluating segmentation
Validation of Image Segmentation > Digital phantoms. Ground truth known accurately. Not so realistic. > Acquisitions and careful segmentation. Some uncertainty in ground truth. More realistic. > Autopsy/histopathology. Addresses pathology directly; resolution > Clinical data? Hard to know ground truth. Most realistic model. Source: S. Warfield
Validation of Image Segmentation > Comparison to digital and physical phantoms: Excellent for testing the anatomy, noise and artifact which is modeled. Typically lacks range of normal or pathological variability encountered in practice. Most common atlas: BrainWeb MRI of brain phantom from Prastawa et al. MedIA 2010 MRI of brain phantom from Styner et al. IEEE TMI 2000
Comparison To Higher Resolution MRI Photograph MRI Provided by Peter Ratiu and Florin Talos. Source: S. Warfield
Comparison To Higher Resolution Photograph MRI Photograph Microscopy Provided by Peter Ratiu and Florin Talos. Source: S. Warfield
Comparison to Autopsy Data > Neonate gyrification index Ratio of length of cortical boundary to length of smooth contour enclosing brain surface Source: S. Warfield
Validation of Image Segmentation > Comparison to expert performance; to other algorithms. Experts are currently doing the segmentation tasks that we seek algorithms for. Surgical planning. Neuroscience research. > What is the appropriate measure for such comparisons? Source: S. Warfield
Measures of Expert Performance > Repeated measures of volume Intra-class correlation coefficient > Spatial overlap Jaccard coefficient: Area of intersection over union. Dice coefficient: increased weight of intersection. Vote counting: majority rule, etc. > Boundary measures Hausdorff, 95% Hausdorff. > Bland-Altman methodology: Requires a reference standard. > Measures of correct classification rate: Sensitivity, specificity ( Pr(D=1 T=1), Pr(D=0 T=0) ) Positive predictive value and negative predictive value (posterior probabilities Pr(T=1 D=1), Pr(T=0 D=0) )
Measures of Expert Performance > Measures of correct classification rate: Sensitivity, specificity ( Pr(D=1 T=1), Pr(D=0 T=0) ) Positive predictive value and negative predictive value (posterior probabilities Pr(T=1 D=1), Pr(T=0 D=0) ) Wikipedia
Thanks for your attention!