Methodological progress in image registration for ventilation estimation, segmentation propagation and multi-modal fusion

Transcription

1 Methodological progress in image registration for ventilation estimation, segmentation propagation and multi-modal fusion Mattias P. Heinrich Julia A. Schnabel, Mark Jenkinson, Sir Michael Brady

2 2 Clinical applications for image registration Monitoring of longitudinal change Multimodal fusion for diagnosis Image-guided interventions Atlas-based segmentation - require compensation of motion/ deformations between scans: non-linear image registration Technical Contributions: Fusion of PET and CT scans for cancer diagnosis Dose planning in radiotherapy Lung segmentation modality independent neighbourhood descriptor (MIND) discrete optimisation with dense displacement sampling (deeds)

3 Motion and ventilation estimation for 4DCT lung scans

4 4 Challenges for lung motion estimation Locally varying image contrast - e.g. due to lung compression Large displacements of small anatomical features Complex motion patterns - discontinuous sliding motion between lungs and rib-cage discrete optimisation with dense displacement sampling (deeds)

5 5 Optimisation for image registration Aim of image registration is to assign motion vector to each voxel - cost function measuring image similarity and smoothness of motion field cost function parameter value continuous (gradient-based) optimisation vs. discrete (MRF-based) optimisation Advantages of discrete optimisation - no derivative of similarity metric is required (more flexibility) - deformation parameters can be intuitively specified - avoids local minima and need of iterative solution

6 MICCAI 2012: Globally Optimal Registration on a Minimum Spanning Tree using Dense Displacement Sampling 6 Dense displacement sampling (deeds) of similarity term Reduced number of control points - assume constant motion within small area Very dense quantisation of displacement space - up={0,±2,..±16} 3 voxels (~5000 displacements) - random subsampling of voxels within block Similarity distribution for each control point high local similarity fixed image moving image low local similarity Related to block-matching techniques - but: regularisation is inferred for all potential displacements (and not only for local optimum) using belief propagation

7 Regularisation on graph using belief propagation Calculate belief vector C (negative log probabilities) for each node in graph C p (u q ) = min u p S(u p )+αr(u p, u q )+ c C c (u p ) - regularisation R(up,uq) depends on all pairwise connected nodes p and q concept of belief propagation edges connect nodes (grey edges are inactive in tree) incoming messages C c calculate new belief vector for active node outgoing messages C p for next (parent) node Belief propagation on graphs with loops has reduced convergence - minimum spanning tree is advantageous (no iterations, global optimum) - image-adaptive tree preserves sliding motion MICCAI 2012: Globally Optimal Registration on a Minimum Spanning Tree using Dense Displacement Sampling 7

8 8 Visual example of registration result for inhale and exhale of 4D-CT 3D non-linear registration with deeds and MIND - 4 scales, symmetric transformations, computation time: ~30 sec. - no linear pre-registration or lung segmentations necessary axial plane coronal plane sagittal plane maximal inspiration (green), expiration (magenta) 4D-CT scan (#8 DIR-Lab)

9 Trans Med Imag 2013: MRF-based Deformable Registration and Ventilation Estimation of Lung CT 9 Ventilation estimation of 4D-CT scans Lung compression/expansion leads to intensity changes in Hounsfield units - explicit modelling as fourth hyper-label (in addition to 3D geometric) - new similarity metric: S(u p )= I(x p ) (1 + ν p ) J(x p + u p ) Simultaneous, regularised ventilation estimation and improved registration relative occurence of intensities inhale exhale CT Hounsfield units coronal plane axial plane anatomical scan in greyscale, local ventilation in false colours (blue red)

10 Atlas-based segmentation propagation

11 Atlas-based segmentation propagation Manual segmentation is time-consuming and error prone Automatic segmentation using registration - hand-labeled scans of a number of subjects (atlases) - registration of unseen scan to all atlases - propagation of segmentation information to new scan

12 12 Probabilistic registration using uncertainty estimates Relying on most probable transformation is problematic - missing one-to-one correspondences of across subjects - mismatch of structures due to local minima - failures of registration due to noise and artefacts example brain segmentation label probabilities (subset of segmentation labels) only most probable probabilistic registration with uncertainty estimates

13 13 Min-marginal energy computation Same concept, belief propagation, as before (for lung motion estimation) - using only one pass ( dynamic programming ) gives only most probable deformation Second (backward) pass of belief propagation gives full distribution - compute messages and marginal energy (for each node and each displacement) 1. Initialise marginals with similarity cost and messages with zeros 2. Forward pass marginals and messages 3. Backward pass parent node! min!!!! min! parent node!! min!!! child node A! child node B! child node (n)! 4. Add messages to marginals for all nodes child node (n-1)!

14 14 Uncertainty estimates for label fusion Dense non-parametric probability distribution is defined by marginal energy E: p(x i, u i )= 1 n exp β E(x i, u i ) std(e(x j, u j )) Parameter β determines sharpness of distribution axial slice of target scan! marginal distribution over all displacements! u j L and x j Ω nodes x and displacements u axial slice of moving scan! negative energies / probability a.u Estimation of non-parametric (possibly multimodal) uncertainty distributions

15 15 Influence of β parameter Segmentation accuracy (DICE) for LPBA40 dataset (Schattuck 08) - small values over-smooth distribution, β is equivalent to argmin segmentation accuracy (DICE) argmin full marginals parameter for probability calculation Example labeling of MRI brain scan certainty of labelling Spatial uncertainty of labeling

16 Label Fusion using Multiple Atlases Same concept is directly applicable to multiple atlases Uninformative atlases contribute less to fusion result relative occurence of DICE score 0.14 argmin (70.1%) full marginal (72.2%) 0.12 multi atlas (76.0%) <= >=0.9 DICE overlap for segmentation label Single atlas with uncertainty estimates improves DICE by 2.1 % Three atlases give further improvement of 3.8 % 16 error of labels argmin multi-atlas with uncertainty estimates

17 Multi-modal deformable registration

18 Concept of Modality Independent Neighbourhood Descriptors (MIND) Definition of similarity within same modality is straightforward - many different image features (intensities, tissue boundaries, texture) Idea: multidimensional image descriptor (MIND) - based on SSD in local neighbourhood of same modality (self-similarity) - independent of local contrast, noise and discriminative for different image features MRI intensities MIND of MRI CT intensities MIND of CT Advantages compared to statistical similarity measures: - no global intensity mapping across modalities required - directly comparable across scans using SSD (efficient optimisation) MICCAI 2011: Non-Local Shape Descriptor: A New Similarity Metric for Deformable Multimodal Registration 18

19 Calculation of MIND descriptors Calculation of self-similarity distances Dp - SSD of central image block and all image blocks in local neighbourhood R MIND SSC Estimation of local noise V - average of self-similarities distances MINDðI; x; rþ ¼ 1 n exp D pði; x; x þ rþ VðI; xþ r 2 R Large influence of central block on calculation of descriptor - only pair-wise distances between blocks in neighbourhood self-similarity context (SSC) Quantisation and Hamming weight for faster distance between descriptors MICCAI 2013: Towards real-time multi-modal fusion for image-guided interventions using self-similarities Med Imag Anal 2012: MIND: Modality Independent Neighbourhood Descriptor for Multi-modal Deformable Registration 19

20 Med Imag Anal 2012: MIND: Modality Independent Neighbourhood Descriptor for Multi-modal Deformable Registration 20 Visual example of registration result for CT and MRI chest scans 11 clinical scan pairs (CT and MRI) of patients with empyema Challenges for multi-modal registration - low scan quality of MRI, pathological changes axial plane coronal plane sagittal plane Visualisation of result, CT in grayscale, MRI with heat colormap

21 Med Imag Anal 2012: MIND: Modality Independent Neighbourhood Descriptor for Multi-modal Deformable Registration 21 Quantitative evaluation for MRI/US fusion 13 neurosurgery scan pairs: pre-treatment MRI intra-operative ultrasound Comparison of different similarity metrics (mutual information, MIND, SSC) - Target registration error (TRE) based on ~20 manual landmark pairs / scan MI MIND SSC coronal plane Visualisation of result, MRI in grayscale, ultrasound with heat colormap target registration error in mm TRE before: 6.76 mm Computation time: ~20 sec.

22 Evaluation of Results and Comparison to State-of-the-Art

23 Results for EMPIRE10 Lung Registration Target registration error (TRE), evaluated with manual anatomical landmarks - 3 best performing algorithms (for average TRE): CMS, deeds+mind, MetaReg - compared to: 10th best contestant (for average TRE) out of 32 and my initial submission CMS MetaReg initial (19th) deeds+mind 10th best 1.2 intra-observer 0.8 error 0.64mm target registration error in mm we estimate motion for whole domain (other methods require lung segmentation) [CMS] X. Han: Feature-constrained Nonlinear Registration of Lung CT Images, MICCAI-Challenge [MetaReg] S. Münzig et al.: On Combining Algorithms for Deformable Image Registration, WBIR

24 Results for EMPIRE10 Lung Registration Fissure overlap error, evaluated with manual (major) fissure segmentations - 3 best performing algorithms (for average error score): gsyn, deeds+mind, CMS - compared to: 10th best contestant out of 32 and my initial submission gsyn CMS initial (25th) deeds+mind 10th best lung fissure error in % deformation fields are free from any singularities (using symmetric approach) [gsyn] B. Avants et al.: Symmetric diffeomorphic image registration with cross-correlation..., Med Imag Anal [CMS] X. Han: Feature-constrained Nonlinear Registration of Lung CT Images, MICCAI-Challenge

25 25 Summary Comprehensive framework for deformable registration for: - motion/ventilation estimation, atlas-based segmentation, multi-modal fusion New self-similarity descriptors (MIND, SSC) for image similarity Efficient discrete optimisation framework, with dense displacement sampling (deeds) and estimation of local registration uncertainty State-of-the-art registration accuracy (often below resolution) - computation times of less than one minute for high-resolution scans Software tools / source code are available to use

26 References and Acknowledgements Towards real-time multi-modal fusion for image-guided interventions using self-similarities. In: Medical Image Computing and Computer Assisted Intervention (MICCAI) Uncertainty Estimates for Improved Accuracy of Registration-Based Segmentation Propagation using Discrete Optimisation. In: MICCAI Challenge Workshop on Segmentation: Algorithms, Theory and Applications ("SATA") MRF-based Deformable Registration and Ventilation Estimation of Lung CT. IEEE Transaction on Medical Imaging Globally Optimal Registration on a Minimum Spanning Tree using Dense Displacement Sampling In: Medical Image Computing and Computer Assisted Intervention (MICCAI) MIND: Modality Independent Neighbourhood Descriptor for Multimodal Deformable Registration Medical Image Analysis Non-Local Shape Descriptor: A New Similarity Metric for Deformable Multimodal Registration In: Medical Image Computing and Computer Assisted Intervention (MICCAI)