Mapping Multi-Modal Routine Imaging Data to a Single Reference via Multiple Templates

Transcription

1 Mapping Multi-Modal Routine Imaging Data to a Single Reference via Multiple Templates Johannes Hofmanninger 1, Bjoern Menze 2, Marc-André Weber 3,4 and Georg Langs 1 1 Department of Biomedical imaging and Image-guided Therapy Computational Imaging Research Lab, Medical University of Vienna, Austria johannes.hofmanninger@meduniwien.ac.at, 2 Department of Computer Science & Institute for Advanced Study Technical University of Munich, Germany 3 Department of Diagnostic and Interventional Radiology University of Heidelberg, Germany 4 Institute of Diagnostic and Interventional Radiology Rostock University Medical Center, Germany Abstract. Population level analysis of medical imaging data relies on finding spatial correspondence across individuals as a basis for local comparison of visual characteristics. Here, we describe and evaluate a framework to normalize routine images covering different parts of the human body, in different modalities to a common reference space. The framework performs two basic steps towards normalization: (1) The identification of the location and coverage of the human body by an image and (2) a non-linear mapping to the common reference space. Based on these mappings, either coordinates, or label-masks can be transferred across a population of images. We evaluate the framework on a set of routine CT and MR scans exhibiting large variability on location and coverage. A set of manually annotated landmarks is used to assess the accuracy and stability of the approach. We report distinct improvement in stability and registration accuracy compared to a classical single-atlas approach. 1 Introduction Analysis of medical routine imaging data is highly relevant, since they provide a realistic sample of the clinical population and are a key to modeling the natural variability of disease progression and treatment response. In order to compare local characteristics across the population, we need to establish spatial correspondence. Unlike in brain imaging studies, routine imaging is not harmonized by protocols, but applied and guided by indication and case specific needs. Here, we describe and evaluate a robust framework for multi-modal registration of routine imaging data. Many medical imaging applications and research studies rely This research was supported by teamplay which is a Digital Health Service of Siemens Healthineers, by the Austrian Science Fund FWF (I2714-B31), by the WWTF (S14-069), and by the DFG (WE 2709/3-1, ME 3511/3-1).

2 2 Mapping Multi-Modal Images to a Single Reference CT CT MRI MRI Fig. 1. Left: Visualization of the coverage of the images in the reference space and their center position. Right: Mean volume, generated by mapping intensities of 49 CT and 28 MR images to the reference space. High contrast of organ and bone borders indicate accurate registration. on the alignment of volumetric images to a standard frame of reference. For example, structural and functional analysis of brain MR images [3] or atlas based labeling of anatomical structures [8]. However, medical routine images exhibit a wide range of inter-subject variability such as age, sex, disease and health status but also varying image characteristics such as quality and field of view not controlled by any study protocol. Due to these properties, spatial alignment of heterogeneous multi-subject datasets poses a challenging task. Related Work True correspondences between images are unknown and have to be inferred by matching boundaries of anatomical structures or visual landmarks [1]. Existing methods, producing high quality alignments, have been proposed for specific organs or organ parts. Especially the normalization of brain MR images is challenging, yet highly relevant. Advanced methods, that are capable to cope with structural variability in the brain have been proposed and vary from geodesic registration using anatomical manifold learning [6], feature based approaches [10] to multi-template alignments [13]. Studies of other organs typically aim for segmentation of distinct organ or organ-parts. Label fusion of multi-atlas segmentations is a popular approach that yields robust and accurate results using a set of multiple manually annotated atlases [8, 9, 12]. Multi-modal alignment of images is mostly relevant for intra-subject registration due to its role for image-guided interventions. Hence, work on inter-subject multi-modal registration is scarce. Mutual information (MI) [11] is often used as similarity function to align multi-modal images. Heinrich et al. propose point-wise matching of local descriptors [7] for matching of 3D ultrasound and MRI brain scans, but also for registration of inter-subject full body MR and CT volumes to generate pseudo-ct scans [2]. Contribution In this work, we address the problem of aligning truncated intersubject multi-modal images with widely different fields of view to a common

3 Mapping Multi-Modal Images to a Single Reference 3 Modality 1 Normalized Images a b c d Modality 2 e f Fig. 2. Multi-modal multi-template normalization (a) The framework facilitates the processing of routine imaging data of different modalities and exhibiting different coverage of the human body. (b) During normalization, an image is aligned to multiple templates (c) of the same modality as the image. All templates are aligned with a modality specific atlas (d) supported by manually annotated landmarks. (e) Atlases of different modalities are carefully aligned to a central reference space using landmarks, body, bone and organmasks. Positions in an image are mapped to the reference atlas by concatenation of the three transformations (t n i,k, t r k,m and t c m) that yield maximal registration quality. (f) After normalization, coordinates and label masks are mapped across the population. whole body reference space (Fig. 1). The contribution of this study is threefold (1) we propose a novel framework, capable of reliably aligning routine images by (2) adapting the idea of multi-atlas segmentation towards multi-template localization and normalization and (3) evaluating the accuracy and effectiveness of the approach on a heterogeneous set of real world routine CT and MR images. 2 Method The normalization framework is based on an offline and online phase. The offline phase is performed in advance and requires carefully supervised registration of multiple pre-defined template images to a common reference space (Sec. 2.2). During the online phase, no manual interaction with the data is required. Novel images, for which no landmarks are available, are registered to the templates (Sec. 2.1) and the best matching template is selected (Sec. 2.2). An overview of the multi-modal multi-template framework is given in Fig. 2. The framework allows for normalization of routine images I = {I 1,..., I N } covering arbitrary regions of the human body and being of different modalities M = {1,..., M} so that µ(i) M. The framework requires a set of templates T = {T 1,..., T K } where each modality is represented and µ(t) M to facilitate an unbiased normalization of a heterogeneous corpus of images. The templates are chosen in a way to cover natural variation such as size and sex. For each modality, a distinct atlas R m is required as well as a central reference space C. Each template is aligned with its modality specific atlas whereas each atlas is aligned with the

4 4 Mapping Multi-Modal Images to a Single Reference central reference space. The alignment between template and atlas is uni-modal but supported by some supervision to overcome a potential registration bias. The alignment of the atlases to the reference space are multi-modal and thus require an even higher degree of supervision. 2.1 Fragment to Template Registration Registration of an image to a template is performed in two steps. (1) Estimation of the coverage and the position of the image in the human body and (2) a non-rigid transformation to the area estimated at step one. The estimation of position and coverage is performed by matching 3D scale invariant features (3DSIFT) according to [10] and performing affine registration by minimizing the Mean Squared Error (MSE) between the feature locations. Subsequently, a refinement step, by conducting an intensity based affine registration with high regularization on translation, optimizing the Normalized Cross Correlation (NCC) is performed. If no matching 3DSIFT features are identified, an intensity based affine registration with no regularization on translation is performed. After estimation of position and coverage, an intensity-based non-rigid registration is conducted [5]. In the following we define t a i,k as the affine transformation between image I i and template T k so that I i Îa i,k = ta i,k (T k). Further, we define t n i,k as the concatenation of the affine and non-rigid transformation so that I i În i,k = tn i,k (T k). 2.2 Template to Reference Registration Each atlas R m is registered to its corresponding templates T k (if m = µ(t k )) with a high degree of supervision. In our case, the registrations are supported by 58 manually annotated anatomical landmarks on specific bone and organ positions. We perform affine and b-spline registrations optimizing the MSE between the landmarks prior to a non-rigid image registration based on image intensities. As a refinement step a final b-spline registration optimizing for landmarks distance is performed. We define the transformations from an atlas R m to the template T k as t r k,m. The reference atlas C is registered to the modality specific atlases R m and has to be performed either fully supervised and modality independent or multi-modal. We perform registrations using 58 landmarks, segmentations covering 20 organs and organ parts, a body mask and segmentations of all skeletal bones so that the registrations can be performed independently of the modality. We define the transformation from the reference space to an atlas R m as t c m. 2.3 Template Selection Template selection for an image I i is performed in two steps: (1) calculation of Î a i,k for every k where µ(t k) = µ(i i ) and selection of the top C templates that yield the highest NCC(I i, Îa i,k ) so that C i {1,..., K} is the set of candidate

5 Mapping Multi-Modal Images to a Single Reference 5 CT MR ALL T D T D T D mean median std misalig #cases #landm Table 1. Mean, median, standard deviation and number of misaligned (misalig.) landmarks. Results are given in mm of landmark distances to the ground truth for the template registration (T) and direct registration (D). Fig. 3. Landmarks used to evaluate registration accuracy. templates for image I i. Subsequently, the non-rigid transformations between the image and templates t n i,k are computed for k C only. In the second step (2) the transformations t n i,k and tr k,m are concatenated so that and we can select the template by I i Îi,k = t n i,k(t r k,m(r m )) (1) maximizencc(i i, Îi,k) subject to µ(i i ) = µ(t k ), k C i (2) k Given a template x i that yields maximum NCC the final transformation from the reference space to an image is t i = t n i,x i t r x i,µ(i) tc µ(i) (3) 3 Experiments Test Data We perform evaluation on a heterogeneous set of CT (#48) and MR- T1 (#28) images recorded in the daily clinical routine. The images cover different regions of the human body and vary in resolution, dose (CT) and sectioning (saggital and axial). We annotated 16 landmarks on specific bone and organ positions (aortic arch, trachea biforcation, cristia iliaca left, crista iliaca right, symphysis, aorta bifurcation, L5, L4, L3, L2, L1, xyphoideus, sternoclavicular left, sternoclavicular right, renalpelvis left and renalpelvis right) covering the chest and abdomen (Fig.3). Depending on coverage, each image may only exhibit a subset of these landmarks. More formally, the evaluation set consists of tuples I i, V i where V i {1,..., 16}. Template Data For the template set and the atlases we use 22 (11 CT and 11 MR) whole body volumes of the VISCERAL Anatomy 3 dataset [4], which provides manually annotated organ masks and landmarks. For each modality 10

6 6 Mapping Multi-Modal Images to a Single Reference 0 Template registration Direct registration Error in mm # LM CT # LM MR # LM total L5 L4 L3 L2 L1 sternoclavicular left sternoclavicular right xyphoideus aortic arch aorta bifurcation trachea bifurcation renalpelvis left renalpelvis right crista iliaca left crista iliaca right symphysis Fig. 4. Comparison of the template approach to direct registration for each landmark and the full dataset (CT + MR). Registration errors are given in mm distance to the ground truth landmarks. volumes are used as templates and one is used as modality specific atlas. We defined the CT atlas to also represent the reference space. Evaluation and Metric For evaluation we assess robustness and accuracy of the approach and study the influence of number of templates used. We compare the proposed template framework to a direct registration approach where each image is registered to the modality specific atlases R m directly. To allow for a fair comparison, the direct registration is performed according to the image to template registration method described in Sec To assess registration quality, we transform the landmark positions from the image space into the reference space. We then calculate the distance between the transformed landmarks and the corresponding landmarks in the reference. We report the registration accuracy for each landmark and compare the performance between MR and CT cases. To assess robustness we define landmarks that are off by more than 100mm as misaligned. Parameters and pre-processing We perform bias-field correction [14] on the MR images and rescale all volumes (images and templates) to an isotropic voxel resolution of 2mm prior to processing. If not stated differently, results reported are produced with parameters K (Number of templates) set to 10 and C (max. number of non-rigid registrations) set to 4. 4 Results Fig. 4 gives a comparison of registration errors to the ground truth when using the template approach compared to direct registration. Results are given for the full dataset(ct+mr) and each landmark. Distinct improvement in accuracy (lower median) but also stability (less outliers) can be seen. Table 1 shows averaged registration errors for the two modalities. For CT, median distance

7 Mapping Multi-Modal Images to a Single Reference 7 Direct reg. Multi-template reg. Fig. 5. (a) Mappings of the L5 vertebrae landmarks to the reference space comparing the direct registration to the robust template approach. (b) Organ labels mapped from the reference space to the images. Average NCC CT Average NCC MR # Templates per modality CT MR Mean Error # Templates per modality CT MR Fig. 6. Effect of varying number of templates on final Normalized Cross Correlation (NCC) (a) and registration error (b). improves from 17.6 to 13.4mm and for MR from 36.2 to 17.1mm. The number of misaligned landmarks drops from 25/476 to 3 for CT and from 28/206 to 0 for MR. This indicates that especially MR benefits from the multiple templates in terms of stability. However, the 3 misaligned landmarks of the CT cases (xyphoideus and symphysis) are close to the image border and are therefore rather sensible for misalignment. Fig. 5a illustrates an exemplary mapping of all landmarks on vertebra L5 to the reference space. Fig. 5b shows labellings of organs mapped from the reference to the images. Fig. 6b shows the effect of a varying number of templates on the mean landmark error and Fig. 6a shows the effect of a varying number of templates on the final NCC values (Eq. 2). Fig. 1 illustrates localization, coverage and averaged intensity values of CT and MR images that were mapped from image to reference space. 5 Conclusion This paper addresses the challenging task of mapping clinical routine images of different modalities to a common reference space. A multi-template approach for localization and deformable image registration is presented. Evaluation has been performed on a representative dataset of CT and MR images using manually

8 8 Mapping Multi-Modal Images to a Single Reference annotated landmarks. The results show a distinct improvement in registration accuracy and stability compared to direct registration to an atlas. We believe, that spatial normalization of routine images provides a useful tool to especially study systematic diseases such as multiple myeloma, metastatic cancer and others that exhibit visual traits throughout the human body in different imaging modalities. References 1. Crum, W., Griffin, L., Hill, D., Hawkes, D.: Zen and the art of medical image registration: correspondence, homology, and quality (2003) 2. Degen, J., Heinrich, M.P.: Multi-Atlas Based Pseudo-CT Synthesis Using Multimodal Image Registration and Local Atlas Fusion Strategies. In: Computer Vision and Pattern Recognition (CVPR). pp (2016) 3. Fischl, B., Salat, D.H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., Montillo, A., Makris, N., Rosen, B., Dale, A.M.: Whole Brain Segmentation: Automated Labeling of Neuroanatomical Structures in the Human Brain. Neuron 33(3), (2002) 4. Goksel, O., Foncubierta-Rodriguez, A., del Toro, O.A.J., Müller, H., Langs, G., Weber, M.A., Menze, B.H., Eggel, I., Gruenberg, K., et al.: Overview of the visceral challenge at isbi 20. In: VISCERAL Challenge@ ISBI. pp (20) 5. Gruslys, A., Acosta-Cabronero, J., Nestor, P.J., Others: A New Fast Accurate Nonlinear Medical Image Registration Program Including Surface Preserving Regularization. IEEE Transactions on Medical Imaging 33(11), (2014) 6. Hamm, J., Ye, D.H., Verma, R., Davatzikos, C.: GRAM: A framework for geodesic registration on anatomical manifolds. Medical Image Analysis 14(5), (2010) 7. Heinrich, M.P., Jenkinson, M., Papiez, B.W., Brady, S.M., Schnabel, J.A.: Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International conference on medical image computing and computer-assisted intervention (MICCAI). pp Springer Berlin Heidelberg (2013) 8. Iglesias, J.E., Sabuncu, M.R.: Multi-atlas segmentation of biomedical images: A survey. Medical image analysis 24(1), (20) 9. Koch, L.M., Rajchl, M., Bai, W., Baumgartner, C.F., Tong, T., Passerat-Palmbach, J., et al.: Multi-Atlas Segmentation using Partially Annotated Data: Methods and Annotation Strategies. arxiv preprint arxiv: , pp (2016) 10. Toews, M., Wells, W.M.: Efficient and robust model-to-image alignment using 3D scale-invariant features. Medical Image Analysis 17(3), (2013) 11. Viola, P., Wells Iii, W.M.: Alignment by Maximization of Mutual Information. International Journal of Computer Vision 9(242), (1997) 12. Wolz, R., Chu, C., Misawa, K., Fujiwara, M., Mori, K., Rueckert, D.: Automated abdominal multi-organ segmentation with subject-specific atlas generation. IEEE Transactions on Medical Imaging 32(9), (2013) 13. Xie, L., Pluta, J.B., Das, S.R., Wisse, L.E., Wang, H., Mancuso, L., Kliot, D., Avants, B.B., Ding, S.L., Manjón, J.V., Wolk, D.A., Yushkevich, P.A.: Multitemplate analysis of human perirhinal cortex in brain MRI: Explicitly accounting for anatomical variability. NeuroImage 1, (2017) 14. Zhang, Y., Brady, M., Smith, S.: Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Transactions on Medical Imaging 20(1), (2001)