LICENCIATURA EM ENGENHARIA BIOMÉDICA Fundamentos de Imagem Diagnóstica e Terapeutica Métodos de fusão e co-registo de imagem Jorge Isidoro Sumário Introdução à imagem médica Registo de imagens Metodologia do registo de imagens Tipos de transformações Graus de liberdade Algoritmos de registo de imagem Comparação dos métodos de registo 1
Introduction Image registration is the process of aligning images so that corresponding features can easily be related. The images might be acquired with different sensors (e.g. sensitive to different parts or the electromagnetic spectrum) or the same sensor at different times. Introduction Image registration has applications in many fields, and the one that is addressed is medical imaging This encompasses a wide range of image usage, but the main emphasis is on radiological imaging 2
Introduction Medical images: In Xray Computed Tomography (CT) images are sensitive to tissue density, atomic composition and the X-ray attenuation coefficient In Magnetic Resonance Imaging (MR) they are related to proton density, relaxation times, flow and other parameters The introduction of contrast agents provides information on the patency and function of tubular structures such as blood vessels, the bile duct and the bowel, as well as the state of the blood brain barrier. Introduction Medical images: In nuclear medicine, radio-pharmaceuticals introduced into the body allow delineation of functioning tissue and measurement of metabolic and pathophysiological processes. Ultrasound detects subtle changes in acoustic impedance at tissue boundaries and diffraction patterns in different tissues providing discrimination of different tissue types. Doppler ultrasound provides images of flowing blood. Endoscopy and surgical microscopy provide images of visible surfaces deep within the body 3
Introduction Image Modalities Anatomical Depicting primarily morphology (MRI,CT,X-ray) Functional Depicting primarily information on the metabolism of the underlying anatomy (SPECT,PET) Introduction These and other imaging technologies now provide rich sources of data on the physical properties and biological function of tissues at spatial resolutions from: 5mm for nuclear medicine 1.0 or 0.5mm for MR and CT 20-100µm for optical systems. 4
Image Registration Since the mid 1980 s medical image registration has evolved from being perceived as a rather minor precursor to some medical imaging applications to a significant subdiscipline in itself. Image registration has also become one of the more successful areas of image processing with fully automated algorithms becoming available in a number of applications. Image Registration Why has registration become so important? Medical imaging is about establishing shape, structure, size, and spatial relationships of anatomical structures within the patient together with spatial information about function and any pathology or other abnormality Establishing the correspondence of spatial information in medical images and equivalent structures in the body is fundamental to medical image interpretation and analysis 5
Image Registration In many clinical scenarios images from several modalities may be acquired and the diagnostician s task is to draw useful clinical conclusions: This generally requires mental compensation for changes in subject position. Image registration aligns the images and so establishes correspondence between different features seen on different imaging modalities allows monitoring of subtle changes in size or intensity over time or across a population Image Registration Cont.: establishes correspondence between images and physical space in image guided interventions. Registration of an atlas or computer model aids in the delineation of anatomical and pathological structures in medical images and is an important precursor to detailed analysis. 6
Image Registration It is now common for patients to be imaged multiple times: either repeated imaging with a single modality, or imaging with different modalities to be imaged dynamically, that is to have sequences of images acquired, often at many frames per second The ever-increasing amount of image data acquired makes it more and more desirable to relate one image to another to assist in extracting relevant clinical information. Image Registration Image registration can help in: intermodality registration enables complementary information from different modalities to be combined intramodality registration enables accurate comparisons to be made between images from the same modality. 7
Image Registration Image registration can be used to align multiple images: from the same individual (intrasubject registration) from different subjects (intersubject registration) Image Registration All the images that we wish to register or manipulate must be available in digital form. For most medical images this means that they are made up of: rectangular array of small square or rectangular elements called pixels each pixel having an associated image intensity value 8
Image Registration 2D images are often stacked together to form a 3D volume: many images are now acquired directly as 3D volumes. each pixel will now correspond to a small volume element of tissue or voxel. The number stored in each voxel, the voxel image intensity, will be some average of a physical attribute measured over this volume Registration Methodology Image registration establishes spatial correspondence: which point on one image corresponds to a particular point on the other. by correspond we mean that these points represent a measurement localised to the same small element of tissue within the patient. yields the appropriate transformation between the co-ordinate systems of the two sets of scans 9
Registration Methodology No measurement is perfectly accurate and there will always be uncertainty, error or tolerance in this estimate of correspondence for many clinical applications it is important to know what this tolerance is so as not to over interpret the registered datasets. Registration Methodology Once correspondence is determined throughout the volume imaged by the two modalities then: one image can be transformed into the coordinate system of the other this calculation can itself lead to further approximations or errors typically there will be some blurring of the transformed image 10
Registration Methodology Finally we need to consider correspondence when combining images from multiple subjects or images between an individual and an atlas derived from one or more other individuals. Registration of images from different individuals requires the use of deformations that can accommodate biological variations Registration Methodology In many applications reported in the literature correspondence is implicitly defined as the points resulting from arbitrary transformations derived using algorithms that plausibly morph one image into another. There are an infinite number of possible transformations that can morph one image into another including those that lead to creation or destruction of complete structures. Medical imaging applications usually require deformations of the images that maintain correspondence (nose to nose, eye to eye etc.) 11
Registration Methodology Morphing Registration Methodology Finally we have the process of combining or fusing the information in the images in some useful or meaningful way: may be left entirely to the clinician in his/her mind s eye or simple visualisation effects may be used, including colour, or interactive fading in and out of one image s contribution overlaid on the other two cursors, so-called linked cursors, might be used to indicate corresponding points in the two images 12
Types of Transformation 2D to 2D If the geometry of image acquisition is tightly controlled, 2D images may be registered purely via a rotation and 2 orthogonal translations. It may also be necessary to correct for differences in scaling from the real object to each of the images. Types of Transformation 2D to 2D 13
Types of Transformation 3D to 3D The assumption is usually made that the internal anatomy of the patient has not distorted or changed in spatial relationships between organs so that the imaged part of the body behaves as a rigid body In this case 3 translations and 3 rotations will bring the images into registration Types of Transformation 3D to 3D Careful calibration of each scanning device is required to determine image scaling, i.e. the size of the voxels in each modality. 3D to 3D registration is the most well developed and widely used 14
Within-subject rigid registration - Example Full 3D Anatomic al Image Conventional Anatomical Image Within-subject rigid registration Example 15
Types of Transformation 2D to 3D 2D to 3D registration may be required when establishing correspondence between 3D volumes and projection images such as X-ray or optical images Time Another class of registration problem concerns registration of image sequences that follow some process that changes with time Degrees of freedom Degrees of freedom of the transformation: number of parameters needed to describe a registration transformation is referred to as the number of degrees of freedom depends on the dimensionality of the images and on the constraints of the imaged structures simplest transformation corresponds to the motion of a rigid body. 16
Degrees of freedom Degrees of freedom of the transformation For 2D to 2D registration there will be three degrees of freedom: 2 translations and 1 rotation Original Image Same after a rigid transformation Same after a linear affine transform Degrees of freedom For 3D to 3D registration: 3 translations and 3 rotations, giving six degrees of freedom. When we do not know the pixel or voxel sizes or the fields of view, the registration algorithm may need to determine these. This will lead to an extra: two degrees of freedom in 2D or three degrees of freedom in 3D equating to scaling in each direction 17
Degrees of freedom When images are coming from same subject only a rigid 6 parameter transformation has to be estimated: 3 translations (along X, Y and Z axis) 3 rotations (around X, Y and Z axis) Degrees of freedom affine transformation. In the affine transformation any straight line in one image will transform to a straight line in the other parallel lines are preserved allowing a combination of rigid body motion, scaling, and skew about any of the 3 axes has 12 degrees of freedom. non-rigid registration many more degrees of freedom are required (~2000!) 18
Image Registration Algorithms Corresponding landmark based registration Is based on identification of corresponding point landmarks or fiducial markers in the two images For a rigid structure, identification and location of three landmarks will be sufficient to establish the transformation between two 3D image volumes provided the fiducial points are not all in a straight line In practice it is usual to use more than three. The larger the number of points used the more any errors in marking the points are averaged out Image Registration Algorithms Corresponding landmark based registration It involves first computing the average or centroid of each set of points The difference between the centroids in 3D tells us the translation that must be applied to one set of points This point set is then rotated about its new centroid until the sum of the squared distances between each corresponding point pair is minimised The square root of the mean of this squared distance is also referred to as the Root Mean Square (RMS) error, residual error or Fiducial Registration Error (FRE) 19
Image Registration Algorithms Surface based registration The Head and Hat Algorithm the contours of the surface are drawn on a series of slices from one modality, the head a set of points that correspond to the same surface in the other modality are identified, the hat Image Registration Algorithms Surface based registration The Head and Hat Algorithm computer then attempts a series of trial fits of the hat points on the head contours At each iteration the sum of the squares of the distances between each hat point and the head is calculated and the process continues until this value is minimised tend to fail when the surfaces show symmetries to rotation and this is often the case for many anatomical structures. 20
Image Registration Algorithms The Head and Hat Algorithm Image Registration Algorithms Surface based registration The Iterative Closest Point Algorithm one surface is represented by a set of points while the other is represented by a surface made up of many triangular patches or facets. 21
Image Registration Algorithms Surface based registration The Iterative Closest Point Algorithm proceeds by finding the closest point on the appropriate triangular patch to each of the points in turn closest points form a set and these are registered using the corresponding landmark based registration and the residual error is calculated. the process is repeated until the residual error drops by less than a preset value. Image Registration Algorithms Registration Based on Voxel Intensities Alone Voxel Similarity Measures use the intensities in the two images alone without any requirement to segment or delineate corresponding structures a way of representing the image intensities of a pair of images that are to be registered is called the joint histogram or joint probability distribution these methods use all the data in each image and so tend to average out any errors caused by the noise or random fluctuations of image intensity 22
Image Registration Algorithms Image Registration Algorithms Registration Based on Voxel Intensities Alone Voxel Similarity Measures Registration of multiple images of the same patient acquired using the same imaging modality Because the images are acquired using the same modality, an approximately linear relationship will exist between the voxel intensities in one image and voxel intensities in the other In these cases the Correlation Coefficient is a good measure of alignment basically involves multiplication of corresponding image intensities One image is moved with respect to the other until the largest value of the correlation coefficient is found 23
Image Registration Algorithms Registration Based on Voxel Intensities Alone Voxel Similarity Measures Registration of multiple images of the same patient acquired using the same imaging modality Instead of multiplying corresponding intensities we may subtract them which leads to another measure, the sums of squared intensity differences (SSD) In this case alignment is adjusted until the smallest SSD is found Image Registration Algorithms Registration Based on Voxel Intensities Alone Voxel Similarity Measures Registration of multiple images of the same patient acquired using the same imaging modality If two images are very similar then their ratio will be most uniform at registration. This is the basis of Woods Ratio Image Uniformity (RIU) algorithm in which the variance of this ratio is calculated Alignment is adjusted until the smallest variance is found. 24
Image Registration Algorithms Registration Based on Voxel Intensities Alone Voxel Similarity Measures Registration of multiple images of the same patient acquired using the same imaging modality The most common technique for aligning these images is to find a rigid body transformation. Prior to carrying out the rigid body registration, it is advisable to correct for any scaling or intensity errors in the images and it may be necessary to carry out additional preprocessing such as segmentation Image Registration Algorithms Voxel similarity measures applied to images from different modalities - entropy as a measure of alignment. while images from different modalities exhibit complementary information there is usually also a high degree of shared information between images of the same structures 25
Image Registration Algorithms Voxel similarity measures applied to images from different modalities - entropy as a measure of alignment. Any algorithm that is being used to register images from two different modalities must be insensitive to modality specific differences in image intensity associated with the same tissue and also accommodate differences in relative intensity from tissue to tissue Image Registration Algorithms Voxel similarity measures applied to images from different modalities - entropy as a measure of alignment. generate a scatter plot of these image intensities, point by point. These are two-dimensional plots of image intensity of one image against corresponding image intensity of the other. The resulting plot is a type of two-dimensional histogram. when divided by the number of contributing pixels, is equal to the joint probability distribution. 26
Image Registration Algorithms aligned MR and 18FDG PET 2mm translation 5mm translation Image Registration Algorithms Voxel similarity measures applied to images from different modalities - entropy as a measure of alignment. the joint probability distribution provides a number that is equal to the probability that those intensities occur together at corresponding locations in the two images 27
Image Registration Algorithms Mutual Information: Entropies of images A,B: H(A), H(B) Joint entropy of A,B: H(A,B) If A and B independent H(A,B) = H(A)+H(B) Else difference is MI -> Minimize difference Problem: estimate pdfs for H(A), H(B), H(A,B) Registration Framework Components: 28
Comparison of registration methods West J, Fitzpatrick JM, Dawant BM, et al. AIR Headmounted fiducials serves as ``Gold standard'' coregistration between the modalities (MR/CT/PET). Coregistration parameters are kept for reference, and fiducials are removed from the datasets and replaced by artificial noise. Methods are tested ``blindly'' - no knowledge of the Gold standard answer. Method Mean(mm) Med.(mm) Max.(mm) Method Mean(mm) Med.(mm) Max.(mm) Pluim 3.04 2.34 10.10 Rohlfing 3 3.72 3.40 10.00 Woods 1 2.67 2.35 5.81 Maintz 3.86 3.48 10.63 Rohlfing 5 3.15 2.60 8.09 Tanacs 1 5.31 3.56 16.80 Rohlfing 4 6.32 2.61 50.90 Luo 2 3.85 3.59 7.45 Rohlfing corr 4.09 2.62 23.45 Collignon 4.63 3.64 12.73 Rohlfing corr 12.29 2.74 115.57 Noz 4.57 3.64 11.37 Harkness 3.74 2.78 12.09 Ding 3.76 3.74 9.44 Pelizzari 3.36 2.89 9.99 Thevenaz 2 9.85 3.97 27.57 Thevenaz 3 4.52 2.98 14.34 Robb 3 4.21 4.01 9.44 Arata 2 3.00 3.01 5.43 Ashburner 2 4.14 4.20 7.46 Hsu 3.70 3.06 12.79 Malandain 4.24 4.21 8.54 Rohlfing 6 4.46 3.11 17.69 Nikou 1 4.24 4.29 8.62 Woods 2 3.00 3.14 5.97 Barillot 4.57 4.58 11.48 Ren 3.54 3.20 9.51 Thevenaz 1 4.86 5.03 10.13 Hill 3.50 3.25 9.32 Nikou 2 6.28 5.06 12.28 Tanacs 2 5.82 3.29 19.74 Ashburner 1 5.57 5.11 11.62 Robb 4 3.48 3.35 5.90 Capek 9.08 9.02 9.84 SPM Bibliografia Medical Image Registration. Edited by Joseph Hajnal, David Hawkes and Derek Hill. CRC Press, 2001 Links: AIR - http://bishopw.loni.ucla.edu/air5/ CISG - http://www-ipg.umds.ac.uk/cisg/vtk-software/ ImageJ - http://rsb.info.nih.gov/ij/ MIPAV - http://mipav.cit.nih.gov/index.php RView - http://www.colin-studholme.net/software/software.html MRIcro - http://www.sph.sc.edu/comd/rorden/ XMedCon - http://xmedcon.sourceforge.net/ 29