Registration and Fusion of Retinal Images An Evaluation Study

Transcription

1 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY Registration and Fusion of Retinal Images An Evaluation Study France Laliberté, Langis Gagnon*, Member, IEEE, and Yunlong Sheng Abstract We present the results of a study on the application of registration and pixel-level fusion techniques to retinal images. The images are of different modalities (color, fluorescein angiogram), different resolutions, and taken at different times (from a few minutes during an angiography examination to several years between two examinations). We propose a new registration method based on global point mapping with blood vessel bifurcations as control points and a search for control point matches that uses local structural information of the retinal network. Three transformation types (similarity, affine, and second-order polynomial) are evaluated on each image pair. Fourteen pixel-level fusion techniques have been tested and classified according to their qualitative and quantitative performance. Four quantitative fusion performance criteria are used to evaluate the gain obtained with the grayscale fusion. Index Terms Image fusion, image registration, medical imaging, ophthalmology. I. INTRODUCTION THE aim of this paper is to present the results of a study on 1) the development of a robust image registration algorithm and 2) the exploration of gray and color pixel-level fusion techniques to human retinal images of various types (i.e., with respect to color or angiographic modalities, resolution, disease, and image quality). Although image fusion is a standard topic in medical imaging, it has not been studied thoroughly for ophthalmic applications. Still much work has to be done in order to evaluate the potential benefits of the technique in ophthalmology. To our knowledge, only one paper [1] presents a feature-based fusion method where the coarse contours of anatomical and pathological features (vessels, fovea, optic disc, scotoma, subretinal leakage) are extracted from scanning laser ophthalmoscope images and superposed on the same image. Ophthalmology is one of the many medical areas for which diagnosis implies the manipulation and analysis of a large number of images, even for a single patient. For instance, mass screening of diseases like diabetic retinopathy can involve a Manuscript received May ; revised November 26, This work was supported in part by the Computer Research Institute of Montreal (CRIM), in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada, and in part by the Fonds québécois de la recherche sur la nature et les technologies. The Associate Editor responsible for coordinating the review of this paper and recommending its publication was M. W. Vannier. Asterisk indicates corresponding author. F. Laliberté is with the CRIM Montréal, QC H3A 1B9, Canada. He is also with the Département de Physique, Université Laval, Laval, QC G1K 7P4, Canada. *L. Gagnon is with the CRIM, rue Sherbrooke West, Suite 100, Montréal, QC H3A 1B9, Canada ( lgagnon@crim.ca). Y. Sheng is with the Département de Physique, Université Laval, Laval, QC G1K 7P4, Canada. Digital Object Identifier /TMI follow-up based on images acquired from different modalities (color, angiogram) over many years. Large parts of the image analysis process are still done by hand which contribute to increase the workload for the ophthalmologists. Therefore, any automatic image manipulation procedure that could help in reducing that workload or even suggest new ways to present the visual information is of potential interest. For instance, temporal registration can help to follow disease evolution while multimodal registration can help to accurately identify the type of lesions. On the other hand, image fusion can be used to combine or enhance pathological information to facilitate the diagnosis. Our study involves the development of a robust registration method in order to cope with the presence of various types of lesions associated with retinal diseases as well as the high visual quality variation between images. The method is essentially a global point mapping one but with a particular point matching search that takes into account the local structural information of the retinal network. Parameters of three transformation types are tested and a quantitative performance measure is given. Other registration methods have been proposed in the literature for retinal images. Correlation methods have been tested for unimodal ophthalmic image registration (see [2] [4]) but they cannot deal with local intensity reversal in multimodal registration ([5] and [6]). Since ophthalmic image deformations are mainly global, elastic model methods are unnecessary. Fourier methods are not suited because rotation and scaling can be present. In this study, a point-matching method is developed for temporal and multimodal registration of ophthalmic images. A point matching method has already been proposed for temporal and multimodal registration of mydriatic color images and fluorescein angiograms of the retina [7]. This method seems to give good results but ours does not require the assumption of a Gaussian shape vessel intensity profile, which is inappropriate for low-resolution optical images [8], and is much easier to implement. Former control point detection methods have been proposed in the literature that include edge detection [9] and color vessel reconstruction [10]. These methods have at least some drawbacks for our application, for example, coarse bifurcation point location or the requirement for color information. Fourteen pixel-level fusion techniques are tested: 12 grayscale [11] and two false color fusion methods ([12] and [13]). All have their qualitative advantages and drawbacks depending on the underlying medical context (application, disease type, subjective color appreciation, etc.). It is not our intention to identify the best method based on those criteria. However, we use some quantitative criteria (standard deviation, entropy, cross-entropy, and spatial frequency) that offer interesting performance indications for the grayscale fusion /03$ IEEE

2 662 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY 2003 TABLE I TECHNICAL CHARACTERISTICS OF THE FIVE DATA SETS USED methods. Our tests are performed on a large set of image pairs of different modalities (color, fluorescein angiogram), different resolutions, and taken at different times (from a few minutes during an angiography examination to several years between two examinations). The paper is organized as follows. Section II presents the data set used. Section III describes the registration technique we have developed and tested. Section IV describes the image fusion methods that have been selected and compared in this study. In Section V, we discuss the results obtained. Finally, conclusions and possible extension to this work are given in the last section. II. DATA SET Retinal images can be obtained with a fundus camera or a scanning laser ophthalmoscope. Fundus cameras have been used for this study. Images can be acquired under many different conditions. First, the pupil can be dilated with mydriatic eye drops. Second, to enhance the blood vessel appearance, a dye (sodium fluorescein) can be injected in the patient s arm, to produce angiograms. These are acquired under blue light because the dye has an absorption peak between 465 and 490 nm. Since the dye has an emission peak between 520 and 530 nm, a filter is placed in front of the camera to keep only the dye contribution. Images are acquired from about 10 s to 15 min after the injection, as the dye circulates through the retinal arteries, capillaries, veins, and is progressively eliminated from the vasculature, while staining in the optic disc and lesions. Finally, images can be taken on film and digitized, or directly with a digital camera. Five data sets, described in Table I, covering many eye diseases (diabetic retinopathy, age-related macular degeneration, anterior ischemic optic neuropathy, retinal vein occlusion, choroidal neovascular membrane, cystoid macular edema, histoplasmosis retinitis, telangiectasia, cytomegalovirus retinitis) are used in this study. The first three sets provide 12 image pairs, the fourth set provides 42 image pairs, and the two last sets provide 16 image pairs. For the color images, only the green band is used because it has the best contrast. Angiograms are very different from green band images: they have a better contrast and some features (blood vessels, microaneurisms) are intensity reversed. They are useful to detect leaks and other artery abnormalities, occluded capillaries, macular edema, microaneurisms and neovascularization. Color images are useful to detect exudates and hemorrhages. Acquiring angiograms at different phases is useful to follow the dye progression and staining which indicate the vasculature and lesion state. Angiograms taken at different phases are also very different, vessels go from black to white to black again eventually. It is important to say that our image data set has a very large variability in term of fundus disease and image quality (see all figures in the paper for specific examples). Some image pairs are of so bad quality regarding the visibility of the retinal network and/or the presence of large lesions (see Figs. 18 and 19 for examples) that we feel no retinal network-based registration techniques can work well on them. Our first idea was to discard those pairs like ophthalmologists do prior performing their diagnosis process. Such image quality check is often a critical step for the development of a reliable computer-aided diagnostic system [8]. Nevertheless, we finally decided to keep those bad-quality image pairs in the data set in order to get a clear idea about the robustness and limit performance of our registration algorithm. III. REGISTRATION Registration methods can be divided in four groups: elastic model, Fourier, correlation and point matching methods [5]. Here we describe the point matching method we have developed. In retinal imagery, image distortion comes from different sources: 1) change in patient sitting position (large horizontal translation); 2) change in chin cup position (smaller vertical translation); 3) head tilting and ocular torsion (rotation); 4) distance change between the eye and the camera (scaling); 5) three dimensional retinal surface (spherical distortion); and 6) inherent aberrations of the eye and camera optical systems. Four different transformation types have been used in the literature to register retinal images: translations ([3] and [4]), similarity transformation ([1], [2], and [14]), affine transformation ([6] and [7]), and second-order polynomial transformation ([15] and [16]). We propose to test the similarity, affine and second-order transformation types to determine which is the best (we reject the transformation composed only of translations because it is visually evident that it is not sufficient to correct the distortions). Our registration algorithm has ten parameters of which seven are dependent on the image resolution. The three left are the main free parameters. A. Control Point Detection A good point matching registration process requires that a sufficient number of corresponding control points be present in both images, uniformly distributed, and not affected by lesions. For these reasons, blood vessel bifurcation points are a natural choice. The control point detection involves two steps: retinal

3 LALIBERTÉ et al.: REGISTRATION AND FUSION OF RETINAL IMAGES AN EVALUATION STUDY 663 Fig. 1. Retinal network (left) and bifurcation point (right) detection algorithm. vessel centerline detection [Fig. 1 (left)] followed by bifurcation point detection [Fig. 1 (right)]. To extract the retinal vessel centerline, we first correct for nonuniform illumination by dividing the image by its medianfiltered version with a kernel of size. Second, a threshold is applied to binarize the image. In order to take into account the average intensity difference between the images, we do not use a fixed threshold but a Gaussian-constant false-alarm rate (CFAR) one satisfying (1) where and are the mean andstandard deviation ofthe image intensity and is a constant [17]. The constant is fixed for a given data set allowing to vary from image to image as and vary. Third, small circular objects like microaneurisms are eliminated by opening with a linear structuring element of size with orientation ranging from 0 to 180 with 15 steps [18]. Fourth, vessel segments are connected by dilating with a circular structuring element of size. Fifth, the image is thinned with a thinning algorithm which guarantees a one pixel width. Finally, a mask is used to hide the optic disc because itcontainstoo many interlaced blood vessels which would not provide reliable bifurcation points. The optic disc mask is automatically created offline using an Hausdorff-based template matching algorithm [19]. On this monopixel binary retinal network, bifurcation points are pixels with three or four neighbors [Fig. 2(a)]. Groups of three adjacent bifurcation points are fused because they correspond to the same bifurcation point [Fig. 2(b)]. Bifurcation points due to small vessels (maximum length ) are eliminated [Fig. 2(c)]. Bifurcation point pairs formed by the juxtaposition of two vessels are also fused maximum distance [Fig. 2(d)]. Finally, the angles of the vessels surrounding each bifurcation point are obtained by computing the intersection between the vessels and a circle of fixed diameter centered on each point [Fig. 2(e)] [7]. B. Point Matching and Transformation In an automatic registration procedure, one does not want to impose the constraint about which of the images is to be used as the reference one. The algorithm proposed in [7] for ophthalmic images provides a registration result that is not the same according to which is the reference image. Our matching proce- Fig. 2. Bifurcation point detection steps: (a) identification of pixels with three or four neighbors, (b) fusion of the groups of three adjacent points, (c) elimination of the points due to small vessels, (d) fusion of the point pairs formed by the juxtaposition of two vessels, and (e) computation of the angles of the vessels surrounding each bifurcation point. dure (Fig. 3) gives results that are independent of the choice of the reference image. Each bifurcation point in image 1 is linked to the bifurcation points in image 2 that are located inside a given distance, with the same number of surrounding vessels and a difference between the angles less than a given threshold [20]. Bifurcation points without matches are eliminated. We then use the following relaxation method to eliminate matches while estimating the transformation parameters. We have the choice to use the similarity, affine and second-order polynomial transformation in the matching process. We have tried all of them and found that the affine transformation gives the best results. Affine parameters are computed in the two directions (to register image 1 to image 2 and vice versa) and applied to the appropriate bifurcation points. A root-mean-square error (RMSE)

4 664 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY 2003 Fig. 3. Flow chart of our matching algorithm. TABLE II USED PARAMETER VALUES FOR TWO IMAGE RESOLUTIONS Fig. 4. Fusion architecture of the MIT method. Step 1 is a within-band contrast enhancement and normalization. Step 2 is a between-band fusion. between the transformed and the reference points is computed for each image. The average RMSE is then computed. Followed by an iterative process where: 1) each initial match is removed; 2) the affine parameters are recalculated; 3) the RMSE is reestimated; and 4) the match that gives the smallest average RMS is eliminated. An initial match is eliminated until the average RMS is smaller than a threshold or when there are only three matches left (minimum number of matches needed to calculate an affine transformation). After this step, if there are some bifurcation points with more than one match, the worst is eliminated until there is only one left. If there are more matches than needed to find an exact solution, the least mean square method is used to estimate the transformation parameters.

5 LALIBERTÉ et al.: REGISTRATION AND FUSION OF RETINAL IMAGES AN EVALUATION STUDY 665 Fig. 5. Example of color nonmydriatic (top-left) and reversed angiographic (top-right) images, retinal networks, and control points (circles). Fig. 7. Checkerboard images showing the registration of Fig. 5 with no (top-left), similarity (top-right), affine (bottom-left), and polynomial (bottom-right) transformation.. Then we compute for each. To determine which must be kept, we compare, for a certain number of images, the retinal network detected with the algorithm for different with the retinal network detected manually. The comparison criterion is the percentage of good vessels detected (true positive) minus the percentage of bad vessels detected (false positive). The length of the opening linear structuring element must be greater than the diameter of the largest circular feature to be removed(the largest microaneurisms have a diameter of 125 m). The size of the dilating circular structuring element must be large enough to connect the vessel segments. The other parameters (small branch length, fusion distance, orientation circle diameter, search distance, orientation difference, RMS threshold) have all been determined experimentally from a subset of the first three data sets. The values of all those parameters are given in Table II for two image data sets. Fig. 6. Color (red) and angiographic registered retinal networks of Fig. 5 with no (top-left), similarity (top-right), affine (bottom-left), and polynomial (bottom-right) transformation. We will now discuss the choice of the values for the ten parameters of the registration algorithm. The median filter size for the illumination correction must be large enough to eliminate the retinal structures and get an average illumination image. The Gaussian-CFAR threshold in (1) is estimated using the following procedure. First, we compute for where N is the number of images in a data set and C. Image Registration Performance Assessment In the fundus image literature, a visual performance assessment of registration is often done by creating a checkerboard of reference and transformed images [6] or by superposing transformed extracted vessels on the reference image [14]. Visual evaluation is useful but may be insufficient and too slow to compare the registration performance for different transformation types. We, thus, developed a quantitative criterion. A good registration is one with a good superposition of blood vessel centerline. However, retinal networks may be slightly different because the source images are different or the blood vessel detection is incomplete. To minimize this effect, the retinal network with least pixels is used as the reference. Superposition is defined as the presence of another network pixel into a fixed size window centered on a reference network pixel. A window of size unity corresponds to the true superposition definition. The larger the window is, the

6 666 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY 2003 TABLE III RESULTS FOR THE 45 IMAGE PAIRS (1) Number of control points, (2) number of matches, (3) registration performance (in %) without transformation, (4) with similarity, (5) affine, and (6) polynomial transformation, (7) fusion performance for the SIDWT method evaluated with the standard deviation, (8) entropy, (9) cross-entropy, and (10) spatial frequency criteria. larger the accepted registration error. We use a 3 3 window. The superposition percentage is the registration performance quantitative criterion. Of course, a 100% superposition can never be reached in practice because that would require that all pixels of the reference retinal network must be present in the other network and be within a one pixel distance. IV. FUSION A. Methods Twelve classical grayscale image fusion methods have been implemented in a Matlab toolbox [11]. We refer to [11] for details about those methods. Here, we only give a brief description. The classical methods can be classified in four groups: linear, nonlinear, image pyramids, and wavelet transform. Linear methods [average, principal component analysis (PCA)] consist in a weighted superposition of the source images. In nonlinear methods (select minimum, select maximum), nonlinear operators are applied that select the minimum or maximum intensity pixel between the images. In image pyramid methods (Laplacian, filter-subtract-decimate (FSD), ratio, contrast, gradient, morphological), each source image is decomposed in a pyramid representing the edge information at different resolutions. The fused image is obtained by combining those two pyramids. Wavelet transform methods [discrete wavelet transform (DWT) with Daubechies

7 LALIBERTÉ et al.: REGISTRATION AND FUSION OF RETINAL IMAGES AN EVALUATION STUDY 667 Fig. 8. Fused images of Fig. 5 with (from top to bottom and left to right) average, PCA, select minimum, maximum, laplacian, FSD, ratio, contrast, gradient, and morphological pyramid, DWT, and SIDWT methods.

8 668 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY 2003 TABLE IV GRAYSCALE FUSION METHODS CLASSIFIED IN DECREASING PERFORMANCE ORDER FOR EACH QUANTITATIVE EVALUATION CRITERION wavelets] are similar to image pyramid methods but constitute nonredundant image representations. For the multiresolution methods, one has to choose the number of decomposition levels (1 7), the high-pass coefficient combination method (max, salience/match measure [21], or consistency check [22]), the area used for the high-pass coefficient combination (3, 5, 7, 9), and the low-pass coefficient combination method (one of the source images or the average of both). We tested these 12 grayscale fusion methods as well as three additional approaches that generate color fused images. The first color approach consists in adding the red and blue bands to the grayscale fused image. The second and third are two false color image fusion methods recently proposed: the so-called The Netherlands Organisation for Applied Scientific Research (TNO) and Massachusetts Institute of Technology (MIT) methods ([12] and [13]). The TNO method is a false color mapping of the unique component of each source image satisfying where is the unique component of the first image, that is, the source image minus the common component of both images. The MIT method is based on a color perception model of the human visual system. Many different mappings have been proposed in the past according to the application. We choose the one proposed in [23] and illustrated in Fig. 4. This mapping is appropriate when fusion involves one image with a better contrast or a more natural appearance, as is the case for temporal and multimodal fusion. As a result, the better image is preserved by its mapping to the green band. Step 1 in Fig. 4 is a within-band contrast enhancement and normalization. Step 2 is a between-band fusion. The concentric circles in Fig. 4 represent a convolution operator defined by (2) (3) (4) where is the value of the th pixel of the processed image,,,, and are constants, and and are Gaussian weighted averaging masks. The other terms are described below. The small circles with the plus sign in Fig. 4 represent the excitatory center. The large circles with the minus sign represent the inhibitory surround. Arrows indicate which image feeds the center and which image feeds the surround of the operator. The upper-left operator in Fig. 4 is characterized by, the lower-left one by, the upper-right one by and ( and being the output images of the upper-left and lower-left operator, respectively), and the lower-right one by and. We use the parameter values suggested in [13]:,,,,, and. B. Image Fusion Performance Assessment In the literature, almost all image fusion evaluations are done qualitatively because of the user perception factor. This is particularly obvious for color image fusion since it is not easy to find a criterion to evaluate color information. However, there are some quantitative criteria that can be used [24]. Some need an ideal fused image (RMSE, mutual information, difference entropy) while others do not (standard deviation, entropy, cross-entropy, spatial frequency). We are restricted to the second group in the current study. The standard deviation and the entropy, respectively, measure the contrast and the information content in an image. The cross-entropy measures the similarity in information content between the source and the fused images. We use those criteria to evaluate the 12 grayscale fusion methods. We also evaluate the false color fusion methods qualitatively on the basis of the brilliance contrast, the presence of complementary features without attenuation, the enhancement of common features, and the color contrast. V. RESULTS A. Registration The registration algorithm has been tested on the 70 image pairs described in Section II. Among those image pairs, 9 have been used as a training set to determine some of the algorithm

9 LALIBERTÉ et al.: REGISTRATION AND FUSION OF RETINAL IMAGES AN EVALUATION STUDY 669 Fig. 10. Examples of angiograms taken at different stages and color image (bottom-right) of an eye with nonproliferative diabetic retinopathy. vessels) and Fig. 7 (checkerboard of reference and transformed image). On average, the registration performance is 22 6% (20 7%) (no transformation), 54 18% (52 19%) (similarity transformation), 56 19% (54 19%) (affine transformation) and 55 18% (54 19%) (polynomial transformation). We recall that to obtain a 100% score, all pixels of the reference retinal network must be present in the other network (which is highly unlikely) and be within a one pixel distance, which is not possible in practice. So, the registration improves the image alignment by 32 18% (32 18%) (similarity), 34 18% (34 18%) (affine), and 34 18% (35 18%) (polynomial). This indicates that, on average, the three transformation types are equally good. The affine transformation is significantly (difference greater than ) better than the similarity one in 10% (10%) of the cases and vice versa in 8% (7%) of the cases. The polynomial transformation is better than the affine one in 8% (9%) of the cases and vice versa in 10% (10%) of the cases. The polynomial transform is better than the similarity one in 10% (9%) of the cases and vice versa in 11% (11%) of the cases. Although the three transformation types are equally good on average, there is a significant number of cases [19% (19%)] for which a transformation type is better than another one. Fig. 9. Fused images of Fig. 5 with SIDWT, red and blue bands added (top), TNO (middle) and MIT (bottom) methods. parameter values. From now on the results in parentheses are those including this training set. The registration succeeded on 36 (45) image pairs. Results are detailed in Table III. Fig. 5 gives an example of a pair of source images along with the resulting thinned retinal networks and the detected control points. On average for each image pair, (90 21) control points were detected leading to 13 7 (14 7) good matches. The registration results for the images in Fig. 5 are shown in Fig. 6 (reference and transformed extracted B. Fusion The 14 fusion algorithms have been tested on the 45 registered image pairs. For each pair, we select the transformation that gives the best registration performance. For the multiresolution methods, we obtain the best results with seven decomposition levels, salience/match measure for high-pass coefficient combination, a value of three for the high-pass coefficient combination area, and the source image average for the low-pass coefficient combination (see [11] for details). Fig. 8 shows the fusion on image pair of Fig. 5 for the 12 grayscale fusion methods. The 12 grayscale fusion methods are evaluated with the four quantitative criteria mentioned in Section IV-B. Table IV shows the classification of the methods in decreasing order of performance for each criterion. The classification varies between the criteria because they do not measure the same image characteristics. If we classify the methods according to the sum of their

10 670 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY 2003 Fig. 11. methods. Fusion of images 1, 2, and 3 of Fig. 10 (top: 1 and 2, middle: 1 and 3, bottom: 2 and 3) with the SIDWT (left), TNO (middle), and the MIT (right) Fig. 12. Fusion of images 3 and 4 of Fig. 10 with the SIDWT (top-left), R and B bands added (top-right), TNO (bottom-left), and the MIT (bottom-right) methods. order of performance for the four criteria, we obtain: laplacian, shift-invariant discrete wavelet transform (SIDWT), contrast, morphological, DWT, PCA, FSD, select maximum, select minimum, ratio, gradient, and average. For example, for the SIDWT method, the average standard deviation is for the source images and for the fused images. Thus, the contrast is improved by 4 4. The average entropy is for the source images and for the fused images. Thus, the information content is improved by The average cross-entropy is for the fused images. The average spatial frequency is Fig. 13. Examples of angiograms taken at different stages and color image (bottom-right) of an eye with cystoid macular edema for the source images and 19 7 for the fused images. Thus, the spatial frequency is improved by 4 2. Fig. 9 shows the color fusion of the same image pair. The fused images gathered the exudates (yellow lesions) of the color image and the microaneurisms (black dot lesions) of the angiogram. The R and B bands addition gives realistic yellow color to the optic disc and exudates. The TNO-fused image shows in red what comes from the color image, in green what comes from the angiogram and in blue the difference between the two images. The MIT-fused image shows in red what comes from the color image too, in blue what comes from the angiogram and in green the normalized and contrast enhanced angiogram.

11 LALIBERTÉ et al.: REGISTRATION AND FUSION OF RETINAL IMAGES AN EVALUATION STUDY 671 Fig. 14. methods. Fusion of images 1, 2, and 3 of Fig. 13 (top: 1 and 2, middle: 1 and 3, bottom: 2 and 3) with the SIDWT (left), TNO (middle), and the MIT (right) Fig. 15. Fusion of images 3 and 4 of Fig. 13 with the SIDWT (top-left), R and B bands added (top-right), TNO (bottom-left), and the MIT (bottom-right) methods. Fig. 10 shows four images of an eye with nonproliferative diabetic retinopathy from the fourth data set: angiograms taken 17, 25, and 640 s after the injection, plus a color image. Fusion results of the angiograms are shown in Fig. 11. The middle and bottom rows show the fusion results obtained, respectively, with the TNO and MIT methods. From left to right, we have the fusion of the first and second, first and third, and second and third angiograms. In the first case, we see the progression of the dye by the laminar filling of the veins. The second case allows us to see the progression of the dye and the lesions very well by the evident staining. The third case is the one where the lesions are the best defined. Fig. 12 shows the fusion of the third angiogram and the color image with the SIDWT (without and with the R and B bands added), TNO, and MIT methods. Figs are the equivalent of the three former figures but for an eye with cystoid macular edema imaged at 18, 22, and 324 s after the injection, respectively. Similarly, Figs. 16 and 17 show the results for an eye with age-related macular degeneration. We can see on these figures that the TNO method provides a better color contrast than the MIT one. VI. CONCLUSION We have presented the results of a study on the application of image registration and pixel-level fusion techniques between

12 672 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 22, NO. 5, MAY 2003 Fig. 16. Angiograms taken at different stages of an eye with age-related macular degeneration. Fig. 17. Fusion of images 1, 2, and 3 of Fig. 16 (top: 1 and 2, bottom: 2 and 3) with the SIDWT (left), TNO (middle), and the MIT (right) methods. pairs of human retinal images of different modalities (color, fluorescein angiogram), different resolutions, and taken at different times (from a few minutes during an angiography examination to several years between two examinations). We proposed an improved registration algorithm for the temporal and multimodal registration of low- and high-resolution retinal images which can be used for any global transformation type. We have tested this algorithm on 61 (70) image pairs of different modalities, eye diseases, and taken at different times. The registration performance has been evaluated with an overlap-based criterion which indicates that the similarity, affine, and second-order polynomial transformations are equally good on average for registering the images, although there is a significant number of cases for which a transformation type is better than another one. We presented the results of 12 grayscale and two false color fusion methods. The fusion performance of the grayscale methods has been evaluated with four quantitative criteria. The fusion provides an average entropy gain of , a standard deviation gain of 4 4, and a spatial frequency gain of 4 2 indicating that the fused images contain more information and have a better contrast than the source images. We have also found that the addition of the R and B bands to the grayscale fused images, provides additional information of potential interest for medical diagnosis. Finally, our tests indicate that the TNO false color mapping method is better than the MIT one regarding the color contrast rendering. Although extensive testing has been done during this work, there are many issues that remain open and should be addressed in the future. Let us mention two of them. First, the robustness of the image registration. Retinal image appearance is highly variable in practice regarding to color, texture, structure, and lesions. This is due to many factors such as human color pigmentation, age, diseases, eye movement, camera calibration, film defects, etc. Developing a robust registration algorithm that can cope with those practical constraints was the main challenge of this work. We failed to register 25 (25) of the 61 (70) image pairs because they are of very bad quality (very few vessels due to illumination problem, zoom on the macula or vessel deterioration caused by the disease). An example is shown in Fig. 18. Also images that are taken at the late angiography stage have vessels that are hardly visible because the dye has left them. Since the vessels are about the same intensity as the background they cannot be extracted. An example is shown in Fig. 19. Although those image pairs are of bad quality in term of network visibility, clinicians can still use them for diagnosis. Thus, there is room for improving the registration robustness but other reference structure seems to be necessary when network quality is poor. The second issue is the color information carried out by the color fusion technique. Color information is an important feature for the diagnosis of some diseases (e.g., diabetic retinopathy). It is clear that the false colors created by the fusion process carry information that cannot be interpreted easily by the clinician at this point. One can then ask: How fused color

13 LALIBERTÉ et al.: REGISTRATION AND FUSION OF RETINAL IMAGES AN EVALUATION STUDY 673 ACKNOWLEDGMENT The authors are would like to thank M.-C. Boucher from the Maisonneuve-Rosemont Hospital (Montreal, QC, Canada) for providing some of the images and clinical inputs. They would also like to thank the reviewers for their valuable comments that help improving the quality of the paper. Fig. 18. Image pair that the algorithm failed to register because one of the images shows very few vessels (bottom image). Fig. 19. Image pair that the algorithm failed to register because one of the images is taken at the late stage (bottom image). information can be adapted to be human readable or used in a further computer-aided diagnosis processing? This is an important issue that necessitates a close relationship with clinicians. We plan to address those issues, and others, in the near future. REFERENCES [1] A. Pinz, S. Bernögger, P. Datlinger, and A. Kruger, Mapping the human retina, IEEE Trans. Med. Imag., vol. 17, pp , Aug [2] A. V. Cideciyan, Registration of ocular fundus images by cross-correlation of triple invariant image descriptors, IEEE Eng. Med., Biol. Mag., vol. 14, pp , [3] E. Peli, R. A. Augliere, and G. T. Timberlake, Feature-based registration of retinal images, IEEE Trans. Med. Imag., vol. MI-6, pp , June [4] J. J.-H. Yu, B.-N. Hung, and C.-L. Liou, Fast algorithm for digital retinal image alignment, in Proc. Int. Conf. IEEE Engineering in Medicine and Biology Society, vol. 11, 1989, pp [5] L. G. Brown, A survey of image registration techniques, ACM Computing Surveys, vol. 24, no. 4, pp , [6] N. Ritter, R. Owens, J. Cooper, R. H. Eikelboom, and P. P. van Saarloos, Registration of stereo and temporal images of the retina, IEEE Trans. Med. Imag., vol. 18, pp , May [7] F. Zana and J. C. Klein, A multimodal registration algorithm of eye fundus images using vessels detection and Hough transform, IEEE Trans. Med. Imag., vol. 18, pp , May [8] M. Lalonde, L. Gagnon, and M.-C. Boucher, Non-recursive paired tracking for vessel extraction from retinal images, in Proc. Vision Interface, 2000, pp [9] D. E. Becker, A. Can, J. N. Turner, H. L. Tanenbaum, and B. Roysam, Image processing algorithms for retinal montage synthesis, mapping, and real-time location determination, IEEE Trans. Biomed. Eng., vol. 45, pp , Jan [10] V. Rakotomalala, L. Macaire, J.-G. Postaire, and M. Valette, Identification of retinal vessels by color image analysis, Machine Graph. Vis., vol. 7, no. 4, pp , [11] O. Rockinger and T. Fechner, Pixel-level image fusion: the case of image sequences, Proc. SPIE, vol. 3374, pp , [12] A. Toet and J. Walraven, New false color mapping for image fusion, Opt. Eng., vol. 35, no. 3, pp , [13] A. M. Waxman, D. A. Fay, A. N. Gove, M. C. Seibert, and J. P. Racamato, Method and apparatus for generating a synthetic image by the fusion of signals representative of different views of the same scene, U. S. Patent , [14] J. W. Berger, M. E. Leventon, N. Hata, W. M. Wells, and R. Kikinis, Design considerations for a computer-vision-enabled ophthalmic augmented reality environment, presented at the CVRMed/MRCAS, Grenoble, France, [15] A. Can, C. V. Stewart, and B. Roysam, Robust hierarchical algorithm for constructing a mosaic from images of the curved human retina, in Proc. Computer Vision and Pattern Recognition, 1999, pp [16] P. Jasiobedzki, Registration of retinal images using adaptive adjacency graphs, in Proc. Computer-Based Medical Systems, 1993, pp [17] L. Gagnon, M. Lalonde, M. Beaulieu, and M.-C. Boucher, Procedure to detect anatomical structures in optical fundus images, Proc. SPIE, vol. 4322, pp , [18] G. Yang, S. Wang, L. Gagnon, and M.-C. Boucher, Algorithm for detecting micro-aneurysms in low-resolution color retinal images, in Proc. Vision Interface, 2001, pp [19] M. Lalonde, M. Beaulieu, and L. Gagnon, Fast and robust optic disk detection using pyramidal decomposition and Hausdorff-based template matching, IEEE Trans. Med Imag., vol. 20, pp , Nov [20] F. Laliberté, L. Gagnon, and Y. Sheng, Registration and fusion of retinal images: a comparative study, in Proc. Int. Conf. Pattern Recognition, 2002, pp [21] P. J. Burt and R. J. Kolczynski, Enhanced image capture through fusion, in Proc. 4th Int. Conf.Computer Vision, 1993, pp [22] H. Li, B. S. Manjunath, and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graph. Models Image Processing, vol. 57, no. 3, pp , [23] M. Aguilar and A. L. Garrett, Neurophysiologically-motivated sensor fusion for visualization and characterization of medical imagery, presented at the Fusion 2001, Montreal, QC, Canada. [24] Y. Wang and B. Lohmann, Multisensor image fusion: Concept, method, and applications, Univ. Bremen, Bremen, Germany, Tech. Rep., 2000.