1 Introduction Image fusion, in a clinical setting, is defined as the effective and meaningful integration of data from associated images to improve d

Transcription

1 Likelihood Maximization Approach to Image Registration Yang-Ming Zhu Λ Member, IEEE and Steven M. Cochoff Nuclear Medicine Division, Marconi Medical Systems 595 Miner Road, Cleveland, Ohio Abstract Alikelihood maximization approach to image registration is developed in this paper which does not rely on data reduction, requires no segmentation, and involves no user interactions. It is assumed that the voxel values in two images in registration are probabilitically related. The principle of maximum likelihood is then exploited to find the optimal registration: the likelihood that given image f, one has image g and given image g, one has image f is optimized with respect to the registration parameters. All voxel pairs in the overlapping volume or a portion of it can be used to compute the likelihood. Aknowledge-based method and a self-consistent technique are proposed to obtain the probability relation. The accuracy and robustness of the likelihood maximization approach isvalidated by single modality registration of single photon emission computed tomographic (SPECT) images and magnetic resonance (MR) images and by multimodality registration (MR/SPECT). The results demonstrate that the performance of the likelihood maximization approach to image registration is comparable to that of the mutual information maximization. Finally the relation of the likelihood approach withentropy, conditional entropy, and mutual information approaches is discussed. Index terms: Image registration, likelihood maximization, probability, mutual information. Λ yang-ming.zhu@marconi.com 1

2 1 Introduction Image fusion, in a clinical setting, is defined as the effective and meaningful integration of data from associated images to improve diagnostic accuracy, assess lesion progression or treatment effectiveness, and aid surgical and/or radiotherapeutic planning. In the case of multi-modality image fusion, it is useful since different imaging modalities provide information that tends to be complimentary in nature. For example, computed tomography (CT) and magnetic resonance (MR) imaging primarily provide anatomic information while single photon emission computed tomography (SPECT) and positron emission tomography (PET) provide functional and metabolic information. Thefirststepof image fusion is to bring the images into spatial alignment such that they are unambiguously linked together, a procedure referred to as image registration. Image registration algorithms have been extensively investigated in the past decades. For a recent review of this area and classifications of these algorithms from a medical imaging perspective, please refer to [1]. Work done in the area of three-dimensional registration before 1993 is reviewed in [2, 3], and for twodimensional registration in [4]. Fitzpatrick and his colleagues evaluated 13 algorithms visually as well as with an external fiducial marker-based registration which typically has a minimum accuracy of 0.5 mm [5, 6, 7]. More recently, two different imaging modalities have been combined in a single imaging device [8]. Given this hardware approach to image registration, the question arises as to the continued need for software registration techniques. It is our opinion, that software image registration will continue to play a vital role in many instances and that the development of registration algorithms shall remain an important research areaforyears to come. In many cases, hardware registration is impractical or impossible and one must rely on software-based registration techniques. For example, when monitoring treatment effectiveness over time, software image registration is necessary since the single or multi-modality images are acquired at different times. In addition, applications involving inter-subject or atlas comparisons require software registration since the images originate from 2

3 different subjects. Other applications for software registration include the correction of motion that occurs between sequential transmission and emission scans in PET and SPECT as well as the positioning of patients with respect to previously determined treatment plans. Even if the hardware approach becomes commercially successful, the affordability remains an issue. The need to offer multiple different combinations of imaging modalities (i.e. PET/MR, SPECT/MR, PET/CT, etc) would quickly become impractical. We have developed a new image registration algorithm based on the likelihood maximization principle that will be the subject of this paper. In recent years, full volume-based registration algorithms have become popular since they do not rely on data reduction, require no segmentation, and involve little or no user interaction. More importantly, they can be fully automated and provide quantitative assessment of registration results. Entropy-based algorithms, the mutual information approach in particular, are among the most prominent ofthe full volume-based registration algorithms. Maintz and his colleagues have provided a summary of the entropy-based algorithms [1]. Most of these algorithms optimize some objective function that relates the image data from two modalities. Our newly developed likelihood maximization approach belongs to this category. The basic assumption underlying our approach isthatthevoxel values in two registered images are probabilitily related, regardless of whether they are both from the same modality or are from different modalities. This probability relation can be obtained based on previous experimental observation or can be estimated solely on the volume data in a self-consistent way. Based on this probability relation, given one volumetric image, the likelihood of having the other image is computed. When this likelihood has its maximal value, the two images are considered to be registered. When two volumes are in registration, the normalized maximal likelihood has a probabilityinterpretation that is far less abstract than mutual information or entropy. The likelihood has a lower bound of zero and an upper bound of one. This is in contrast to the entropy or mutual information techniques which have fixed low bounds but unfixed upper bounds which may be image content dependent. In addition, the maximum likelihood registration is symmetric, i.e. rather than saying image f is reg- 3

4 istered to image g or vice versa, we say images f and g are in registration. This new algorithm can also easily incorporate segmentation whereby the probability relation can be estimated as stated above, but the likelihood is calculated using a subset of the volume data. This subset can be based on spatial segmentation of the volume or on voxel-value based segmentation. By emphasizing the importance of a subset of the volume, one would expect a better registration result in some cases. This flexibility isnotavailable in most of today's full volume-based methods. The remainder of this paper is organized into the following sections. Section 2 presents the theory of the likelihood maximization approach. Section 3 discusses two different methods that can be used to calculate the likelihood. Other implementation issues including optimization strategies are discussed in Section 4. Section 5evaluates the accuracy and robustness of this new algorithm by illustrating a set of single and multi-modality registration examples. Section 6 discusses the implementation aspects and relation of this algorithm to other entropy-based algorithms. Section 7 concludes the whole paper. 2 Theory Assume two images f(x; y; z) andg(x; y; z) with gray scales (u1;u2; ;u n ) and (v1;v2; ;v m ). In a single modality case due to noise or changes in the imaged object itself, or in a multi-modality case due to the intrinsic properties of the different modalities, a gray scale value u i in image f can correspond to gray scale value v j in image g; j =1; 2; m. A similar idea was exploited by Wells et al [9] to illustrate that the joint entropy in alignment is a local minimum. We also note that, a PET image was simulated (mapped) from a MR image by using a simple segmentation scheme and assigning a plausible radioactivity toeachsegment[10]. In the maximum likelihood approach, we do not assume any concrete relationship between the voxel values in different modality images nor impose any constraint on the image content of the modalities involved. Rather we describe their relationship on a purely statistical basis that could be knowledge-based or estimated from the 4

5 involved data in a self-consistent way as will be discussed in Section 3. Let the transition probability foravoxel u i to v j be w ij. Obviously, P m j=1 w ij =1: Given the image f and the transition probability w ij, the conditional likelihood that one has image g is L 0 gjf = Y w ij : Since w ij» 1, this product can be very close to zero. To ease the computation, its logarithmic value is used instead. The logarithmic likelihood is just LL 0 gjf = X log w ij Note that the logarithmic likelihood has a maximum value under some condition if and only if the likelihood has a maximum value under the same condition. The likelihood can be calculated using the entire overlapping volume or a subset of this volume (utilizing only selected (u i ;v j ) pairs or (u i ;v j ) pairs from selected regions/volumes). Thus the above-defined logarithmic likelihood depends on the number of gray value pairs. To avoid this dependence, a normalized logarithmic likelihood is used: LL gjf = 1 N LL0 gjf = 1 X log wij ; N where N is the number of gray value pairs. N is the total overlapping volume in number of voxels if all pairs in the overlapping volume are used to calculate the logarithmic likelihood. Considering the voxel transition from g to f, one has LL f jg. Adding these two pieces of logarithmic likelihood together (multiplication of likelihood) gives us LL = LL gjf + LL f jg (1) The likelihood maximization approach to image registration is thus to find a transformation that maximizes the logarithmic likelihood. Hereafter the normalized logarithmic likelihood defined in Eq. (1) is used. If two volume images are registered, the normalized likelihood has a probability interpretation, which hasalow bound 0 and a upper bound 1. The normalized likelihood is the average chance that 5

6 a voxel in the first image corresponds to the voxel in the second image and a voxel in the second image corresponds to the voxel in the first image. By looking at the optimized likelihood, one has a quantitative estimate of the registration quality. If an image is registered to itself, the normalized likelihood is 1. With this in mind, one would then expect that intra-modality registration would result in a higher normalized likelihood than in cases of inter-modality registration. 3 Implementations As we discussed above, the likelihood maximization approach to image registration needs to know the transition probability for a voxel value in one image to a voxel value in another image. Two implementations of this likelihood maximization approach, knowledge-based and self-consistent, are discussed in this section. 3.1 Knowledge-based implementation Based on previous experience, one may have the statistics on the transition probabilities between the gray scale pairs (u i ;v j );i=1; 2; ;n;j =1; 2; ;m. One can change the relative orientation and position of these two images, and based on the gray scale pairs (u i ;v j ), compute the logarithmic likelihood using the prior transition probabilities. By some optimization technique, an optimal registration can be found. 3.2 Self-consistent implementation If prior knowledge of the transition probability is not available, one can implement thelikelihood maximization approach in a self-consistent fashion as discussed in this subsection. We call it selfconsistent since the overlapping volume produced by the current registration is used to estimate the transition probability which is then used to calculate the likelihood. The optimized likelihood then yields the optimum registration. 6

7 Under the current registration, the transition probabilities can be estimated. The gray value pairs (u i ;v j )arechecked over the whole overlapping volume. Given the pixel value u i, the transition probability can be estimated in the following manner: Assume there are N i pairs (u i ;v j );j = 1; 2; ;m, among which aren ij pairs for (u i ;v j ). Then w ij is estimated as w ij = N ij N i Similarly, one can estimate the transition probabilities, given pixel value v j. Given these transition probabilities and the voxel value pairs (u i ;v j ), the computation of the logarithmic likelihood becomes a simple matter. 3.3 Implementation using segmented volume data In both implementations it is not necessary to calculate the likelihood over the whole overlapping volume. As a matter of fact, the likelihood can be calculated over a pre-selected set of points or a portion of the overlapping volume which may be segmented. The emphasis can also be placed on the voxel values, for example, the voxels whose values are in the upper 25%. By emphasizing specific segments, one would expect improved registration accuracy. 4 Algorithm The details of the algorithm are discussed in this section. They are generally applicable to both implementation as well as their variations using segmentation. The difference will be pointed out wherever appropriate. 4.1 Transformation In this paper we restrict the transformation to rigid-body transformations although the likelihood maximization approach can be applied to more general transformations as well. For a rigid-body transformation, the registration parameter is a six-dimensional vector, ( x ; y ; z ;t x ;t y ;t z ), where 7

8 x ; y,and z are rotation angles in degrees around the x-, y-, and z-axis respectively and t x ;t y,and t z are translational offset in mm along the x-, y-, and z-axis respectively. For each rotation, there is a corresponding 4 4 matrix in a homogeneous coordinate system [12]. A successive application of the rotation amounts to matrix multiplication. Since the matrix multiplication is not commutative, the order of these rotations is important. It is assumed in this paper that the rotations are applied around the x, y, and z-axis, in that order. We also assume that the rotation is applied before the translation. 4.2 Interpolation After a transformation is applied, a grid point in one volume will typically not coincide with another grid point in the transformed space. Since the voxel values of the reference image are given at grids, to compute the likelihood, one needs to interpolate the voxel values at grids in the transformed space of the floating image. There are different interpolation methods: nearest neighbor, tri-linear, and tri-linear partial volume distribution [13]. Since it is insensitive to the translation up to one voxel, the nearest neighbor interpolation is not sufficient in order to achieve sub-voxel accuracy. For simplicity the tri-linear interpolation was used in our study. 4.3 Computation of logarithmic likelihood In the knowledge-based implementation, to compute the logarithmic likelihood, one can scan the overlapping volume or part of it satisfying specific segmentation requirements, and add up the logarithmic likelihood on the fly. Alternatively, one can collect the information on the frequencies of voxel pairs and then compute the logarithmic likelihood. We use the latter approach since less computations are needed. In that latter case, the voxel pair frequencies can be stored in a 2-d array. In the self-consistent implementation, to compute the logarithmic likelihood, one needs to know the transition probabilities, which can be estimated from marginal and joint histograms, as dis- 8

9 cussed in Section 3.2. In the calculation of logarithmic likelihood, one can also store in a 2-d array the information on voxel pair frequencies. Note that, if the volumes of interest are the whole overlapping volumes or are specified in the voxel values, the above-mentioned collective information can be derived from the joint histogram. Otherwise this 2-d array has to be different from the joint histogram array. To compute the marginal and joint histograms, the maximum voxel value of image f is first found. The voxel values in image f are then divided into n f discrete levels. Similarly, the voxel values in image g are divided into n g discrete values. Here n f and n g can be different. In the overlapping volume, the histograms of voxel values in images f and g, and of the voxel pairs are calculated by binning the voxel values and value pairs. The bin size of the histogram for f is n f, the bin size of the histogram for g is n g, and the bin size of the joint histogram is n f n g. In our software implementation, we keep n f = n g = 64 in almost all the cases (except the knowledge-based implementation). Since the transition probabilities are associated with the binning method, the same binning information is used to compute the frequencies of voxel pairs. As mentioned earlier, under some circumstances, the work can be simplified. 4.4 Optimization Under a transformation, Powell's multidimensional direction set optimization is used to minimize the negated logarithmic likelihood (maximize the likelihood), using Brent's method in onedimensional search [11]. The direction matrix is initialized to a unitary matrix. The vector is ( x ; y ; z ;t x ;t y ;t z ), as explained before. A different order of these registration parameters is possible which may improve the optimization speed [13]. We did not try to optimize the parameter order in this paper since the order may be image content dependent and an exhaustive trial seems impractical (there are 6!=720 different combinations although one may just try a subset of them). Furthermore, Powell's optimization may useother six independent directions which does not nec- 9

10 essarily correspond to the six directions one intended as the search proceeds. 4.5 Multiresolution Powell's optimization method cannot guarantee that a global optimal solution is to be found. It can be easily trapped by a local optimal solution. To find a true global optimal value, simulated annealing [11] can be exploited. Simulated annealing has been successfully applied to 2-dimensional image registration [14]. It is a stochastic method and is slow, which limits its application to 3- dimensional image registration. In practice, the multiresolution or subsampling approach proves to be helpful. It is a robust algorithm that can improve the optimization speed and increase the capture range [15]. We used the multiresolution optimization in our implementation. The images are folded down to an image as the most coarse image. The resolutions of the successive images are doubled until the full image resolution is reached in all three dimensions. When the volume is large, such as , the computation is demanding and the registration is slow. In some cases we did not use the full resolution images. Instead, the finest image was or , which can still yield a very good registration, as we will see later. To obtain the coarse images, the voxel values within a sampling volume are averaged. Although it is a little slower than the subsampling approach, in practice it gives a better registration result. 5 Examples In this section the likelihood maximization approach to image registration is evaluated. A SPECT and a MR dataset are used as test volumes. The SPECT image and the MR images were selfregistered after various misregistrations were introduced, with and without the presence of noise in the voxel values. Then the SPECT volume was registered to the MR volume. The registration results are assessed. All those registrations were done by using the Image Volume Registration soft- 10

11 ware (Nuclear Medicine Division, Marconi Medical Systems) with some appropriate modifications. The workstation used was a Digital AlphaStation with 192 MB memory. The operating system was Digital UNIX, V.4.0D. 5.1 Image data description The image data consisted of slices. The x-axis is directed horizontally from right to left, the y-axis horizontally from front toback, and the z-axis vertically from bottom to top. The patient is a 21-year male. Technetium-99m hexamethyl-propolamine-oxime (HMPAO Tc- 99m) was used as the pharmaceutical for the SPECT image acquisition. The acquisition was performed using the Picker Prism 3000XP camera. The image has a size of voxels with a voxel size of 7:12 7:12 7:12mm 3. The minimum voxel gray value is 0 and the maximum voxel value is The average voxel gray value is The MR image (T1 saggital) has voxels with a voxelsizeof1:0 1:0 1:5mm 3. The minimum voxel value is 0 and the maximum voxel value is 504, with an average voxel value of Single modality registration We first registered an image to itself, from various starting misregistrations. In this single modality registration, the self-consistent implementation is exploited and all voxel pairs in the overlapping volume are used to compute the logarithmic likelihood. To assess the robustness against noise, one image was corrupted by different levels of noise and the registration was done on the same set of randomly generated misregistrations. To generate randomly misregistered image pairs, an image was first rotated around the x-axis by an angle, then by the y-axis by another angle, and by the z-axis by yet another angle. These angles have a uniform distribution over a certain range. The rotated image was then translated to a new position. The offsets in the x, y, and z directions have a uniform distribution over some 11

12 range. A white noise was added to the image to generate a corrupted image. The noise follows a Gaussian distribution with a mean value of zero and a standard deviation. For these experiments with known ground truth, the registration results are then inspected. As a well-trained observer can detect a translational misregistration in the x- and y-axis of 2 mm or more, in the z-axis of 3 mm or more, and a rotational misregistration around the z-axis of 2 o or more and around the x- and y-axis of 4 o or more [16], the registration is regarded as a failure if any of the misregistration parameters is beyond these ranges Nuclear image registration A set of 50 randomly misregistered volume pairs were registered. To generate those misregistrations, one image was rotated around x-, y- and z-axis. Those rotation angles were uniformly distributed over [-20, 20] degrees. The image was then translated. The translation offsets along the three axes were uniformly distributed over [-56.96, 56.96] mm. For the nuclear image without noise, the likelihood approach fails 10 out of 50 times. For a comparison, those starting misregistered volumes were also registered by mutual information maximization. The mutual information approach fails 41 out of 50 times. For those successful registrations, the average mis-rotations, mis-translations as well as their standard deviations of those registration parameters are calculated and given in Table 1 under heading N(0; 0). The misregistration parameters are defined as the difference between the actual one and the computed one. They all achieved a subvoxel accuracy, with remarkably small bias and deviations. The likelihood maximization method is about 25% faster than the mutual information approach. When the images are registered, the likelihood is exactly one, showing the voxel has a one-to-one correspondence. To assess the robustness of the likelihood maximization algorithm, the image was corrupted by additive noise. The registration was done again with the same set of initial misregistrations. 12

13 The statistics of the misregistration parameters are given in Table 1, again for the successful registrations. The Gaussian noise has a variance of 50 and 100, respectively. It is worth noting that these algorithms behave well even with the presence of a noise as strong as N(0; 100). The robust behavior is probably due to the strong signal present in the image (The average gray value is 163.8). When the noise is N(0; 50), the likelihood is at registration when the resolution is and is when the resolution is Similarly, when the noise is N(0; 100), the likelihoods are and respectively. The voxel range is very large, so each bin can accommodate a relatively large voxel gray value. As a consequence, the noise corrupted voxel may fall into the same bin as the noise-free voxel. That is the reason why wehave a large likelihood here. When the coarse image is used, the voxel is smoothed. That explains the higher likelihood in coarse image. The data here also reveals that higher noise reduces the likelihood MR image registration The same experiment was done on the MR image. However, the rotation angles are uniformly distributed over [-30, 30] degrees and the translational offsets are uniformly distributed over [-24, 24] mm. The MR volume has voxels. As we mentioned earlier, if the full resolution image is used, the registration speed is slow (ca. 1 hr). Thus, the maximum resolution used is It turns out that it works quite well as the following data shows. The statistics of the misregistration parameters are shown in Table 2. The success rates are in the same order although mutual information correctly registered all the image pairs. We did not study how the resolution would affect the registration results since all failed registrations have a large translation offset such that there is no overlapping volume. This situation will not change in the further finer resolution optimization. For the noise-free images in registration, the likelihood is one as expected. For the noised 13

14 images in registration, when the resolution is , it is 0.082, and when the resolution is , As in the previous example, the likelihood is increased when the resolution is decreased. We also note the likelihood here is smaller than that in the previous example since the average voxel gray value is lower here and the influence of the noise is more remarkable. 5.3 Multimodality registration For the multimodality registration, the correct registration parameters are unknown. Various evaluation methods are used to assess its accuracy, including phantom validation, observer assessment [16], fiducial marks [5, 6, 7], among others. Here we use a modified observer assessment approach. The MR/SPECT image pair was first registered by four clinical experts using an interactive (manual) registration method which is available in the software. Their average results were then used as a reference. Three sets of misregistrations were generated. Each set has 50 misregistration configurations generated around the manual reference. For Set 1, the rotation angle differences are uniformly distributed over [-10, 10] degrees and the translation offset differences are uniformly distributed over [-10, 10] mm. For Set 2, the angle and translation differences are uniformly distributed over [-20, 20] degrees (mm). For Set 3, those differences are uniformly distributed over [-30, 30] degrees (mm). Since the ground truth is unknown, the manual result is not reliable, and each registration criterion may have its own optimum solution, we compare the optimal solution of the likelihood maximization approach to other solutions to evaluate its accuracy, and study the distribution of resultant registration parameters to assess its robustness. It is preferred to use exhaustive search to pinpoint the true optimum registration of the likelihood maximization. But it is time consuming and beyond our capacity. Instead we use the mean of successful registration in each set as an approximation to the optimum result, assuming the registration results are aggregated around the optimal solutions. We did not use the average value of all registration results since the statistic outliers generated by failure registrations may have an adverse effect. To judge if a 14

15 registration succeeds or not, we use the threshold vector (4 o, 4 o, 2 o, 2mm, 2mm, 3mm) which is the misregistration detection threshold of a trained clinician, and if the absolute difference between any registration parameter and its median value is larger than the corresponding threshold, the registration is considered a failure. The mean and standard deviation of successful registrations are calculated and reported. We compare the registration results of likelihood maximization to those of manual registration and mutual information maximization. The reported misregistration parameters are measured against the manual results Knowledge-based When prior knowledge on the voxel value transition probabilities is available, one can directly use this information in the calculation of likelihood. Although this prior knowledge is hard to gather, we report an artificial experiment here to validate the idea. The averaged manual registration gives a good match. We compute the marginal and joint histograms under this set of parameters, from which the transition probabilities are estimated. All the voxel pairs in the overlapping volume are taken into account to compute the likelihood. As one may expect, some transition probabilities are zero which can cause problem in the likelihood calculation. To avoid this situation, one can assign an arbitrarily small value to the transition probability. Alternatively, one can assume a parametric form of the transition probability. Based on the available points the shape of the transition probability is estimated and the missing transition probability can be interpolated. Unfortunately there is no justification for a certain shape of the transition probability, particularly in the multimodality cases. One can also ignore the occurrence of the voxel pairs in the likelihood computation if their transition probabilities are zero. We found that this latter approach has a low success rate. Rather than conjecturing a parametric form of the transition probability, we assign a small number to the transition probability if it is zero. If 64 bins are used, a uniform transition probability would be We arbitrarily replace the 15

16 zero transition probabilities with the one hundardth of this value. We did not try to normalize the transition probability after this modification since in the worst case, the total transition probability is Therefore its influence on other transition probability can be neglected. A multiresolution optimization strategy is employed to find the optimal solution. For coarse images, if we use 64 bins to collect the voxel pair information, then there would be a lot of empty bins. To avoid this situation, the bin size is adaptively reduced. We keep the bin size as the resolution (i.e. if the image is 8 8 8, then the bin size is 8). The prior transition probabilities are estimated accordingly. Table3gives the statistics of the registration results. The median of each registration parameter in each group is close to the average value of successful registration and they are in a vector form (-0.03, 0.17, 0.07, 0.04, 0.01, -0.09), (-0.03, 2.14, 1.66, 0.48, 0.02, -0.63), and (-0.03, 2.01, 1.45, 0.06, 0.02, -0.23) for Sets 1, 2, and 3, respectively. As expected, the registration results are close to the manual results since the prior is derived from the manual registration. In Set 1, the registration parameters are more closer to those of the manual results, but the success rate is somewhat low. In other 2 sets, the success rate is high, but the registration parameters are deviated from manual ones. Inspecting the results reveals that there seems another attractive point around the manual one, which explains the large difference and deviation in y and z. Nevertheless, all the differences from the manual registration are within the detection threshold of a trained expert. The normalized likelihood of the manual registration is , while those at the average registration for Sets 1, 2, and 3 are , , and , respectively. They are virtually the same. Further discussion on the magnitude of these likelihood is deferred to a later section Self-consistent In the self-consistent implementation, the transition probability is estimated from the underlying image data. To compute the likelihood, one may have different options: (1) Use all the voxel pairs in the overlapping volume. We denote this approach aslh1. (2) Use partial voxel pairs whose voxel 16

17 values satisfy some conditions. We denote this strategy as LH2. (3) Use partial voxel pairs. The positions of these pairs satisfy some conditions. This third option is denoted as LH3. LH2 and LH3 can be used together. By emphasizing those voxel pairs in strategies LH2 and LH3, the likelihood maximization approach can be tailored to suit different conditions or to incorporate various prior experience and knowledge. For example, in SPECT and PET images, lesions may have avery large voxel values. Thus one may exclude those hot spots in the likelihood calculation with LH2. In other cases, two images may cover different parts of body. In the likelihood calculation, one may exclude the part in one image which does not appear in the other image. This can be done with LH3. In our examples discussed here, we excluded the voxel pairs if the SPECT voxel values fall within the upper 6.25% (LH2). In practice one can exclude the voxels based on the features of a histogram. The MR image covers part of neck which does not appear in the SPECT image. We excluded the lower 1 4 part of the MR image in the likelihood computation (LH3). The MR image was used as a reference and the SPECT as a floating image. The statistics of the misregistration are shown in Table 4. Again, the median of the registration parameters in each group are close to the average of successful registration and thus not shown here. The results of mutual information registration are also included for comparison. The success rate of the mutual information maximization approach is slightly better than the likelihood approach. Most of the failed registrations by likelihood maximization are due to large translations so that there is no overlap in two volumes. As the initial misregistration becomes larger, the success rate of these two algorithms decreases. Also revealed by the data is that, as the initial misregistration becomes larger, the statistics do not change noticeably, indicating that both approaches are robust and reliable. Comparison of LH2 and LH3 against mutual information may be unfair since there is no such strategies in the latter algorithm. Both registration results show some systematic difference from the manual one, but all these differences are within the detection threshold of an expect. The differences between mutual infor- 17

18 mation maximization and likelihood maximization (LH1 and LH2) are also within the detection threshold. The noticeable differences are in x, t y, and t z, which are about 2 degrees, 1.0 and 1.0 mm, respectively. For three variations of likelihood computation, LH1 and LH2 give almost identical results in this particular case. LH3 gives slightly different results. The noticeable differences are in x and t y,which are about 1.5 degrees and 1.0 mm, respectively. The normalized likelihoods in three sets were computed at the average registration. For each likelihood variation, those likelihoods are close to each other and were averaged. They are , , and for LH1, LH2, and LH3, respectively. The values for LH2 and LH3 are slightly larger than that of LH1. While these values may appear small in magnitude, they do indicate a significant likelihood. As we will discuss later, if there is no relation between two images, one would expect likelihoods of Although we did not expect any remarkable difference when the SPECT image is used as a reference, the experimental results do show some disparity, with reduced success rate (data not shown). When the MR image is used as the floating image, the voxel values of MR image are interpolated. Since the MR image voxel values have a smaller dynamic range, the effect of interpolation error could be severe. 6 Discussion In the knowledge-based implementation the priori transition probability can be obtained from previously registered images. Leventon and Grimson proposed a similar idea [17]. However, they model the joint probability ofvoxel pairs and use this probability directly in the likelihood computation. As we mentioned earlier, for multimodality registration, a reasonable model is hard to get, if not impossible. Their formulation is closely related to entropy maximization, as we will discuss later. We incorporate the segmentation and prior knowledge into likelihood maximization. The tran- 18

19 sition probability iscomputed from all overlapping voxel pairs, while likelihood computation can be confined to selected voxel pairs. The similar idea can be employed in mutual information registration [18], where the marginal and joint distributions are estimated based on the segmented data. However, if part of the overlapping volume is excluded from the estimation of marginal and joint probability density functions, the estimated joint probabilities may have large noise or even biased due to data reduction. The application of the likelihood maximization principle to image registration was attempted before [19, 20, 17, 21]. According to this principle, Mort and Srinath [19] assume a Gaussian conditional probability of an augmented pixel data vector, given the displacement. Maximizing this probability yields an estimate of the displacement vector. This technique was successfully applied to interframe displacement estimation. Costa et al [20] proposed a particular formula for PET emission and attenuation map registration, where the attenuation map is segmented into foreground and background classes. Roche et al [21] tried to derive a likelihood maximization framework and regard other popular registration measures as special cases of it. They concluded that, the mutual information maximization registration is a special case of likelihood estimation. Unfortunately what they derived is not mutual information, but the condition entropy. Our approached developed here is applicable to 3d registration in general. We tested it on the rigid-body transformation. In principle, however, it is applicable to any transformation. No assumption on the shape of the probability is needed. Segmentation is not the prerequisite of our algorithm. Rather the user can use it to tailor the algorithm to different needs as well as to incorporate different prior knowledge or experience. In the remaining of this section we discuss the relation of our algorithm to entropy-based measures. Hereafter we assume all data in the overlapping volume are used to compute the normalized log likelihood. In that particular situation (LH1), the normalized log likelihood can be expressed in a simple formula. Let the frequencies of gray scales (u1;u2; ;u n ) and (v1;v2; ;v m ) be (p1;p2; ;p n ) and 19

20 (q1;q2; ;q m ), respectively. Let the transition probability forvoxel u i to v j be w ij. p and q are then related by nx q j = p i w ij i=1 w ij can be estimated from the marginal and joint histograms (see Section 3.2). Substituting this estimation into LL gjf, one has LL gjf = 1 N = 1 X N = X i = X i = i X i;j XX i j X j X j X log w ij N i w ij log w ij p i w ij log w ij h ij log w ij h ij log h ij X j i p i log p i where p i = N i N and p iw ij = h ij are used and h ij is the estimation of the joint probability. Note that this is the negated conditional entropy H(f jg). Similarly, one has X LL f jg = i X h ij log h ij X j j which is the negated conditional entropy H(gjf). Finally, LL =2 X i X h ij log h ij X q j log q j q j log q j X j j i p i log p i which is a combination of mutual information and negated joint entropy. Under this particular scheme (LH1), the log likelihood can be computed as above. Note that this log likelihood is proposed in [22] as an abstract measure and used in [13]. Here we provide a likelihood interpretation. In the derivation, if we treat image f as a random field, u i would be a random variable. Instead of computing the conditional likelihood, we can compute the likelihood. The formula is then the negated joint entropy. Since the image is given, it would be better to compute the conditional likelihood. 20

21 The concept of entropy was proposed in the 1940's. The conventional use of entropy is to exploit it as an objective function to be maximized, for example, in statistical physics [23], estimation theory, spectral analysis [24], image reconstruction [25], queue theory [26], and city planning [27]. In the middle of the 1980's, entropy maximization was formally established as a principle based on a set of axioms [28]. However, entropy minimization as a principle is never formally posed. One can still informally, or intuitively justify entropy minimization in the image registration context, but a sound foundation is still missing. Likelihood maximization is an established principle and easy to understand. The approach presented here therefore has a solid theoretical foundation and is conceptually easy to appreciate. Mutual information has a lower bound of 0, but the upper bound is not a fixed number ([13] summarizes its properties). Entropy has a lower bound 0 and the upper bound is also not fixed. Likelihood is a probability and has a lower bound of 0 and an upper bound of 1. Although the calculation of the likelihood depends on the resolution and also the bin size, the upper bound of 1 never changes. If the two images have no correlation, then one would expect that the transition probability fora voxel in f to a voxel in g is 1 n f, and the transition for a voxel in g to a voxel in f is 1 n g. The likelihood is then 1 n f n g. By looking at the likelihood, one knows to what extent, the two images match. 7 Conclusion We have presented the likelihood maximization approach to image registration. Two implementations, one knowledge-based and the other self-consistent, are proposed and tested. In the latter implementation, three variations are studied. One can incorporate various prior knowledge and experience into these schemes. The algorithm and its variations are compared to mutual information maximization approach in mono- and multi-modality registrations. The results indicate that the performance of this new approach is comparable to that of the mutual information approach. 21

22 Further work is required to examine its absolute accuracy using the gold standard such as that describedin[5]. It would also be interesting to understand when the likelihood approach outperforms the mutual information approach and when it does not. Acknowledgements: The authors express their sincere thanks to Lou Arata who implemented the mutual information, Dawn Bray, Zeeba Mercer, Chuck Nortmann who manually registered the MR/SPECT image pair. The image files were provided by Michael Hartshorne. 22

23 References [1] J.B.A. Maintz and M.A. Viergever, A survey of medical image registration", Medical Image Analysis, vol. 2, no. 1, pp.1-36, [2] P.A. van den Elsen and M.A. Viergever, Medical image matching - a review with classification", IEEE Engng Med. Biol. vol. 12, pp.26-39, Mar [3] C.R. Maurer and J.M. Fitzpatrick, A review of medical image registration", in R.J. Maciunas, Ed., Interactive Image-guided Neurosurgery, Parkridge, IL: Amer. Assoc. Neurological Surgeons, 1993, pp [4] L.G. Brown, A survey of image registration techniques", ACM Computing Surveys, vol.24, no.4, pp , Dec [5] J. West, J.M. Fitzpatrick, M.Y. Wang, B.M. Dawant, C.R. Maurer, Jr., R.M. Kessler, R.J. Maciunas, C. Barillot, D. Lemoine, A. Collignon, F. Maes, P. Suetens, D. Vandermeulen, P.A. van den Elsen, S. Napel, T.S. Sumanaweera, B. Harkness, P.F. Hemler, D.L.G. Hill, D.J. Hawkes, C. Studholme, J.B. A. Maintz, M.A. Viergever, G. Malaandain, X. Pennec, M.E. Noz, G.Q. Maguire, Jr., M. Pollack, C.A. Pelizzari, R.A. Robb, D. Hanson, and R.P. Woods, Comparison and evaluation of retrospective intermodality brain image registration techniques", J. Computer Assisted Tomography, vol.21, pp , [6] J.M. Fitzpatrick, D.L.G. Hill, Y. Shyr, J. West, C. Studholme, and C.R. Maurer, Jr., Visual Assessment of the accuracy of retrospective registration of MR and CT images of the brain", IEEE Trans. Med. Imag., vol.17, no.4, pp , Aug [7] J. West, J.M. Fitzpatrick, M.Y. Wang, B.M. Dawant, C.R. Maurer, Jr., R.M. Kessler, and R.J. Maciunas, Retrospective intermodality registration techniques for images of the head: 23

24 Surface-based versus volume-based", IEEE Trans. Med. Imag., vol.18, no.2, pp , Feb [8] D. Kotz, Nuclear medicine in the 21st century: Integration with other specialities", J. Nuc. Med., vol.40, no.7, pp.13n-25n, July [9] W.M. Wells, III, P.V. Viola, H. Atsumi, S. Nakajima, and R. Kikinis, Multi-modal volume registration by maximization of mutual information", Medical Image Analysis, vol.1, no.1, pp.35-51, [10] J.L.R. Andersson, A. Sundin, and S. Valind, A Method for coregistration of PET and MR brain images", J. Nucl. Med., vol.36, pp , [11] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C: the Art of scientific computing (2nd ed.), Cambridge: Cambridge Univ. Press, Chapter 10. [12] J.D. Foley, A. van Dam, S.K. Feiner, and J.F. Hughes, Computer Graphics: Principles and Practice (2nd ed), Addison-Wesley, [13] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, Multimodality image registration by maximization of mutual information", IEEE Trans. Med. Imag., vol. 16, no.2, pp , Apr [14] N. Ritter, R. Owens, J. Cooper, R.H. Eikelboom, and P.P. van Saarloos, Registration of stereo and temporal images of the retina", IEEE Trans. Med. Imag., vol.18, no.5, May [15] F. Maes, D. Vandermeulen, and P. Suetens, Comparative evaluation of multiresolution optimization strategies for multimodality image registration by maximization of mutual information", Med. Image Anal., vol.3, no.4, pp ,

25 [16] C. Studholme, D.L.G. Hill, and D.J. Hawkes, Automatic three-dimensional registration of magnetic resonance and positron emission tomography brain images by multiresolution optimization of voxel similarity measures", Med. Phys., vol.24, no.1, pp.25-35, Jan [17] M.E. Leventon and W.E. Grimson, Multi-modal volume registration using joint intensity distribution", in Proc. MICCAI'98 (also Lecture Notes on Computer Sciences vol.1496), pp , Cambridge, MA, Oct [18] A. Carrillo, J.L. Duerk, J.S. Lewin, and D.L. Wilson, Semiautomatic 3-D image registration as applied to interventional MRI liver cancer treatment", IEEE Trans. Med. Imag., vol.19, no.3, pp , Mar [19] M.S. Mort and M.D. Srinath, Maximum likelihood image registration with subpixel accuracy", SPIE Applications of Digital Image Processing XI, vol.974, pp.38-45, [20] W.L.S. Costa, D.R. Haynor, R.M. Haralick, T.K. Lewellen, and M.M. Graham, A maximum-likelihood approach to pet emission/attenuation image registration", IEEE Nuclear Science Symposium and Medical Imaging Conference, pp , [21] A. Roche, G. Malandain, N. Ayache, and S. Prima, Towards a better comprehension of similarity measures used in medical image registration", in Proc. MICCAI'99 (also Lecture Notes on Computer Sciences vol.1679), pp , Cambridge, UK, Sept [22] T.M. Cover and J.A. Thomas, Elements of Information Theory, New York: Wiley, [23] D. Giordano, Equivalence of energy, entropy, and thermodynamic potentials in relation to the thermodynamic equilibrium of multitemperature gas mixtures", Phys. Rev. E vol.58, no.2, pp , Sept [24] B. Choi, An information-theoretic derivation of the 2d maximum entropy spectrum", IEEE Signal Processing Letters vol.5, no.10, pp , Oct

26 [25] G. Minerbo, MENT: A maximum entropy algorithm for reconstructing a source from project data", Computer Graphics and Image Processing, vol10, pp.48-68, [26] J.S. Wu, Maximum entropy analysis of open queuing networks with group arrivals", J. Opera. Res. Soc. vol.43, no.11, pp , Nov [27] V. Kumar, U. Kumar, and K. Hoshino, An application of the entropy maximization approach in shopping area planning", Int'l J. General Systems vol.16, no.1, pp.25-42, [28] J.E. Shore and R.W. Johnson, Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy", IEEE Trans. Info. Theory, vol.26, no.1,

27 Alg x y z t x t y t z Time Success N(0; 0) MI -0.01± ± ± ± ± ± ± % LH -0.00± ± ± ± ± ± ± % N(0; 50) MI -0.00± ± ± ± ± ± ± % LH 0.01± ± ± ± ± ± ± % N(0; 100) MI 0.02± ± ± ± ± ± ± % LH 0.01± ± ± ± ± ± ± % Table 1: Statistics of the misregistration parameters for successful registration. Anuclear image was registered to itself with the presence of zero-meaned Gaussian noise with different levels of standard deviation (0, 50, and 100), from a set of misregistrations. In each entry, the first number is the average and the second one is the standard deviation. The angles are in degrees, the translations in mm, and the times in seconds. Two algorithms are evaluated: mutual information maximization (MI), and likelihood maximization (LH). 27

28 Alg x y z t x t y t z Time Success N(0; 0) MI -0.01± ± ± ± ± ± ± % LH 0.00± ± ± ± ± ± ± % N(0; 50) MI -0.01± ± ± ± ± ± ± % LH 0.02± ± ± ± ± ± ± % Table 2: Statistics of the misregistration parameters for successful registration. A MR image was registered to itself with and without the presence of noise, from a set of misregistration. See the caption of Table 1. Set x y z t x t y t z Time Success ± ± ± ± ± ± ± % ± ± ± ± ± ± ± % ± ± ± ± ± ± ± % Table 3: Statistics of the misregistration parameters for successful registration. A SPECT image is registered to a MR image using likelihood maximization. The transition probabilities are calculated from a manual registration. See the caption of Table 1. 28