Global optimization of weighted mutual information for multimodality image registration

Transcription

1 Global optimization of weighted mutual information for multimodality image registration C. E. RodrIguez-Carranza and M. H. Loew Department of Electrical Engineering and Computer Science Institute for Medical Imaging and Image Analysis The George Washington University, Washington DC ABSTRACT Failure to align images accurately often is due to the optimization algorithm being trapped in local maxima or spurious global maxima of the mutual information function. Strategies contemplated to improve registration involve modifying the optimization scheme or the registration measure itself. We recently found that normalized mutual information (for 2D image registration) provides a larger capture range and that is more robust, with respect to the optimization parameters, than the non-normalized measure. In this paper we assessed the utility of a stochastic global optimization technique for image registration using normalized and non-normalized mutual information. By conducting large-scale studies with patient data in 2D, we established a success rate baseline with the local optimizer only. Formal proof has not yet been found that incorporating the global optimizer does not impair performance. However, experiments to date indicate that its inclusion leads to better (i.e., higher probability of correct convergence) overall performance. More over, studies now underway show good effectiveness of our approach in a variety of 3D cases. Keywords: multimodality image registration, mutual information, topographical global optimization 1. INTRODUCTION Mutual information was first introduced as a measure for image alignment by Collignon, et al.' and Viola, et al.2 in Since then it has become a popular method for registration of mono- and multimodality images because of its robustness and accuracy. Reports of alignment studies using mutual information have come from a variety of sites.36 Yet there are still situations in which algorithms employing mutual information do not align images correctly. Failure to align images is due to the optimization algorithm being trapped in local maxima or spurious global maxima of the mutual information function. Strategies to improve registration can involve modifying the optimization scheme or the registration measure itself. In an earlier attempt to improve the behavior of mutual information, we introduced the use of a deterministic and weighted entropy measure.7 It was tested in 2D images (one modality) and results obtained indicate that normalized (or weighted) mutual information (using either Shannon's entropy or the deterministic entropy) provides a larger capture range and is more robust, with respect to the optimization parameters, than the non-normalized mutual information. For this paper, we look into the optimization part of the registration algorithm. A variety of optimization schemes have been applied to image registration using similarity measures: Powell's,"3 simplex,4'8 gradient descent, multiresolution exhaustive searches,9 stochastic optimization using derivative estimation,2 genetic algorithms,8 etc. In most cases the optimization strategy is local and requires a starting point (in other words, a starting transformation vector); the typical choice for it is the zero vector. Sometimes, the alternative is to select multiple starting points, and a decision has to be made of how to select such points. Some techniques, such as topographical global optimization, allow high-quality sampling of a large number of points. We wanted to determine whether it is possible that the incorporation of this global optimization scheme in image registration can refine the selection of the starting point and hence increase the probability of correct alignment. Therefore, in this paper we explore the use of topographical global optimization for 2D and 3D multimodality image registration using normalized mutual information (computed with either Shannon's entropy or the deterministic entropy). Other author information: (Send correspondence to C.ER.C.) M.H.L.: loew seas.gwu.edu C.E.R.C.: claudia seas.gwu.edu Part of the SPIE Conference on Image Processing San Diego, California February 1999 SPIE Vol X/99/$

2 2.1. Mutual information 2. BACKGROUND Mutual information has its roots in information theory. We refer the interested reader elsewhere' '11 for an in-depth cover of the subject. The basic equation for mutual information is the following: I(a,i3) = = H(a) H(aJ$) where H(a) refers to the marginal entropy or uncertainty of a, H(a, 3) refers to the joint entropy of a and j3, and H(a/13) is the conditional entropy. Entropy refers to the average information of an event. Mutual information indicates to what extent the realization of /3 lowers the uncertainty of a. The expression for the Shannon's entropy of event a with j outcomes A, each with probability of occurrence p(a), is defined as: H(a) = H(A1,..., A) = p(a)( log p(a)) Tn the context of image registration, mutual information has been used as a measure of alignment; the assumption is that mutual information between two images is maximized when both are aligned. The entropy of an image is computed based on the intensity distribution; this can be estimated either by a simple normalization of the histogram or by Parzen windowing. A totally homogeneous image will have zero entropy; an image with a different gray value at each pixel will have maximum entropy. Recently7 we proposed the use of a deterministic entropy measure to compute mutual information. The expression of the two-dimensional discrete case is: : 2f(x,y) 2x. I 1 L2f(x,y) H(f(,.); ) = (1) 2f(x,y) 2x For details in the derivation of this expression we refer the reader to the earlier paper Topographical global optimization The purpose of global optimization is to find the smallest of a number of local minima that a multimodal function I attains in the region of interest A C Stochastic algorithms for global optimization'2 consist of three distinct major steps: a sampling step, an optimization step, and a check of some stopping criterion. The sampling step consists of the generation of random points in the region of interest A, and the computation of the associated function value. As the number of sampled points approaches inffnity, the probability of finding the global minimum approaches one. The drawback, of course, is that the more trial points we use the longer the running time of the algorithm. This is something that should be avoided especially if the objective function is costly to evaluate. The optimization step consists of applying a local optimization routine to some of the sampled points. We can start a local minimization from each sampled point, but this simple strategy is highly inefficient. For example, a local optimizer will normally arrive at the same local solution several times. A more efficient strategy is to try to group (cluster) points belonging to basins of local minima and then start a local minimization from just one point in each identified basin. There are several stopping criteriathat can be used to stop the algorithm : (1) when there's sufficient evidence that the global optimum has been detected, (2) when the cost connected with searches for a better estimate has been exceeded, (3) when the number of prespecified number of steps or function evaluations has been reached. 90

3 In this paper we focus in the stochastic algorithm proposed by Torn and Vittanen'3 called topographical global optimization. In their algorithm, points are uniformly sampled (a good random number generator is vital to guarantee a uniform distribution of points in multiple dimensions). Ideally, sampling should be such that at least one of the sampled points lies in each basin of the global minima. The way TGO identifies clusters of points belonging to the same basin is through a directed graph of the topographical information of the objective function; this graph is called k-topoyraph. Each node in the graph represents a sampled point; the graph connects neighboring points with directed arcs pointing towards points of higher function values. The Euclidean distance is used to establish which are the closest neighbors. Figure 1 shows a very simple example. The function is one-dimensional and consecutive k-topographs are shown; thin arcs represent arcs from graphs with smaller k. Observe that the larger k is, the further away the neighbors are. For a k-topograph, nodes with no incoming arcs represent local minima. f(x) /R\\//1 ;1u' RE c3 1-topograph 2-topograph 3-topograph 0 = graph minimum = sampled point Figure 1. Example of a one-dimensional function and three k-topographs. When the sampling is terminated, the minima of the graph are presented and a local minimizer (the choice is not a central question) is started from a number of them. Minimization is started from, say, the three best minima in the graph and the corresponding minima found by the local minimization algorithm are presented; the smallest of the solutions corresponds to the global minima. The user has then the option to start additional local minimizations from other graph minima. The algorithm is simple but it has some disadvantages. The underlying assumption is that a lower function value implies a local minimum, which is not always the case. The algorithm will not perform too well if the problem function is very ragged within small areas. Large search areas exaggerate the effect of even small raggedness and renders the algorithm useless for problems of this nature unless the number of points, N, is chosen very large. Most other algorithms, however, will also have difficulties in finding the global solution for such problems Iterative topographical global optimization Torn and Viitanen'4 recently proposed an iterative version of topographical global optimization (I-TGO) which allows sampling of a large number of points in an efficient way. There are two versions of the algorithm, and we present one of them here. Both require constant storage complexity and linear computational complexity in the number of total points used (N). 91

4 In the I-TGO algorithm, points are iteratively sampled and minima are identified through the k-topograph. The parameters of the algorithm are (1) number of desired iterations, I, (2) number of points to sample per iteration, N1, and (3) the number of nearest neighbors k to determine the graph minima. A sketch of the I-TGO1 algorithm is shown next: 1. mo = 0, j = 1, and the k topograph is empty. 2. While the number of iterations j is less than I (a) sample N1 m11 points (b) compute Euclidean distances, form the k-topogragh, and identify the graph minima (m3) (c) eliminate all points except for the m3 graph minima 3. For each point in the graph (i.e. graph minima), run the local optimization algorithm. 4. Select the smallest of the solutions to be the global minima. As can be seen, the only information carried from one iteration to the next is the set of minima in the graph. The authors recommend that a relatively small k be used to avoid losing information. Only Nj(Nj 1)/2 distances need to be stored at any time. The number of distance calculations is I x N1(Nj 1)/2. Since the number of points sampled in the.7th iteration is N1 m_1, (where m3 is the number of minima in the th iteration), the total number of sampled points, N, can be written as: N = Nj m1_1 = IxNj m_1 = IxN1 Ixii where m: s the mean value of the m3's. Therefore, 1= N N1 ii Using this value of I in the expression for the number of distance calculations, we obtain implies linearity in N. x Ni( which 3. METHODS For our experiments, we assumed the following context. Given a pair of misaligned images, the idea is to present to a human observer the best set of starting points so that the images can be aligned with the smallest number of attempts. The observer picks one starting point, runs the registration algorithm, and then judges whether the registration is good enough or not; if it is not, then she/he repeats the sequence of selection and evaluation until satisfied with the alignment. We wanted to determine whether I-TGO could improve registration performance by providing a better selection of starting points; in other words, whether I-TGO could increase the probability of finding the global maxima of mutual information with fewer points. With this in mind, we compared the performance of registration with and without I-TGO by computing the probability that we need to sample j "bad" starting points (points that lead to a failed registration) before we 92

5 sample the first "good" starting point (points that lead to a successful registration). For registration without I-TGO, the baseline experiment consisted of running N registrations, each with a different starting point. The probability p of selecting a "good" starting point is the proportion of successful registrations out of the total N. With p we calculated the probability of selecting j "bad" sample points before the first "good" one as: P(F=j)=(1 _p)jp (2) For registration with I-TGO, we run M registration experiments. Each of them consisted of: (1) sampling points with I-TGO; the output of 1-TGO is the graph maxima, which constitute the list of starting points for registration, and (2) applying the local optimizer to each of the graph maxima. For each of this M lists of graph maxima, we observed the rank (or position in the list) of the first "good" maxima. With this information we computed the probability of j "bad" samples before the first "good" one as: P(F = j) = {number of "good" maxima with rank j + 1}/M (3) For example, the probability of no "bad" maxima corresponds to the proportion of "good" maxima with rank 1; the probability of one "bad" maxima corresponds to the proportion of "good" maxima with rank 2, etc. For all the experiments, the range of the starting points (ti, ti,, 0) was ([ 60mm, 60mm], [ 60mm, 6Omm}, [ 75, 75 ]). The number of registrations for the experiment without I-TGO was N= The parameters required for I-TGO are (a) number of iterations I, (b) number of samples per iteration N1, and (c) the extent of the topograph, k. For reasons explained next and based on results from the baseline experiment without T-TGO, we selected 1=4, N1=50, and k=3. The number of registrations with I-TGO was M = 100. The cost of incorporating I-TGO into image registration was measured in terms of number of function evaluations. The total cost of registration with I-TGO consists of two parts: (1) the cost of the sampling stage of I-TGO, and (2) the cost of running the local optimizer for each of the final graph maxima. For our experiments, we established that the cost of the sampling stage should not be more than the cost of running the local optimizer on one point. The latter was determined from the baseline experiment without I-TGO. The image dataset we used consisted of two slices of a registered pair of MR and CT images; they were reformatted to a size of 256x256. We applied a shift of 15mmin the y direction, +20mm in the x direction and +20 degrees to the MR image. This transformation vector becomes the reference transformation and we used it to establish registration accuracy of the experiments. Accuracy was calculated as the difference between the reference transformation vector and the computed transformation vector. A registration result was considered successful if the difference between the reference transformation and the computed transforniation was no more than 1mm for translations and 1 for the rotation. The local optimizer used in all experiments was Powell's direction set method. The parameters for this algorithm were the same for all experiments. The transformations were assumed to be rigid body. We used trilinear interpo lation, and only pixels that lie within the volume of overlap were retained; pixels that lied outside were assigned a value of zero. The number of histogram bins used to compute mutual information via Shannon's entropy was 256. For the case of normalized mutual information, we multiplied mutual information (using either Shannon's entropy measure or the deterministic entropy measure) by the ratio of the area of of overlap to the area of the smallest image. 4. RESULTS The results of the baseline experiment of registration without I-TGO are shown in Table 1. The mean number of function evaluations during optimization with Powell's algorithm is 243 and 301. As noted before, the cost of sampling in I-TGO should not exceed 301. Hence, by selecting the number of iterations to be 4 and the number of sampled points per iteration to be 50, we restricted the number of function evaluations in the sampling stage of I-TGO to be at most 4 x 50 =

6 Measure No. of successful starting points Function evaluations (mean stdfl from Powell's iterations Weighted Mldet Weighted Mlshan Table 1. Baseline results for registration without I-TGO Table 2. Registration with I-TGO In Table 2 we show the ranks of the "good" maxima obtained for the registration with I-TGO experiment. The average number of function evaluations during ITGO was 180 for mutual information using the deterministic entropy measure, and 179 for mutual information using Shannon's entropy measure. For registration without I-TGO, using Table 1 we calculate the probability of selecting a "good" starting point to be p = 0.35 for the deterministic entropy measure and p = for Shannon's entropy measure (the total number of registration experiments was 1000). With these values, we compute the probabilities of sampling j"bad" points before the first "good" point using Equation 2. The results are shown in rows 3 and 5 of Table 3. For the case of registration with I-TGO, we use Table 2 to compute the probabilities from Equation 3 (the total number of registration experiments was 100). The results are shown in rows 2 and 4 of Table 3. We look at up to a maximum of 5 samples; the last column corresponds to the probability of at least five consecutive failures. Measure Registration strategy p(f = 0) p(f = 1) p(f = 2) p(f = 3) p(f = 4) p(f > 4) weighted Midet with 1-TGO without I-TGO weighted Mlsha with I-TGO without I-TGO Table 3. Comparison of P(F = j) probabilities for registration with and without I-TGO. Table 3 indicates that, for mutual information with the deterministic entropy measure, the probability of a successful registration with one point is higher for the strategy with I-TGO than without I-TGO: 0.51 vs 0.35; the same is true for two points: 0.67 vs (adding the columns p(f = 0) and p(f = 1)). When we look at more than three points, the difference between the probabilities is minimal. The effect of I-TGO is more dramatic when using Shannon's entropy. The probability of selecting a "good" starting point at the first sample is 0.70 with I-TGO and without 1-TGO. The same holds if selecting two points (0.86 vs 0.374), three points (0.90 vs 0.505), etc. 5. DISCUSSION Topographical global optimization is a stochastic optimization scheme that provides a high-quality selection of points among which to search for a global optimum. The current results show that I-TGO increases the probability of finding the global maximum of mutual information with fewer points. This can be observed in Table 3, where normalized mutual information (using either entropy measure) with I-TGO shows higher probability of selecting a "good" starting point with less samples than when using the normalized mutual information without I-TGO. 94

7 The effect of I-TGO on the performance of registration is more important for mutual information based on Shannon's entropy measure than for mutual information based on the deterministic entropy measure. Taking as example the case of successful registration with only one sample, we observe that the improvement for the deterministicentropy-based measure with I-TGO is 1.45-fold; for the Shannon's entropy-based measure, it is 3.35-fold. Overall, Shannon's entropy based normalized mutual information with I-TGO shows the best performance of the four possibilities; it is followed by the deterministic-entropy-based measure with I-TGO and without I-TGO, and finally the Shannon's-entropy-based measure without I-TGO. For the case of sampling only one point, the probabilities are respectively, (0.7,0.51,0.35,0.209); when sampling two points the probabilities are (0.86,0.67,0.578,0.374). The cost of incorporating I-TGO into image registration was based on the number of function evaluations. We established that the maximum cost to pay for the sampling stage of I-ITGO was the cost of running the local optimizer on one point. For our experiments, the mean cost of sampling in I-TGO was 180 function evaluations, and the mean number of function evaluations for running the local optimizer were 243 and 301, respectively, for mutual information based on the deterministic entropy and Shannon's entropy. Hence, even though there is a (small) cost for using I-TGO, we believe that the observed improvement in the registration performance well justifies its use. Nevertheless, we need to perform further experiments to determine if the improvement in the performance is significant and if it can be generalized. We are currently testing normalized mutual information and the 1-TGO strategy for 3D image registration. Experiments now underway are showing good effectiveness of the I-TGO approach in a variety of 3D cases. ACKNOWLEDGMENTS C.E.R.C. thanks Josien Pluim for her kind help and fruitful discussions. REFERENCES 1. A. Collignon, F. Maes, D. Delaere, D. Vandermeulen, P. Suetens, and G. Marchal, "Automated multimodality image registration using information theory," Proceedings Information Processing in Medical Imaging (IPMI'95), (Dordrecht, The Netherlands), P. Viola and W. M. Wells, "Alignment by maximization of mutual information," Proceedings of the 5th International Conference on Computer Vision, pp , F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, "Multimodality image registration by maximization of mutual information," IEEE Transactions on Medical Imaging 16(2), pp , C. R. Meyer, J. L. Boes, B. Kim, P. H. Bland, K. Zasadny, P. V. Kison, K. Koral, K. Prey, and R. L. Wahl, "Demonstration of accuracy and clinical versatility of mutual information for automatic multimodality image fusion using affine and thin plate spline warped geometric deformations," Medical Image Analysis 1(3), C. Studholme, D. L. G. Hill, and D. J. Hawkes, "Automated 3-d registration of mr and ct images of the head," Medical Image Analysis 1(2), J. West, M. Fitzpatrick, M. Wang, B. Dawant, C. Maurer, R. Kessler, R. Maciunas, C. Barillot, D. Lemoine, A. Collignon, F. Maes, P. Suetens, D. Vandermeulen, P. van den Elsen, S. Napel, T. Sumanaweera, B. Harkness, P. Hemler, D. Hill, D. Hawkes, C. Studholme, J. Maintz, M. Viergever, G. Malandain, X. Pennec, M. Noz, G. Maguire, M. Pollack, C. Pelizzari, R. Robb, D. Hanson, and It. Woods, "Comparison and evaluation of retrospective intermodality brain image registration techniques," Journal of Computer Assisted Tomography 21(4), pp , C. RodrIguez-Carranza and M. Loew, "Weighted and deterministic entropy measure for image registration using mutual information," in Image Processing, Proceedings SPIE 3338, pp , D. Hill, D. Hawkes, N. A. Harrison, and C. F. Ruff, "A strategy for automated multimodality image registration incorporating anatomical knowledge and imager characteristics," Proceedings Information Processing in Medical Imaging (IPMI'93), pp , Springer-Verlag, Berlin, P. A. van den Elsen, E. J. D. Pol, T. S. Sumanaweera, P. Hemler, S. Napel, and J. Adler, "Grey value correlation techniques used for automatic matching of Ct and mr brain and spine images," in Visualization in Biomedical Computing, Proceedings SPIE 2359, pp , T. M. Cover and J. A. Thomas, Elements of information theory, John Wiley and Sons, Inc., New York, A. M. Yaglom and I. M. Yaglom, Probability and Information, D. Reidel Publishing Company, Moscow,

8 F. Schoen, "Stochastic techniques for global optimization: a survey of recent advances," Journal of Global Optimization 1, pp , A. Torn and S. Viitanen, "Topographical global optimization," in Recent advances in global optimization, C. Floudas and P. Pardalos, eds., pp , Princeton University Press, A. TOrn and S. Viitanen, "Iterative topographical global optimization," in State of the art in global optimization, C. Floudas and P. Pardalos, eds., pp , Kiuwer Academic Publisher, 1996.