A Discrete Search Method for Multi-modal Non-Rigid Image Registration

Transcription

1 A Discrete Search Metho for Multi-moal on-rigi Image Registration Alexaner Shekhovtsov Juan D. García-Arteaga Tomáš Werner Center for Machine Perception, Dept. of Cybernetics Faculty of Electrical Engineering, Czech Technical University, Prague, Czech Republic Abstract We consier the problem of image matching uner the unknown statistical epenence of the signals, i.e. a signal in one image may correspon to one or more signals in the other image with ifferent probabilities. This problem is wiely known as multimoal image registration an is commonly solve by the maximization of the empirical mutual information between the images. The eformation is typically represente in a parametric form an optimization w.r.t. it is performe using graient-base methos. In contrast, we represent the eformation as a fiel of iscretize isplacements an optimize w.r.t. it using pairwise Gibbs energy minimization technique. This has potential avantage of fining goo solutions even for problems having many local minima. In experiments we emonstrate that the propose metho working on a single scale achieves comparable performance to a state-of-the-art multi-scale metho.. Introuction Image registration is efine as the process of fining the corresponence between the points in two or more images of the same scene. Its application to meical images has been well stuie 9, 3, 2, an extensively use in the fusion of complimentary ata acquire from multiple scanning evices, the comparison of images from ifferent iniviuals, an the follow-up of the changes within the same iniviual at ifferent times. When imaging biological structures, rigi eformations cannot, in general, compensate for morphological variations between iniviuals, the graual changes over time or the natural elasticity of image tissues. In these cases, espite its inherent ifficulty, non-rigi matching is the preferre image registration option. The task of estimating the non-rigi eformation that best aligns two images is usually formulate as an optimization problem that minimizes an image issimilarity criterion between the eforme (registere) images. The eformation, usually taken from a parametric family (e.g. B-Splines), is expecte to realistically moel the physical variations without introucing artifacts. As the eformation is typically escribe by a high number of parameters, the optimal solution is normally foun by a ifferential or local search metho 2. Because the objective function comparing the images uner the eformation has many local minima, it can only be rather poorly optimize by local methos. The problem becomes even more significant when aitional parameters nee to be estimate; specifically, we consier the case where the statistical epenence between intensities is unknown. Maximizing the likelihoo of the observe ata w.r.t. both the eformation an the parameters of color epenence moel naturally yiels an alternating optimization algorithm. Taking the non-parametric form of the unknown statistical color epenence it is known that this maximum likelihoo approach is equivalent to registration by maximization of empirical mutual information 4. In this article we propose to search for the best eformation aligning multi-moal meical images using, instea of local search methos, a global approximate metho. Our approach is to consier the eformation as a combination of the local translations of small blocks with the constraint that eviation of translations of neighboring blocks is restricte. Relate Work. Kim et al. 7 applie mutual information criterion to solve the stereo corresponence problem. They searche for the best next estimate of the isparity fiel (eformation) by solving a iscrete search problem in the form of pairwise Gibbs energy minimization. They showe that taking the first orer Taylor approximation of the variant of empirical mutual information with Parzen winow ensity estimator yiels the same objective as alternating the estimation of the eformation an color statistical moels in the maximum likelihoo approach. Glocker et al. 4 an Shekhovtsov et al. 8 applie linear programming relaxation to cope with ifficult iscrete search problem of the optimal 2D/3D eformation. While iscrete optimization methos were propose earlier for recovering nonrigi eformation 5,,, methos 4, 8

2 propose several approximation tricks which mae it realistically applicable for large eformations. While 4 consiere normalize mutual information as a possible measure of image issimilarity they iscuss it just very briefly. In particular, it is not clear if it was applie to local winows inepenently, like normalize cross-correlation, or otherwise. Both 4 an 8 apply similar piecewise translational approximations of the eformation (moving blocks): 4 applies soft regularization on the eformation fiel an proposes a graual multi-scale approach while 8 consiers har constraints an focuses on the efficient implementation of single-scale optimization. Hufnagel et al. 5 applie a block matching scheme with inepenent blocks, with spline post-interpolation. This approach is similar to that of 4, 8, but the blocks are matche inepenently, meaning that they must be rather large to make a goo match an this strongly restricts the class of eformations for which the approach is applicable. Roche et al. 4 presente a systematic stuy of the image registration problem from the point of view of the maximum likelihoo approach, in particular they showe that the non-parametric estimate of statistical color epenence (joint histogram) in maximum likelihoo formulation leas to a maximization of the simplest variant of the empirical mutual information. 2. Probabilistic Moel We follow the stanar computer vision approach an formulate the registration problem as a maximum a posteriori inference in a constructe probabilistic moel. A etaile treatment of image registration within maximum likelihoo approach may be foun in 4, whereas we focus on a special case. I Figure. A non-symmetric formulation of registration of image I an image J. Let I an J be two images to be registere. Let I : T C an J : T 2 C 2, where T an T 2 are sets of locations an C an C 2 are sets of signals. The eformation is represente by a mapping : T T 2. We assume that is an injective mapping, meaning that not all pixels from T 2 will have a preimage. This is illustrate in Fig.. This nonsymmetric formulation is convenient since it allows image I to be matche to a subregion of J. J Assuming that I an J are geometrically istorte observations of an unknown but equal unerlying physical escription, one erives (as in e.g. 4) that there is a statistical epenence of how a particular hien element is image in I an how it is image in J. In other wors, there is a statistical epenence between signals in the two images, which is moele as p(c,c 2 ; θ), c C, c 2 C 2, where θ is a parameter efining this istribution. Treating images I, J an the mapping as ranom variables, we moel their statistical epenence as p(i, J) =p(i, J)p(). () Assuming that pixel signals in I are conitionally inepenent given an J, we write p(i, J) = t T p(i t, J). (2) When pixel t is mappe to pixel (t) we consier that its signal I t epens only on the corresponing signal J (t) an oes not epen on the rest of signals in J. This assumption is expresse probabilistically as p(i t, J) =p(i t J (t) ; θ). (3) Prior istribution p(), etaile later, expresses how likely is eformation. In particular, it is often assume that highly curve eformations are unlikely. The maximum likelihoo estimate of eformation an parameters θ is formulate now as (,θ ) = argmax p(i t J (t) ; θ)p(). (4),θ t T We consier that sets C an C 2 are finite. This coul mean that signals in I (resp. J) woul be quantize into C (resp. C 2 ) bins, if the continuous signal spaces were one imensional. Alternatively, we can eal with multi-imensional signals by representing images I (resp. J) by C (resp. C 2 ) representative examples (much like a color palette). Our moel of the epenence p(c,c 2 ; θ) is then a Gaussian mixture moel with C C 2 Gaussians place in each possible location (c,c 2) having iagonal covariance matrix. Formally, p(c,c 2 ; θ) = θ c,c g 2 (c,c )g 2 (c 2,c 2), (5) c,c 2 where g k (c, c )= exp ρ k(c, c ) 2, Z k 2 k =, 2; (6) an ρ k (c, c ) is a metric in the space C k, k =, 2. Metrics ρ (resp. ρ 2 ) captures the important information of

3 how close are two ifferent elements of C (resp. C 2 ). A Gaussian istribution with a ifferent variance can be obtaine by changing the metric an the normalization constant accoringly. The metrics are assume to be preefine an fixe. For (5) to efine a istribution the constraint c θ,c 2 c,c =must hol. Fining the maximum 2 likelihoo estimate of parameters θ an evaluating the istribution p(c,c 2 ; θ) may be seen as very similar to Parzen winow kernel ensity estimation with Gaussian kernel. 2.. Relation to Mutual Information Mutual information was simultaneously propose as an image similarity criterion by Viola an Wells 9 an Collignon et al. 3. Since then, it has rapily become the preferre choice when registering multi-moal meical images (see the surveys by Pluim et al. 3 an Maes et al. ). Before we make any further assumptions an iscuss our approach to the optimization problem (4) we will escribe its relation to the registration by maximization of mutual information. The erivation is similar to the one presente in 4 an in 7. Taking the logarithm of (4) we can write that: =argmax log p()+max log p(i t J (t) ; θ). θ t T (7) Introucing n c,c 2 = t T δ {It=c }δ {J(t) =c 2} the maximum over θ with aitional factor may be expresse as max θ n c,c2 log p(c c 2 ; θ), (8) c C where = T = n c,c 2. Denoting ˆθ a maximizer c,c 2 of (8), the objective of (8) expresses as n c,c2 log p(c,c 2 ; ˆθ) c C p(c,c 2 ; n c,c2 log ˆθ) c C p(c 2 ; ˆθ) = n c2 log p(c 2 ; ˆθ) = Ĥ(n C,C 2 )+Ĥ(n C 2 ), (9) where n c2 = c n c,c 2 an n C,C 2 stans for the collection of numbers {n c,c 2 c C,c 2 C 2 } an n C2 for the collection of numbers { c C n c,c 2 c 2 C 2 }. The term Ĥ(n C,C 2 ) then enotes the empirical estimate of the entropy from samples represente by counts n C,C 2. The estimate is obtaine by fitting parameters of the istribution an using the sample mean instea of the expectation. It shoul be note that this is only one particular choice of how to estimate the entropy from samples an that other choices are possible. ow problem (7) can be expresse as: =argmax =argmax log p() Ĥ(n C,C 2 ()) + Ĥ(n C 2 ()) log p()+ĥ(n C ) Ĥ(n C,C 2 ()) +Ĥ(n C 2 ()) =argmax log p()+î(n C,C 2 ()), (0) where Ĥ(n C ) is the empirical entropy of n C which oes not epen on, an Î(n C,C 2 ()) is the empirical mutual information. Allowing p() to be uniform over a subset of eformations D (an zero outsie this subset), it is =argmaxî(n C,C 2 ()), () D Thus our objective may be state as the maximization of mutual information. We prefer to work with maximum likelihoo formulation because it explicitly clarifies the assumptions about the moel an allows a wier range of choices, e.g. to specify a nontrivial prior on Optimization We follow the stanar approach to optimize (4): the objective is alternatively optimize w.r.t. an θ. In this section we iscuss the etails of these two steps. Uner fixe, maximization of the likelihoo over θ is obtaine by solving (8). For this problem we apply a simple iterative algorithm 6, which in our notation reas: θc new,c 2 = θc ol n c,c 2,c 2 p(c c,c,c 2 ; θol ) g (c,c )g 2 (c 2,c 2). 2 (2) The estimate is initialize to θ c,c 2 = nc,c 2, which is easily seen to be the Parzen winow estimate, an then upate by several iterations of (2). It shoul be notice that the improvement over the initial estimate is not very significant; (2) is use instea of the simple Parzen winow estimate to be consistent with maximum likelihoo formulation. Uner fixe θ, maximizing the log-likelihoo over reas argmax log p()+ log p(i t J (t) ; θ). (3) t T We apply the metho 8 to search for the best eformation. This metho assumes that the eformation is a mapping B K, where B is the set of small blocks into which

4 the image I is subivie an K is the set of possible isplacements each block is allowe to move to (see Fig. 2). The prior on is moele as a prouct of pairwise potentials over the set E of pairs of neighboring blocks: p() = φ bb ( b, b ), (4) Z bb E where φ bb ( b, b ) is or 0 epening on whether the pair of isplacements ( b, b ) is allowe ( D) or not ( / D). This is a so-calle har moel, in which normalization constant Z oes not influence the results since all feasible eformations from D will have the same constant probability Z, an thus the optimal will not epen on Z. When moel (4) is substitute into (3), the problem can be expresse as =argmin q b ( b ) b B bb E log φ bb ( b, b ), (5) where we enote q b ( b )= t b log p(i t J b (t); θ) an switch to minimization, following the convention of energy interpretation. Problem (5) is known as a minimization of pairwise Gibbs energy which is known to be har. A linear programming relaxation approach is applie 8. For the scope of this work it is important to notice that 8 takes explicitly into account that the problem has many local minima an a certain convex relaxation of it is solve (in a certain sense suboptimally, for the sake of spee). evertheless, we refer to this metho as a global one. ote that unlike the approach of e.g. 5, there are pairwise constraints on the neighboring locations of blocks, implying that fining the best translations can only be one jointly an oes not ecompose into fining the best position for each block inepenently. After the best piece-wise translational eformation is foun, we interpolate it to obtain a smooth eformation. I b b b b Figure 2. An approximation of eformation as a piece-wise translational eformation. Image I is regularry subivie into small blocks, each block b B may be move solily by isplacement b, a pair of neighboring blocks is constraine to have small relative isplacement, thus preventing fol-overs an iscontinuities in the eformation fiel. 3. Experiments For a quantitative test we use two synthetic images shown in Fig. 3(b), which correspon to two axial cuts taken J Figure 3. A synthetically generate 2 pair of multi-moal MRI images Figure 4. Convergence plot of the log likelihoo objective function (empirical mutual information). Circles inicate optimization steps w.r.t. parameters θ of the statistical epenence of signals an boxes inicate optimization steps w.r.t. eformation. from a simulate MRI brain scan 2 with ifferent relaxation times. Figure 5(last) shows that there is a non-trivial statistical epenence between colors of the two input images when they are perfectly aligne. A typical run with secon input image eforme like it is exemplifie in Fig. 6 an both images corrupte by (0, 0. 2 ) noise (the signal space is 0, ) is as follows. The progress in log-likelihoo objective uring alternating optimization steps is shown in Fig. 4. ote that the plot starts by estimating the signal epenence parameters θ assuming that the initial eformation is the ientity. It is seen that the metho converges in a few alternating steps. Figure 5 shows the evolution of the conitional ensity estimate p(c c c ; θ). We o not show a visual comparison of registere images, since the registration is quite accurate an it is very ifficult to see the ifferences irectly. What we will be intereste in instea is to measure quantitatively how the recovere eformation iffers from the true eformation r. We teste two types of noise: Gaussian i.i.. ae to each pixel an a clutter-like noise, B(σ 2 ), which makes bright spots appear in ranom positions. Here, σ is the variance (in pixels) of the spatial Gaussian mask efining the size of the spots. We have compare our metho against an elastic metho base on free-form eformations using B-splines an local

5 Ourmetho ITK P{err a} a,px Figure 6. Left: an example of ranomly eforme an corrupte by Gaussian noise of variance The other input image is not eforme but perturbe with the noise as well. Right: the error statistics. For each value of the error the plot shows the estimate probability of the event that an error larger than that woul be encountere on a ranomly taken sample. Soli line shows our results an ashe like shows results of ITK 6, Figure 5. Estimates of the log conitional ensity log p(c c 2; θ) uring first 2 an the last iterations. Bottom: the estimate uner the groun truth eformation an the estimate for noiseless images uner groun truth eformation. P{err a} Ourmetho ITK graient escent optimization of mutual information implemente by the Image Registration Toolkit Software (ITK) 6, 7. For each recovere eformation the maximum error was measure as err = max t r t, (6) t A which correspons to the maximum eviation from the groun-truth eformation. The maximum is compute over the subset A T, which exclues the ark backgroun of the template original image. The eformation of the black backgroun is, inee, unrecoverable. By measuring this error over 00 recovere eformations, we estimate the probability P {err α} that on a ranom sample the error will be α or greater. This probability equals P {err <α} an P {err <α} is compute as empirical cumulative istribution function estimate. Results of our measurements are shown in Figs. 6 an 7. ote that although the errors may seem large (over 0 pixels for some examples), these are maximal errors, whereas average errors over pixels are shown in the Table. The registration by both methos align the images intensities very well an it is left to the regularization prior to solve any ambiguity. Actually, if the groun truth error was not known for every pixel, the error in the estimate eformation woul not be iscovere. Our current implementation takes aroun 30 secons for 37x28 pixel images a,px Figure 7. Similar to figure (6). Gaussian noise with variance 0. 2 an clutter-type noise ae to both input images. The eforme an corrupte by noise secon image is shown on the left. (0, 0. 2 ) B(2 2 ) + (0, 0. 2 ) our ITK our ITK AE Mean AE Meian AE St MOD Mean MOD Meian MOD St Table. Statistics of Angular Error (AE, egrees) an Magnitue of Difference (MOD, pixels) errors for ifferent amount of noise. Errors are compute over the area excluing the ark backgroun of the template original image. 4. Discussion an Conclusions We have presente a metho for image registration base on iscrete search techniques for mutual information an have given an empirical evience of its usefulness as an alternative for local optimization of the mutual information criterion in multi-moal images. Tests on simulate 2D meical images show that the metho is able to consistently recover large eformations in a robust manner an feasible time. When comparing the precision of the metho to

6 a specialize meical image registration system using local optimization (Image Registration Toolkit) the results show that the two systems have similar errors with ITK having a slightly better performance. One of the problems with the metho is that the approximation technique we use in the -optimization step is of course not guarantee to fin a global optimum. Furthermore, reestimating the eformation sometimes ecreases the objective. This happens because 8 fins new approximate optimal solution without taking into account the estimate from the previous step an thus is not guarantee to improve it. It shoul be note that cubic B-spline base methos have a strong avantage by using physically-inspire regularization on the eformation fiel. An open research irection is to a to the current har constraints on the eformation fiel a regularization term, e.g. a bening energy type of regularization, which may be efine using secon erivatives of the eformation fiel. It is clear that with pairwise interactions only, our iscrete energy function cannot moel secon erivatives. It is hence natural to consier higher orer Gibbs energy moels which coul impose this prior. 5. Acknowlegements The authors have been supporte by European Commission grant (DIPLECS), the Czech government grant MSM , an Czech Science Founation Project 02/07/37. The Image Registration Toolkit use to register the images by free-form eformations was use uner Licence from Ixico Lt. References Y. Boykov, O. Veksler, an R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis an Machine Intelligence, 23(): , ov C. CA, K. Kwan, an R. Evans. Brainweb: online interface to a 3-D MRI simulate brain atabase. euroimage, 5:425, A. Collignon, F. Maes, D. Delaere, D. Vanermeulen, P. Suetens, an G. Marchal. Automate multi-moality image registration base on information theory. Information Processing in Meical Imaging, pages , B. Glocker,. Komoakis,. Paragios, G. Tziritas, an. avab. Inter an intra-moal eformable registration: Continuous eformations meet efficient optimal linear programming. In Information Processing in Meical Imaging, pages 8 4, H. Hufnagel, X. Pennec, G. Malanain, H. Hanels, an. Ayache. on-linear 2D an 3D registration using blockmatching an b-splines. In Bilverarbeitung fuer ie Meizin, pages , March M. Jones an D. Henerson. Maximum likelihoo kernel ensity estimation. Research Report 05/0, The Open University, UK, an University of ewcastle, UK, January J. Kim, V. Kolmogorov, an R. Zabih. Visual corresponence using energy minimization an mutual information. In International Conference on Computer Vision, pages 033 0, Komoakis an G. Tziritas. A new framework for approximate labeling via graph cuts. In International Conference on Computer Vision, pages , H. Lester an S. R. Arrige. A survey of hierarchical non-linear meical image registration. Pattern Recognition, 32():29 49, January H. Luan, F. Qi, Z. Xue, L. Chen, an D. Shen. Multimoality image registration by maximization of quantitativequalitative measure of mutual information. Pattern Recognition, 4(): , January 08. F. Maes, D. Vanermeulen, an P. Suetens. Meical image registration using mutual information. Proceeings of the IEEE, 9(0): , J. Maintz an M. Viergever. A survey of meical image registration. Meical Image Analysis, 2(): 36, J. P. W. Pluim, J. B. A. Maintz, an M. A. Viergever. Mutualinformation-base registration of meical images: a survey. Meical Imaging, IEEE Transactions on, 22(8): , A. Roche, G. Malanain, an. Ayache. Unifying maximum likelihoo approaches in meical image registration. International Journal of Imaging Systems an Technology, ():7, S. Roy an M.-A. Drouin. Mrf solutions for probabilistic optical flow formulations. In International Conference on Pattern Recognition, volume 3, Septembre D. Rueckert, L. Sonoa, C. Hayes, D. Hill, M. Leach, an D. Hawkes. on-rigi registration using free-form eformations: Application to breast MR images. IEEE Transactions on Meical Imaging, 8(8):72 72, J. A. Schnabel, D. Rueckert, M. Quist, J. M. Blackall, A. D. Castellano-Smith, T. Hartkens, G. P. Penney, W. A. Hall, H. Liu, C. L. Truwit, F. A. Gerritsen, D. L. G. Hill, an D. J. Hawkes. A generic framework for non-rigi registration base on non-uniform multi-level free-form eformations. In International Conference on Meical Image Computing an Computer-Assiste Intervention, pages , 0. 8 A. Shekhovtsov, I. Kovtun, an V. Hlavac. Efficient MRF eformation moel for non-rigi image matching. In IEEE Transactions on International Conference on Pattern Recognition, P. Viola an W. Wells, III. Alignment by maximization of mutual information. International Journal of Computer Vision, 24(2):37 54, September 997. J. Winn an. Jojic. LOCUS: Learning object classes with unsupervise segmentation. In International Conference on Computer Vision, volume, pages , 05.