Performance of Mutual Information Similarity Measure for Registration of Multi-Temporal Remote Sensing Images

Transcription

1 bstract Performance of Mutual Information Similarity Measure for Registration of Multi-emporal Remote Sensing Images Hua-mei Chen Department of Computer Science and Engineering University of exas at rlington rlington X 7609 US Pramod K. Varshney and Manoj K. rora Department of Electrical Engineering and Computer Science Syracuse University Syracuse NY 3244 ccurate registration of multi-temporal remote sensing images is essential for various change detection applications. Mutual information MI has recently been used as a similarity measure for registration of medical images because of its generality and high accuracy. Its application in remote sensing is relatively new. here are a number of algorithms for the estimation of joint histogram to compute mutual information but they may suffer from interpolation-induced artifacts under certain conditions. In this paper we investigate the use of a new joint histogram estimation algorithm called generalized partial volume estimation GPVE for computing mutual information to register multi-temporal remote sensing images. he experimental results show that higher order GPVE algorithms have the ability to significantly reduce interpolation-induced artifacts. In addition mutual information based image registration performed using the GPVE algorithm produces better registration consistency than the other two popular similarity measures namely mean squared difference MSD and normalized cross correlation NCC used for the registration of multi-temporal remote sensing images. Index erms- multi-temporal images image registration mutual information joint histogram estimation registration consistency. Corresponding author: varshney@syr.edu

2 I INRODUCION emporal change detection study is based on the processing of data collected at different times i.e. multi-temporal data and is important in many applications that include land use land cover change detection deforestation and environmental monitoring etc. variety of change detection algorithms based on tools such as image differencing [] principal component analysis [2] change vector analysis [3] Markov Random Fields [4] and neural networks [5] may be used. he basis of all these algorithms is however accurate registration of images taken at different times. For example registration accuracy of less than one-fifth of a pixel is required to achieve a change detection error of less than 0% [6]. he necessity of accurate image registration and geometric rectification arises due to the presence of a number of distortions errors in remote sensing images that occur as a result of variations in platform positions rotation of earth and relief displacements etc. he behavior of most of these distortions is systematic and thus can be easily removed at data acquisition centers. It is also generally expedient to procure systematic-corrected images as these are already geometrically rectified to a map projection system such as universal transverse mercator UM. However systematic corrections are normally performed on the basis of the platform ephemeris data obtained from the header information which may be relatively inaccurate. herefore some random distortions may still be present in the systematic-corrected data. his can be illustrated with the help of agency supplied Landsat thematic mapper M images taken at two different times as shown in Fig.. he images were systematically corrected and geometrically rectified to UM at the data acquisition center. It can however be seen that the two points marked by "+" having the same UM coordinates in both the images are located at different positions indicating the presence of non-systematic registration errors. o perform change detection

3 studies these images have to be registered with each other to correct for non-systematic errors. his is typically done by selecting a few features also known as ground control points GCP in both images floating image taken at time I and reference image taken at time II. he GCP are matched by pair and used to compute transformation parameters to register the images. resampling procedure is then followed to estimate the intensity values at the new locations of the floating image. he feature based registration technique via manual selection of GCP is however laborious time intensive and a complex task. Some automatic algorithms have been developed to automate the selection of GCP to improve the efficiency [7 8]. However the extraction of GCP may still suffer from the fact that sometimes too few a points will be selected and further the extracted points may be inaccurate and unevenly distributed over the image. his may lead to large registration errors. Hence automatic intensity based registration techniques may be more appropriate than the feature based techniques. In this paper we adopt mutual information MI as the similarity measure for automatic registration of multi-temporal remote sensing images. he major requirement to compute the MI between two images is the accurate estimation of the joint histogram. number of interpolation algorithms such as the linear and partial volume interpolation PVI may be used to estimate the joint histogram. However we have found that the performance of the existing joint histogram estimation algorithms may be limited due to a phenomenon called interpolation-induced artifacts which not only hampers the global optimization process but also limits the accuracy [9 0]. ypical artifact patterns resulting from linear [] and partial volume interpolation [2] are shown in Fig. 2a and 2b. he multiple peaks in these figures are the result of artifacts. o overcome the problem of interpolation-induced artifacts we have developed a new joint histogram estimation algorithm called generalized partial volume estimation GPVE [0]. Here

4 we investigate its application for multi-temporal remote sensing image registration. We also compare the performance of MI based registration that employs GPVE with other intensitybased registration algorithms using mean square difference MSD and normalized crosscorrelation NCC as similarity measures. In Section II we review the general intensity-based image registration approach and point out one common problem encountered when MSD and NCC are used as similarity measures. In Section III we briefly review the MI based registration approach and describe the artifacts phenomenon. he 2D implementation of the GPVE algorithm is described in Section IV. In the absence of ground data i.e. GCP registration consistency [] has been used as a measure to evaluate the performances of different registration algorithms and is reviewed in Section V. Experimental results their discussion and conclusions are provided in Sections VI and VII respectively. II REVIEW OF INENSIY SED IMGE REGISRION PPROCH he main principle behind any intensity based registration technique is to find a set of transformation parameters that globally optimizes a similarity measure. his can be expressed mathematically as * α = arg opt S F R = arg opt S F R α α where F is the floating image R is the reference image α is the transformation model α is the set of parameters involved in α and S represents the chosen similarity measure. From we can observe that to register images F and R using a transformation model α we need to find the * associated transformation parameters α that optimize the selected similarity measure S.

5 wo commonly used similarity measures are MSD and NCC [3]. o use these as valid similarity measures images are assumed radiometrically corrected. However in some cases even though accurate radiometric correction has been applied the registration results using MSD or NCC as similarity measures may still not be reliable. his occurs for example when the scene undergoes a significant amount of change between two times. his can be illustrated by registering small portions [28 28] pixels and [ ] pixels of Landsat M band images taken in years 995 and 998 see Fig. 3. he images are systematically corrected and thus contain no radiometric errors. From Fig. 3 a significant change between the intensity values of the two images can be observed which is probably due to the change in crop types in the region. he 2D registration functions obtained by using MSD and NCC as similarity measures to register images in Fig. 3 are shown in Fig. 4a-b. If both MSD and NCC were suitable similarity measures in this case we would expect a global minimum in Fig. 4a and a global maximum in Fig. 4b both at the position Instead it occurs at 3020 for MSD and at 7030 for NCC. hus these two similarity measures fail to accurately register the two images. In order to overcome this potential difficulty pertaining to MSD and NCC a more robust similarity measure is required. Mutual information MI has been proposed as a suitable similarity measure for many multi-modal image registration problems [24]. nalogous to NCC the objective now is to maximize the MI between the two images. Since its introduction MI has been used widely in many medical image registration applications because of its high accuracy and generality [5 6]. Only recently some work has been initiated on MI based registration of remote sensing images [7-9]. Fig. 5 illustrates the 2D registration function using MI as the similarity measure for the registration of images shown in Fig. 3. It is clear from Fig. 5 that the maximum occurs at

6 the position 5050 thereby demonstrating that MI is able to successfully register the two images which was not the case with either NCC or MSD. III COMPUION OF MUUL INFORMION EWEEN WO IMGES MI of two random variables and can be obtained as [20] I = H + H H 2 where H and H are the entropies of and and H is their joint entropy. Considering and as two images the MI based registration criterion states that the images shall be registered when I is maximal. he entropies and joint entropy can be computed from H = P alog P a 3 a H = P blog P b 4 b H = P a blog P a b 5 a b where P a and P b are the marginal probability mass functions and P a b is the joint probability mass function. MI measures the degree of dependence of and by measuring the distance between the joint distribution P a b and the distribution associated with the case of complete independence P a P b. he probability mass functions can be obtained from P h a b a b = 6 h a b a b P a = P a b 7 b P b = P a b 8 a

7 where h is the joint histogram of the image pair. It is a 2D matrix of the following form: h00 h0 h =... h M 0 h0 h... h M h0 N h N... h M N he value hab a [0 M-] b [0N-] is the number of corresponding pairs having intensity value a in the first image and intensity value b in the second image. It can thus be seen from 2 to 8 that the joint histogram estimate is sufficient to determine the MI between two images. wo commonly used algorithms to estimate the joint histogram of two images are linear interpolation [] and PVI [2]. However when the two images have the same spatial resolution along at least in one direction both algorithms may suffer from interpolation-induced artifacts [9]. he reason is that under this condition the number of grid-aligned pixels may change abruptly when the displacement involved in the geometrical transformation along that direction changes. However if a rotational difference is involved in the two images to be registered artifacts will not appear even when displacements change. his is due to the fact that when a rotational difference is present except for multiples of 90 degrees the number of grid-aligned pixels is always small no matter how the displacements change. n extensive discussion on the mechanisms causing the artifacts and the underlying theory may be found in [9 0]. he artifact patterns may however be present while registering multi-temporal remote sensing images see Fig. 2. o combat this problem we use a new joint histogram estimation algorithm called GPVE which we have successfully employed earlier for the registration of medical brain C and MR images [0]. Its 2D implementation is described in the next section.

8 IV GENERLIZED PRIL VOLUME ESIMION GPVE LGORIHM FOR JOIN HISOGRM ESIMION Let α be the transformation characterized by the parameter set α that will be applied to image see equation. ssume that α maps a grid point with coordinates x y of a pixel in image onto the corresponding point i + j + in image where i j are the coordinates i j of a pixel in and i and j are small displacements with 0 i j <. he unit here is interpixel distance. See Fig. 6 for a geometrical illustration. For each grid point the joint histogram h is updated as: h x y i + p j + q + = f p f q p q Z 9 where Z is the set of all integers and f is a real valued kernel function that satisfies the following two conditions i f x 0 where x is a real number. 0 i j n= ii f n + = where n is an integer 0 < he first condition ensures that the joint histogram values are non-negative while the second condition makes the sum of the updated values equal to one for each corresponding pair of points in image and image. In this paper we employ -spline functions as the kernel functions. he details on -spline functions can be found in [2 22]. It may be noted that the PVI algorithm is a special case when the first order -spline function is used as the kernel function. Fig. 7 shows the 2D MI registration function using the second order GPVE algorithm second order -spline function is used for the same data as was used to produce Fig. 2. Clearly the artifacts have been reduced significantly.

9 V REGISRION CONSISENCY In the absence of ground data registration consistency [] may be used as a measure to evaluate the performance of different intensity-based image registration algorithms. Defining as the transformation obtained using image as the floating image and image as the reference image the registration consistency dp of and can be formulated as 2b / 2a / = = I I y x I I y x y x y x N dp y x y x N dp o o where the composition o represents the transformation that applies first and then. he overlap region of images and is defined as I. he discrete domains of images and are I and I respectively N and N are the number of pixels of image and image within the overlap region. he registration consistency defined in 2a specifies the mean distance of the two mapped points p and p where p is a pixel in image. Similarly 2b represents the mean distance of the two mapped points p and p where p is a pixel in image. In general it is expected that the two values from 2a and 2b will practically be the same. Similarly a three-date registration consistency can be defined as 2 3 y x y x y x y x N dp C C I I y x C C C o o o o + = 3 where I C is the overlap region of images and C. In cases where the transformation required to bring two images in alignment is just a displacement the transformation can be represented as a 2-D vector y x v = where

10 x is the vertical displacement and y is the horizontal displacement. For this case a simplified two-date registration consistency can be defined as dp v + v 2 = 4 Similarly the three-date registration consistency can be defined as dp = v + v + v + v + v + v / C C C C VI. EXPERIMENL RESULS ND DISCUSSION hree multi-temporal image data sets are used in our experiments. hese are Landsat M band images of three different dates and 998 IRS PN images of two different dates 997 and 998 and Radarsat SR images of two different dates 997 and 998. s an example only the Landsat M images are shown in Fig. 8. he area covered by all the images of sizes [52 52] pixels belongs to a portion of the San Francisco ay in California. he images obtained from the source were already systematically corrected. he header files associated with each image indicate that there is no rotational difference in Landsat M images a slight rotational difference of 0.02 degree in IRS PN images and a significant amount of rotational difference i.e degree in the SR images. herefore it is anticipated that interpolation-induced artifacts may be more pronounced while registering multi-temporal Landsat M and IRS PN images than while registering multi-temporal Radarsat SR images see Section III.

11 We evaluate the performance of four algorithms to estimate the joint histogram to compute mutual information. hese are linear interpolation the first order GPVE algorithm which is actually the PVI algorithm and the second and third order GPVE algorithms. Simplex search procedure [2324] is used to find the global optimum in all cases. Since the simplex search procedure is just a local optimizer there are instances when it may fail to find the position of the global optimum. Under those circumstances we change the initial search points until the correct optimum is obtained. ables -6 show all the experimental results of registering different pairs of multi-temporal remote sensing images. he first date in each pair serves as the floating image and the second date as the reference image. Since Landsat M images are obtained for three dates three sets of registration are performed. ables to 3 show these registration results in the form of transformation parameters i.e. displacements in x and y directions along with the corresponding two-date registration consistency. able 4 shows the corresponding three-date registration consistencies of each algorithm to register these images. glance at the first four rows of tables -4 shows no significant difference between the performances of each algorithm as all of them perform fairly well with the linear interpolation resulting in the worst registration consistency. Further on close inspection of transformation parameters for the four joint histogram estimation algorithms in ables -3 we notice that the PVI algorithm results in almost perfect registration consistency. However if we plot the registration function of each algorithm for a pair of images to be registered we can observe the differences in these algorithms because of interpolation-induced artifacts. One such illustration is provided in Fig. 9 a to c for the registration of Landsat M images of 995 and 997. he artifact patterns can be clearly seen in these plots. In fact the PVI algorithm which has produced perfect registration consistency as observed from the transformation parameters also shows the

12 presence of artifacts Fig. 0a and b. In contrast the second and higher order GPVE do not result in any artifact patterns see Fig. 0c and d for second order as an example. hus higher order GPVE implementation clearly has an advantage over linear and PVI algorithms since the resultant registration function is very smooth. Similar conclusions can be drawn from the registration result of IRS PN images reported in able 5. Linear and PVI algorithms result in artifact patterns which are significantly reduced on implementing higher order GPVE algorithms as can be seen from plots of D registration function in y direction in Fig. a to c. In case of registration of Radarsat SR images able 6 we did not observe any artifact patterns when linear and PVI algorithms are used. his is because of the rotational difference between the two images as described in Section III. In this case the linear interpolation has again resulted in relatively poor registration consistency while PVI and higher order GPVEs have similar performance. hus joint histogram estimation using higher order GPVE algorithms produces more reliable registration results than using linear or PVI algorithms. Further in order to evaluate the performance of our MI based registration algorithm the registration results of 3 rd order GPVE algorithm are compared with those obtained from image registration using MSD and NCC as similarity measures. Linear image interpolation is used when implementing the algorithms based upon these two measures. he registration results obtained via MSD and NCC are shown in the last two rows of ables -6. From our experiments although it is hard to determine which similarity measure results in better registration accuracy due to a lack of ground data. However on the basis of registration consistency it can be clearly seen that MI based registration implemented through higher order GPVE algorithms outperforms the registration obtained with MSD and NCC as similarity measures.

13 VI. CONCLUSIONS MI based image registration approach is investigated in this paper for three multitemporal remote sensing data sets. In cases when the images to be registered have the same spatial resolution and orientation the application of MI based registration through conventional implementations like linear interpolation and partial volume interpolation may create problems due to the presence of artifacts as shown by the results in this paper. We have introduced a new joint histogram estimation algorithm called GPVE to calculate the mutual information. y choosing the 2 nd order or the 3 rd order -splines as the kernel functions involved in GPVE we have shown that the artifacts can be reduced significantly thereby improving registration accuracy. lthough a precise evaluation of registration accuracy is not possible without the availability of accurate ground data we have shown that MI based registration implemented through the higher order GPVE algorithm results in better registration consistency than the registration performed using MSD and NCC as the similarity measures. CKNOWLEDGEMENS his research was supported by National eronautics and Space dministration NS under grant number NG he remote sensing data was provided by the Eastman Kodak Company under the multi modality image fusion study program.

14 REFERENCES [] V. Castelli C. D. Elvidge C. S. Li and J. J. urek Classification-based change detection: heory and applications to the NLC data set in Remote Sensing Change Detection: Environmental Monitoring Methods and pplications R. S. Lunetta and C. D. Elvidge eds. Michigan: nn rbor Press 998 pp [2] J. F. Mas Monitoring land-cover changes: a comparison of change detection techniques Int. J. Remote Sensing vol. 20 pp [3] E. F. Lambin and. H. Strahler Indicators of land-cover change for change-vector analysis in multitemporal space at coarse spatial scales Int. J. Remote Sensing vol. 5 pp [4]. Kasetkasem and P. K Varshney " n image change detection algorithm based on Markov random field models" IEEE rans. Geosci. Remote Sensing vol. 40 pp ug [5] X. Liu and R. G. Lathrop Urban change detection based on an artificial neural network Int. J. Remote Sensing vol. 23 pp [6] X. Dai and S. Khorram "he effects of image misregistration on the accuracy of remotely sensed change detection" IEEE rans. Geosc.Remote Sensing vol. 36 pp Sept [7] X. Dai and S. Khorram " feature-based image registration algorithm using improved chain-code representation combined with invariant moments" IEEE rans. Geosci. Remote Sensing vol.37 pp Sept [8] C. K. oth and. Schenk "Feature-based matching for automatic image registration" IC Journal vol. pp [9] J. P. W. Pluim J... Maintz and M.. Viergever " Interpolation artifacts in mutual information-based image registration" Computer Vision and Image Understanding vol. 77 pp [0] H. Chen and P. K. Varshney "Mutual Information ased C-MR rain Image Registration Using Generalized Partial Volume Joint Histogram Estimation" to appear in IEEE rans. Medical Imaging. [] M. Holden D. L. G. Hill E. R. E. Denton J. M. Jarosz. C. S. Cox. Rohlfing J. Goodey and D. J. Hawkes "Voxel similarity measures for 3-D serial MR brain image registration" IEEE rans. Medical Imaging vol. 9 pp Feb

15 [2] F. Maes. Collignon D. Vandermeulen G. Marchal and P. Suetens Multimodality image registration by maximization of mutual Information IEEE rans. Medical Imaging vol. 6 pp [3] L. G. rown survey of image registration techniques CM Computing Surveys vol. 24 pp [4] P. Viola and W. Wells lignment by maximization of mutual information in Proc. 5 th International Conference on Computer Vision pp [5]. K. Erdi Y. Hu and C. Chui "Using mutual information MI for automated 3D registration in the pelvis and thorax region for radiotherapy treatment planning" in Proc. Society of Photo-Optical Instrument Engineers SPIE vol pp [6] V. Zagrodsky R. Shekhar and J.F. Cornhill "Mutual information-based registration of cardiac ultrasound volumes" in Proc. Society of Photo-Optical Instrument Engineers SPIE vol.3979 pp [7] H. Chen and P. K. Varshney pyramid approach for multimodality image registration based on mutual information in Proc.Fusion' 2000 pp. MoD3-9 MoD [8] K. Johnson. C. Rhodes I Zavorin and J. LeMoigne Mutual information as a similarity measure for remote sensing image registration in Proc. SPIE vol pp [9] J. Inglada Similarity measures for multisensor remote sensing images in Proc. IGRSS 02 CD ROM. [20]. M. Cover and J.. homas Elements of Information heory John Wiley and Sons 99. [2] M. Unser. ldroubi and M. Eden "-spline signal processing: Part I-theory" IEEE rans. Signal Processing vol. 4 pp [22] M. Unser. ldroubi and M. Eden "-spline signal processing: Part II-efficient design" IEEE rans. Signal Processing vol. 4 pp [23] J.. Nelder and R. Mead simplex method for function minimization he Computer Journal vol. 7 pp [24] H. Chen Mutual Information ased Image Registration with pplications Ph.D. dissertation Syracuse University Syracuse NY 2002.

16 able. Registration results for Landsat M 995 and 997 images. denotes the transformation parameters vertical and horizontal displacements lgorithm 95-> > date consistency [pixel pixel] [pixel pixel] Linear [ ] [ ] st order [ ] [ ] nd order [ ] [ ] rd order [ ] [ ] MSD [ ] [ ] NCC [ ] [ ] able 2. Registration results for Landsat M 995 and 998 images. denotes the transformation parameters vertical and horizontal displacements lgorithm 95-> > date consistency [pixel pixel] [pixel pixel] Linear [ ] [ ] st order [ ] [ ] nd order [ ] [ ] rd order [ ] [ ] MSD [ ] [ ] NCC [ ] [ ] able 3. Registration results for Landsat M 997 and 998 images. denotes the transformation parameters vertical and horizontal displacements lgorithm 97-> > date consistency [pixel pixel] [pixel pixel] Linear [ ] [ ] st order [ ] [ ] nd order [ ] [ ] rd order [ ] [ ] MSD [ ] [ ] NCC [ ] [ ]

17 able 4. 3-date registration consistency for registration of Landsat M and 998 images. lgorithm 3-date Registration Consistency Linear 0.87 st order nd order rd order MSD NCC able 5. Registration results for IRS PN 997 and 998 images. denotes the transformation parameters rotation angle vertical and horizontal displacements lgorithm date consistency [degree pixel pixel] [degree pixel pixel] Linear [ ] [ ] st order [ ] [ ] nd order [ ] [ ] rd order [ ] [ ] MSD [ ] [ ] NCC [ ] [ ] able 6. Registration results for Radarsat SR 997 and 998 images. denotes the transformation parameters vertical and horizontal displacements lgorithm date consistency [degree pixel pixel] [degree pixel pixel] Linear [ ] [ ] 0.50 st order [ ] [ ] nd order [ ] [ ] rd order [ ] [ ] MSD [ ] [ ] NCC [ ] [ ] 0.064

18 a b Fig. a Landsat M band image taken in 997. b Landsat M band image taken in 998. he two crosses mark the points of the same UM coordinates in each frame. he noticeable shift demonstrates the presence of registration error in systematic-corrected data. a b Fig. 2. rtifact patterns in the MI registration function using a linear interpolation and b partial volume interpolation.

19 a b Fig. 3. wo small portions of Landsat M band images taken in 995 and 998 respectively. he white box in image b indicates the corresponding sub-set of image a. a b Fig. 4. wo dimensional registration functions while registering images shown in Fig. 3 using a MSD and b NCC as similarity measures.

20 Fig. 5. 2D registration function while registering images shown in Fig. 3 using MI as the similarity measure. x y i j j i i + i j + α j Fig. 6. Geometrical illustration of the transformation α

21 Fig. 7. MI registration function using 2 nd order GPVE. Compare this with Fig. 5 and notice the reduction in artifacts. a b c Fig. 8. Landsat M images used in the experiments. Images were acquired in a 995 b 997 and c 998.

22 .5 MI y-displacement x-displacement 66 a MI MI x-displacement y-displacement c b Fig. 9. Registration functions obtained through linear interpolation for registration of Landsat images of 995 and 997. a landscape of the 2D MI registration function. b D registration function in x direction. c D registration function in y direction.

23 MI MI x-displacement y-displacement a b MI MI x-displacement y-displacement c d Fig. 0. D registration functions resulting from PVI a and b and 2 nd order GPVE c and d in registering Landsat M images of 995 and 997.

24 MI y-displacement a MI y-displacement b MI y-displacement c Fig.. D registration functions in y direction resulting from different algorithms in registering IRS PN images of 997 and 998. a Linear interpolation b PVI c 2 nd order GPVE