Wavelet for Image Fusion

Transcription

1 Wavelet for Image Fusion Shih-Gu Huang ( 黃世谷 ) Graduate Institute of Communication Engineering & Department of Electrical Engineering, National Taiwan University, Abstract Image fusion is the process that combines information from multiple images of the same scene. The result of image fusion is a new image that retains the most desirable information and characteristics of each input image. The main application of image fusion is merging the gray-level high-resolution panchromatic image and the colored low-resolution multispectral image. It has been found that the standard fusion methods perform well spatially but usually introduce spectral distortion. To overcome this problem, numerous multiscale transform based fusion schemes have been proposed. In this paper, we focus on the fusion methods based on the discrete wavelet transform (DWT), the most popular tool for image processing. Due to the numerous multiscale transform, different fusion rules have been proposed for different purpose and applications. In this paper, experiment results of several applications and comparisons between different fusion schemes and rules are addressed. 1. Introduction 1.1. Image fusion Image fusion is the process that combines information from multiple images of the same scene. These images may be captured from different sensors, acquired at different times, or having different spatial and spectral characteristics. The object of the image fusion is to retain the most desirable characteristics of each image. With the availability of multisensor data in many fields, image fusion has been receiving increasing attention in the researches for a wide spectrum of applications. We use the following four examples to illustrate the purpose of image fusion: (1) In optical remote sensing fields, the multispectral (MS) image which contains color information is produced by three sensors covering the red, green and blue spectral wavelengths. Because of the trade-off imposed by the physical constraint between spatial

2 and spectral resolutions, the MS image has poor spatial resolution. On the contrast, the panchromatic (PAN) images has high spatial resolution but without color information. Image fusion can combine the geometric detail of the PAN image and the color information of the MS image to produce a high-resolution MS image. Fig. 1.1 shows the MS and PAN images provided by IKONOS, a commercial earth observation satellite, and the resulting fused image [1]. (a) (b) (c) Fig Image fusion for an IKONOS scene for Cairo, Egypt: (a) multispectral low resolution input image, (b) panchromatic high resolution input image, and (c) fused image of IHS. (2) As the optical lenses in CCD devices have limited depth-of focus, it is often impossible to obtain an image in which all relevant objects are in focus. To achieve all interesting objects in focus, several CCD images, each of which contains some part of the objects in focus, are required. The fusion process is expected to select all focused objects from these images. An experiment is shown in Fig. 1.2 [2]. (a) (b) (c) Fig CCD visual images with the (a) right and (b) left clocks out of focus, respectively; (c) the resulting fused image from (a) and (b) with the two clocks in focus. (3) Sometimes the in focus problem is due to the different characteristics of different types of optical sensors, such as visual sensors, infrared sensors, Gamma sensors and X-Ray

3 sensors. Each of these types of sensors offers different information to the human operator or a computer vision system. An experiment is shown in Fig. 1.3 [3]. The image in Fig. 1.3(a), captured from a visual sensor, provides most visual and textural details, while the image in Fig. 1.3(b), captured from an infrared sensor, can highlight the man hiding behind the smoke. Therefore, the image fusion process can combine all the interesting details into a composite image (see Fig. 1.3(c)). (a) (b) (c) Fig Images captured from the (a) visual sensor and (b) infrared sensor, respectively; (c) the resulting fused image from (a) and (b) with all interesting details in focus. (4) In medical imaging, different medical imaging techniques may provide scans with complementary and occasionally conflicting information, such as magnetic resonance image (MRI), computed tomography (CT), positron emission tomography (PET), and single photon emission computed tomography (SPECT). In Fig. 1.4, the MRI and PET images are fused [4]. The PET is a functional image displaying the brain activity, but without anatomical information. The MRI, having higher spatial resolution than the PET, provides anatomical information but without functional activity. The object of image fusion is to achieve a high spatial resolution image with functional and anatomical information.

4 (a) (b) (c) Fig (a) MRI and (b) PET images; (c) fused image from (a) and (b) Image fusion methods There are various methods that have been developed to perform image fusion. Some well-known image fusion methods are listed below [5-9]. (1) Intensity-hue-saturation (IHS) transform based fusion (2) Principal component analysis (PCA) based fusion (3) Arithmetic combinations (a) Brovey transform (b) Synthetic variable ratio technique (c) Ratio enhancement technique (4) Multiscale transform based fusion (a) High-pass filtering method (b) Pyramid method (i) Gaussian pyramid (ii) Laplacian Pyramid (iii) Gradient pyramid (iv) Morphological pyramid (v) Ratio of low pass pyramid (vi) Contrast pyramid (vii) Filter-subtract-decimate pyramid (c) Wavelet transform (i) Discrete wavelet transform (DWT) (ii) Stationary wavelet transform (iii) Dual tree discrete wavelet transform (iv) Lifting wavelet transform

5 (v) Multi-wavelet transform (d) Curvelet transform (5) Total probability density fusion (6) Biologically inspired information fusion In this paper, we first introduce the IHS based and PCA based fusion schemes because they are classical, simple and probably the most popular. Then we focus our attention on the DWT based fusion methods and hybrid methods which combine the IHS and DWT. As for the other fusion methods mentioned above, a brief introduction of various pyramid methods, total probability density fusion and biologically inspired information fusion is available in [6]. Comparison of different wavelet transform in image fusion is available in [8], while the arithmetic combinations have discussed in [9] Wavelet based image fusion The standard image fusion techniques, such as IHS based method, PCA based method and Brovey transform method operate under spatial domain. However, the spatial domain fusions may produce spectral degradation. This is particularly crucial in optical remote sensing if the images to fuse were not acquired at the same time. Therefore, compared with the ideal output of the fusion, these methods often produce poor result. Over the past decade, new approaches or improvements on the existing approaches are regularly being proposed to overcome the problems in the standard techniques. As multiresolution analysis has become one of the most promising methods in image processing, the wavelet transform has become a very useful tool for image fusion. It has been found that wavelet-based fusion techniques outperform the standard fusion techniques in spatial and spectral quality, especially in minimizing color distortion [7,10-11]. Schemes that combine the standard methods (HIS or PCA) with wavelet transforms produce superior results than either standard methods or simple wavelet-based methods alone. However, the tradeoff is higher complexity and cost Organization The rest of this paper is organized as follows. Section 2 describes some image processing backgrounds that would be used in the fusion schemes. In Section 3, we give a brief introduction to the discrete wavelet transform (DWT). In Section 4, the standard image fusion schemes and some DWT based fusion schemes are introduced, while several basic and simple fusion rules are introduced in Section 5. Section 6 presents some experimental results and comparisons between different fusion schemes and rules. Finally, this paper is concluded in Section 7.

6 2. Background 2.1 Image registration (IR) [16] IR is the process that transforms several images into the same coordinate system. For example, given an image, several copies of the image are out-of-shape by rotation, shearing, twisting, etc. With the given image as reference, IR can align the out-of-shape images to be the same as the given image. Therefore, IR is essential preprocessing operation for image fusion. 2.2 Image resampling (RS) [17] RS is the procedure that creates a new version of the original image with a different width and height in pixels. Simply speaking, RS can change the size of the image. Increasing the size is called upsampling, for example, the left image in Fig. 2.1 is enlarged by a factor of 10 as shown in the right image in Fig On the contrast, decreasing the size is called downsamplig. Note that the spatial resolution would not change after the RS procedure, either upsampling or downsamplig. Fig Image upsampling with a factor of Histogram matching [18] Consider two images X and Y. If Y is histogram-matched to X, the pixel values of Y is changed by a nonlinear transform such that the histogram of the new Y is the as that of X.

7 3. Discrete Wavelet Transform: A Review In this paper, we only introduce the discrete wavelet transform (DWT) based fusion schemes because DWT is the basic and simplest transform among numerous multiscale transform and other type of wavelet based fusion schemes are usually similar to the DWT fusion scheme. Therefore, we give a brief introduction of the DWT [15] in this section Continuous wavelet transform (CWT) Given a input signal x(t), the CWT of x(t) is defined as 1 t a Xw ( a, b) = x() t ψ dt b, b where location factor a can be any real number, and scaling factor b can be positive real number. The mother wavelet ψ(t) is a well-designed function so that the CWT has low computation complexity and is reversible. It is obvious that as b is larger, (( t a)/ b) ψ is more like a high-frequency signal, and thus output X w (a, b) would represent the high-frequency component of x(t) after inner product with (( t a)/ b) implies the window size of (( t a)/ b) ψ. Also, larger b ψ is smaller; that is, the location (time) resolution is smaller. Therefore, the location-scaling (time-frequency) resolution plane is as Fig Fig Time-frequency resolution plane for wavelet transform.

8 3.1. Continuous wavelet transform with discrete coefficients (DC-CWT) Although the CWT performs well in mathematics, it is hard to implement. Thus, it is not useful in practical. As we restrict the values of parameters a and b as a = n2 m and b = 2 m, the CWT can be rewritten as /2 (, ) 2 m m w = ( ) ψ(2 ) X n m x t t n dt. We call this special case the CWT with discrete coefficients (DC-CWT). The main reason of this setting is easy in implementation. As the mother wavelet ψ(t) satisfies k k, k k () t = 2 h ( 2 t k) and () t = 2 g ( 2t k) ψ φ φ φ X w (n, m) can be computed from X w (n, m 1) by digital convolution, 1 2 w kχw k χ ( nm, ) = 2 g (2 n+ km, + 1) ; 1 2 w kχw k X ( n, m) = 2 h (2 n+ k, m+ 1). The φ(t), called scaling function, can be deemed as a low-pass filter compared to the high-pass filter ψ(t). Although the setting of a = n2 m and b = 2 m, we can only obtain some coefficients in the particular positions of the time-frequency distribution, as shown in Fig However, these coefficients are enough for most acoustics and image processing. Fig Time-frequency positions of the DC-CWT coefficients.

9 3.1. Discrete wavelet transform (DWT) D discrete wavelet transform The DWT is similar to the DC-CWT except that the input signal is discrete. Therefore, the design rules for ψ(t), φ(t), g[k] and h[k] are similar as in the DC-CWT. The block diagram of the 1-D DWT is illustrated in Fig Fig The block diagram of the 1-D DWT Multi-level 1-D discrete wavelet transform Furthermore, the 1-D DWT confidents can be decomposed again using the 1-D DWT. This scheme is called multi-level 1-D DWT (see Fig. 3.4 for 2-level 1-D DWT). Fig The block diagram of the 2-level 1-D DWT Inverse discrete wavelet transform (IDWT) The reconstruction process from the DWT coefficients is shown in the right part of Fig. 3.5, called inverse DWT (IDWT). The filters h[n], g[n], h 1 [n] and g 1 [n] in the figure can be design by quadrature mirror filter (QMF) method or Orthonormal filter method.

10 Fig The block diagrams of DWT and IDWT D discrete wavelet transform 2-D DWT is very useful for image processing because the image data are discrete and the spatial-spectral resolution is dependent on the frequency. An example is shown in Fig The DWT has the property that the spatial resolution is small in low-frequency bands but large in high-frequency bands. The left-top sub-image (the band with lowest frequencies) has the smallest spatial resolution and represents the approximation information of the original image. Thus, the DWT is suitable for image compression. On the contrast, the other sub-images (the bands with high frequencies) show the detailed information of the original image. Therefore, these sub-images can be used for edge detection or corner detection. Fig The 2-D DWT of a square image.

11 4. Image Fusion Schemes In this section, we use two examples of image fusion to some basic fusion schemes, including intensity-hue-saturation (IHS) transform fusion, principal component analysis (PCA) fusion and wavelet-based fusion schemes. One example is the fusion of panchromatic (PAN) and multispectral (MS) images. The fused image is requested to combine the high-resolution spatial information of the PAN and the color information of the MS image. See Fig. 1.1 in Section 1. Another example is the fusion of multifocus images. In this example, several images with different focus regions are combined to produce a fused image that contains all these focus regions. See Fig. 1.2 in Section Panchromatic and multispectral image fusion Consider that the PAN and MS images are acquired from a remote IKONOS sensor. IKONOS is a commercial earth observation satellite. It offers MS and PAN imagery characterized by 4-meter and 1-meter spatial resolution respectively. The PAN image is without color information while the MS image covers three spectral bands (MS1 red, MS2 green, MS3 blue) as shown in Table 4.1. In order to take benefit of the high spatial information of the PAN image and the essential spectral information of the MS image, we introduce HIS based, PCA based, DWT based and DWT combined with HIS fusion schemes. Band Spectral resolution Spatial resolution PAN µm 1 meter MS3 (Blue) µm 4 meters MS2 (Green) µm 4 meters MS1 (Red) µm 4 meters Near IR µm 4 meters Table 4.1. The spectral and spatial resolution of IKONOS.

12 Standard IHS fusion method As the MS image is represented in RGB color space, we can separate the intensity (I) and color information, hue (H) and saturation (S), by IHS transform. The I component can be deemed as an image without color information. Because the I component resembles the PAN image, we match the histogram of the PAN image to the histogram of the I component. Then, the I component is replaced by the high-resolution PAN image before the inverse IHS transform is applied. The main steps, illustrated in Fig. 4.1, of the standard HIS fusion scheme are [7,12-13]: (1) Perform image registration (IR) to PAN and MS, and resample MS. (2) Convert MS from RGB space into IHS space. (3) Match the histogram of PAN to the histogram of the I component. (4) Replace the I component with PAN. (5) Convert the fused MS back to RGB space. Fig Standard IHS fusion scheme. In step (1), image resampling (RS) is utilized so that the MS image has the same spatial resolution as the PAN image. IR and RS have introduced in Section 2. In step (3), because the mean and variance of the I component (I=(R+G+B)/3) are different to those of the PAN image, histogram matching is employed to prevent the change of the histogram of the MS image.

13 Fig Spectral response of the PAN and MS sensor of IKONOS Standard PCA fusion method An alternative to IHS-based method is principal component analysis (PCA). In Fig. 4.2 [14], it is found out that the MS bands are somewhat correlated. The PCA transform can convert the correlated MS bands into a set of uncorrelated components, say PC1, PC2, PC3 The first principle component (PC1) also resembles the PAN image. Therefore, the PCA fusion scheme is similar to the IHS fusion scheme [9-10,12]: (1) Perform IR to PAN and MS, and resample MS. (2) Convert the MS bands into PC1, PC2, PC3, by PCA transform. (3) Match the histogram of PAN to the histogram of PC1. (4) Replace PC1 with PAN. (5) Convert PAN, PC2, PC3, back by reverse PCA. Fig Standard PCA fusion scheme. In general, the PC1 collects the spatial information which is common to all the bands, while

14 PC2, PC3 collect the spectral information that is specific to each band [7]. Therefore, PCA based fusion is very suitable for merging the MS and PAN images. Compared to the IHS fusion, the PCA fusion has the advantage that the MS images is allowed to contain more than three bands. For instance, the near infrared component (Table 4.1) is also taken into account, or the MS image is form from more than one (satellite) sensors. In this case, the MS bands cannot be exactly separated into R, G and B bands Substitutive DWT fusion method The standard IHS and PCA based fusion methods is suitable for the case that the I component (or PC1) of the MS image is highly correlated with the PAN image. This is possible only if the PAN band covers all the MS bands, and also if both the I component (or PC1) of the MS image and the PAN image has similar spectral response. The second condition is usually impossible (see Fig. 4.2). Thus, the two standard fusion methods usually introduce spectral distortion. A As mentioned in Section 3, the one-level DWT can decompose an image into a set of low-resolution sub-images (DWT coefficients), LL, H1, H2 and H3. The LL sub-image is the approximation image while the H1, H2 and H3 sub-images contains the details of the image. It is straightforward to have the fusion method to retain the LL sub-image of the MS image and replace the H1, H2 and H3 by those of the PAN image. Therefore, the fused image contains the extra spatial details from the high resolution PAN image. Also, if we downsample the fused image, the low-resolution fused image will be approximately equivalent to the original low-resolution MS image. That is, the DWT fusion method may outperform the standard fusion methods in terms of minimizing the spectral distortion. The main steps, illustrated in Fig. 4.4, of the substitutive DWT fusion scheme are [10]: (1) Perform IR to PAN and MSi, and resample MSi. (2) Match the histogram of PAN to the histogram of MSi. (3) Apply DWT to both the histogram-matched PAN and MSi. (4) Replace the detail sub-images (H1, H2 and H3) of MSi with those of PAN. (5) Perform IDWT on the new combined set of sub-images.

15 Fig Substitutive DWT fusion scheme. If the MS image contains three bands, i.e. MS1-MS3, the steps mentioned above require to be repeated 3 times. That is, four DWTs and three IDWTs are required. Since the resolution of the MS image is 1/4 of that of the PAN image in KONOS, we can modify the scheme as: (1) Perform IR to PAN and MSi. (2) Match the histogram of PAN to the histogram of MSi. (3) Transform the histogram-matched PAN using the n-level DWT (n=2 in the case). (4) Replace the approximation (LL) sub-image of PAN with MSi. (5) Perform IDWT on the new combined set of sub-images. This modification can save 3 DWT computations. This modification can be applied to any DWT based fusion methods. However, in some applications, the RS process is still necessary in step (1), and we will discuss it in Section Substitutive IHS-DWT fusion method In this subsection, we introduce the hybrid method that combines the IHS fusion and the DWT fusion. A great deal of research has focused on incorporating the IHS transform into wavelet methods, since the IHS fusion methods performs well spatially while the wavelet fusion methods perform well spectrally [10]. Besides, as IHS transform is employed, the fusion process is reduced to the fusion of the I component and the PAN image, and thus we do not need to repeat the fusion process for all MS bands. However, again the restriction of using IHS-DWT fusion method is that the MS image should contain only three bands. The main steps, illustrated in Fig. 4.5, of the substitutive IHS-DWT fusion scheme are [10,12]: (1) Perform IR to PAN and MS, and resample MS. (2) Convert MS from RGB space into IHS space. (3) Match the histogram of PAN to the histogram of the I component. (4) Apply DWT to both the histogram-matched PAN and the I component.

16 (5) Replace the detail sub-images (H1, H2 and H3) of the I component with those of PAN. (6) Perform IDWT to obtain fused I component. (7) Convert the resulting MS back to RGB space. Fig Substitutive IHS-DWT fusion scheme Multifocus image fusion As the optical lenses in CCD devices have limited depth-of focus, it is often impossible to obtain an image in which all relevant objects are in focus. One possible solution to this problem is to combine several CCD images, each of which contains some part of the objects in focus. The fusion schemes for this example (Fig 4.6 [4]) are similar to the DWT fusion scheme and the modified one in Section Consider there are two CCD images X and Y. If X and Y have similar spatial resolution, a RS followed by an IR are previously required, as shown in Fig 4.6 (a). If the resolution of one image, say Y, is smaller than 1/2 n resolution of another image, say X, a n-level DWT is only applied to Y. If Y and the approximation (LL) sub-image of X still have different resolution, IR and RS is still required (see Fig. 4.6 (b) with n=1). Besides, the histogram matching is neglected since the histogram reference is not determined. (a)

17 (b) Fig Schemes of multifocus image fusions where (a) input images have similar resolution and (b) one input image has resolution (at least 1/2) smaller than another one. For merging the DWT coefficients, the substitution strategy used in all fusion methods in Section 4.1 is no longer suitable in this example. A simple and straightforward method is choose-max (CM) [4], i.e. choosing the maximal DWT coefficients between X and Y. The steps of this method using the scheme in Fig. 4.6 (a) are: (1) Perform IR and RS to X and Y. (2) Apply DWT to both X and Y, and their coefficients in pixel p are D X (p) and D Y (p), respectively. (3) The output DWT coefficient in pixel p is D Z (p) given by DX( p), if DX( p) DY( p) DZ ( p) =. DY( p), if DX( p) DY( p) (4) Perform IDWT to D Z. 5. Fusion Rules In this Section, we focus on the DWT fusion methods. Most DWT fusion schemes are similar to the schemes illustrated in Figs. 4.4 and 4.6. However, various fusion rules are proposed for a wide variety of applications. The fusion rule includes two parts, activity-level measurement method and coefficient combining method. In Section 4, we have introduced substitution fusion rule in the example of PAN and MS image fusion. In this fusion rule, the activity-level measurement is not used while the coefficient combining method is simply the substitution. In the example of multifocus image fusion, the activity-level measurement is DI ( p ) while the coefficient combining method is choose-max (CM). In the following, some simple and basic fusion rules, picked out from [4], are introduced. First, consider two source images X and Y, and the fused image Z. Generally, an image I has its

18 DWT coefficients denoted as D I and the activity level as A I. Thus, we shall encounter D X, D Y, D Z, A X and A Y. The index of each DWT coefficient is denoted by p = (m, n, k, l), illustrated by Fig. 5.1, where k is the decomposition level while I is the index of the frequency band. So (m, n, k, l) implies the (m,n)-th coefficient in LL (k=0, l=0) or Hkl band. Therefore, D I (p) and A I (p) are the value and activity level of p DWT coefficient. Note that coefficients in different level can use different fusion rule. (m,n, 0,0) (m,n, 1,1) (m,n, 1,2) (m,n, 1,3) (m,n,2,2) (m,n,2,1) (m,n,2,3) Fig The two-level DWT coefficients Activity-level measurement methods There are three categories of methods for computing the activity level A I (p) at position p: coefficient-based, window-based and region-based measures. (1) Coefficient-based activity (CBA) In CBA, the activity level is given by A( p) = D( p) or A( p) = D( p). I I I I 2 (2) Window-based activity (WBA) The WBA employ a small (typically 3 3 or 5 5) window centered at the current coefficient position. Thus, the activity level A I (p) is determined by the coefficients surrounding p using a small. One option of the WBA is the weighted average method (WA-WBA) AI( p) = w( s, t) DI( m+ s, n+ t, k, l), s S, t T where wst (,) is the weighting function, and the value of S and T is determined by the

19 window size. Compared with the CBA, the WBA has the benefit of smaller interference from noise. Other possible options of the WBA include the rank filter method (RF-WBA), the spatial frequency method (SF-WBA) and statistical method (ST-WBA). (3) Region-based activity (RBA) The regions used in RBA measurement are similar to windows with odd shapes. In the following, we list the procedure for calculating the activity level of RBA. (a) Perform image segmentation on the LL band sub-image. Therefore, the LL band sub-image is divided into several regions. (b) Any region R v in the low-frequency band has a corresponding group of coefficients, defined as C(R v ), in each high-frequency band, as illustrated in Fig (c) (d) 1 1 A R D p A R D p V V 2 I( ) = I( ) or I( ) = I( ). N V V V p C( R ) NV p C( R ) CBA or RF-WBA, if p is on an edge AI ( p) = V V. AI ( R ), if p is inside R Fig The different black boxes, associated to each decomposition level, are coefficient corresponding to the same image spatial representation in each original image, i.e. the same pixel or pixels positions in the original images Coefficient combining methods Selection and averaging are probably the most popular coefficient combining methods. Selection method can collect the largest DWT coefficients between two images. Therefore, it is suitable for collecting the edges or corners, i.e. detailed information. Averaging method is employed while the DWT coefficients between two images are both important. Therefore, averaging method is suitable for combing the low-frequency bands because the approximation images of the images to be fused usually look similar. (1) Selection The simplest selection method is choose-max (CM),

20 DX( p), if AX( p) AY( p) DZ ( p) = {. D ( p), if A ( p) A ( p) Y X Y In the high-frequency bands (H11, H12, H13, H21 ), the larger DWT coefficients correspond to sharper brightness changes and thus to the salient features in the image such as edges, lines, and region boundaries [4]. Therefore, the CM method is useful in the collection of the detailed information. In multifocus image fusion, the input images X and Y have diffident regions out of focus. It is obvious the detailed DWT coefficients of the out-of-focus region in X would be smaller than those of the corresponding region in Y. Thus, the CM scheme is very suitable for merging the detail information of X and Y. (2) General weighted average (WA) For each p, the composite D Z is obtain by D ( p) = w ( p) D ( p) + w ( p) D ( p). Z X X Y Y The weighting factors w X (p) and w Y (p) can be deterministic or dependent on the activity levels of X and Y, given by (index p is omitted) wx = 1 and wy = 0, if AX AY, MXY < α M XY wx 1 wy and w = Y =, if AX AY, MXY α α, wx = 0 and wy = 1, if AX AY, MXY < α M XY w X = and wy = 1 wx, if AX AY, MXY α α where A X and A Y are obtained by WA-WBA method, and M XY (p) is defined as the normalized cross-correlation between X and Y over a neighborhood of p, M XY ( p) = s S, t T wstd (,) ( m+ sn, + tkld,,) ( m+ sn, + tkl,,) X A ( p) + A ( p) 2 2 X Y wstd (,) ( m+ sn, + tkld,,) ( m+ sn, + tkl,,) X Y s S, t T =. 2 2 wst (, ) D( m sn, tkl,, ) wst (, ) DY( m+ sn, + tkl,, ) X s S, t T s S, t T If the cross-correlation M XY (p) is too small (< α), it implies that the regions neighboring p in X and Y are quite different, and thus CM method (w X (p)=0, w Y (p)=1 or w X (p)=1, w Y (p)=0) is used. If M XY (p) is larger than α, we average the DWT coefficients with weights dependent on the cross-correlation. Since M XY (p) 1, we have w X (p) 1/2 and w Y (p) 1/2 Y

21 as A X A Y. The WA method is useful for merging the LL bands of X and Y because the approximation images usually looks similar, i.e. have high cross-correlation. However, since the DWT coefficients in high-frequency bands would have negative values, the WA may introduce the pattern cancellation problem if employed in high-frequency bands. (3) Adaptive weighted average (AWA) The AWA scheme is a special WA scheme that the weight w X (p) is not deterministic or dependent on the cross-correlation but only relevant to the neighborhood around p in X, w ( p) = D ( p) D ( p) a, X X X where D ( p ) is average value over the neighborhood (say N M) centered at p. Simply X speaking, the weight represents the degree of interest of p. For example, the warmer and cooler pixels in the thermal image will be assigned larger weights. Thus, the AWA scheme is useful to distinguish objects having special characteristics compared to their neighborhoods. Other coefficient combining methods include fusion by energy comparison (EC), region-based fusion by multiresolution feature detection (RFD), background elimination (BE) and variance area based (VA) [4]. 6. Experimental Results and Comparison In this section, we first give an experiment to each of the three typical fusion applications: PAN and MS image fusion, Multifocus image fusion and Medical image fusion. Next, we give a comparison between the fusion schemes discussed in Section 4.1 and a brief comparison between some fusion rules PAN and MS image fusion Consider the remote sensing images acquired from an IKONOS sensor: panchromatic (PAN) image with 1 m spatial resolution and multispectral (MS) image with 4 m spatial resolution.

22 Three bands (MS1 red, MS2 green, MS3 blue) form the color MS image as displayed in Fig As the IHS-DWT fusion scheme using substitution fusion rule is employed, the resulting fused MS image is shown in Fig. 6.2 [4]. Fig (a) MS1 band, (b) MS2 band, (c) MS3 band and (d) color MS image with 4 m spatial resolution; (e) PAN image with 1 m spatial resolution. Fig Fused color MS image using IHS-DWT fusion scheme and substitution fusion rule Multifocus image fusion Due to the depth of focus problem in CCD optical lenses, one possibility to obtain an image with all interesting objects are in focus is performing image fusion to several images with partial objects in focus. An example is shown in Fig 6.3. The image in Fig 6.3 (a) has the left vehicle out of focus while the image in Fig 6.3 (b) has the right vehicle out of focus. As the two

23 images have the same spatial resolution, the fusion scheme used in this experiment is the DWT multifocus image fusion scheme (Fig 4.6 (a)) with (i) the CM fusion rule for merging the approximation coefficients; and (ii) the AWA fusion rule for merging the details coefficients. Fig. 6.3 (c) displays the fusion result. (a) (b) (c) Fig Images with the (a) left and (b) right vehicles out of focus, respectively; (c) resulting fused image by DWT multifocus image fusion scheme using CM and AWA fusion rules Medical image fusion As introduced in Section 1, the PET is a functional image displaying the brain activity, but without anatomical information. The MRI, having higher spatial resolution than the PET, provides anatomical information but without functional activity. An example is shown in Figs. 6.4 (a) and (b). Since the spatial resolution of the PET image is 1/2 of that of the MRI image, we use the DWT multifocus image fusion scheme (Fig 4.6 (b)) with WA fusion rule. The weights of WA in this experiment are deterministic, 0.4 and 0.6, respectively. Fig The (a) MRI image, (b) PET image and (c) resulting fused image using DWT multifocus image fusion scheme and WA fusion rule

24 (a) (b) Fig (a) Colored MS image with 4 m spatial resolution; (b) PAN image with 1 m spatial resolution Comparison of IHS, PCA, and IHS-DWT fusion methods Again, we use the IKONOS MS and PAN images to analyze the performance of the IHS, PCA, and IHS-DWT fusion methods. The images used here are shown in Fig The fused MS image is compared to the original high-resolution MS image (i.e. 1 m spatial resolution). The mean square error (MSE) and root mean square error (RMSE) values of the MS1 red, MS2 green, MS3 blue are shown in Table 6.1 [12]. We can found out that IHS-DWT method outperforms the PCA method, while the IHS method has the worst performance. It has been found in [7] that PCA performs better than IHS, and in particular, that the spectral distortion in the fused bands is usually less noticeable, even if it cannot be completely avoided. The wavelet based method can further reduce the spectral distortion, and thus the IHS-DWT fusion method is the best choice in most cases. Method MS3 ( µm) MS2 ( µm) MS1 ( µm) MSE RMSE MSE RMSE MSE RMSE IHS PCA

25 IHS-DWT Table 6.1. MSE and RMSE for IKONOS image fusion. 6.5 Comparison of different fusion rules In this comparison, we use the example mentioned in Section 6.3, the medical image fusion. Different fusion rules are applied to this example to compare the performance Comparison of different activity-level measurement methods Recall all the activity-level measurement methods mentioned in this paper: coefficient-based activity (CBA), window-based activity (WBA) including WA-WBA, RF-WBA, SF-WBA and ST_WBA, and Region-based activity (RBA). It has been concluded in [4] that the performance comparison of these method is CBA = WA-WBA = RF-WBA > SF-WBA = ST-WBA > RBA. The RBA is worse than the CBA and WBA because it requires good region segmentation, thus for some applications it is ineffective Comparison of different coefficient combing methods We have introduced three types of different coefficient combing method, including choose-max (CM), weighted average (WA) and adaptive weighted average (AWA) methods. Note that the approximation coefficients (APX) and details coefficients (DET) can use different combing methods. In this experiment, the Biorthogonal wavelet family ( bior ( NN, ) each set of bior ( NN, ) ) is used. For and number of DWT levels, we evaluate the RMSE of all possible combination of combining methods for APX and DET, i.e. CM-CM, CM-WA, CM-AWA, WA-CM,. The best results are shown in Table 6.2 [4]. For example, as 5-level DWT with bior(1,1) is used, the best choice is AWA-CM. From the table, we can easily conclude that the CM method is the best choice for combing details coefficients while WA and WAW methods are better choice for combing approximation coefficients than the CM method. Furthermore, we can find out that the AWA method is more suitable than the WA method.

26 7. Conclusions Table 6.2. RMSE for the wavelets Biorthogonal family.

27 In this paper, the object and definition of image fusion is addressed. We also give a brief introduction to the wavelet transform which is the main tool for image fusion. A number of different fusion schemes have been proposed in terms of the two fusion applications: the panchromatic (PAN) and multispectral (MS) image fusion and multifocus image fusion. In the former application, the object of image fusion is generating a new image that enjoys the high-spatial resolution of the PAN images and the color information of the MS image. Schemes includes the intensity-hue-saturation (IHS) fusion scheme, principle component analysis (PCA) fusion scheme, discrete wavelet transform (DWT) fusion scheme and IHS-DWT hybrid scheme are introduced. We have concluded in experimental results that the PCA scheme outperforms the IHS scheme while IHS-DWT scheme has the best performance because its spectral distortion is minimal. In another application, the object of image fusion is collect the all the objects in focus from several CCD images of the same scene. Since the input image are gray-level, and thus only the DWT scheme is suitable. However, there are numerous fusion rules for merging the DWT coefficients of the input images. A fusion rules includes the choice of an activity-level measurement and the choice of a coefficient combing method. It has been shown in simulation results that CBA is the best activity-level measurements, and choose-max (CM) is the best method for combining approximation coefficients while weighted average (WA) and adaptive WA (AWA) are good for combing detail coefficients. 7. References

28 [1] S. Ibrahim and M. Wirth, Multiresolution region-based image fusion using the Contourlet Transform, in IEEE TIC-STH, Sept [2] W. Huang, Zh.L. Jing, Multi-focus image fusion using pulse coupled neural network, Pattern Recognition Letters, vol. 28, no. 9, 2007, pp [3] Dr. Nikolaos Mitianoudis, Image fusion: theory and application, [4] G. Pajares and J. M. Cruz, A wavelet-based image fusion tutorial, Pattern Recognit., vol. 37, no. 9, pp , [5] T. Stathaki, Image Fusion: Algorithms and Applications. New York: Academic, [6] F. Sadjadi, Comparative image fusion analysis, in Proc. IEEE Conf. Comput. Vision Pattern Recogn., San Diego, CA, Jun. 2005, vol. 3. [7] M. Gonzáles Audícana and A. Seco, Fusion of multispectral and panchromatic images using wavelet transform. Evaluation of crop classification accuracy, in Proc. 22nd EARSeL Annu. Symp. Geoinformation Eur.-Wide Integr., Prague, Czech Republic, 4 6 June 2002, T. Benes, Ed., 2003, pp [8] Performance Comparison of various levels of Fusion of Multi-focused Images using Wavelet Transform [9] Y. Zhang, Understanding image fusion, Photogramm. Eng. Remote Sens., vol. 70, no. 6, pp , Jun [10] Krista Amolins, Yun Zhang, and Peter Dare, Wavelet based image fusion techniques An introduction, review and comparison, ISPRS Journal of Photogrammetric and Remote Sensing, Vol. 62, pp , [11] J. Nuñez, X. Otazu, O. Fors, A. Prades, V. Palà, and Romàn Arbiol, Multiresolution-based image fusion with additive wavelet decomposition, IEEE Trans. Geosci. Remote Sensing, vol. 37, pp , May [12] V. Vijayaraj, N. H. Younan, and C. G. O hara, Quantitative analysis of pansharpened

29 images, Opt. Eng., vol. 45, no. 4, pp , [13] Haixia Liu, Bing Zhang, Xia Zhang, Junsheng Li, Zhengchao Chen and Xiaoxue Zhou, AN improved fusion method for pan-sharpening Beijing-1 Micro-Satellite images, in IEEE IGARSS, 2009 [14] K. A. Kalpoma and J. Kudoh, Image fusion processing for IKONOS 1-m color imagery, IEEE Trans. on Geosci. Remote Sens., vol. 45, no. 10, pp , Oct [15] Jian-Jiun Ding, Time-Frequency Analysis and Wavelet Transform, [16] Isaac Bankman, Handbook of Medical Imaging: Medical image processing and analysis, 1st edition, Academic Press, 2000 [17] Jonathan Sachs, Image Resampling, [18] RC Gonzalez and RE Woods, Digital Image Processing, 2nd Ed., Englewood Cliffs,NJ: Prentice-Hall, Inc., 2002.