Survey on Multi-Focus Image Fusion Algorithms

Proceedings of 2014 RAECS UIET Panjab University Chandigarh, 06 08 March, 2014 Survey on Multi-Focus Image Fusion Algorithms Rishu Garg University Inst of Engg & Tech. Panjab University Chandigarh, India Preeti Gupta, Harvinder Kaur University Inst of Engg & Tech. Panjab University Chandigarh, India Abstract Image fusion is a technique of combining source images i.e. multi-modal, multi-focus etc. to obtain a new more informative image. Multi-focus image fusion algorithm combines different images having different parts in focus. Applications of image fusion includes remote sensing, digital camera etc. This paper describes various multi-focus image fusion algorithms which uses different focus measure such as spatial frequency, energy of image laplacian, morphological opening and closing etc. The performance of these algorithms is analyzed based on how focused regions in images are determined to get a fused image The method used for multi-focus image fusion is identifying the focused regions and combine them together to get an enhanced image. Keywords image fusion; multi-focus; focus measure. I. INTRODUCTION Multi-focus image fusion algorithm combines two or more images to obtain an image with every part in focus. Main aim of image fusion is to obtain information having greater quality [15]. Three levels of image fusion process are pixel, feature and decision levels. Pixel-level is low level of image fusion, deals with pixels obtained at imaging sensor output. Pixellevel is mainly concerned with visual enhancement. Image fusion process deals with intensity of source images at pixel level. Advantage of pixel level fusion is to detect undesirable noise, low complexity. Pixel based image fusion methods have ease of implementation, less complex but these are much sensitive to mis-registration and causes blurring effects. Region based methods can be used to overcome problem of mis-registration and sensitivity to noise. Feature-level fusion operation is performed on features extracted from source images. Image is segmented in contiguous regions and fused using fusion rule. In feature-level, features of images are combined such as size, shape, contrast. Decision-level deals with image descriptors. Pixel-level image fusion provides detailed information than region and decision level image fusion [14]. Combining objects or regions rather than pixels give good results. Region level is less explored. Fusion process is categorized in two domains: spatial domain and frequency domain. Spatial domain deals directly with pixels to combine relevant information. In frequency domain, image is transformed in frequency domain and frequency coefficients are combined to get fused image. Spatial domain techniques has disadvantage that it produces spatial distortion in new fused image. This spatial distortion problem is solved in frequency domain. Spatial domain contains high spatial information, but causes image blurring problem. Basic fusion steps are image registration and combination of images using fusion rule. Pre-processing step before image fusion is image registration. Image registration process transforms several images into the same coordinate system. There are two categories for measuring sharpness. One is spatial domain based in which sharpness is measured directly from the pixel values. Second one is frequency domain based measure, which measure sharpness using the sub-band coefficients of host image. Sharpness measures based on spatial domain include: variance, laplacian and spatial frequency [10]. Image fusion process is used in many applications like in medicine, defense, remote sensing. Fusion algorithm is problem dependent. Image fusion system has advantages such that higher signal to noise ration and increased robustness. The paper describes various multi-focus image fusion techniques such as image fusion using genetic algorithm, differential evolution algorithm, quad-tree structure, weighted image fusion algorithm, neighbor distance and multi-resolution algorithms. The performance of these algorithms is analyzed based on how focused regions in images are determined to get a fused image. II. ALGORITHMS S. Li [1] introduced a new block based multi-focus image fusion method. In this method original images are divided in blocks. All parts in focus. Selection of clear blocks is based on spatial frequency measure. Spatial frequency measures the overall active level in an image. For M*N image block F, with gray value F (m, n), the row and column frequency is defined in equation 1 and 2 as R= 1/MN ( m=0 to M-1 n=1 to N-1 ([f (m, n)-f (m, n-1)] 2 )) (1) C= 1/MN ( n=0 to N-1 m=1 to M-1 ([f (m, n)-f (m-1, n)] 2 )) (2) R and C are the row and column frequency respectively. The total spatial frequency of the image is defined in equation 3 as S= (R) 2 + (C) 2 (3) Spatial frequency of corresponding blocks is compared. Block with high spatial frequency is selected and used to construct a 978-1-4799-2291-8/14/$31.00 2014 IEEE

Fig. 1 Block diagram of image fusion process [1] fused image. Too small or too large block size is not desirable. For evaluating the final fused image root mean square measure is used. Block size 32*32 yield best results. This method is simple by computation and is used in many applications. The method is compared with wavelet transform. Method described above is better by considering human visual system. I. De [2] proposed a region-based multi-focus image fusion technique. Reduced DOF (depth-of-focus) in light optical imaging system can be enhanced by using this technique. After registering the multi-focus images, morphological filters and multi-scale top-hat transformation are used to detect the focused regions in each image. Finally, the fused image is constructed by combining the focused regions. This regionbased method has no problem of mis-registration. The new image is obtained by copying the detected regions in resultant image. This method is better than the Haar wavelet and weighted averaging methods but the computational cost of this method is higher than the weighted averaging method and Haar wavelet. W. Huang [3] proposed an image fusion algorithm which uses pulse coupled neural network (PCNN). The source images are decomposed into 8*8 size blocks. Energy of image laplacian is used as a focus measure to check clarity of image blocks. For image f(x, y), the energy of image laplacian (EOL) is calculated in equation 4 and 5 as EOL= x y (f xx+ f yy ) 2 (4) Where f xx +f yy = -f(x-1,y-1)-4f(x-1,y)-f(x-1,y+1)-4f(x,y-1)+20f(x,y)- 4f(x,y+1)-f(x+1,y-1)-4f(x+1,y)-f(x+1,y+1) (5) In PCNN, linking strength β of each neuron is represented by clarity of each pixel. The value of focus measure is used to obtain feature maps. The feature maps obtained is given into inputs of PCNN. An iterative method is used to obtain a proper value of β. Based on the comparison of outputs of PCNN, image blocks are selected to construct a final fused image. This method outperforms DWT and Li [1] s method. S. Li [4] proposed a multi-focus image fusion method which is region-based and is implemented in the spatial domain. Fusion process consists of three main steps: segmentation, checking clarity of region and construction of fused image. Firstly source images are combined to form a temporary fused image by averaging method. Then the image segmentation is done using the normalized cuts algorithm. Then the source images are partitioned based on the result of temporary fused image. Spatial frequency measure of each region is calculated to check clear regions and these regions are fused together. Normalized cut criterion can measure total similarity and total dissimilarity within different groups. To compare fusion results, two objective criteria are used. One is quality matrix which gives the amount of edge information transferred from source images to fused image. Second criterion is mutual information. For both criteria, higher values give better fusion result. This method is compared wavelet transform based technique. Result obtained using wavelet transform is worse than the above method but this method is more time consuming. Ji. Zhang [5] proposed a region based image fusion scheme. Source images are divided into various blocks, and then fused image is constructed based on higher value of quality assessment value of each block. In multi-focus image fusion process, the first step is to identify whether the region of image is clear or not. Quality assessment can be categorized into two ways, first is the frequency domain and second is the spatial domain. Spatial domain assessment is used in this algorithm. GA (genetic algorithm) is used to determine the suitable block size. This method outperforms Haar wavelet approach and morphological wavelet approach visually. The method has simple computation, performance is better. V. Aslantas [6] presented a new image fusion scheme using differential evolution algorithm. Images are divided into blocks. Optimal block size is selected using differential evolution algorithm. This algorithm is fast. DE involves 3 operations: initial population generation, mutation, crossover to choose optimum block size. Then focus measure of block calculated using spatial frequency, variance or sum-modified-

laplacian measure. Higher value of focus measure, sharper is the image block. Thus sharpness value is compared of 2 corresponding blocks and shaper blocks are selected to construct a fused image. Then global sharpness value of image is calculated. Larger the sharpness value better is the fused image. The process is repeated till a predefined condition is satisfied. For performance measure of fusion process, peak signal to noise ratio (PSNR) and mutual information (MI) are used. Larger value of MI MSE and PSNR gives better fusion. Differential evolution (DE) algorithm is compared with genetic algorithm (GE) and transform based methods like laplacian and wavelet. DE is reliable than GA based method. Fused image has higher quality than previous techniques. Yi. Chai [7] proposed image fusion scheme based on focused region detection and multi-resolution. Firstly, initial fused image is obtained by using lifting stationary wavelet transform (LSWT). To select focused region, morphological opening and closing similarity measure is used and border pixels are selected from initially fused image. These focused regions are combined to get a fused image. This method reduces problem of information loss in multi-scale transform domain method. Fusion algorithm is implemented by integrating spatial domain and transforms domain methods. LSWT has many advantages such as it reduces complexity, it is shift-invariant, reliable, and reduces amount of distortion artifacts and loss of contrast information. But its limitation is that many erroneous results may occur at boundary of focused regions. Xi. Bai et al. [8] proposed multi-focus/multi-modal image fusion scheme using weighted image fusion algorithm based on multi-scale top-hat transform to get fused image. Top-hat transform is used to extract bright and dim features of image. For appropriately importing the extracted bright and dim features of image into fused image, three weighted image fusion methods are described. The weighted image fusion algorithms based on the multi-scale to-hat transform can obtain satisfying results. From the results it has found that mean value weighted image fusion algorithm obtains an image which is clearer and more informative than other algorithms and achieves good performance on all measures. Weighted image fusion algorithms based on multi-scale top-hat transform may be widely used in different applications, such as target detection, object recognition and security surveillance. Ishita De [9] proposed a block-based algorithm for multifocus image fusion. Images are partitioned into blocks. It doesn t use fixed block size as fixed block size has undesirable effects. Large and small block size may select defocused regions. For optimal sub-division of blocks, quadtree structure is used. A new focus measure called energy of morphological gradients is introduced to select clear blocks. In quad-tree structure image is divided into four-blocks. Focus measure is calculated for each block and compared with corresponding block. Higher focus measure value is compared with threshold value. If it is greater than threshold value then block is copied in resultant image otherwise blocks are further subdivided in blocks. Detected focused blocks of different sizes are merged. Two evaluations are done for performance analysis. First is quality index. Second is structure similarity index. Greater the value of these parameter means better fusion. This method is having ease of implementation, fast and simple. This method is compared with a region-based fusion technique presented by Li [2] in 2008. Results are better than that of given by Li s. H. Zhao [10] proposed a multi-focus image method based on the neighbor distance. Gray image is considered to be a 2-D surface. Pixel is surrounded by eight pixels has eight oriented distances. Sum of these eight oriented distance is neighbor distance. Neighbor distance is used as a measure of pixel s sharpness. Multi-resolution transform-based fusion scheme is used as the fusion scheme. Image is divided in high and low frequency components using multi-resolution algorithm. High frequency component is calculated by considering neighbor distance image. Low frequency component of image is obtained by subtracting the neighbor distance image from the original image. Low frequency/ high frequency components are combined together to create fused image. Select maximum fusion rule is used to create fused image. The multi-focus neighbor distance analysis is shift-invariant. For performance analysis, standard deviation average gradient and spatial frequency is used as evaluation criterion. The standard deviation is given in equation 6 as: SD= 1/MN ( m=0 to M-1 n=0 to N-1 ([f (m, n)-µ] 2 )) (6) where F is the fused image, µ is the image s mean. The average gradient is given in equation 7. AG= 1/MN ( m=0 to M-1 n=0 to N-1 ( (f x (m, n)) 2 + (f y (m, n)) 2 ) /2) (7) where ΔFx and ΔFy are the differences in x and y direction of the fused image. Lesser the value of these parameters, the sharper is the image. Neighbor distance based image fusion method provides good fusion performance and is better than the image fusion methods including DWT (discrete wavelet transform), LAP (laplacian pyramid). Fig.2 Quad-tree structure [9]

Q. Li [11] proposed a region-based multi-focus image fusion scheme based on local spatial frequency. It overcomes the problem of sensitivity to noise in pixel-based image fusion methods. Multi-focus image contain clear and blurry parts. Fusion process extracts clear parts and combines them to construct a fused image. The algorithm involves 3 steps: image segmentation, calculation of region clarity and construction of fused image. Source images are combined using averaging algorithm to get the temporary fused image. This temporary fused image is segmented using entropy rate super pixel segmentation method. Source images are also partitioned using region map of temporary fused image. Then region spatial frequency is calculated for every region to decide which region is more suitable for fusion. Finally, selected regions are combined to get a fused image. Mutual information and quality index is used as the evaluation criteria. Quality index gives amount of edge information transferred from source images to fused image Mutual information indicates the amount of information that the fused image contains from host images. The larger the measure value, the better is the fusion performance. The method described has better performance than the wavelet transform and laplacian pyramid. Limitation of this method is that it is more time consuming. R.Maruthi [12] gives a novel fusion scheme for combining images using index of fuzziness used as a focus measure. Fusion process consists of three steps. Source images are partitioned into sub-blocks and focus measure were calculated for each block of the image. Then block with higher value of focus measure is selected to construct a fused image. Index of fuzziness (I) is used as the focus measure. The index of fuzziness quantifies the information level in an image. For an M*N images block index of fuzziness is defined in equation 8. I= (2/ (M*N)) m n min (µ mn, 1- µ mn ) (8) where µ mn is the mean value of all pixel values in an image. Higher is the I, higher is the quality of image. Increase in blurring causes decrease in the value of I. I determine the clarity of an image. Fusion performance is compared with the existing spatial domain methods like spatial frequency, variance and energy of image gradient. The performance analysis for estimating quality of fused image is checked using spatial frequency and quality index. A. Saha [13] proposed a multi-focus image fusion scheme to get a fused image by combining focused part of input images. To select focused part of image mutual spectral residual (MSR) approach is used. First of all images are transformed in frequency domain using Fourier transform. In spectral residual, log spectrum of Fourier transform of image is calculated. MSR of each image is calculated which forms saliency maps. This MSR represent the unique feature of image. Saliency maps determine focused and defocused part of image. The focused parts are combined to get a fused image. Fusion rule is applied in spatial domain. III. PERFORMANCE MEASURE To check evaluation performance of fusion process, qualitative and quantitative measures can be used. Quantitatively it is measured by visual analysis statistical analysis. Qualitative assessment can be evaluated using reference and non-reference based metrics. Reference based metrics includes mean square error, peak signal to noise ratio. Non-reference based metrics includes mutual information, structural similarity index, spatial frequency and quality index. RMSE= (1/ (M*N)) i=1to M j=1to N ((R (i, j) F (i, j)) 2 (9) PSNR= 10log 10 L 2 / (RMSE) 2 (db) (10) Mutual information is defined in equation 11 as MI RF = i=1to M j=1to N p RF (i, j) 10log 2 (p RF (i, j) / p F (j) * p R (i)) (11) Where p RF is the normalized joint gray level histogram of images F and R having size M*N, p F and p R are the normalized marginal histograms of the two images, L is number of gray levels and F (i, j) and R (i, j) are the pixel values of the fused and the reference image. IV. CONCLUSION Image fusion techniques have been widely used in many applications such as computer vision system, remote-sensing etc. This paper gives introduction to some multi-focus image fusion techniques and their performance assignment techniques. Various issues have been found on the basis of the study of the reviewed papers. Image fusion algorithm is problem dependent. Pixel-based algorithms leads to misregistration and are sensitive to noise, causes blurring problems Region-based algorithms are complex than pixelbased algorithms, but gives good performance and removes problems of pixel-based algorithms. Region-based image fusion schemes are less explored than pixel based, thus lots of work can be extended in region based multi-focus image fusion algorithms in order to reduce time complexity and computational complexity. Problem of fixed image block size is solved by using genetic algorithm and quad-tree structure. Spatial domain methods are better than frequency domain methods in terms that these are shift-invariant and does not cause lose of information. By using technique with varying image block size and by modifying focus measures, better results can be achieved. Hybrid technique using spatial and frequency domain methods can be used to obtain better results. ACKNOWLEDGMENT We sincerely thank to all those who helped us in completing this task.

REFERENCES [1] S. Li, A. T. Kwok and Y. Wang, Combination of images with diverse focuses using the spatial frequency, International Conference on Information Fusion, vol. 2, no. 3, pp. 169-176, 2001. [2] I. De, B. Chanda and B. Chattopadhyay, Enhancing effective depth-offield by image fusion using mathematical morphology, International Conference on Image and Vision Computing, vol. 24, no. 12, pp. 1278-1287, 2006. [3] W. Huang and Z. Jing, Multi-focus image fusion using pulse coupled neural network, International journal of pattern recognition, vol. 28 no. 9, pp. 1123-1132, 2007. [4] S. Li and B. Yang, Multi-focus image fusion using region segmentation and spatial frequency, International Conference on Image and Vision Computing, vol. 26, no. 7, pp. 971-979, 2008. [5] J. Zhang, X. Feng, B. Song, M. Li and Y. Lu, Multi-focus Image Fusion Using Quality Assessment of Spatial Domain and Genetic Algorithm, IEEE Conference on Human System Interaction, pp. 71-75, 2008. [6] V. Aslantas and R. Kurban, Fusion of multi-focus images using differential evolution algorithm, IEEE conference on human system interaction, vol.37, no. 12, pp-8861-8870, Dec. 2010. [7] Y. Chai, H. Li and Z. Li, Multi-focus image fusion scheme using focused region detection and multi-resolution, International Journal of Optics Communications, vol. 284, no. 19, pp. 4376-4389,2011. [8] X. Bai, S. Gu, F. Zhou and B. Xue, Weighted image fusion method based on multi-scale top-hat transform: Algorithms and a comparison study, International Journal of Optik, vol. 124, no. 13, pp. 1660-1668, 2013. [9] Ishita De and Bhabatosh Chanda, Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure, International Journal of Information Fusion, vol.14, no.2, pp-136-146, 2013. [10] H. Zhao, A. Shang, Y. Y. Tang and B. Fang, Multi-focus image fusion based on the neighbor distance, International Journal of Pattern Recognition, vol. 46, no.3, pp. 1002-1011, 2013. [11] Q. Li, J. Du, F. Song, C. Wang, H. Liu and C. Lu, Region-based Multifocus image Fusion Using the Local Spatial Frequency, IEEE control and decision Chinese conference, pp-3792-3796, 2013. [12] R.Maruthi and C. Jayakumari, Fusion scheme for fusing images with diverse focuses using index of fuzziness as a focus measure, International Conference on Communication and Technology and System Design, pp. 584-590, 2012. [13] A. Saha, G. Bhatnagar and Q. M. J. Wu, Mutual spectral residual approach for multi-focus image fusion, Digital Signal Processing, vol. 23, no. 4, pp.1121-1135, 2013. [14] Z. Mengyu and Y. Yuliang, A new image fusion algorithm based on fuzzy logic, IEEE international conference on intelligent computation technology and automation, vol. 2, pp-83-86, 2008. [15] Z. Wang, D. Ziou, C. Armenakis, D. Li and Q. Li, A comparative analysis of image fusion methods, IEEE Transactions on Geosciences and Remote Sensing, vol. 43, no. 6, 2005.