Optics Communications

Transcription

1 Optis Communiations 284 (2011) Contents lists available at SieneDiet Optis Communiations jounal homepage: Multi-fous image fusion using a bilateal gadient-based shapness iteion Jing Tian a,, Li Chen a, Lihong Ma b, Weiyu Yu a Shool of Compute Siene and Tehnology, Wuhan Univesity of Siene and Tehnology, , PR China b Guangdong Key Lab. of Wieless Netwok and Teminal, Shool of Eletoni and Infomation Engineeing, South China Univesity of Tehnology, Guangzhou, , PR China Shool of Eletoni and Infomation Engineeing, South China Univesity of Tehnology, Guangzhou, , PR China atile info abstat Atile histoy: Reeived 21 May 2010 Reeived in evised fom 16 August 2010 Aepted 23 August 2010 Keywods: Image fusion Gadient Shapness The aim of multi-fous image fusion is to ombine multiple images with diffeent fouses fo enhaning the peeption of a sene. The hallenge is to how evaluate the loal ontent (shap) infomation of the input images. To takle the above hallenge, a new bilateal shapness iteion is poposed to exploit both the stength and the phase oheene that ae evaluated using the gadient infomation of the images. Then the poposed bilateal shapness iteion is futhe exploited to pefom weighted aggegation of multi-fous images. Extensive expeimental esults ae povided to demonstate that the poposed bilateal shapness iteion outpefoms onventional seven shapness iteions Elsevie B.V. All ights eseved. 1. Intodution Multi-fous image fusion aims to fuse two o moe images that ae aptued using diffeent amea settings (e.g., at diffeent fouses) of the same sene to fom anothe supeio image with unifom fous and shap ontent [1]. Due to the limited depth-of-fous of optial lenses, it is usually impossible to aquie an image that ontains all elevant objets in-fous. Theefoe, the multi-fous image fusion tehnique is desiable to eate a single image whee all objets ae in fous. It has been used as an effetive tool fo many impotant appliations, suh as medial imaging and miosopi imaging. The basi idea of pefoming image fusion is to hoose the leae image pixels (o bloks/egions) fom soue images, o adaptively aveage obseved images aoding to thei espetive shapness, to onstut the fused image, sine the optial out-of-fous is the majo soue of image quality degadations in multi-fous image fusion [2]. Vaious tehniques have been developed in the liteatue, inluding spatial fequeny [3,4], enegy of gadient [5], phase oheene [6,7], et. A detailed eview will be povided in Setion 3. The othe kind of popula appoah is to exploit multi-esolution analysis [8 10]. Its idea is to pefom multi-esolution deomposition on eah input image, integate the deompositions to fom a omposite epesentation based on etain fusion ules, and then eonstut the fused image via an invese multi-esolution tansfom. This sheme is usually ompliated and time-onsuming to implement [11]. The key hallenge of multi-fous image fusion is how to evaluate the blu of eah image and then selet infomation fom the most infomative (shap) image [12 14]. To takle the above hallenge, the gadient Coesponding autho. addesses: eejtian@gmail.om (J. Tian), henli@ieee.og (L. Chen), eelhma@sut.edu.n (L. Ma), yuweiyu@sut.edu.n (W. Yu). infomation of the images is examined in this pape to popose a new iteion to measue the loal shapness of the image by onsideing bilateal statistis of gadient infomation the stength and the phase oheene. To be moe speifi, an image with highe stength and highe phase oheene is onsideed as shape and moe infomative; onsequently, it should ontibute moe to the fused image. Futhemoe, the poposed iteion is exploited to develop a weighted aggegation appoah to pefom image fusion. The pape is oganized as follows. Setion 2 pesents the fomulation fo the multi-fous image fusion poblem. Setion 3 eviews vaious shapness iteions developed in the liteatue. Then a bilateal shapness iteion using statistis of image's gadient infomation is poposed in Setion 4. Extensive expeimental esults ae pesented in Setion 5 to ompae the poposed bilateal shapness iteion with onventional iteions. Finally, Setion 6 onludes this pape. 2. Poblem fomulation Given a set (say, N) of2-dimagesi 1 (,),I 2 (,),,I N (,), whih have been aquied using diffeent imaging setting and aligned well, the goal of multi-fous image fusion is to integate the infomation ontent of the individual images into a single fused image f(,). A simple but effetive method fo image fusion is to pefom a simple nomalized aggegation of the images. This an be mathematially expessed as fð; Þ = 1 N N n =1 I n ð; Þ: ð1þ The main dawbak with this simple nomalized aggegation of images is that all infomation ontent within the images ae teated the same. Theefoe, impotant image egions, whih yield moe detailed infomation (edge o high-fequeny) and ae moe infomative, ae /$ see font matte 2010 Elsevie B.V. All ights eseved. doi: /j.optom

2 J. Tian et al. / Optis Communiations 284 (2011) teated no diffeently than unimpotant egions. To oveome this dawbak, a nomalized weighted aggegation appoah to image fusion an be used. It an be mathematially expessed as fð; Þ = N n =1w n ð; ÞI n ð; Þ N n =1 w : ð2þ nð; Þ whee w n (,) is the weight assigned to infomation ontent at (,)in the n-th image. The hoie of the weighting sheme (i.e., the detemination of w n (,) in Eq. (2)) is vey uial to the pefomane of the image fusion algoithm. Sine the optial out-of-fous is the majo soue of image quality degadations in multi-fous image fusion [12 14], the weighting sheme is equied to be sensitive to the blu of eah image and be loally adaptive to image's ontent. In view of this, a bief eview on the existing shapness measues will be povided in Setion 3, followed by the development of the poposed bilateal shapness iteion using two kinds of statistis of image's gadient infomation in Setion Existing shapness iteions In view of that the detemination of the weighting sheme is the key issue of designing multi-fous image fusion tehniques, a eview on the existing shapness iteions is povided in this setion. Sine the degee of de-fous vaies invesely with the amount of high spatial fequeny enegy pesent in the spatial fequeny spetum, the amount of high-fequeny infomation (oesponding to edge infomation in images) is usually used as the basis to measue the degee of image's blu [14]. Moe speifially, the well foused image has shape edges and is expeted to have highe fequeny ontent than those that ae blued. In the following analysis, denote I(,) be the intensity value at the position (,)oftheimagei. Vaiane [12]. Fo an M N blok of the image, its vaiane is defined as [12] S VAR = 1 M N ði; ð Þ μþ 2 ; ð3þ whee μ is the mean intensity value of the image blok and it is defined as μ = 1 M N ði; ð ÞÞ. Enegy of image gadient [12]. Fo an M N blok of the image, it is measued as [12] S EG = I 2 + I 2 ; ð4þ whee I and I epesent image gadients at the ow and olumn dietions, espetively. They ae usually defined as I =I(+1,) I(,) and I =I(,+1) I(,). Tenenbaum [12]. Fo an M N blok of the image, it is measued as [12] S TNG = ð Ið; ÞÞ 2 ; ð5þ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi whee Ið; Þ = I 2 + I 2, in whih I and I ae gadients (obtained using Sobel opeatos) along the ow and olumn dietions, espetively. Enegy of Laplaian [12]. Fo an M N blok of the image, it is measued as [12] S EL = 2 2; I; ð Þ ð6þ whee 2 I(,) epesents image gadient obtained by Laplaian opeato [ 1, 4, 1; 4,20, 4; 1, 4, 1]. Sum-modified-Laplaian [12]. It diffes fom the usual Laplaian opeato in that the absolute values of the patial seond deivatives ae summed instead of thei atual values. That is, it an be mathematially expessed as [12] S SML = 2; 2 I; ð Þ ð7þ whee 2 I(,)= 2I(,) I(+1,) I( 1,) + 2I(,) I(,+1) I(, 1). Fequeny seletive weighted median filte [14]. It measues the shapness of the image as S FSWM = I 2 + I 2 ; ð8þ whee I ¼ medfi 1; ð Þ; I; ð Þ; I+1; ð Þg 1 med I 3; 2 f ð Þ; I 2; ð Þ; I 1; ð Þg 1 med I+1; 2 f ð Þ; I+2; ð Þ; I+3; ð Þg; I ¼ medfi; ð 1Þ; I; ð Þ; I; ð +1Þg 1 med I; 3 2 f ð Þ; I; ð 2Þ; I; ð 1 Þg 1 med I; ð +1Þ; I; ð +2Þ; I; +3 2 f ð Þg:

3 82 J. Tian et al. / Optis Communiations 284 (2011) Fig. 1. Thee sets of test images: (a) Lena, (b) Tissue and () Calenda. This iteion is faily obust to against the noise inued in the obseved images. Phase oheene model [6]. It is onsistent to the peeptual signifiane of the image, and it an be detemined at a patiula position (, ) as S PCM ð; Þ = 1 2 jðh; ð ; θþsinðþ θ Þ 2 + ðh; ð ; θþosðθþþ 2 j θ sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðh; ð ; θþsinðþh; θ ð ; θþosðþ θ Þ + ðh; ð ; θþosðþ θ Þ 2 ðh; ð ; θþsinðþ θ Þ 2 ; 2 θ θ ð9þ whee h; ð ; θþ = n W; ð ; θþja n ð; ; θþδφ n ð; ; θþj n A n ð; ; θþ + ξ ; ð10þ Δφ n ð; ; θþ = os φ n ð; ; θþ φ n ð; ; θþ jsin φn ð; ; θþ φ n ð; ; θþ j; ð11þ in whih W epesents the fequeny spead weighting fato, A n and φ n epesent the amplitude and phase at the wavelet sale n, espetively, φ n epesents the weighted mean phase, ξ is a small onstant used to avoid the division by zeo. All of these paametes ae as same as that used in [15]. 4. Poposed bilateal gadient-based shapness iteion The statistis of image's gadient is examined in this setion to popose a new shapness measuement iteion, whih exploits bilateal statistis of image's gadient infomation. Image stutue an be measued effetively by using the image gadients. Conside an image of inteest I(, ). The gadient ovaiane matix of a egion within an M N loal window is defined as [16] C =! W I 2 ð; Þ W I ð; ÞI ð; Þ ; ð12þ W I ð; ÞI ð; Þ W I 2 ð; Þ whee I (,) and I (,) epesent image's gadient at the ow and olumn dietions, espetively. Futhemoe, the above gadient ovaiane matix an be deomposed as! C = VDV T λ = ðv 1 v 2 Þ 1 0 v T 1 ; ð13þ 0 λ 2 v T 2 whee V epesents a 2 2 matix whose olumn vetos ae eigenvetos v 1 and v 2, D denotes the 2 2 diagonal matix whose diagonal elements ae eigenvalues λ 1 and λ 2 (λ 1 Nλ 2 ) that oespond to eigenvetos v 1 and v 2, espetively, and the supesipt T denotes the tanspose. The geometial stutue at a pixel in an image an be desibed by the eigenvalues λ 1 and λ 2 of the above gadient ovaiane matix [17]. Motivated by this, the fist iteion is poposed to measue the stength of the image's gadient, whih is defined as A; ð Þ = λ 1 λ 2 : ð14þ On the othe hand, loal phase oheene is onsistent with the peeptual signifiane of image's haateistis. This has been suppoted by the physiologial evidene that showed high human peeption esponse to signal haateistis with high loal phase oheene [15]. Anothe advantage is the fat that it is insensitive magnitude vaiations aused by illumination onditions o noise

4 J. Tian et al. / Optis Communiations 284 (2011) Fig. 2. Plots of esponses of Eqs. (14)) ((16)) fo iteation numbes, whee a 5 5 aveage blu is iteatively applied to the test image Lena. (a) iteion (14); (b) iteion (15); () iteion (16). Fig. 3. Plots of esponses of Eqs. (14) (16) fo iteation numbes, whee an aveage blu, whose size stats fom 5 5 and ineases with a step of 2, is iteatively applied to the test image Lena. (a) iteion (14); (b) iteion (15); () iteion (16). inued in image signals. In view of this, the seond iteion is to onside the phase oheene of the image's gadient, that is P; ð Þ = os θð; Þ θð; Þ ; ð15þ whee θ(,) is the phase infomation at the position (,) detemined by the piniple eigenveto v 1 assoiated with the lagest eigenvalue λ 1 defined as (13), θ ; is the aveage of phases of the neighboing positions. This measue ahieves the maximal value when the loal phase oheene is wost, whih is usually aused by an edge. Finally, the above two iteions (14) and (15) ae jointly onsideed to develop a bilateal shapness iteion as S BSC = A α ð; ÞP β ð; Þ; ð16þ whee α and β ae two fatos to adjust ontibutions of two iteions. Expeiments ae onduted to justify that the poposed shapness iteions (14), (15) and (16) ae faily good to measue the shapness Table 1 The esponses of iteions (14) (16) fo test images with diffeent manually-adjusted fous levels. Test image Citeion (14) Citeion (15) Citeion (16) Tissue Shap Blued Calenda Shap 17, Blued of the image. Theefoe, they an be used as the weighting iteion of the image fusion appoah in Eq. (2). In the fist expeiment, a 5 5 aveage blu is iteatively applied to the well-known test image Lena (as shown in Fig. 1(a)), the esulting iteion values ae eoded, nomalized with thei espetive maximum values, and pesented in Fig. 2, whee one an see that the shapness values dop steadily, when the images ae moe blued. In the seond expeiment, an aveage blu, whose size stats fom 5 5 and ineases with a step of 2, is iteatively applied to the wellknown test image Lena (as shown in Fig. 1(a)). The pupose is hange the size of the aveaging blu filte to geneate a numbe of

5 84 J. Tian et al. / Optis Communiations 284 (2011) Fig. 4. Thee sets of test images: (a) Clok, (b) Bottle, and () Book. Fig. 5. A ompaison of fused images (Clok) using diffeent shapness iteions: (a) two soue images; (b) (h) ae esults obtained using iteions defined in Eqs. (3) (9), espetively; (i) poposed bilateal gadient-based shapness iteion (16).

6 J. Tian et al. / Optis Communiations 284 (2011) Fig. 6. A ompaison of fused images (Bottle) using diffeent shapness iteions: (a) two soue images; (b) (h) ae esults obtained using iteions defined in (3) (9), espetively; (i) poposed bilateal gadient-based shapness iteion (16). images with diffeent bluing levels. Moe speifially, the image geneated using a lage blu size is onsideed to be less shape than that geneated using a smalle blu size. Then the esulting shapness values ae eoded, nomalized with thei espetive maximum values, and pesented in Fig. 3, whee one an see that the values dop steadily, when the images ae moe blued. In the thid expeiment, two sets of images ae aptued using a amea Olympus SP-500 with diffeent manually-adjusted fous levels; one is in fous, while the othe one is out of fous, as shown in Fig. 1(b) and (). Thei espetive shapness values ae ompaed in Table 1, whee one an see that the values of blued images ae smalle than that of the shap images. 5. Expeimental esults Expeiments ae onduted to ompae the pefomane of the poposed bilateal shapness iteion with othe seven shapness iteions (3) (9), by individually inopoating them as the weighting sheme of Eq. (2) to pefom image fusion. The paametes of the poposed iteion (see Eq. (16)) ae expeimentally set as α=1 and β=0.5. Also, the size of the neighbohood is set to be 5 5. The above paamete settings ae expeimentally seleted. Ou setting might not be globally optimal, but this setting yields faily good pefomane in ou simulations. The fist expeiment is to ondut image fusion using thee sets of images with diffeent fous levels: Clok, Bottle, and Book, as shown in Fig. 4. The ompaison of vaious fused images ae pesented in Figs. 5 7, espetively. One an see that the fused images obtained using the poposed method yield bette image quality than that of onventional appoahes. Sine thee is no gound tuth image to evaluate the pefomane of image fusion algoithms using PSNR, two image quality evaluation iteions (without need fo gound tuth) ae used to povide objetive pefomane ompaison in this pape. These two metis ae: i) mutual infomation meti [18] and ii) spatial fequeny meti [19], whee lage metis values indiate bette image quality. The objetive pefomane ompaisons ae pesented in Tables 2 and 3, whee one an see that the poposed appoah always outpefoms othe seven onventional shapness iteions by poduing the best objetive pefomane. The seond expeiment is to ompae the omputational omplexity of image fusion appoah using vaious shapness iteions. These image fusion appoahes ae implemented using the Matlab pogamming language and un on a PC with a Pentium 1.66 GHz CPU and a 2048 MB RAM. Ten expeiments ae onduted fo eah of the abovementioned appoahes, then thei espetive aveage un-times ae ompaed in Table 4, whee the omputational omplexity of the image fusion appoah with the inopoation of the poposed shapness iteion is ompaable to that of othe shapness iteions.

7 86 J. Tian et al. / Optis Communiations 284 (2011) Fig. 7. A ompaison of fused images (Book) using diffeent shapness iteions: (a) two soue images; (b) (h) ae esults obtained using iteions defined in (3) (9), espetively; (i) poposed bilateal gadient-based shapness iteion (16). Table 2 The mutual infomation pefomane [18] ompaison of image fusion using vaious shapness iteions. Test image Citeion Citeion Citeion Citeion Citeion Citeion Citeion Poposed iteion (3) (4) (5) (6) (7) (8) (9) (16) Clok Bottle Book Table 3 The spatial fequeny pefomane [19] ompaison of image fusion using vaious shapness iteions. Test image Citeion Citeion Citeion Citeion Citeion Citeion Citeion Poposed iteion (3) (4) (5) (6) (7) (8) (9) (16) Clok Bottle Book Conlusions A multi-fous image fusion appoah using a new shapness iteion that depends on statistis of image's gadient infomation is poposed in this pape. The poposed appoah exploits a bilateal shapness iteion to adaptively pefom image fusion by seleting most infomative (shap) infomation fom the input images. The poposed bilateal shapness iteion outpefoms seven onventional shapness iteions, as veified in ou extensive expeiments using fou sets of test images unde two objetive metis. Thee ae a few dietions fo futue eseah. Fist, the poposed appoah is onduted in spatial domain only in this pape, it ould be futhe exploited into multi-esolution analysis [9]. Seond, the noises inued in obseved images is negleted in this pape. Howeve,

8 J. Tian et al. / Optis Communiations 284 (2011) Table 4 The un-time (in seonds) ompaison of image fusion using vaious shapness iteions. Citeion Citeion Citeion Citeion Citeion Citeion Citeion Poposed iteion (3) (4) (5) (6) (7) (8) (9) (16) Run time images ae usually oupted with noises in image aquisition o image ommuniation. Suh noise may ause misalulation of shapness values, whih onsequently intodue signifiant eos in the esults of image fusion. Theefoe, how to pefom image fusion fo noisy images would be a hallenge, whee the shapness iteion is equied to be obust to handle the noisy images [14,20]. Thid, the neighbohood used in the poposed appoah to alulate the shapness value is expeimentally seleted to be a squae-shape window. Cetainly, it should be adaptive to loal image ontent [21]. All of the above issues need futue investigation to futhe impove the appoah poposed in this pape. Aknowledgment This wok was suppoted by the National Natual Siene Foundation of China (Gant No , ). Refeenes [1] A.A. Goshtasby, S. Nikolov, Infomation Fusion 8 (Ap. 2007) 114. [2] H.B. Mithell, Image fusion: theoies, tehniques and appliations, Spinge, [3] S. Li, J.T. Kwok, Y. Wang, Infomation Fusion 2 (Sep. 2001) 169. [4] S. Li, B. Yang, Image Vision Computing 26 (Jul. 2008) 971. [5] H. Li, B.S. Manjunath, S.K. Mita, Gaphial Models and Image Poessing 57 (May 1995) 235. [6] A. Wong, W. Bishop, Patten Reognition Lettes 29 (Feb. 2008) 173. [7] R. Hassen, Z. Wang, M. Salama, Multifous image fusion using loal phase oheene measuement, Po. Int. Conf. on Image Analysis and Reognition, Jul. 2009, p. 54. [8] G. Pajaes, J.M. Cuz, A wavelet-based image fusion tutoial, Patten Reognition vol. 37 (Sept. 2004) [9] S. Aivazhagan, L. Ganesan, T.G. Subash Kuma, Signal Image and Video Poessing 3 (Jun. 2009) 137. [10] J. Tian, L. Chen, Multi-fous image fusion using wavelet-domain statistis, Po. IEEE Int. Conf. on Image Poessing, [11] Z. Wang, Y. Ma, J. Gu, Patten Reognition (Jun. 2010) [12] W. Huang, Z. Jing, Patten Reognition Lettes 28 (Ap. 2007) 493. [13] Y. Zhang, L. Ge, Digital Signal Poessing 19 (Ma. 2009) 186. [14] V. Aslantas, R. Kuban, Optis Communiations 282 (Aug. 2009) [15] P. Kovesi, Videe: A Jounal of Compute Vision Reseah 1 (1999) 2. [16] A. Agawal, R. Raska, R. Chellappa, Edge suppession by gadient field tansfomation using oss-pojetion tensos, Po. IEEE Int. Conf. on Compute Vision and Patten Reognition, 2006, p [17] C.-Y. Wee, R. Paamesan, Infomation Sienes 177 (Jun. 2007) [18] G. Qu, D. Zhang, P. Yan, Eletonis Lettes 38 (Ma 2002) 313. [19] S. Li, B. Yang, Patten Reognition Lettes 29 (Jul. 2008) [20] V. Aslantas, R. Kuban, Evaluation of iteion funtions fo the fusion of multifous noisy images, Po. Int. Conf. on Signal Poessing and Communiations Appliations, Ap. 2009, p [21] P.W. Huang, C.-I. Chen, P.-L. Lin, Multi-fous image fusion based on salient edge infomation within adaptive fous-measuing windows, Po. Int. Conf. on Systems, Man and Cybenetis, Ot. 2009, p