Image Quality Assessment for Performance Evaluation of Image Fusion

Transcription

1 Image ualit Assessment for Performance Evaluation of Image Fusion Erik Blasch AFRL/RYAA Evaluation Branch WPAFB OH 45433, U.S.A. Xiaokun L Genshe Chen, Wenhua Li DC Research Resources, LLC 00 Centur Boulevard, Suite 50 D 0874, U.S.A. {l gchen, wli}@dcmresearchresources.com Abstract - We present a novel approach on objective non-reference image fusion performance assessment. The Global-Local Image ualit Analsis (GLIA) approach takes into account local measurements to estimate how well the important information in the source images is represented b the fused image. The metric is an etended version of the Universal Image ualit Inde (UII) and uses the similarit between blocks of piels in the input images and the fused image as the weighting factors. When the difference of an image piel in the input images and its correspondence in the fused image is larger than a threshold and difficult to assess the fusion qualit, global measurements will be applied to assist the judgment. The global measurement metric considers a set of properties of human Gestalt visual perception, such as image structure, teture, and spectral signature, for image qualit assessment. Preliminar stud results confirm that the performance scores of the proposed metrics correlate well with the subjective qualit of the fused images. Kewords: Fusion performance evaluation, image fusion, non-reference qualit measures, objective qualit measures. Introduction For the performance evaluation of different image fusion methods, subjective criteria are most commonl used as the human perception of the fused image is of fundamental importance. Objective image fusion performance evaluation is a challenging task due to different application scenarios and the lack of a clearl defined and documented ground truth. Scenarios tpicall include comple structure over varing operating conditions such as illumination changes in the environment, varing parameters in sensor designs and eploitation algorithms, and a host of target sizes. Each of these variations require fusion algorithms to adapt in real time. Some fusion algorithms have been evaluated objectivel b constructing an ideal fused image and using it as a reference to compare with the eperimental results. ean squared error (SE), peak signal to noise ratio (PSR), and signal to noise ratio (SR) based metrics were widel used for these comparisons. In recent ears, several objective performance measures [-] for image qualit analsis were proposed where the knowledge of ground truth is not assumed. Unfortunatel, none of them has shown an obvious advantage over the simple measures such as RSE and PSR under strict testing conditions and with different levels of image distortion [3-5]. Despite this, more efforts on objective fusion metrics without knowing displa equipment or audience have been made [6-]. One such evaluation approach is based on the Universal Image ualit Inde [7] where local image statistics are used to define a similarit between all corresponding 8 8 blocks across input and fused images. Information theoretic measures based on global image statistics such as entrop and mutual information have also been considered within the contet of fusion evaluation [8], [0]. But, onl considering either local information or global information cannot work well on qualit analsis of the fused images in a variet of conditions. Obviousl, we need to incorporate them (local and global) together for more reliable and comprehensive performance evaluation. The new Global-Local Image ualit Analsis (GLIA) performance metric will also provide helpful guidance not onl to the performance assessment of image fusion with the same-tpe imaging sensors, but also to the image fusion with multi-modal sensors worked in 4/7 operational environments [6-7]. In this paper, we propose the global-local image qualit analsis (GLIA) approach to incorporate local image qualit evaluation and global qualit analsis together for more reliable performance analsis to image fusion 583

2 Report Documentation Page Form Approved OB o Public reporting burden for the collection of information is estimated to average hour per response, including the time for reviewing instructions, searching eisting data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or an other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 5 Jefferson Davis Highwa, Suite 04, Arlington VA Respondents should be aware that notwithstanding an other provision of law, no person shall be subject to a penalt for failing to compl with a collection of information if it does not displa a currentl valid OB control number.. REPORT DATE JUL 008. REPORT TYPE 3. DATES COVERED to TITLE AD SUBTITLE Image ualit Assessment for Performance Evaluation of Image Fusion 5a. COTRACT UBER 5b. GRAT UBER 5c. PROGRA ELEET UBER 6. AUTHOR(S) 5d. PROJECT UBER 5e. TASK UBER 5f. WORK UIT UBER 7. PERFORIG ORGAIZATIO AE(S) AD ADDRESS(ES) Air Force Research Laborator,AFRL/RYAA Evaluation Branch,Wright Patterson AFB,OH, PERFORIG ORGAIZATIO REPORT UBER 9. SPOSORIG/OITORIG AGECY AE(S) AD ADDRESS(ES) 0. SPOSOR/OITOR S ACROY(S). DISTRIBUTIO/AVAILABILITY STATEET Approved for public release; distribution unlimited. SPOSOR/OITOR S REPORT UBER(S) 3. SUPPLEETARY OTES th International Conference on Information Fusion, June 30? Jul 3, 008, Cologne, German. 4. ABSTRACT see report 5. SUBJECT TERS 6. SECURITY CLASSIFICATIO OF: 7. LIITATIO OF ABSTRACT a. REPORT unclassified b. ABSTRACT unclassified c. THIS PAGE unclassified Same as Report (SAR) 8. UBER OF PAGES 6 9a. AE OF RESPOSIBLE PERSO Standard Form 98 (Rev. 8-98) Prescribed b ASI Std Z39-8

3 without knowing the ground truth. The approach consists of two qualit metrics: local qualit metric and global qualit metric. In our framework, we use local metric and global metric collaborativel for image qualit analsis. The final value of the qualit measure will be the values from the two metrics. The paper highlights an eample of image fusion performance evaluation with a focus on reliable and robust metrics. Section details the local qualit metric, while Section 3 defines the global qualit metric. Section 4 defines the eperiment as an eemplar case in image fusion performance analsis. Section 5 draws conclusions. Definition of the Local Image ualit Inde Wang and Bovik [3] proposed a universal objective image qualit assessment metric which is eas to calculate and applicable to various image processing applications. In this section, we first give a brief introduction to the method. Then, we present a modified metric which is based on Wang-Bovik s metric.. A universal image qualit inde (UII) Instead of using traditional error summation methods, the method proposed b Wang and Bovik was designed to model an image distortion via a combination of three factors: loss of correlation, luminance distortion, and contrast distortion. ore specificall, given two real valued sequences =,..., } and =,..., }, is the mean of, { n σ is variance of, covariance of,. { n σ is variance of and σ is the σ = ( i ), = n n n n σ ( i ) () n σ = ( i )( i ) () n Then, we can compute 4σ = (3) ( + )( σ + σ ) The can be decomposed into three components as σ σ σ = (4) σ σ + σ + σ = correlation (luminance) contrast The first component is the correlation coefficient between and, which measures the degree of linear correlation between and and. The second component measures how close the mean luminance is between and. The third component measures how similar the contrasts of the images are as σ and σ can be viewed as estimate of the contrast of and. The value of the three components is in the range of [0, ]. Therefore, the final value of the qualit metric is normalized between [0, ]. Wang and Bovik appl the metric to local regions using a sliding window for objective image qualit analsis. Started from the top-left corner of the image, a sliding window with the size of B B is moved piel b piel horizontall and verticall through all piels of the image. At the position of (, the local qualit inde is computed. If the row number and column number of the image are and, then the overall normalized qualit inde is: = j= ij. Image fusion qualit metric The qualit inde proposed b Wang-Bovik has been proven ver efficient on image fusion performance evaluation as it considers three factors which are crucial in image qualit measurement. Besides these three factors, man studies show that in human visual sstem (HVS) gradient (edge) information plas a ver important role when human subject judges the qualit of an image. To take the advantage of known characteristics of the human perception, we add the local gradient information into the UII metric. Before performing qualit analsis, we use an edge detector (such as Sobel operator) to quickl process the image and get gradient information for each image piel, which is denoted as g. Therefore, the new UII metric can be presented as: σ = σ σ + σ σ σ + σ g g g + g Here, we give the brief introduction on how to appl the new UII metric to image fusion performance evaluation. For the fusion of input image A and image B resulting in a fused image F. When the sliding window moves piel b piel over the image A and image F at the same time, we can get the following: A / F = j= We do the same procedure to the image B and image F. Then, we can obtain: ij ij (5) (6) 584

4 B / F = j= Then, the fusion qualit inde can be given as AB / F = ( λ ( A / F( + ( λ( B / F( ) (7) j where λ ( is a local weight between 0 and. To automaticall set a suitable value to λ ( is also ver important. In our research, we calculate λ( according to the local saliencies of the sliding window of image A and image B. The value of saliencies is computed from the local contrast and sharpness. If we denote the value of saliencies as s, we can have the following equation to compute λ ( : s( A( ) λ ( = (8) s( A( ) + s( B( ) 3 Definition of the Global Image ualit Inde The fundamental problem of the local measure based approaches is the definition of image qualit. In particular, it is not clear that the error/difference visibilit based analsis should be equated with loss of qualit. For qualit measurement via HVS, the evaluation processing heavil depends on both pschophsical and phsiological eperience of the human subject. an studies, such as [5], show that the properties of the whole image act ver important role in HVS on qualit assessment. Another important reason for considering global properties is when the difference of an image piel in one input image and its correspondence in the fused image is significantl different from each other, while the image piel in another input image is ver close to its correspondence in the fused image, we cannot sa the fusion qualit is bad or not. In this case, we need to go back to a larger region or the whole image to decide the fusion qualit through analzing the properties of the entire image. In this section, we propose a global image qualit metric for fusion performance measurement. In the metric, the important properties to HVS, such as image structure, teture, and spectral signature, are all considered. 3. Image Structure In computer vision, finding objects or regions of a given tpe (such as woods, grass, building, sk, etc.) in a photograph, is known as image segmentation. After the segmentation, the image structure can be easil obtained ij and presented. Finding was of doing both automaticall is of great interest to researchers. Among the current image segmentation methods, graph cut is recognized a ver efficient and fast algorithm which can optimall segment an image into different patches as illustrated in Fig.. Graph cut provides a clean, fleible formulation for image segmentation. It provides a convenient language to encode simple local segmentation cues, and a set of powerful computational mechanisms to etract global segmentation from the simple local (pairwise) piel similarit. According different parameter settings, the patch number is controllable, which means the user can control the level of image segmentation. (a) Original image (c) Original image (b) Image after segmentation (d) Image after segmentation Fig. Image structure (From [8]) 3. Image Teture In image processing, simplifing assumptions are made about the uniformit of intensities in local image regions. However, images of real objects often do not ehibit regions of uniform intensities. For eample, the image of a wooden surface is not uniform but contains variations of intensities which form certain repeated patterns called visual teture. The patterns can be the result of phsical surface properties such as roughness or oriented strands which often have a tactile qualit, or the could be the result of reflectance differences such as the color on a surface. Fig. shows an eample of image teture. (a) Original image Fig. Image teture (b) Teture pattern 585

5 Image teture, defined as a function of the spatial variation in piel intensities (gra values), is useful in a variet of applications as well as in image fusion evaluation. Teture is the most important visual cue in identifing tpes of homogeneous regions. Based on tetural properties, we can identif a variet of materials and their pattern in the image as shown in Fig 3.. inde b modifing Equation (6) to the following epression. s s t t f f = (0) s + s t + t f + f = structure (teture) spectral where s, t, and f are the qualit value of image structure, teture, and spectral signature respectivel. During image qualit assessment, when the difference of an image piel in the input image and its correspondence in the fused image is significantl large and difficult to estimate the fusion qualit, global measurements will be applied to asses the image qualit. (a) (b) (c) Fig. 3 Three eamples for spectral signature (From [9]):(a) Original images; (b) Global magnitude of Fourier Transform; (c) the magnitude of the windowed Fourier Transform 3.3 Spectral Signature Spectral properties of an image can be described and studied b discrete Fourier transform (DFT) or wavelet transform. Since there are man real-time codes for DFT, we select DFT for our stud. The DFT of an image can be defined as: I( f ) = = 0 = 0 i( ) h( ) e jπ < f, > where i() is the intensit distribution of the image along the spatial variables = (, ) ; j = ; and the spatial frequenc variables are defined b (9) f = f, f ) [-0.5, ( 0.5] [ 0.5, 0.5] (units are in ccles per piel); and h () is a circular window that reduces boundar effects. The amplitude spectrum is defined as the magnitude of the DFT: A ( f ) = I( f ). The amplitude spectrum reveals the dominant orientations and tetural patterns in the image as shown in Fig Global Image qualit inde With the information of image structure, teture, and spectral signature, we can construct a global image qualit 4 Eperimental results Our proposed Global-local Image ualit Analsis (GLIA) method for image fusion performance evaluation was tested b several representative samples of multisensor image sequences including both urban and natural scenes. Sequences were fused b three representative fusion schemes: Laplacian-Pramid (LP) based image fusion method, Discrete-Wavelet-Transform (DWT) based image fusion method, and Dual-Tree-Comple-Wavelet- Transform (DT-CWT) based image fusion method. The evaluation based on our proposed GLIA metric was compared to two traditional image fusion evaluation metrics: UII metric [3] and SSI (structural similarit) metric [4]. Some eperimental results selected from multiple-sensor (CCD camera and IR camera) data are shown in Fig. 4 and Fig. 5. These testing data are provided b OCTEC [0] and TO Human Factors Research Institute []. (a) EO image (b) IR image (c) Fused image Fig. 4 Eample fused frames from the OCTEC sequence b using three representative image fusion methods 586

6 with subjective criteria as with other eisting performance measures. Our measures are eas to calculate, and applicable to man sceanrion particular, our measures provide good results on variable qualit of input images as it takes into account the local information as well as the global properties of these images. References [] T.. Pappas and R. J. Safranek, Perceptual criteria for image qualit evaluation, in Handbook of Image and video Processing, Academic Press, a 000. [] Special issue on image and video qualit metrics, Signal Processing, (Edited b urat Kunt), Vol. 70, ov (a) EO image (b) IR image (c) Fused image Fig. 5 Eample fused frames from the maritime sequence b using three representative image fusion methods The mean performance scores of UII, SSI, and GLIA are listed in Table to Table. The mean subjective scores made b people are in the range of 0.60 to 0.65 for the fused images. From the scores in the tables, we can see that UII has the biggest deviation to the scores made b human subjects because it onl considers local information of an image. SSI includes imagestructure information in its image fusion qualit metric. Therefore, SSI has much better performance than that of UII. Since our proposed GLIA metric considers both global and local information of an image, its performance scores are ver close to the HVS (human visual sstem) values made b people. From the scores in the tables, the overall performance of GLIA is the best of the three image qualit evaluation methods. Table : OCTEC Scheme DT-CWT LP DWT UII SSI GLIA Table : TO Scheme DT-CWT LP DWT UII SSI GLIA Conclusion In this paper we have discussed some new objective qualit measures, which do not require a reference image, for image fusion performance evaluation. The global-local image qualit analsis (GLIA) method correlates well [3] J.-B. artens and L. eesters, Image dissimilarit, Signal Processing, Vol. 70, pp , ov [4] VEG, Final report from the video qualit eperts group on the validation of objective models of video qualit assessment, ar [5] A.. Eskicioglu and P.S. Fisher, Image qualit measures and their performance, IEEE Trans. Communications, vol. 34, pp , Dec [6] C. Xdeas, V. Petrovic, Objective image fusion performance measure, Electronics Letters, vol. 36(4), pp , IEE, 000 [7] G. Piella, H. Heijmans, A new qualit metric for image fusion, IEEE Conference on Image Processing, vol. 3, pp , 003 [8] G. u, D. Zhang, P. Yan, Information measure for performance of image fusion, Electronics Letters, vol. 38(7), pp , IEE, 00. [9] V. Tsagaris, V. Anastassopoulos, Global measure for assessing image fusion methods, Optical Engineering, Vol. 45, SPIE, 006. [0] V. Petrovic, T. Cootes, Information Representation for image fusion evaluation, Proceedings of Fusion, Florence, ISIF, 006. []. Cvejic, D. Bull, C. Canegarjah, A new metric for multimodal image sensor fusion, Electronics Letters, Vol. 43(), pp , IEE, 007 [] V. Petrovic, C. Xdeas, Objective evaluation of signal-level image fusion performance, Optical Engineering, SPIE, Vol. 44(8), ,

7 [3] Z. Wang and A. C. Bovik, A universal image qualit inde, IEEE Signal Processing letters, vol. 9, no.3, pp. 8-84, arch 00 [4] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncell Image qualit assessment: from error visibilit to structural similarit, IEEE Trans. on Image Processing, Vol. 3, o.4, April 004. [5] Z. Wang, A. C. Bovik, and L. Lu, Wh is image qualit assessment so difficult, in Proc. IEEE Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp , FL, 00. [6] Y. Chen and R. S. Blum, "Eperimental Tests of Image Fusion for ight Vision", Eighth International Conference on Information Fusion, Philadelphia, PA, Jul 005. [7] B. Kahler, E. Blasch, D. J. Pikas, and T. Ross, EO/IR ATR Performance odeling To Support Fusion Eperimentation, SPIE Defense and Securit Smposium. [8] C. Galleguillos, A. Rabinovich, S. Belongie, Object categorization using co-occurrence, location and appearance, IEEE Conference on CVPR, 008. [9] A. Torralba and A. Oliva, Depth estimation from image structure, IEEE Trans. PAI, Vol. 4, o.9, Sept. 00. [0] The Online Resource for Research in Image Fusion, [] Toet, A., Detection of dim point targets in cluttered maritime backgrounds through multi-sensor image fusion, In: W.R. Watkins, D. Clement & W.R. Renolds (Ed.), Targets and Backgrounds: Characterization and Representation VIII, The International Societ for Optical Engineering, pp. 8-9, Bellingham, WA,