Approaches used to measure quality of multifocus image fusion

Size: px

Start display at page:

Download "Approaches used to measure quality of multifocus image fusion"

Jemima Kelly
5 years ago
Views:

1 Approaches used to measure quality of multifocus image fusion Marek Vajgl Institute for Research and Applications of Fuzzy Modeling University of Ostrava Ostrava, Czech Republic January 30, 2014 Marek Vajgl (IRAFM) Meausrement of multifocus fusion 1 / 42

2 1 Problem introduction Multifocus image F-Transform and multifocus fusion Fusion evaluation 2 Measures and evaluation Mosaic images Non-mosaic images 3 Expert decision 4 Conclusion 5 References Marek Vajgl (IRAFM) Meausrement of multifocus fusion 2 / 42

3 Problem introduction Multifocus image Multifocus image fusion I Multifocus images Multiple images obtained with dierent focus (No rotation, no shift, no scale change) Example Marek Vajgl (IRAFM) Meausrement of multifocus fusion 3 / 42

4 Problem introduction Multifocus image Multifocus image fusion II Types Mosaic - each pixel is the "best" information in at least one image Real-life - there might be no ideal image for pixel Example Marek Vajgl (IRAFM) Meausrement of multifocus fusion 4 / 42

5 Problem introduction F-Transform and multifocus fusion F-Transform schematically Original function f : [a, b] [c, d] Fuzzy partition A 1,..., A n of [a, b] F-Transform Components F 1,..., F n Transformation: f F n [f] x A 1 A 2 A n F n [f] F 1 F 2 F n Result: F n [f] = (F 1,..., F n ) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 5 / 42

6 Problem introduction F-Transform and multifocus fusion F-transform formally Direct and inverse F-Transform F [u](a k B l ) = u nm (i, j) = Basic usage in image processing M j=1 u(i, j)a k(i)b l (j) N M i=1 j=1 A, (1) k(i)b l (j) N i=1 n k=1 l=1 1 Calculate components c of input image u m F [u] kl A k (i)b l (j). (2) 2 Calculate reconstructed image u r from components c 3 Calculate residuals r = u r u Marek Vajgl (IRAFM) Meausrement of multifocus fusion 6 / 42

7 Problem introduction F-Transform and multifocus fusion Image fusion using F-Transform Algorithm description 1 Calculate residuals for each input image 2 Calculate residuals from residuals for each input image (obtained next "level") 3... continue interatively until residuals are not signicant 4 For each "level" take highest residuals in absolute values 5 Reconstruct image as sum highest residuals Properties Very time consuming Precise for complex images May contain artifacts in case of simple images improvements done Marek Vajgl (IRAFM) Meausrement of multifocus fusion 7 / 42

8 Problem introduction F-Transform and multifocus fusion Fusion example - input Marek Vajgl (IRAFM) Meausrement of multifocus fusion 8 / 42

9 Problem introduction F-Transform and multifocus fusion Fusion example - output Marek Vajgl (IRAFM) Meausrement of multifocus fusion 9 / 42

10 Problem introduction Fusion evaluation Fusion example - which is the best? Marek Vajgl (IRAFM) Meausrement of multifocus fusion 10 / 42

11 Problem introduction Fusion evaluation Fusion - evaluation Possible defects Result image (or its parts) is out of focus Artifacts existing in result image How to measure quality? Solution based on automatically calculated measures Is there "best" image to compare There is no best image to compare Solution based on human experts (Subjective? Who is expert?) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 11 / 42

12 Problem introduction Fusion evaluation Fusion - evaluation Possible defects Result image (or its parts) is out of focus Artifacts existing in result image How to measure quality? Solution based on automatically calculated measures Is there "best" image to compare There is no best image to compare Solution based on human experts (Subjective? Who is expert?) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 11 / 42

13 Problem introduction Fusion evaluation Fusion - evaluation Possible defects Result image (or its parts) is out of focus Artifacts existing in result image How to measure quality? Solution based on automatically calculated measures Is there "best" image to compare There is no best image to compare Solution based on human experts (Subjective? Who is expert?) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 11 / 42

14 Problem introduction Fusion evaluation... quick remark about measures... about measures Distortion measures - higher means lower quality (bigger distortion) Quality measures - higher means better quality (lower distortion) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 12 / 42

15 Mosaic images Automatically calculated measures - mosaic images I Commonly used measures Mean square error MSE = 1 n m Peak signal to noise ratio n x=1 y=1 m (u(x, y) û(x, y)) 2 P SNR = 20 log MSE Structural similarity SSIM(x, y) = (2µ xµ y + c 1 )(2σ xy + c 2 ) (µ 2 x + µ 2 y + c 1 )(σ 2 x + σ 2 y + c 2 ) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 13 / 42

16 Mosaic images Automatically calculated measures - mosaic images I Commonly used measures Mean square error MSE = 1 n m Peak signal to noise ratio n x=1 y=1 m (u(x, y) û(x, y)) 2 P SNR = 20 log MSE Structural similarity SSIM(x, y) = (2µ xµ y + c 1 )(2σ xy + c 2 ) (µ 2 x + µ 2 y + c 1 )(σ 2 x + σ 2 y + c 2 ) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 13 / 42

17 Mosaic images Automatically calculated measures - mosaic images I Commonly used measures Mean square error MSE = 1 n m Peak signal to noise ratio n x=1 y=1 m (u(x, y) û(x, y)) 2 P SNR = 20 log MSE Structural similarity SSIM(x, y) = (2µ xµ y + c 1 )(2σ xy + c 2 ) (µ 2 x + µ 2 y + c 1 )(σ 2 x + σ 2 y + c 2 ) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 13 / 42

18 Mosaic images Automatically calculated measures - mosaic images II Information-weighted MSE & PSNR [Wang (2004)] M [ i IW-MSE = w j,i(u j,i û j,i ) i w j,i j=1 IW-PSNR = 20 log IW MSE ] βj Marek Vajgl (IRAFM) Meausrement of multifocus fusion 14 / 42

19 Mosaic images Automatically calculated measures - mosaic images III Multiscale structural similarity [Wang (2004)] Evaluated SSIM in multiple scales Brightness (luminance) l(x, y) = 2µxµy+C 1 µ 2 x +µ2 y +C 1 Contrast c(x, y) = 2σxσy+C 2 σ 2 x+σ 2 y+c 2 Similarity s(x, y) = σxy+c 3 σ xσ y+c 3 Local SSIM j = 1 [l(u j,i, û j,i )c(u j,i, û j,i )s(u j,i, û j,i )] N j i Result M MS-SSIM = (SSIM j ) β j j=1 Marek Vajgl (IRAFM) Meausrement of multifocus fusion 15 / 42

20 Mosaic images Automatically calculated measures - mosaic images III Multiscale structural similarity [Wang (2004)] Evaluated SSIM in multiple scales Brightness (luminance) l(x, y) = 2µxµy+C 1 µ 2 x +µ2 y +C 1 Contrast c(x, y) = 2σxσy+C 2 σ 2 x+σ 2 y+c 2 Similarity s(x, y) = σxy+c 3 σ xσ y+c 3 Local SSIM j = 1 [l(u j,i, û j,i )c(u j,i, û j,i )s(u j,i, û j,i )] N j i Result M MS-SSIM = (SSIM j ) β j j=1 Marek Vajgl (IRAFM) Meausrement of multifocus fusion 15 / 42

21 Mosaic images Automatically calculated measures - mosaic images III Multiscale structural similarity [Wang (2004)] Evaluated SSIM in multiple scales Brightness (luminance) l(x, y) = 2µxµy+C 1 µ 2 x +µ2 y +C 1 Contrast c(x, y) = 2σxσy+C 2 σ 2 x+σ 2 y+c 2 Similarity s(x, y) = σxy+c 3 σ xσ y+c 3 Local SSIM j = 1 [l(u j,i, û j,i )c(u j,i, û j,i )s(u j,i, û j,i )] N j i Result M MS-SSIM = (SSIM j ) β j j=1 Marek Vajgl (IRAFM) Meausrement of multifocus fusion 15 / 42

22 Mosaic images Automatically calculated measures - mosaic images IV Multiscale weighted structural similarity [Wang (2004)] Evaluated information weighted SSIM in multiple scales Brightness, contrast, similarity stays the same Local value now weighted i IW-SSIM j = w j,ic(u j,i, û j,i )s(u j,i, û j,i ) i w, j < M j,i Result IW-SSIM = M (IW-SSIM j ) β j j=1 Marek Vajgl (IRAFM) Meausrement of multifocus fusion 16 / 42

23 Mosaic images Automatically calculated measures - mosaic images IV Multiscale weighted structural similarity [Wang (2004)] Evaluated information weighted SSIM in multiple scales Brightness, contrast, similarity stays the same Local value now weighted i IW-SSIM j = w j,ic(u j,i, û j,i )s(u j,i, û j,i ) i w, j < M j,i Result IW-SSIM = M (IW-SSIM j ) β j j=1 Marek Vajgl (IRAFM) Meausrement of multifocus fusion 16 / 42

24 Mosaic images Commonly used measures - results Commonly used measures Image Size MSE PSNR SSIM reference 256x (Inf) input A 256x input B 256x fused 256x reference 1024x (Inf) input A 1024x input B 1024x fused 1024x Marek Vajgl (IRAFM) Meausrement of multifocus fusion 17 / 42

25 Mosaic images Information weighted measures - results Information weighted measures Image Size SSIM IW-PSNR MS-SSIM IW-SSIM reference 256x input A 256x input B 256x ,944 fused 256x reference 1024x input A 1024x input B 1024x fused 1024x ,975 blur x blur x blur x Marek Vajgl (IRAFM) Meausrement of multifocus fusion 18 / 42

26 Mosaic images Mosaic fusion results A1 obtained by DCT [Haghighat (2011)] Marek Vajgl (IRAFM) Meausrement of multifocus fusion 19 / 42

27 Mosaic images Mosaic fusion results A2 obtained by DCT [Haghighat (2011)] Marek Vajgl (IRAFM) Meausrement of multifocus fusion 20 / 42

28 Mosaic images Mosaic fusion results B obtained by wavelet-based statistical sharpness measure [Tian (2012)] Marek Vajgl (IRAFM) Meausrement of multifocus fusion 21 / 42

29 Mosaic images Mosaic fusion results C obtained by bilateral gradiend-based sharpness [Tian (2011)] Marek Vajgl (IRAFM) Meausrement of multifocus fusion 22 / 42

30 Mosaic images Mosaic fusion results D obtained by laplacian-based fusion [Xiao (2009)] Marek Vajgl (IRAFM) Meausrement of multifocus fusion 23 / 42

31 Mosaic images Mosaic fusion results Original F-Transform based fusion (origin) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 24 / 42

32 Mosaic images Mosaic fusion results Residual F-Transform based fusion (residual) Marek Vajgl (IRAFM) Meausrement of multifocus fusion 25 / 42

33 Mosaic images Mosaic fusion results Invasive A - F-Transform based fusion Marek Vajgl (IRAFM) Meausrement of multifocus fusion 26 / 42

34 Mosaic images Mosaic fusion results Invasive B - F-Transform based fusion Marek Vajgl (IRAFM) Meausrement of multifocus fusion 27 / 42

35 Mosaic images Mosaic fusion results Invasive C - F-Transform based fusion Marek Vajgl (IRAFM) Meausrement of multifocus fusion 28 / 42

36 Mosaic images Mosaic fusion results Invasive D - F-Transform based fusion Marek Vajgl (IRAFM) Meausrement of multifocus fusion 29 / 42

37 Mosaic images Fused images - results Fused images - results Image SSIM IW-PSNR MS-SSIM IW-SSIM orig.bmp D.bmp Forigin.bmp FinvasiveB.bmp A1.bmp , FinvasiveD.bmp FinvasiveA.bmp Fresidual.bmp A2.bmp FinvasiveC.bmp B.bmp C.bmp Marek Vajgl (IRAFM) Meausrement of multifocus fusion 30 / 42

38 Non-mosaic images Automatically calculated metrics - non-mosaic images How to evaluate focus Camera uses focus measure to autofocus Phase detection Contrast detection - intensity dierences between neighbor pixels increasing with correct focus no default image needed Marek Vajgl (IRAFM) Meausrement of multifocus fusion 31 / 42

39 Non-mosaic images Autofocus measurement methods Many approaches, references implementations [Petruz (2012)] Image contrast / curvature [Nanda (2001), Helmli (2001)] Gray level variance/normalized/locality [Krotkov (1986), Santos (1997), Pech (2000)] Histogram range / entrophy [Firestone (1991), Krotkov (1986)] Energy of gradient [Subbarao (1992)] Thresholded gradient, squared gradient [Santos (1997), Eskicioglu (1995)] Steerable lter-based measure [Minhas (2009)] Gaussian derivative [Geusebroek (2000)] Laplacian energy / variance / diagonal [Subbarao (1992), Pech (2000), Thelen (2009)] DCT energy mesasure / ratio [Shen (2006), Lee (2009)] Wavelet sum / variance / ratio [Yang (2003), Yang (2003), Xie (2004)] more Marek Vajgl (IRAFM) Meausrement of multifocus fusion 32 / 42

40 Non-mosaic images Autofocus measurement methods - results I Autofocus measures - demo images Image SFIL ( ) GRAE WAVV ACMO reference input A input B fused blur blur blur SFIL Steerable lters-based [Minhas (2009)] GRAE Energy of gradient [Subbarao (1992)] WAVV Wavelet variance [Yang (2003)] ACMO Absolute central moment [Shirvaikar (2004)] Marek Vajgl (IRAFM) Meausrement of multifocus fusion 33 / 42

41 Non-mosaic images Autofocus measurement methods - results II Autofocus measures - fused images Image SFIL ( ) GRAE WAVV input A.bmp input B.bmp orig.bmp D.bmp Forigin.bmp FinvasiveB.bmp FinvasiveD.bmp B.bmp A1.bmp Fresidual.bmp A2.bmp FinvasiveC.bmp Marek Vajgl (IRAFM) Meausrement of multifocus fusion 34 / 42

42 Non-mosaic images Autofocus methods - summary Acceptable measures (in case of no artifacts) Steerable ltering-based approach DCT energy, DCT ratio Gaussian derivative Gray level local variance Unacceptable measures Detecting directly sharpness of image Wavelets... Contrast... Histogram... Laplacian... Marek Vajgl (IRAFM) Meausrement of multifocus fusion 35 / 42

43 Expert decision Who is human expert? Who is human expert? Multiple subjects evaluate images dierently Some prefer improved contrast in image Some do not see (or ignore) artifacts Sometimes dierence is too small So which one is better? Marek Vajgl (IRAFM) Meausrement of multifocus fusion 36 / 42

44 Expert decision Just Noticeable Dierence Just Noticeable Dierence Based on perceptual sensitivity (E.H. Webber, G.T. Fecher) Test based on img test = img ref + k (img dmg img ref ) Subject must decide if img test or img ref is better With success > 75%, dierence is 1 JND, with success >93.75%, dierence is 2 JND Dierence 2 JND is "noticeable" This may be useful to dene minimal signicant measure dierence. Marek Vajgl (IRAFM) Meausrement of multifocus fusion 37 / 42

45 Conclusion Conclusion Future In case of reference image, IW-SSIM characteristic can be used In case of non-mosaic input, DCT based measures, gaussian derivative or gray-level-local-variance can be used Artifacts cause results should be processed carefully Measure the robustness of selected algorithms Look for other types of algorithms than focus-oriented Marek Vajgl (IRAFM) Meausrement of multifocus fusion 38 / 42

46 References References Aslantas (2009) Aslantas, V. and R.Kurban: A comparison of dierent focus measures for use in fusion of multi-focus noisy images, The 4th International Conference on Information Technology (ICIT 2009), Amman, Jordan, 2009 Wang (2004) Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli: Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, vol. 13, no. 4, pp , Apr Petruz (2012) Said Pertuz, Domenec Puig, Miguel Angel Garcia: Analysis of focus measure operators for shape-from-focus, Pattern Recognition, Volume 46, Issue 5, May 2013, Pages , ISSN Tektronics (2008) Tektronics: Understanding PQR, DMOS, and PSNR Measurements [online ] Perlieva (2011) Perljeva I., Dankova M., Hodakova P., Vajgl M.: F-Transform based Image Fusion. In: Image Fusion. InTech, Rijeka, Croatia (2011) pp Perljeva (2012) Perljeva I., Vajgl M., Hodakova P.: Advanced F-Transform-based Image Fusion. In: Advances in Fuzzy Systems (2012) 1-9. Marek Vajgl (IRAFM) Meausrement of multifocus fusion 39 / 42

47 References Fusion algorithms references Haghighat (2011) M.B.A. Haghighat, A. Aghagolzadeh, H. Seyedarabi, Multi-Focus Image Fusion for Visual Sensor Networks in DCT Domain. Computers and Electrical Engineering, vol. 37, no. 5, pp , Sep Tian (2012) J. Tian and L. Chen: Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure. Signal Processing. Vol. 92, No. 9, Sep. 2012, pp Tian (2011) J. Tian, L. Chen, L. Ma and W. Yu: Multi-focus image fusion using a bilateral gradient-based sharpness criterion. Optics Communications, Vol. 284, Jan. 2011, pp Xiao (2009) QU Xiao-bo, YAN Jing-wen, YANG Gui-de: Sum-modied-Laplacian-based Multifocus Image Fusion Method in Sharp Frequency Localized Contourlet Transform Domain. Optics and Precision Engineering, 17(5): , June Marek Vajgl (IRAFM) Meausrement of multifocus fusion 40 / 42

48 References Measure algorithm references I Helmli (2001) Helmli, F. & Scherer, S. Adaptive shape from focus with an error estimation in light microscopy Nanda (2001) Nanda, H. & Cutler, R. Practical calibrations for a real-time digital omnidirectional camera Krotkov (1986) Krotkov, E. Range from focus Santos (1997) Santos, A.; de Solorzano, C. O.; Vaquero, J. J.; Pena, J. M.; Mapica, N. & Pozo, F. D. Evaluation of autofocus functions in moleclar cytogenetic analysis. Journal of Microscopy, Vol. 188, Issue 3, pages , December 1997 Pech (2000) Pech, J.; Cristobal, G.; Chamorro, J. & Fernandez, J. Diatom autofocusing in brighteld microscopy: a comparative study Firestone (1991) Firestone, L.; Cook, K.; Culp, K.; Talsania, N. & Jr., K. P. Comparison of autofocus methods for automated microscopy Subbarao (1992) Subbarao, M.; Choi, T. & Nizkad, A. Tech. report: Focusing techniques Eskicioglu (1995) Eskicioglu, A. M & Fisher, P. S. Image quality measures and their performance Thelen (2009) Thelen, A.; Frey, S.; Hirsch, S. & Hering, P. Interpolation Improvements in Shape-From-Focus for Holographic Reconstructions With Regard to Focus Operators, Neighborhood-Size, and Height Value Marek Vajgl (IRAFM) Meausrement of multifocus fusion 41 / 42

49 References Measure algorithm references II Shen (2006) Shen, C. & Chen, H. Robust focus measure for low-contrast images Lee (2009) Sang-Yong Lee, Jae-Tack Yoo, K. Y. S. K Reduced Energy-Ratio Measure for Robust Autofocusing in Digital Camera Yang (2003) Yang, G. & Nelson, B. Wavelet-based autofocusing and unsupervised segmentation of microscopic images Xie (2006) Xie, H.; Rong, W. & Sun, L. Wavelet-Based Focus Measure and 3-D Surface Reconstruction Method for Microscopy Images Gausenbroek (2000) Geusebroek, J.; Cornelissen, F.; Smeilders, A. & Geerts, H Robust autofocusing in microscopy Minhas (2009) Minhas, R.; Mohammed, A. A.; Wu, Q. M. & Sid-Ahmed, M. A. 3D Shape from Focus and Depth Map Computation Using Steerable Filter Marek Vajgl (IRAFM) Meausrement of multifocus fusion 42 / 42

The method of Compression of High-Dynamic-Range Infrared Images using image aggregation algorithms

The method of Compression of High-Dynamic-Range Infrared Images using image aggregation algorithms More info about this article: http://www.ndt.net/?id=20668 Abstract by M. Fidali*, W. Jamrozik* * Silesian