Deconvolution of 3D Fluorescence Microscopy Images by Combining the Filtered Gerchberg-Papoulis and Richardson-Lucy Algorithms

Transcription

1 Deconvolution of 3D Fluorescence Microscopy Images by Combining the Filtered Gerchberg-Papoulis and Richardson-Lucy Algorithms Moacir P. Ponti-Junior Nelson D.A. Mascarenhas Marcelo R. Zorzan Claudio A.T. Suazo Murillo R.P. Homem Universidade Federal de São Carlos Rod. SP-310, São Carlos-SP - Brazil moacir@dc.ufscar.br Abstract The problem of deconvolution in fluorescence microscopy deals at the same time with the diffraction limit that cuts off some of the frequencies, blurring the image, and photon noise, that corrupts the image by inserting elements that are not present in the real object and also distorting the contrast. This distortions hampers the possibility of using the 3D images for recognition and analysis applications. In addition, the algorithms developed, in general, assume absence of noise or a white additive noise. This work presents an approach to deconvolve the images and deal with the noise present in real images. The Gerchberg-Papoulis algorithm and a smoothing operator were combined with the Richardson-Lucy iterative algorithm. The results of the method for simulated data are compared with the ones obtained by the original algorithm. The method improved the results by obtaining higher signal-to-noise ratio and quality index values, performing a better band extrapolation. 1. Introduction Fluorescence microscopy is a powerful tool for medical and biological applications and research. The images obtained with these instruments, however, are distorted by the acquisition system. The wide-field microscope, the most used fluorescence microscope, is subject to the diffraction limit as any optical system due to the finite aperture of the lens. As a result, the light originated from a point in the focal plane is not exactly imaged as point. The widefield microscope uses an excitation light that illuminates the whole specimen, so that the camera captures light provenient from the focal plane and also from the planes above and below, yieding an out-of-focus, blurred image. The microscopy images are also often degraded by noise, as a result of the photonic statistical nature of the process. These characteristics make it difficult to use the images to obtain three-dimensional (3D) images by computational optical sectioning (COSM), a technique used in microscopy to obtain a 3D image from a series of 2D images of different focal planes [1]. Since the formation of an image changes the recorded information content with respect to that of the original object, there is interest directed to processing images so that the results closely match the original object. The algorithms that allow a reconstruction of the distorted images and the recovery of frequencies lost in the acquisition process are called restoration or super-resolution restoration algorithms. Fourier optics demonstrates that there exists a cut-off spatial frequency which is directly determined by the shape and size of the limiting pupil in the optical system. This distortion of the spatial frequency components is governed by the OTF (Optical Transfer Function), the normalized Fourier transform of the PSF (Point Spread Function) of an imaging system (see Figure 1). The PSF describes the spatial spread for a single point input. The presence of noise is another distortion caused by the low exposure acquisition time. This noise is well modeled by a Poisson distribution, details of which will be discussed later. A probabilistic algorithm was developed by Richardson and Lucy [2, 3] to restore images using a Poisson process to model the image formation system such as in telescopic, the focus of their work, and also microscopic systems. At the time those works were written, the ability to restore frequencies beyond the image or signal passband was controversial. However, Gerchberg and Papoulis [4, 5] showed that the imposition of known constraints on frequency and time domains could extrapolate a signal. This algorithm, however, was not suitable to images with a more complex formation model, since that algorithm was not concerned with blur and noise, but it led to the use of constraints in image restoration problems, that was also the focus of the POCS (Projection Onto Convex Sets) methods, an approach to the image restoration problem that was introduced by Youla and Webb [6]. The band extrapolation possibility was

2 analysed, and a Maximum a Posteriori algorithm was developed to performed superresolution, stating the real possibility of band extrapolation [7, 8, 9]. A POCS-Taylor expansion extrapolation algorithm was developed in 2003, still assuming absence of noise [10]. The pre-filtering of 3D microscopy images has been explored recently as a means of dealing with noise [11]. Previous work also applied a combination of Richardson-Lucy and the Gerchberg-Papoulis for 2D images [12] The method proposed in this work combines the Gerchberg-Papoulis (GP) with the Richardson-Lucy (RL) algorithm. The iterative procedure also includes a filtering step between the iterations. The filtering is carried out just in the extrapolated frequencies of the spectrum, the known portion being re-inserted after the filtering. This procedure makes use of the prior information imposed by the GP algorithm to reduce the variation of the noise, stopping the noise amplification without smoothing the signal. An averaging filter was used in this context. The rationale behind the use of these filters is further explained and basically one takes advantage of the smoothing properties of the operators to prevent undesirable artifacts [13, 11]. Our goal is simultaneous restoration and extrapolation improvement. The main contribution of this work is to show that better superresolution results can be achieved with the combination of the previous methods and also improving the contrast of the 3D images in a faster iterative procedure. Figure 1. Contour line of the frequency domain support of a OTF of the non-confocal microscope. 2. Microscopy Images The general imaging system for an optic wide-field microscope can be modeled as: g(x, y, z) = f(x, y, z) h(x, y, z) + n(x, y, z), (1) where g is the observed image, h the point spread function (PSF) that models the blurring due to the optical system, f is the original image, is a three-dimensional convolution operator and n is the additive Gaussian noise. The model can be also efficiently expressed in terms of the Fourier transform: G(u, v, w) = F (u, v, w) H(u, v, w) + N(u, v, w), (2) where the capital letters indicate the Fourier transform of each object described on equation 1. The model described is suitable for an incoherent imaging system [14]. Under this approach, the natural solution would be the inverse filter: G(u, v, w) N(u, v, w) F (u, v, w) = H(u, v, w) H(u, v, w). (3) Unfortunately, practical OTF have zero valued regions (beyond the diffraction limit) often called missing cone regions, resulting in no information on these areas. Figure 1 shows a frequency domain support for a wide-field microscope OTF showing the ρ = (u, v) 1/2 against w to illustrates how the z axis is affected and defines the band limit Ω [15]. For instance, microspheres images appear diamond shaped when viewed in an (x, z) or (y, z) planar view. Besides, there are low valued regions that amplify the noise. So, under these conditions the inverse filter does not exist. Due to the photon counting nature of light based sensors, a Poisson distribution can model the noise [16]. Considering each value of the function g(x, y, z) to be a realization of a random variable described by a Poisson process, the model can be written as: g(x, y, z) = N {f(x, y, z) h(x, y, z)}, (4) where N {} represents a random process, more specifically a Poisson process. This kind of process produces a noise that is correlated with the image spanned by f(x, y, z) h(x, y, z). The images are obtained under a low exposure to avoid photobleaching, a process that causes loss of fluorescence; therefore it causes the Poisson noise to be higher. This model does not consider the additive Gaussian noise as in the Equation 1. Indeed, the electronic CCD adds Gaussian noise but it is not significant compared to the Poisson photon counting noise [17]. 3. Richardson-Lucy Algorithm The Richardson-Lucy (RL) algorithm [2, 3] uses a probabilistic approach: given a degraded image g, what is the im-

3 age ˆf that maximizes the probability of observing the image g? Considering the image as an observation of a Poisson process, the likelihood function would be: p(g ˆf) = x h(x) ˆf(x) g(x) h(x) ˆf(x) e, (5) g(x)! where x represents (x, y, z) for a 3D signal. The RL algorithm minimizes the functional L( ˆf) = log p(g ˆf), giving the maximum likelihood estimation: L( ˆf) = [ g(x) log (h ˆf)(x) ] + (h ˆf)(x). (6) x The RL iteration is given by: [( ) ] g(x) ˆf n+1 (x) = ˆf(x) h(x) h(x) ˆf n (x). (7) This algorithm is generally stopped after a finite number of iterations. When the deconvolution is ill-posed (a common situation in real applications), the signal-to-noise ratio becomes increasingly poorer as the number of iterations n. 4. Gerchberg-Papoulis algorithm The Gerchberg-Papoulis (GP) algorithm [4, 5] assumes that there is some knowledge about the bandwidth and iteratively imposes the requirements that the signal is bandlimited and matches the known portion of the signal. Let g(x) have a spectrum G(u) and Ω the region where G(u) is nonzero, u represents (u, v, w) for a 3D signal in the frequency domain. Since g(x) is known within a region T, since it is space limited, the spatial support can be defined as: { 1, (x) T B T (x) = (8) 0, (x) / T The spectral pupil can be defined in the same context: { 1, (u) Ω B Ω (u) = (9) 0, (u) / Ω The algorithm consists in imposes the constraints above, as follows; ê 0 (x) = B T g(x), ê n+1 (x) = ê 0 + (1 B T ) F 1 {B Ω E n (u)}, (10) where ê 0 (x) is the first estimation of the extrapolated image, ê n+1 (x) is the estimation at interation n+1, and Ên(u) is the current estimation in the frequency domain. The signal is clipped in the frequency domain by the spectral pupil and the known portion of the signal is imposed in the space domain. It constrains the frequencies to extend the spatial limit. The convergence of this method is established in the absence of noise [5]. 5. Method 5.1. Filtered Gerchberg-Papoulis The presence of noise, specially non-additive non- Gaussian noise, can easily led the reconstruction algorithm to diverge after a number of iterations. So, a filtering step can be introduced to reduce the variation in the image and make the restoration and spectrum extrapolation easier. The filtering of data before running a restoration method for a non-linear iterative method was explored by Van Kempen et al. [18] and for linear restoration by Colicchio et al. [11], showing good results. To prevent over-smoothing, and to preserve the frequencies already present in the observed image, the filtering step is restricted to the extrapolated frequencies beyond the limit Ω. This represents an attempt to preserve the signal while smoothing the high-frequencies, since the signal information in extrapolated frequencies are often corrupted by noise as pointed by Hunt, Nadar and Sementilli [7]. The algorithm that includes the filtering of the extrapolated frequencies can be written as follows: Ĉn+1(u) F = Filter [ (1 B Ω ) F { B T ê f n(x) }] }, ê F n+1(x) = F {B 1 Ω Ên F (u) + Ĉf n+1, (11) where ê F n (x) and ÊF n (x) are the previous estimates of the filtered-extrapolated image and its Fourier transform, respectively; and Ĉf n+1 (x) is the filtered extrapolated frequencies. So, the extrapolated portion of the spectrum is filtered and the frequencies within the band-limit are reinserted. The spectral pupil can be defined as a circular area defined by the frequency boundaries where the spectrum goes approximately to zero, or the practical bandwidth. The space support must be wide enough so that the object lies completely on it. The modified algorithm constrains the spatial limit in order to extrapolate the frequency limit Algorithm The proposed method simultaneously applies the RL regularized algorithm and the GP algorithm as follows: ˆf n+1 = RL [e n ] ê n+1 = FGP [f n+1 ], (12) where RL is the Richardson-Lucy algorithm and FGP stands for Filtered Gerchberg-Papoulis, referring to the al-

4 gorithm defined. Since the GP constraints are applied, the band extrapolation is expected to be improved. levels. This image was convolved with a theoretic microscope PSF of size 128x128x128. The PSF was constructed following the theoretical model proposed by Gibson and Lanni [19]. A montage of sections of the original phantom is shown in Figure 2 and the correspondent sections of the image blurred by the PSF and also degraded by Poisson noise are presented in Figure 3. A 3D real image obtained with a wide-field fluorescence microscope was also used in the experiment, this image has the same resolution and gray levels as the phantoms. Both phantom and real images were restored using the RL algorithm; the RL algorithm combined with the classical GP, to which we refer as RL-GP; and the RL algorithm combined with the filtered GP, or RL-FGP. The filter used for the RL-FGP algorithm was a [3x3] moving average filter Evaluation Figure 2. Axial sections (x,z) number 32, 42, 54 and 64 of the original bead image The restored images were evaluated by observing ISNR (Improvement on Signal-to-Noise Ratio) and the UIQI (Universal Image Quality Index) [20]. The ISNR is given by: g f ISNR = 20 log 10 ˆf f, (13) where g is the degraded image, f the original image and ˆf the restored image. The ISNR compares the restored images with the original, and yields a number that measures the relative improvement. The UIQI is given by: UIQI = 4σ f ˆf f ˆf ( ) σf 2 + σ2ˆf (f 2 + ˆf ), (14) 2 Figure 3. Axial sections (x,z) number 32, 42, 54 and 64 of the degraded bead image 6. Experiments A series of experiments were carried out using the methods described in the previous sections. A bead phantom image was built, with 128x128x128 voxels and 256 gray where letters with bars are averages; σ 2ˆf and σf 2 are the variance of the original and restored images, respectively, and σ f ˆf is the correlation coefficient between f and ˆf. The dynamic range of UIQI is [ 1, 1]. The best value, 1, is achieved if and only if f = ˆf. Also, to assess the spectral extrapolation, we use a method described by Conchello [15], estimating the practical bandpass of the image by as the region of the frequency domain where the modulus of the Fourier transform is larger than 1% of its peak value. That is, the bandpass is the region where: ˆF (u)/ ˆF (0) > 0.01, (15) This measure can give a numerical information on how much the algoritms extrapolated the frequencies, and can be used to compare different methods.

5 7. Results The degraded phantom image was restored using the three algorithms as described in section 6. The section number 54 of the bead phantom image as restored with each algorithm is shown in Figure 4 the colormap was adjusted in all images for better visualization of the solutions. The section number 54 of a real image restored with the methods is also shown in Figure 5. With the proposed method, the images were better restored, confirmed by an improvement in both the ISNR and UIQI; the practical bandwidth of the images also increased with use of the GP constraints as shown in Table 1, showing the best results obtained by each method, stopped when the ISNR decreased (RL with 380 iterations, RL-GP with 320 iterations and RL-FGP with 360 iterations) An undesirable effect of this method is ringing artifacts when the spectral support is taken too small or the filter oversmooths the frequencies. One example is shown in Figure 6, with the colormap adjusted to enhance the visualization of the artifacts. Images ISNR UIQI Bandpass Degraded Image RL RL-GP RL-FGP Table 1. Bead image restoration evaluation Images Bandpass Degraded Image 731 RL 1983 RL-GP 2485 RL-FGP 2831 Table 2. Real image band extrapolation 8. Conclusion Noise is always a difficult obstacle to overcome. Many well-known restoration algorithms yield good results in the absence of noise, but show poor performance when dealing with real or noisy data. The proposed algorithm performs a good restoration and spectrum enhancement using the known constraints and filtering. The spatial support constraint allows a better spectral match to the original object and the filtering prevents noise amplification. Because of Figure 4. Axial sections n.54 (x,z) of the best restoration results for the bead image: (a) degraded; (b) RL with 380 iterations; (c) RL-GP with 320 iterations; (d) RL-FGP with 360 iterations the superior regularization, a larger number of iterations can be used before a high level of noise amplification occurs. In some cases, the proposed method achieves a better signalto-noise ratio sooner than the competing methods. The proposed algorithm leads to better ISNR and UIQI than the other methods. The results can be explained by the fact that the FGP method includes an extra regularization in the frequency domain due to the filtering step. The results with the real image do not show differences as marked, possibly because the image is relatively noise-free. However, the contrast was improved. Further tests with images acquired with lower exposure times and, consequently more noise can show how the algorithm behaves under high level noise conditions. The band extrapolation is still an open problem, specially in the presence of noise. The increasing of the practical bandwidth of the images indicates that the RL-FGP algorithm is expected to perform a better spectrum extrapolation when compared to the classical RL algorithm, since it enforces both frequency and spatial support constraints. The improvement in the band extrapolation do not assure the correct estimation of the spectrum beyond the diffraction limit. However, the restored images are visually better, and are appropriate to be used in recognition and analysis applications. Future work could analyse this feature by observing how the spectrum spreads, and how reliable the extrapolation is. The design of more accurate filters to deal with the noise without loss of detail is a point left for future work.

6 Figure 5. Axial sections n.54 (x,z) of the best restoration results for the real image: (a) degraded; (b) RL with 290 iterations; (c) RL-GP with 240 iterations; (d) RL-FGP with 260 iterations Figure 6. Ringing artifacts in case of oversmoothing References [1] P. Sarder and A. Nehorai. Deconvolution methods for 3-D fluorescence microscopy images. Signal Processing Magazine, 23(3):32 45, [2] W.H. Richardson. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am., 62(1):55 59, [3] L.B. Lucy. An iterative technique for the rectification of observed distributions. The Astronomical Journal, 79(6): , [4] R.W. Gerchberg. Super-resolution through error energy reduction. Opt. Acta, 21: , [5] A. Papoulis. A new algorithm in spectral analysis and bandlimited extrapolation. IEEE Trans. Circuits and Systems, 22(9): , [6] D.C. Youla and H. Webb. Image restoration by the method of convex projections: part 1-theory. IEEE Trans., MI-1(2):81 94, [7] P. Sementilli, B. Hunt, and M. Nadar. Analysis of the limit to super-resolution in incoherent imaging. J. Opt. Soc. Am. A, 10: , [8] B.R. Hunt. Super-resolution of images: algorithms, principles, performance. International Journal of Imaging Systems and Technology, 6: , [9] B.R. Hunt. Super-resolution of imagery: understanding the basis for recovery of spatial frequencies beyond the diffraction limit. In Information, Decision and Control, IDC 99. Proceedings., pages IEEE, [10] S. Bhattarcharjee and M.K. Sundareshan. Mathematical extrapolation of image spectrum for constraint-set design and set-theoretic superresolution. J. Opt. Soc. Am. A, 20(8): , [11] B. Colicchio, O. Haeberle, C. Xu, A. Dieterlen, and G. Jung. Improvement of the LLS and MAP deconvolution algorithms by automatic determination of optimal regularization parameters and pre-filtering of original data. Optics Communications, 224:37 49, [12] M. P. Ponti-Junior, N.D.A. Mascarenhas, and C.A.T. Suazo. A restoration and extrapolation iterative method for bandlimited fluorescence microscopy images. In IEEE Proceedings of 20th Brazilian Symposium on Computer Graphics and Image Processing, pages IEEE, Oct [13] C. Preza, M.I. Miller, and J.-A. Conchello. Image reconstruction for 3-D light microscopy with a regularized linear method incorporating a smoothness prior. In R.S. Acharya and D.B. Goldgof, editors, Proceedings of the IS&T/SPIE symposium on Electronic Imaging, Science and Technology, Biomedical Image Processing and Biomedical Visualization, volume 1905, pages SPIE, Feb [14] J.W. Goodman. Introduction to Fourier Optics. McGraw Hill, 2 edition, [15] J.-A. Conchello. Superresolution and convergence properties of the expectation-maximization algorithm for maximumlikelihood deconvolution of incoherent images. J. Opt. Soc. Am. A, 15(10): , [16] D.L. Snyder and M.I. Miller. Random Point Processes in Time and Space. Springer Verlag, [17] L.J. van Vliet, F.R. Boddeke, D. Sudar, and I.T. Young. Image detectors for digital image microscopy. In M.H.F. Wilkinson and F. Schut, editors, Digital Image Analysis of Microbes: imaging, morphometry, fluorometry and motility techniques and applications. Modern Microbiological Methods, pages John Wiley & Sons, [18] G.M.P. van Kempen, L.J. van Vliet, and P.J. Verveer. Application of image restoration methods for confocal fluorescence microscopy. In C.J. Cogswell, J.-A. Conchello, and T. Wilson, editors, 3-D Microscopy: Image Acquisition and Processing IV, volume 2984, pages SPIE, [19] F.S. Gibson and F. Lanni. Experimental test of an analytical model of aberration in an oil-immersion objective lens used in three-dimensional light microscopy. J. Opt. Soc. Am. A, 8(11): , [20] Z. Wang and A.C. Bovik. A universal image quality index. IEEE Signal Processing Letters, 9(3):81 84, 2002.