The influence of the background estimation on the superresolution properties of non-linear image restoration algorithms

Transcription

1 The influence of the background estimation on the superresolution properties of non-linear image restoration algorithms G.M.P. van Kempen a,2, L.J. van Vliet b a Spectroscopy & Microscopy Unit, Unilever Research Vlaardingen, Olivier van Noortlaan 20, 333 AT Vlaardingen, The Netherlands b Pattern Recognition Group, Faculty of Applied Physics, Delft University of Technology, Lorentzweg, 2628 CJ Delft, The Netherlands ABSTRACT The essential difference of non-linear image restoration algorithms with linear image restoration filters is their capability to restrict the restoration result to non-negative intensities. The iterative constrained algorithm () algorithm, for example, incorporates the non-negativity constraint by clipping each iteration of its conjugate gradient descent algorithm. This constraint will only be effective when the restored intensities have near zero values. Therefore the background estimation will have an influence on the effectiveness of the non-negativity constraint of these non-linear restoration algorithms. We have investigated the effect of the background estimation on the performance of the, Carrington, and Richardson- Lucy algorithms and compared it to the performance of the linear restoration filter. We found that an underestimation of the background will make the non-negativity constraint ineffective which results in a performance that does not differ much from the performance obtained by the linear restoration filter. An overestimation of the background however is even more dramatic since it results in a clipping of object intensities. We show that this will dramatically deteriorate the performance of the non-linear restoration algorithms. We propose a novel method to estimate the background based on the dependency of non-linear restoration algorithms on the background. Keywords: image restoration, background estimation, non-linear iterative restoration algorithms, superresolution. INTRODUCTION One of the essential differences between non-linear image restoration algorithms such as the iterative constrained Tikhonov- Miller algorithm (), the Carrington algorithm and the algorithm with linear image restoration filters is their capability to restrict the restoration result to non-negative intensities. The algorithm, for example, incorporates the non-negativity constraint by clipping each iteration of the conjugate gradient descent algorithm used to minimize the functional. This constraint will only be effective when the intensities of the iterations have near zero values. We model the image formation in a confocal microscope as the convolution of the original image with the microscope s point spread function on a background and distorted with noise. The and Carrington algorithms subtract the estimated background from the first estimate before they start their iteration. Therefore the background estimation will have an influence on the effectiveness of the non-negativity constraint of these algorithms. Correspondence: geert-van.kempen@unilever.com; Telephone: ; Fax: The presented research has been performed while the author was still a member of the Pattern Recognition Group G.M.P. van Kempen and L.J. van Vliet, The influence of the background estimation on the superresolution properties of non-linear image restoration algorithms, Proceedings of Three Dimensional and Multi-Dimensional Microscopy: Image Acquisition and Processing VI, BIOS 99, SPIE Photonics West 99, January 999, San Jose, CA, USA, pp

2 We have investigated the effect of the background estimation on the performance of the, Carrington, and Richardson- Lucy algorithms and of the regularized EM-MLE algorithm (as proposed by Conchello). We compared the performance of the non-linear methods to the performance of the linear restoration filter. The removal of the background from the acquired image is incorporated differently in the,, Carrington, and the Conchello algorithms. In the, and Carrington algorithms the assumption of additive Gaussian noise removes the dependency of the noise from the object and background. Therefore the background can be removed by subtracting it from the acquired image. The and Conchello algorithms model the noise as Poisson noise. This prevents the removal of the background by subtraction. Instead the background is incorporated in the conditional expectation of the translated Poisson process used to model the object intensities in the image. The EM algorithm uses the relative weights of the intensity of the object and of the background to estimate the object intensity from the intensities found in the acquired image. In both approaches an overestimation of the background will lead to large errors in the estimation of the object intensity. In the case of the and Carrington algorithms the subtraction of an overestimated background reduces low object intensities to negative values which are clipped to zero after the first iteration. This leads to an incorrect estimation of the object s shape reducing the performance of the algorithm. Similarly, an overestimation of the background in the EM case will lead to an underestimation of the object s intensities producing a poor performance. On the other hand an underestimation of the background yields undesired properties as well. Not only will it lead to an incomplete removal of the background but low object intensities will - in the case of the and Carrington algorithms - not be set to near zero values in the first estimate. Therefore, the number of points in the object being clipped in each iteration is considerably reduced. Since the clipping operation of these algorithms is the only non-linear operation that distinguishes these algorithms from the linear filter, a reduction of the number of pixels being clipped will decrease the performance benefits of these algorithms. In the extreme case where no points are being set to zero, both the and the Carrington algorithms reduce to an iterative, conjugate-gradient algorithm producing a solution that is identical to the one produced by the linear Tikhonov- Miller restoration filter. In other words, without clipping the and Carrington algorithms produce a solution equal to the linear solution, thus without restoring information beyond the bandwidth of the point spread function. In that case the and Carrington algorithms will not produce the sometimes claimed super-resolution results. In section 2 we derive, based on a linear model for the image formation, the linear restoration algorithm, and discuss four (, Carrington,, Conchello) non-linear image restoration algorithms. In section 3 we describe how we performed our simulation experiments, and present the results of these experiments in section 4. Based on these results, we propose in section 5 a novel method the estimation of the background. We conclude in section IMAGE RESTORATION 2.. Classical Image Restoration: The restoration filter We assume that the image formation in a confocal fluorescence microscope can be modeled as a linear translation-invariant system distorted by noise mxyz,, = Nhxyz,, f xyz,, + bxyz,, () b g c b g b g b gh In this equation f represents the input signal, h the point spread function, b a (constant) background signal, N a general noise distortion function, and m the recorded fluorescence image. For scientific-grade light detectors N is dominated by Poisson noise. In classical image restoration, the signal-dependent Poisson noise is approximated by additive Gaussian noise. Using this additive Gaussian noise model for N we rewrite () as gxyz,, = mxyz,, bxyz,, = hxyz,, f xyz,, + nxyz,, (2) After sampling equation (2) becomes b g b g b g b g b g b g M M M i= j= k = x y z gxyz,, = hx i, y jz, kf i, jk, + nxyz,, (3)

3 with M x, M y and M z the number of sampling points in respectively the x, y and z dimension. For conveniences we will adopt a matrix notation g = Hf + n (4) where the vectors f, g and n of length M d M = M xm ym zi denote respectively the object, its image and the additive Gaussian noise. The MxM matrix H is the blurring matrix representing the point spread function of the microscope. The filter, a classical image restoration filter, is a convolution filter operating on the measured image. It can be written as f ˆ = Wg (5) with W the linear restoration filter and f ˆ its result. The filter is derived from a least squares approach. This approach is based on minimizing the squared difference between the acquired image and a blurred estimate of the original object, Hˆ f g 2 (6) However a direct minimization of (6) will produce undesired results since it does not take into account the (high) frequency components of f ˆ that are set to zero by the convolution with H. Finding an estimate f ˆ from (6) is known as an ill-posed problem []. To address this issue Tikhonov defined the regularized solution f ˆ of (4) the one that minimizes the well-known Tikhonov functional [] Φ f Hf g 2 λ Cf 2 (7) ej= + with 2 the Euclidean norm. In image restoration λ is known as the regularization parameter and C as the regularization matrix. The Tikhonov functional consists of a mean square error fitting criterion and a stabilizing energy bound which penalizes solutions of f ˆ that oscillate wildly due to spectral components which are dominated by noise. The minimum of Φ can be found by solving ˆ f Φ ()= ˆ f 2H T ( Hˆ f g)+ 2λC T Cˆ f = 0 (8) which yields the well-known (TM) solution W TM H T W TM = (9) H T H + λc T C The linear nature of the makes it incapable of restoring frequencies for which the PSF has a zero response. Furthermore, linear methods cannot restrict the domain in which the solution should be found. This property is a major drawback since the intensity of an imaged object represents light energy, which is non-negative. The algorithm, the Carrington algorithm and the algorithm are frequently used in fluorescence microscopy [2-8]. These iterative, non-linear algorithms tackle the above mentioned problems in exchange for a considerable increase in the computational complexity Constrained Tikhonov Restoration The iterative constrained algorithm The iterative constrained () [2, 9] finds the minimum of (7) using the method of conjugate gradients [0]. The non-negativity constraint is incorporated by setting the negative intensities after each iteration to zero The Carrington algorithm Like the algorithm the Carrington algorithm [3, ] minimizes the Tikhonov functional under the constraint of nonnegativity. However the algorithm is based on a more solid mathematical foundation. Carrington used the Kuhn-Tucker conditions [3], ˆ f Φ i = 0 and f ˆ i > 0 or f ˆ Φ i 0 and f ˆ i = 0 (0) to transform the Tikhonov functional with the added non-negativity constraint to the functional Ψ (on the set H T c > 0)

4 bg e j Ψ c = P H T c c T g+ λ c () 2 Since Ψ is strictly convex and twice continuously differentiable, a conjugate gradient algorithm similar to can be used to minimize Ψ Maximum Likelihood Restoration: The algorithm In contrast with the two algorithms discussed previously the is not derived from the image formation model (4) which assumes additive Gaussian noise. Instead the general noise distortion function N is assumed to be dominated by Poisson noise. A fluorescence object can be modeled as a spatially inhomogeneous Poisson process F with an intensity function f [2], 2 P( F i f i )= f i F i e f i F i! The image formation of such an object by a fluorescence microscope can be modeled as a translated Poisson process [2]. This process models the transformation of F into a Poisson process m subjected to a conditional probability density function H, E[ m]= Hf + b (2) with b the mean of an independent (background) Poisson process. The log likelihood function of such a Poisson process is given by [2] L()= f Hf m T ln( Hf + b) (3) where we have dropped all terms that are not dependent on f. The maximum of the likelihood function L can be found iteratively using the EM algorithm as described by [3]. This iterative algorithm was first used by [4] in emission tomography. Holmes [4] introduced the algorithm to microscopy. Applying the EM algorithm to (3) yields [4, 7, 5-7] ˆ f k+ = ˆ f k H T Hˆ f k m (4) + b The EM algorithm ensures a non-negative solution when a non-negative initial guess f 0 is used. Furthermore the likelihood of each iteration of the EM algorithm will strictly increase to a global maximum [2]. The EM algorithm for finding the maximum likelihood estimator of a translated Poisson process (often referred to as EM-MLE) is identical to the Richardson- Lucy algorithm [8]. The algorithm is a constrained but unregularized iterative image restoration algorithm. The and Carrington algorithms, however, incorporate Tikhonov regularization to suppress undesired solutions. Conchello has derived an algorithm that incorporates Tikhonov regularization into the algorithm [5]. Incorporation of Tikhonov regularization yields f k + regularized f k = + + 2λ λ with f k + given by (4). Using l Hopital s rule it is easy to show that (5) becomes (4) when λ 0. We will refer to this algorithm as the algorithm. 3. EXPERIMENTS In this section we present results from a simulation experiment in which the performance of several image restoration algorithms is measured as function of the estimated background. We not only measured the overall performance but also measured the performance in specific regions in both the frequency and spatial domain. We used the footprint of the optical transfer function (OTF) [9] as a mask to measure the performance inside and outside the bandwidth of the point spread function. This gives insight to what extent these non-linear algorithms restore information beyond the microscope s resolution. In the spatial domain we measured the performance in three regions in the image: inside the object, at its boundary, and in the background of the image. To avoid aliasing effects, we used bandlimited masks to select these regions. The mask for the + (5)

5 boundary region has been generated by normalizing the gradient magnitude of the acquired image (see Figure ). The gradient of the image has been computed by convolving the image with Gaussian derivatives. We used a sigma of 0.9 pixels for the derivatives. We segmented the inverse of the boundary mask into two masks: an object mask and a background mask (see Figure ). image gradient magnitude mask inverse mask boundary region object & background region Figure Generation of object, boundary and background region grey-weighted masks using the gradient magnitude of the image. We have tested the linear filter, the algorithm, the algorithm and the Conchello algorithm. We also included the algorithm with the clipping after each iteration disabled to show that this variant of the algorithm is nothing but a conjugate gradient algorithm to find the linear result. The algorithm performs better than linear filters by clipping after each iteration. One could question whether the result obtained this way is different from clipping the result of the linear filter. We have therefore included the result of this procedure as well (referred to as clipped ). mean square error 00 clipped TM Carrington unclipped Unrestored Figure 2 Mean-square-error performance of the,,,, clipped, Carrington and unclipped algorithms together with the performance of the unrestored data as a function of the estimated background value. All images have been generated with the same constant background intensity of 6.0 ADU. However, we have varied the estimate of the background intensity, which is an input to the restoration algorithms, from 0.0 to 32.0 ADU. We have used a simulated confocal point spread function with a NA of.3, a refractive index of.55, an excitation wavelength of 488 nm, an emission wavelength of 520 nm, and a pinhole diameter of 300 nm. The images, sampled at twice the Nyquist rate, are 64x64x32 pixels large which corresponds to an image size of.55x.55x2.73 µm. We used spheres with an intensity of ADU and a diameter of 500 nm as objects. In this experiment we used a signal-to-noise ratio of.0 which corresponds to a conversion factor of 0.22 ADU/photon.

6 mean square error clipped TM Carrington unclipped Figure 3 Enlargement of Figure 2 around the. 4. RESULTS Figure 2 and Figure 3 clearly show that the performance of the tested non-linear restoration algorithms is strongly dependent on the estimation of the background. A (severe) underestimation of the background yields a performance of these algorithms that is not significantly better than the linear filter. An overestimation has an even more dramatic influence on the performance. The performance drops 3 quite significantly for relatively small overestimations (< 25 %), drops below the performance of the linear filter for an overestimation of 25-50%, and even drops below the performance of the unrestored (acquired) image for an overestimation larger than 50%. 00 unclipped Figure 4 Mean-square-error performance of the,,,, and clipped algorithms measured inside the bandwidth of the OTF. The figures also show that the performance of the unclipped is identical to the linear filter, and that the Carrington algorithm yields a performance characteristic very similar to the algorithm. We have therefore not included the results of the unclipped algorithm and the Carrington algorithm in further analysis. 3 A lower performance corresponds with a higher mean-square-error, which is plotted in Figure 4.

7 The performance inside the bandwidth of the OTF, as shown in Figure 4, shows a similar characteristic as found for the overall performance. Roughly ninety percent of the total mean-square-error is measured in this region. This is simply explained by the fact that most of the object energy is found in this region. The remaining ten percent, found outside the bandwidth of the OTF shows a slightly different characteristic (see Figure 5). The mean-square-error of the linear is here lower than all the non-linear algorithms except for the RL- Conchello algorithm. The mean-square-error of the filter is simply the energy of the object in this region of the Fourier domain. As expected, the non-linear algorithms add frequency components outside the bandwidth of the OTF to improve the performance inside the bandwidth. These restored components however do not always improve the performance outside the bandwidth as well. 0 unclipped Figure 5 Mean-square-error performance of the,,,, and clipped algorithms measured outside the bandwidth of the OTF unclipped Figure 6 Mean-square-error performance of the,,,, and clipped algorithms measured inside the object.

8 The performances in the spatial domain on the object, edge and background are shown in Figure 6, Figure 7 and Figure 8. We conclude from these figures that the largest gain in performance is found in the background, the region that contains the most pixels and the lowest pixel intensities. 00 unclipped Figure 7 Mean-square-error performance of the,,,, and clipped algorithms measured around the edges of the object unclipped Figure 8 Mean-square-error performance of the,,,, and clipped algorithms measured in the background. The algorithm produces the best performance in the object and edge region for an overestimation of %, when some of the low intensity edges of the object are being clipped introducing (correlated) high frequency components (Figure 5). These figures also clearly show that only the performance of the clipped algorithm in the background region is influenced by the background estimation.

9 5. BACKGROUND ESTIMATION The experiments presented in the previous paragraph show the strong influence of the background estimation on the performance of non-linear restoration algorithms. In this section we discuss how the background can be estimated in real images. Given the diversity of the samples being imaged with fluorescence (confocal) microscopy, the characteristics of the acquired images are similarly diverse. Therefore it is impossible to construct a single background estimation algorithm suitable for such a wide range of images. Not only do the samples vary from sparse (like our images of spheres) to very dense, the background can be different as well counts intensity (ADU) Figure 9 Histogram of a simulated confocal image as used in the experiment described in section 3. In general, the background can be characterized as being of low intensity and of a low spatial frequency. In images of sparse objects the majority of the pixels are background pixels, and a histogram-based algorithm can be used to estimate the background. Figure 9 shows the histogram of one of the simulated confocal images used in the experiments as described in section 3. Assuming that the distribution of the dominant noise source in the image is unbiased and unimodal (which holds for Poisson noise), we can estimate the background using the position of the maximum of the histogram. In images of dense objects or in images with a non-constant background, the histogram-based approach will not be very accurate (still the maximum of the lowest intensity peak might give a reasonable estimate). In these cases an approach based on mathematical morphology or fitting of a polynomial might work. In the first approach an opening operation can be used to estimate the shape of the background [20]. The second approach fits a low order polynomial through the pixels [2]. The accuracy and the bias of the fit will be improved if the fit is done through background pixels only. Therefore the image needs to be segmented (coarsely) in object and background pixels. One way of doing this is to use a criterion based on the noise variance. Given a first (histogram-based) estimate of the background intensity a pixel can be labeled as object pixel if its intensity is more that n (for example 2 or 3) times the standard deviation of the noise. We propose however an alternative method for estimating the background using the dependency of the non-linear restoration algorithms on the background estimation. As illustrated by Figure 4, the performance of the non-linear algorithms inside the bandwidth of the OTF is (strongly) dependent on the background. Therefore a measure that uses this part of the frequency domain can be used to measure the performance of a non-linear algorithm as function of the background estimation. By optimizing this measure as function of background, the optimal background can be determined. We propose to use the mean-square-error between the acquired image and the restoration result blurred by the microscope s OTF with the added background, g Hf + b e e jj 2 (6)

10 to measure the performance as function of the background. The values of this measure as discussed in the experiments in the previous section are shown in Figure 0 for the, clipped-tm, and algorithms clipped TM Figure 0 The mean-square-error between the acquired image and the blurred restoration result as function of the background. Using the proposed measure we can find the optimal background value by increasing the background value until the meansquare-error increases significantly. For a constant background, the optimal background value will be found in the interval between zero and to the mean intensity of the acquired image. A disadvantage of the proposed method for background estimation is that it requires a few restoration results obtained with, for example, the algorithm. This could lead to a high computational complexity, resulting in an unacceptably long processing time. This could be solved by using the clipped filter instead. As can be observed from Figure 2 the mean-square-error of the clipped filter is also minimal for the correct value of the background. 6. CONCLUSIONS We model the deterministic part of the image formation as a convolution of the original object with the point spread function on a background. Since we require that restoration algorithms restore the original object from the acquired image, both the blurring and the background need to be removed from the image. The iterative restoration algorithms discussed in this paper are non-linear since they constrain their result to non-negative values. This constraint will only be effective when the intensities in the restoration result have values near zero. We have shown that the effectiveness of this constraint is strongly influenced by the background estimation, which is an input parameter in all restoration algorithms. We showed that an underestimation of the background will make the constraint ineffective which results in a performance of these nonlinear algorithms which does not differ much from the performance obtained by linear restoration filters. An overestimation of the background however is even more dramatic since it results in a clipping of object intensities. We showed that this will dramatically deteriorate the performance of these non-linear restoration algorithms. Finally, we showed that the dependency of these non-linear restoration algorithms on the background can be used to estimate the background. ACKNOWLEDGMENTS This work was partially supported by the Royal Netherlands Academy of Arts and Sciences (KNAW) and by the Rolling Grants program of the Foundation for Fundamental Research in Matter (FOM).

11 REFERENCES [] A. N. Tikhonov and V. Y. Arsenin, Solutions of Ill-Posed Problems. New York: Wiley, 977. [2] H. T. M. v. d. Voort and K. C. Strasters, Restoration of confocal images for quantitative image analysis, J. Micros., vol. 78, pp. 65-8, 995. [3] W. A. Carrington, Image Restoration in 3D Microscopy with limited data, presented at Bioimaging and Two- Dimensional Spectroscopy Proc SPIE, 990. [4] T. J. Holmes, Maximum-likelihood image restoration adapted for noncoherent optical imaging,, vol. 5, pp , 988. [5] J.-A. Conchello and J. G. McNally, Fast regularization technique for expectation maximization algorithm for optical sectioning microscopy, presented at Three-Dimensional Microscopy: Image Acquisition and Processing III, San Jose, 996. [6] G. M. P. v. Kempen, L. J. v. Vliet, and P. J. Verveer, Application of image restoration methods for confocal fluorescence microscopy, presented at 3-D Microscopy: Image Acquisition and Processing IV, San Jose, 997. [7] G. M. P. v. Kempen, Image Restoration in Fluorescence Microscopy, in Department of Applied Physics. Delft: Delft University of Technology, 999, pp. 52. [8] P. J. Verveer, M. J. Gemkow, and T. M. Jovin, A comparison of image restoration approaches applied to threedimensional confocal and wide-field fluorescence microscopy, Journal of Microscopy, vol. submitted, 998. [9] R. L. Lagendijk and J. Biemond, Iterative identification and restoration of images, vol. IP. Boston/Dordrecht/London: Kluwer Academic Publishers, 99. [0] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C, 2nd edition, vol. CS. Cambridge: Cambridge University Press, 992. [] W. A. Carrington, R. M. Lynch, E. M. Moore, G. Isenberg, K. E. Fogarty, and F. S. Fay, Superresolution Three- Dimensional Images of Fluorescence in Cells with Minimal light exposure., Science, vol. 268, pp , 995. [2] D. L. Snyder and M. I. Miller, Random Point Processes in Time and Space, vol. SP. Berlin: Springer Verlag, 99. [3] A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum Likelihood from Incomplete Data via the EM Algorithm., J of the Roy. Statist. Soc. B, vol. 39, pp. -37, 977. [4] Y. Vardi, L. A. Shepp, and L. Kaufman, A Statistical Model for Positron Emission Tomography, J. Am. Statistical Association, vol. 80, pp. 8-35, 985. [5] L. A. Shepp and Y. Vardi, Maximum Likelihood Reconstruction for Emission Tomography, IEEE Trans. Med. Imaging, vol. MI-, pp. 3-2, 982. [6] D. L. Snyder, A. M. Hammoud, and R. L. White, Image recovery from data acquired with a charge-coupleddevice camera, J. Opt. Soc. of Am. A, vol. 0, pp , 993. [7] G. M. P. v. Kempen, L. J. v. Vliet, P. J. Verveer, and H. T. M. v. d. Voort, A Quantitative Comparison of Image Restoration methods for confocal microscopy, J. Micros., vol. 85, 997. [8] W. H. Richardson, Bayesian-based iterative method of image restoration, J. Opt. Soc. of Am., vol. 62, pp , 972. [9] T. Wilson and J. B. Tan, Three dimensional image reconstruction in conventional and confocal microscopy, BioImaging, vol., pp , 993. [20] P. W. Verbeek, H. A. Vrooman, and L. J. v. Vliet, Low-level image processing by max-min filters, Signal Processing, vol. 5, pp , 988. [2] I. T. Young, Automated Leukocyte Recognition, in Automated Cell Indentification and Cell Sorting, G. L. Wied and G. F. Bahr, Eds. New York: Academic Press, 970, pp