Image restoration by deconvolution

Transcription

1 Image restoration by deconvolution 17/12/2014 (part) Slides courtesy: Sébastien Tosi (IRB Barcelona)

2 A few concepts related to the topic Convolution Deconvolution Point Spread Function Noise Fourier Transform Spatial resolution Pixel size Rayleigh Criterion Airy disk Numerial Aperture Refractive Index Wavelength

3 Image formation (in Fluorescence microscopy) Image from [9] In fluorescence microscopy (in all its modes including widefield, confocal, and multi-photon): the imaging process can be mathematically described by a convolution

4 Imaging, Convonlution, Deconvolution Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting threedimensional image All microscopy techniques that include directly or indirectly a convolution in their image formation processes can benefit from image deconvolution.

5 2D Convolution Original image Filtered image Original image Filtered image Filter kernel The convolution can also be computed by stamping the kernel on each pixel of the image: the kernel is scaled (multiplied) by the intensity of the central pixel and accumulated (summed) in the output image.

6 3D Convolution = = A 3D kernel is a stack holding the filter coefficients Convolution extends to more than two dimensions: in 3D the kernel is a small volume (stack) and the sum is triple (inside a volume around each voxel).

7 Fourier transform (FT) If we take the FT of the equation, the is replaced by multiplication, thus image restoration might be achievable by: dividing the FT of the image by the FT of the kernel and then taking the inverse Fourier transform. X = = Spatial Domain Fourier Domain Image from Sébastien Tosi (IRB Barcelona)

8 Point Spread Function (PSF) An image resulting from a single small spherical fluorescent bead (smaller than the optical resolution, thus forms effective a point source of light) A record of how much the microscope has spread or blurred a single point Simplified diagram to visualize how a light-emitting point would be imaged using a widefield microscope, Image from [9] Widefield PSF, Image from [3] More slides from Math-Clinic BioImage Analysis website:

9 PSF (in the focal plane) Spatial resolution: distance by which two objects must be separated to be distinguished, i.e. the radius of the smallest point source in the image (defined as the first minimum of the Airy disk) the Rayleigh criterion: r radial 0.61λ 2λn = raxial = λ : fluorophore emission wavelength 2 NA NA Notes: 1. Rayleigh criterion has not taken into account the effects of: brightness, pixel size, noise 2. High NAs are possible when the immersion refractive index is high Widefield PSFs obtained by imaging 100-nm fluorescent beads (excitation 520nm; emission 617nm), Image from [2] NA : objective numerical aperture n : refractive index of the objective lens immersion medium NA = n sinθ Image from [9] NA can never exceed n, which itself has fixed values (e.g. 1.0 for air, 1.33 for water, or 1.52 for oil)

10 Theoretical PSF Quotes from [3]: Most methods are based on the work of Born and Wolf (1980). A good description for a confocal PSF is given by van der Voortand Brakenhoff (1990), in which the PSF is calculated from: the NA of the objective, the illuminating and emitted wavelengths, and the refractive index of the immersion medium in either (simpler) paraxial forms or with WF integrals. Theoretical PSF gives an indication of the best possible resolution for a given objective but these limits are not achievable. In our experience, real PSFs are typically >20% bigger than calculated versions. Experimental PSF (measured PSF) Theoretical PSF image from [9]

11 What else does (measured) PSF tells us? Asymmetry: radial (x-y): commonly misalignment of optical components about the z-axis, either as tilt or decentration along the optical axis (z-axis): commonly due to spherical aberration, which may result from refractive index mismatches between the objective, immersion medium, and sample or tube length/coverslip thickness errors. Notes: 1. The immersion refractive index should match the refractive index of the medium surrounding the sample, to avoid spherical aberration 2. Item 1 is often strongly preferable to using the highest NA objective available, as it is usually better to have a larger PSF than a highly irregular one. Image from [3]

12 Deconvolution Principle AWGN source Object (sample) Microscope PSF Digital filter Deconvolved image The deconvolution filter F should undo the effect of the microscope PSF H by processing the sampled image R, ideally D = S. Assuming H known, F linear (convolution) and no noise (N = 0) leads to: or or D H = S H ( R F) H = R R ( F H ) = R F H = 1 In this context 1 is a black image holding a single point in its center In practice the noise N and the error on the estimation of H (measurement or model) are impeding a perfect deconvolution and we can at best hope for an approximate solution

13 Noise Sources in Digital Microscopy

14 More/Details RECOMMEND! It is a pleasant reading! e-image-analysis-intro (Part III) Ref [2] More on reference slide

15 When to do deconvolution? Wide field microscopy (WFM) Less affected by out-of-focus light: Confocal laser scanning microscopy (CLSM) Two-photon excitation microscopy (TPEM) Selective plane illumination microscopy (SPIM) Super-resolution fluorescence microscopy Stimulated emission depletion microscopy (STED) All microscopy techniques that include directly or indirectly a convolution in their image formation processes can benefit from image deconvolution. Any 2D or 3D image obtained from almost any fluorescence microscope is expected to be deconvolved before being analysed.

16 Why to do deconvolution? Attenuation of the out of focus light - increase contrast Reduce noise Increase of the spatial resolution

17 Deconvolution Algorithms Used in Biological Fluorescence Image Processing

18 Deblurring - subtractive Nearest neighbour No neighbours Linear inverse filter Not for quantitative intensity measurements Do not count for noise Regularized Inverse filter Object smoothness e.g. Wiener filter Constrained Iterative Nonnegative Quantitative Blind deconvolution No PSF as input May not be absolutely quantitative Ref [2]

19 Linear Deconvolution: Inverse Filter Deconvolution As convolution in the spatial domain can be performed as a multiplication in the frequency domain, inverse filtering can be performed as a division in the frequency domain! But in practice Noise enhancement ruins our efforts! F u v H u v 1 (, ) = (, ) 4, , A very simple model for the PSF H (Gaussian std = 1 pixel) H power spectrum (log display) overlaid with raw values 1 H -1 power spectrum (log display) overlaid with raw values 1

20 Inverse Filter Deconvolution H is a Gaussian with std = 2 pixels H +N H -1 Original image S Original image S after convolution by H Blurred image Noise std = 10-4 Noise std = No noise

21 Second try: Regularized Inverse F( u, v) = 1 1 H ( u, v), H ( u, v) t 1 t, H ( u, v) > t 4, , A very simple model for the PSF H (Gaussian std = 1 pixel) H power spectrum (log display) overlaid with raw values (H -1 ) reg (1% clipping) power spectrum (log display) overlaid with raw values

22 Regularized Inverse Filter Deconvolution H +N Original image S Original image S after convolution by H Blurred image (H -1 ) trunc Restoration Noise of Blurred, std Noisy = Image 10-4 Using regularized inverse

23 Third Try: Wiener Filter The Golden Linear Deconvolution Trade-off Coming back to: ( ) Sest = F S H + N E = S S. est Minimizing the expectation of E over all possible noise realizations assuming a white Gaussian noise: F( u, v) = H * ( u, v) S( u, v) H ( u, v) S( u, v) + N( u, v) Wiener filter 2 Bands free of noise: N(u,v) = 0 F(u,v) = H(u,v) -1 Strong noise bands: N(u,v) F(u,v) 0 Intermediate bands: (inverse filter) (cut-off) best trade-off Wiener filter attenuates frequencies dependent on their signal-to-noise ratio.

24 Wiener Deconvolution Restoration Regularized of Blurred, Noisy inverse Image Using filter regularized result inverse Noise std = 10-4 Wiener filter result Noise std = 10-4 Restoration of Blurred, Noisy Image Using Wiener filter for known noise variance

25 Non-Linear Deconvolution The best deconvolution algorithms for 3D microscopy are typically non-linear. Principle of Maximum A Priori algorithms (MAP): ( MAP) Pr( R S) Pr( S) S = arg max Pr( S R) = arg max. S S Pr( R) The second equality comes from Bayes theorem. In the optimization S is usually constrained to be positive and somehow spatially smooth (TV regularization term) Pr(S). The statistical distribution of the noise has to be known to derive the maximum likelihood term Pr(R S) the algorithm is tuned to a particular noise (e.g. Poisson or Gaussian noise). There is usually no known analytical solution to the problem, the algorithms proceeds by iterations (candidate S i at iteration i) to refine the estimate of the data at each iteration. The Richardson-Lucy algorithm is among the most well known MAP deconvolution algorithm. Some algorithms also simultaneously estimate the PSF from the sampled image (blind deconvolution).

26 Quantification with deconvolution Ideally: relocate signal to the point of origin in 3D, thus conserve the sum of fluorescence signal. It improves quantification! In practice: different algorithms have more or less compromises Quantitative intensity measurements, e.g. intensity ratio: controls, also report on un-deconvolved data for comparison Quantitative positional or structural analysis, e.g. centroid, tracking, volume analysis, (object based) colocalisation, etc: relatively less critical the choice For all analysis: Deconvolution process comparable between datasets Compare with control/un-deconvolved data Understand algorithm used and choose most suitable Report possible artifacts and confirm it, if possible

27 Software tools PSF generators Deconvolution

28 Deconvolution tools (not exhaustive!) Fiji plugins Parallel iterative deconvolution (fiji.sc/parallel_iterative_deconvolution): 4 deconvolution algorithms Parallel spectral deconvolution (fiji.sc/parallel_spectral_deconvolution) Not iterative, no constraint e.g. nonnegativity Iterative Deconvolve 3D (fiji.sc/iterative_deconvolve_3d) : non-negative, iterative, similar to WPL algorithm. The execution is way slower on modern (multicore) computers but the memory requirement is less stringent DeconvolutionLab( different algorithms including a custom version of the thresholded Landweber algorithm Commercial software - SVI Huygens - MC AutoquantX - I could not comment on commercial software, due to access issue.

29 Examples Original PID (WPL, Wiener Gamma 0.1, 50IT, bead PSF) AutoquantX (30IT, bead PSF) Huygens (50IT, bead distilled PSF) Original PID (WPL, 50IT, bead PSF) AutoquantX (30IT, bead PSF) Huygens (50IT, bead distilled PSF) Original PID (WPL, 50IT, true PSF) AutoquantX (30IT, true PSF) Huygens (50IT, distilled true PSF) Courtesy of Sébastien Tosi (IRB Barcelona) + Free & Open source & full control + Reasonably fast + Support for spatially-variant PSF (un-tested) - High memory usage - Visually less crispy + Fast convergence + Robust algorithms + Very simple to use + Visually appealing results + 2D mode for thin samples - Expensive & Closed source + Microscope specific PSF + depth-varying PSF + supports spinning disk M. + Visually appealing results - Expensive & Closed source

30 Theoretical PSF generator Diffraction PSF 3D: using Fraunhofer diffraction PSF generator: >15 models

31 Parallel Iterative Deconvolution The plugin provides 4 deconvolution algorithms: - Wiener Filter Preconditioned Landweber (WPL) - Modified Residual Norm Steepest Descent (MRNSD) nonnegative - Conjugate Gradient for Least Squares (CGLS) - Hybrid Bidiagonalization Regularization (HyBR) regularized

32 Parallel Iterative Deconvolution affects the scaling of the result attempts to reduce artifacts from features near the boundary of the imaging volume. stops the iteration if the changes appear to be increasing. Increase the low pass filter size if this problem occurs. fraction of the largest Fourier coefficent of the PSF higher value increases convergence

33 Example Image courtesy: K Peng

34 PSF (Fiji -> Plugins -> Diffraction PSF 3D) Image 1 Image 2

35 Deconvolution(Fiji -> Plugins -> Parallel Iterative Deconvolution->3D Iterative Deconvolution)

36 Original Image 1 Image 3

37 Deconvolved Image 1 Image 3 Objects look brighter -> higher contrast Better separation between close objects

38 Deconvolved+ background subtracted Image 1 Image 3

39 Other resources/tools: A Matlab software Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy, Scientific Reports 2013.

40 Other resources/tools: Fiji Squassh segmentation / colocalization deconvolved (subpixel) segmentations in 2D&3D through prior knowledge of PSF Intensity within each object is homogeneous Bright foreground & dark background Noise model: Gaussian (wide field) or Poisson (confocal) (more robust) Joint deconvolution-segmentation procedure Rizk et al., Segmentation and quantification of subcellular structures in fluorescence microscopy images using Squassh, Nature Protocol, 2014.

41 Summary Deconvolution is a computational technique allowing to (partly) compensate for the image distortion created by an optical system Correct deconvolution should improve: attenuation of the out of focus light quantitative measurements the spatial resolution Incorrect deconvolution could: Introduce (more) artifacts -> reduce image quality It works best for thin (<50 um), optically transparent, fixed, bright samples. Challenging for live microscopy: short exposure (limit motion blur), objective adapted to medium (limit spherical aberrations).

42 References (to name a few ) Good reviews (overviews): 1. Waters, Accuracy and precision in quantitative fluorescence microscopy, JCB Parton et al., Lifting the fog: Image restoration by deconvolution, Cell biology Pawley, Chapter 25: Image enhancement by deconvolution, Handbook of biological confocal microscopy, McNally et al., Three-Dimensional Imaging by Deconvolution Microscopy, Methods 1999 Technical articles: 5. Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy, Scientific Reports Bertero et al., Image deconvolution, Proc. NATO A.S.I Thiébaut, Introduction to image reconstruction and inverse problems, Proc. NATO A.S.I On the web: 8. Olympus microscopy center (overview): 9. Textbook: