Pupil Phase Encoding for Multi-Aperture Imaging

Transcription

1 Pupil Phase Encoding for Multi-Aperture Imaging V. P. Pauca a, D. Chen c, J. van der Gracht d R. J. Plemmons a,c, S. Prasad g, T. C. Torgersen a, a Department of Computer Science, Wake Forest University, Winston-Salem, NC c Department of Mathematics, Wake Forest University, Winston-Salem, NC d Holospex, Inc., 6470 Freetown Rd., Suite , Columbia, MD g Center for Advanced Studies and Department of Physics and Astronomy, University of New Mexico, Albuquerque, NM ABSTRACT Digital super-resolution refers to computational techniques that exploit the generalized sampling theorem to extend image resolution beyond the pixel spacing of the detector, but not beyond the optical limit (Nyquist spatial frequency) of the lens. The approach to digital super-resolution taken by the PERIODIC multi-lenslet camera project is to solve a forward model which describes the effects of sub-pixel shifts, optical blur, and detector sampling as a product of matrix factors. The associated system matrix is often ill-conditioned, and convergence of iterative methods to solve for the high-resolution image may be slow. We investigate the use of pupil phase encoding in a multi-lenslet camera system as a means to physically precondition and regularize the computational super-resolution problem. This is an integrated optical-digital approach that has been previously demonstrated with cubic type and pseudo-random phase elements. Traditional multi-frame phase diversity for imaging through atmospheric turbulence uses a known smooth phase perturbation to help recover a time series of point spread functions corresponding to random phase errors. In the context of a multi-lenslet camera system, a known pseudo-random or cubic phase error may be used to help recover an array of unknown point spread functions corresponding to manufacturing and focus variations among the lenslets. 1. INTRODUCTION Digital super-resolution refers to the technique of extending the resolution of an image beyond that allowed by the detector pixel size or pixel pitch (but still below the limit provided by the lens aperture). This technique is most often related to the use of a set of low-resolution images of a subject obtained with a single lens over time or obtained through a multi-lens imaging system in a single exposure. The latter approach has recently been explored by a number of research efforts and notably by the thin observing module with bounded optics (TOMBO), 7 MONTAGE, 5 and PERIODIC 9 projects. These multi-lenslet camera systems enable the use of many types of observational diversities, including phase, polarization, neutral density, and wavelength diversities. Figure 1 illustrates an example 3 3 PERIODIC camera system. A lenslet array forms multiple images of a subject on a single detector. Under nearly all observing conditions, the multiple images focused on the detector will be sampled with slightly different alignment of the image with respect to the detector pixels. The ensemble of sub-pixel shifted low-resolution images encodes subject information at higher spatial frequencies than present in each image individually. 10 Numerical methods for digital super-resolution reconstruct a single image from the ensemble, combining the available high frequency information to produce a high-resolution or super-resolved image. Recent PERIODIC imager prototypes have been designed to allow post-processing multiple observations made with differing neutral density, IR, polarization, and cubic phase filters, for multi-modality imaging. 2, 3 Figure 2(a) shows a picture of the array imaging component. The lenslet array consists of 18 2mm sub-apertures containing eight clear lenses, three lenses at different focal planes, three neutral density, two polarized, and two spectral (650nm and near-ir) filters that are able to capture an information-rich image over a single 10 megabit detector plane. The pixel pitch is 7.9µm. The digital component (see Figure 2(b)) consists of a Matlab-based graphical user interface front-end and a computational back-end. The front-end follows a flexible object-oriented design that is self-reconfigurable with respect to the camera lenslet and detector specifications. The back-end consists of a set of numerical methods for efficiently solving sub-image registration, digital super-resolution reconstruction,

2 Figure 1. A visual overview of a PERIODIC imaging system. (a) PERIODIC array imaging prototype. (b) PERIODIC digital component front-end. Figure 2. Hardware and software components of a PERIODIC imager. and deconvolution problems. This GUI enables realtime imaging through any user-selected lenslet and near-real time super-resolution and dynamic range reconstruction using an appropriately selected subset of lenslets. In spite of these advancements in observational diversity, the full computational imaging system design space has not been fully explored. Recently, Ashok and Neifeld1 described a way to exploit the pupil phase to obtain additional diversity from a set of subpixel shifted measurements, in a process called PSF engineering. In this paper, we explore PSF engineering through the use of cubic phase masks to physically precondition the underlying computational problem, leading to improved super-resolution performance. Cubic and other types of polynomial phase masks have been traditionally used to extend a single channel imaging system s depth of field, without incurring on a loss of light influx, see e.g. Prasad et al.11 and references therein. The present work is based on the observation that polynomial phase masks can also increase the restorability of an out-of-focus measurement by boosting the singular values of the system matrix above the noise floor.8 This process is akin to numerical preconditioning methods employed in the solution of many imaging and other inverse problems, and constitutes

3 a form of physical preconditioning (through optical lenses) that takes place before the data is sampled. From a practical standpoint, a cubic phase mask also helps mitigate the effects of aberrations and focal length variations due to manufacturing errors, from one lenslet to the next. 11 The remainder of this paper is structured as follows. We introduce the forward image formation model for a multi-lens imager and the associated digital superresolution inverse problem in Section 2. The approach of using cubic phase masks in the design of extended optical PSFs to improve the resolution performance of multi-lens imaging systems is outlined in Section 3. Resolution performance results from this approach on simulation data are given in Section 4, followed by conclusions and final remarks The Image Formation Model 2. IMAGING SYSTEM MODEL We consider a multi-lens imaging system consisting of a maximum of l sub-apertures of similar optical characteristics. The continuous signal formed in the image plane, with coordinate x = (x 1, x 2 ), through each of these sub-apertures is given by ĝ j (x) = f c (u)h(x u + j u)du, (1) where f c represents the continuous object, j u is the subpixel shift due to the relative position of the jth subaperture, and h is the optical system s point spread function (PSF). We model the PSF in the image plane for the jth lenslet as 2 h j (x) = 1 λf 2π ıφj(r)+ı P (r)e λf xtr dr A, (2) where r = (r 1, r 2 ) is a coordinate in the two-dimensional pupil plane, P (r) is the pupil function equal to 1 inside the pupil and 0 outside for a clear pupil, φ j (r) is the pupil phase, λ is the wavelength of the illuminating light, f is the focal length, and A is the area of the pupil. For a diffraction-limited imaging system we have φ j (r) = 0. If the lenslet includes a cubic phase mask then, φ j (r) = α(r r 3 2), (3) where α is a parameter (in units of λ) that denotes the strength of the phase mask. Typical cubic phase masks for depth of field extension have α > 20. A detector discretizes ĝ j with a sampling rate inversely proportional to the the pixel pitch. We use a rectangle function as the sampling pixel function, resulting in a discrete signal given by ( x g j (i) = ĝ j (x ) it ) rect dx + η(i), (4) T where i denotes the ith pixel in the detector, T is the pixel pitch, and η is additive measurement noise. For the purpose of this paper, we neglect nonlinearities inherent in the quantization of amplitude into a finite number of discrete intensity values, determined by the number of bits available on each pixel. If ĝ j is a bandlimited signal ( ω B), where B is essentially determined by the optical subsystem, then to avoid aliasing the pixel pitch must satisfy T 1/2B. For a multi-aperture imaging system consisting of a maximum of l nonredundant linear channels we set T = l/2b, undersampling the signal at 1/l the Nyquist rate. For computer simulation purposes we discretize g j in Equation (4), leading to the following representation in terms of discrete quantities, g j = DS j H j f + η j, (5) where, f is a target representation for the object, stacked as an n 2 1 vector of high-resolution pixel values, H j is an n 2 n 2 matrix blurring operator associated with the optical PSF h j (x) of the jth lenslet,

4 S j is an n 2 n 2 matrix representing the translation of the imaging system due to the relative position of lenslet j with respect to the object and the detector, D is an (n 2 /l) n 2 decimation matrix operator associated with the detector PSF, transforming the target from n n high resolution to the (n/ l) (n/ l) (low) resolution of the detector, g j is the low-resolution discrete image measurements associated with lenslet j stacked as an (n 2 /l) 1 vector and, η j describes a noise process associated with the j th image. Assuming k l lenslets are available for digital super-resolution, the image measurements from Equation (5) can be arranged together to describe an overall imaging equation of the form, where, g = Kf + η, (6) K = (I k D) S 1 H 1.. S k H k (7) is a matrix of size k(n 2 /l) n 2. Equation (6) leads to a least squares inverse problem whose solution is the desired high resolution reconstruction f. For simulation purposes, we also adopt a widely-used noise model 6 given by: η j = g j η j1 + ση j2 (8) where g j = DS j Hf. Both η j1 and η j2 are standard normally distributed random variables with mean equal to zero and variance equal to one. The parameter σ is the standard deviation of the image-independent noise term. The normally distributed term g j η j1 in equation (8) approximates the Poisson noise associated with the light detection process The Inverse Problem The numerical solution of the inverse problem associated with equation (6) is attracting a good deal of attention in the image processing community. Most approaches are based on the minimization of a least-squares cost functional of the form: J(f) = g Kf constraints, (9) where constraints are used to enforced desired characteristics in the computed solution, such as smoothness, sparsity, etc. We note that (9) is a non-trivial computational problem as K is not only ill-conditioned but in many cases it is also singular when k < l or when at least two channels contain a redundant amount of subpixel shift. In practice, we find often that k < l and the system is under-determined. For values of k reasonably close to l, acceptable reconstructions can still be obtained. Smaller values of k also reduce the computational cost of the reconstruction. In addition, full knowledge of matrix K assumes determination of the subpixel shifts (for S j ), and the lenslet optical PSFs (for H j ), which in practice are not always accurately known. These and other practical issues are dealt within our software component of the PERIODIC imager. Figures 3 and 4 show some digital super-resolution reconstructions from real data obtained with the PERIODIC prototype shown in Figure 2. Figure 3(b) shows the digital super-resolution reconstruction of a USAF target using a set of eight low-resolution sub-images. A typical low-resolution sub-image is shown in Figure 3(a). Figure 4(b) shows marked improvement of facial features over the low-resolution imagery in Figure 4(a).

5 (a) Low-resolution subimage with linear interpolation. (b) Digital super-resolution reconstruction from a set of eight sub-images. Figure 3. Digital super-resolution reconstruction of an USAF target. (a) Low-resolution subimage with linear interpolation. (b) Digital super-resolution reconstruction from a set of eight sub-images. Figure 4. Digital super-resolution reconstruction of facial features. 3. PSF ENGINEERING USING CUBIC PHASE MASKS In a recent paper, Ashok and Neifeld 1 showed that for imagers with pixel-limited resolution an impulse-like optical PSF maybe suboptimal as a mechanism for encoding object information while tolerating distortions introduced by the detector array. In such systems, relatively large photodetectors may image two point sources separated by a subpixel amount onto the same pixel, producing blurring or aliasing in the measured data as a result of undersampling. Their proposal employs pseudo-random phase masks placed in the aperture stop of each lenslet in a conventional multi-lens imager to extend the optical PSF over a pixel amount, enabling the extraction of subpixel position information from the sampled detector outputs. Here, we explore the use of cubic phase masks in the design of extended optical PSFs for use within a multilens imaging system with pixel-limited resolution. Our approach is motivated by the observation that under certain conditions carefully designed cubic phase masks can boost the singular values of an imaging operator above the noise floor. Figure 5 illustrates this assertion where cubic phase masks are shown to improve the information content when an imager is affected by defocus blur. For the present work we expect that, like pseudo-random phase masks, 1 cubic-type phase masks may increase or precondition the information content

6 10 0! = 0, " = ! = 20, " = 10! = 60, " = 10! = 0, " = r " (!) singular values x 10 5 (a) Plot of singular values of blurring operator H, showing how a cubic phase mask (α = 20 or 60) can boost singular values above a 10 3 noise floor in the presence of defocus blur (α = 0, τ = 10) cubic parameter! (b) Sum of singular values of a blurring operator H as a function of defocus and cubic parameter α (x-axis), showing that optimal information content can be obtained with a cubic phase mask given a fixed defocus amount. Figure 5. Effect of cubic phase masks on the singular value spectrum of blurring matrix operators. contained within the singular values of the system matrix K in equation (6), leading to increased digital superresolution reconstruction performance. For the purpose of our simulation study, we consider only 1D objects and image measurements. 1 The base imaging system under consideration is modeled after recent PERIODIC prototypes as having an array of lenslets of F/# = 3.5 and back focal length of 5 mm. The detector is modeled as having pixels of 7.5 µm in size, with a full-well capacity of 45,000 electrons and 100% fill factor. The distance from the imager to the plane of best focus is 1m, giving an airy disk of 4.7 µm or spatial resolution of 2.3 µm. We assume a spectral bandwidth centered at λ = 550 nm. The undersampling factor for this system is l = Rayleigh Resolution Performance 4. SIMULATION RESULTS Our simulation experiment is set up as follows. We use equations (1-6) to construct low resolution imagery for both a conventional PERIODIC imager and a multi-lens imager containing identical cubic phase masks in all lenslets. The base system for both imagers is specified in the previous section. We use both image-dependent (shot noise) and image-independent (readout noise) noise terms (see equation (8)), with σ = 0.01 max{ g j }. The target high-resolution image vector f is of size With an undersampling factor of l = 8, the low-resolution images g j are of size The subpixel shift for each sub-image is chosen from a uniformly random distribution. We adopt the Rayleigh s criterion to measure the performance of the conventional PERIODIC imager and that of the multi-lens imager with a cubic phase mask in the aperture stop. We quantify their trade-off between resolution and signal-to-noise ratio (SNR) for varying cubic phase mask strength α. To estimate the Rayleigh resolution we use Ashok and Neifeld s 1 method and image a point-source object to obtain 1 k l lowresolution measurements of the form (4). We then apply the conjugate gradient least squares method (CGLS) to the aggregate (6), with fixed number of iterations. Note that here we assume full knowledge of the system optical PSF, which may be obtained in a laboratory setting. The reconstructed point-source is then used to find the overall PSF of the imager. A diffraction-limited sinc 2 PSF is fitted in the least squares sense to the overall PSF and the Rayleigh spatial resolution (distance from the origin to the first zero-crossing) is computed from it.

7 The Rayleigh resolution metric is computed and averaged over 30 randomly chosen subpixel shift sets for each value of k. Figure 6 shows a sample computed overall PSF and its least squares fitted sinc 2 PSF computed overall PSF sinc 2 LS fit spatial dimension (µm) Figure 6. Sample computed overall PSF (dashed) and least squares fitted sinc 2 PSF. The Rayleigh spatial resolution is the distance from the origin to the first zero-crossing of the sinc 2 PSF. A large number of cubic phase enhanced multi-lens imagers were simulated and their performance tested using the Rayleigh spatial resolution criterion. It was quickly evident that cubic phase masks with relative large steep surfaces (large α value) extended the PSF far too broadly, spreading much of the energy over too many pixels. In these situations, noise makes it nearly impossible to extract useful subpixel shift information and the overall performance was above that of a conventional PERIODIC imager. Thus, we see no clear opportunity for simultaneously improving resolution performance while extending the depth of field. However, resolution performance was significantly improved with respect to the conventional PERIODIC imager for small α values. Figure 7 shows the resolution performance of a cubic phase mask enhanced imager, for an optimal value of α = 1.6, versus that of the conventional PERIODIC imager. This plot shows that using only two frames of a cubic phase mask enhanced imager can produce imagery of similar quality to that of a conventional PERIODIC imager of five frames. Moreover, the smallest resolution achievable with the cubic phase mask enhanced imager is finer, by roughly 0.5µm, than that of the conventional imager. However, at k = 1 frames the conventional system outperforms the cubic phase mask enhanced imager. Figure 8 shows the extended cubic PSF corresponding to the resolution performance plot of Figure 7, zoomed over its center region Overall PSF Reconstruction from Low-Resolution PSFs In this section we study the effectiveness of numerically estimating the overall optical PSF of a multi-lens imager containing relatively large photodetectors and hence significant undersampling factor. The overall optical PSF may be accurately estimated in a lab setting, however its reconstruction from low-resolution realizations is of practical interest. We assume an array of lenslets with identical cubic phase mask on the aperture stop. In this situation, the overall imaging equation becomes, g = (I k D) S 1. S k Hf + η. (10) We suspect this to be a result of using CGLS with a constant, instead of optimal, number of iterations.

8 9 8 Conventional imager Cubic phase enhanced imager 7 spatial resolution (µm) number of frames (k) Figure 7. Rayleigh spatial resolution versus number of frames for the conventional PERIODIC imager and multi-lens imager with a cubic phase mask in the aperture stop Diffraction limited PSF Extended Cubic PSF amplitude spatial dimension (µm) Figure 8. Diffraction-limited versus extended cubic PSF. Assuming f is a point source then h = Hf is the overall PSF. We thus solve a least squares problem of the form, J(h) = g (I k D) S 1. S k h constraints. (11) We use Chung et al. 4 weighted generalized cross-validation method with Lanczos hybrid regularization (HyBR) method to optimally minimize J(h). Figure 9 shows a surface plot of the relative error with respect to the true optical PSF obtained using HyBR, as a function of number of frames k and cubic parameter α. The surface on the top shows is for multi-lens systems with undersampling factor l = 11, corresponding to lenses of F/# = 2.5. The smaller surface below is for multi-lens systems with undersampling factor of l = 5. As expected, the relative error decreases with the number of frames. Also notice that a lower undersampling factor leads to increased accuracy in the determination of the optical PSF. Figure 10 shows a reconstructed PSF for α = 20, and using one

9 relative error number of frames (k) 0 cubic parameter (α) Figure 9. Surface plot of relative error in the determination of overall optical PSF from undersampled realizations. frame (left) and five frames (right). Notice that clearly high resolution features are better reconstructed as the number of frames k is increased. We note that reconstruction of the cubic PSF is challenging due to its smooth and rough features, which cannot easily be recovered simultaneously using standard regularization approaches. 0.1 diffraction limited cubic PSF reconstructed PSF k=1 0.1 diffraction limited cubic PSF reconstructed PSF k= spatial dimension (µm) (a) One-frame PSF reconstruction spatial dimension (µm) (b) Five-frame PSF reconstruction. Figure 10. Reconstruction of optical PSF for a cubic phase mask of α = CONCLUSIONS AND FUTURE WORK We have demonstrated the viability of cubic phase masks for improving the resolution performance of multi-lens imaging systems. Like pseudo-random phase masks, a weak cubic can produce extended PSFs that facilitate the

10 extraction of subpixel shift information from the sample detector outputs. Such a cubic phase mask can boost the singular values of the overall system matrix above the noise floor, improving restorability of the signal. This process constitutes a form of physical preconditioning that takes place before the signal is discretized by the detector. While the cubic is not able to improve resolution performance to the extent achieved with pseudorandom phase masks, 1 in practice it can additionally help mitigate aberrations and focal length variations resulting from imperfections in the manufacturing and assembly process. Other candidate phase masks may be used for improving the resolution performance of multi-lens imaging systems, beyond that achieved by the cubic and pseudo-random phase masks. The design space is large and a method for efficiently determining potential candidates is needed. Information theoretic approaches may be useful in this regard. 6. ACKNOWLEDGMENTS The research described in this paper was supported in part by grants from the Intelligence Advanced Research Projects Agency (IARPA) through the Defense Microelectronics Activity (DMEA) under cooperative agreement number H , the Army Research Office (ARO), with award numbers W911NF and DAAD , and the Air Force Office of Scientific Research (AFOSR), with award number FA Their kind support is sincerely appreciated. The authors also wish to thank our PERIODIC project collaborators Ryan Barnard, Scott Mathews, Mark Mirotznik, Jim Nagy, and Peter Zhang. REFERENCES 1. Ashok, A., and Neifeld, M. Pseudorandom phase masks for superresolution imaging from subpixel shifting. Applied Optics 46, 12 (2007), Barnard, R., Gray, B., Pauca, V., Torgersen, T., Mirotznik, M., van der Gracht, J., Plemmons, R., Behrmann, G., Mathews, S., and Prasad, S. PERIODIC: State-of-the-art array imaging technology. In Proc ACM Southeast Conference (Winston-Salem, NC, March 2007), ACM, pp Barnard, R., Pauca, V., Torgersen, T., Plemmons, R., Prasad, S., van der Gracht, J., Nagy, J., Chung, J., Behrmann, G., Mathews, S., and Mirotznik, M. High-resolution iris image reconstruction from low- resolution imagery. In Proc. SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations XVI (San Diego, CA, Aug. 2006), vol. 6313, SPIE, pp Chung, J., Nagy, J., and O Leary, D. P. A weighted GCV method for Lanczos hybrid regularization. Elec. Trans. Numer. Anal. 28 (2008), DISP. The Duke Imaging and Spectroscopy Program Jain, A. K. Fundamentals of digital image proccessing. Information and System Sciences. Prentice-Hall International, Kitamura, Y., Shogenji, R., Yamada, K., Miyatake, S., Miyamoto, M., Morimoto, T., Masaki, Y., Kondou, N., Miyazaki, D., Tanida, J., and Ichioka, Y. Reconstruction of a high-resolution image on a compound-eye image-capturing system. Applied Optics 43, 8 (2004), Pauca, V., Plemmons, R., Prasad, S., Torgersen, T., and van der Gracht, J. Integrated opticaldigital approaches for enhancing image restoration and focus invariance. In Proc. SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations XIII (San Diego, CA, Aug. 2003), vol. 5205, SPIE, pp PERIODIC. The Practical Enhanced-Resolution Integrated Optical-Digital Imaging Camera (PERIODIC) Project Prasad, S. Digital superresolution and the generalized sampling theorem. J. Opt. Soc. Am. A 24 (2007), Prasad, S., Pauca, V., Plemmons, R., Torgersen, T., and van der Gracht, J. Pupil-phase optimization for extended-focus aberration-corrected imaging systems. In Proc. SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations XIV (Denver, CO, Aug. 2004), vol. 5559, SPIE, pp