Ultra-fast Lensless Computational Imaging through 5D Frequency Analysis of Time-resolved Light Transport

Transcription

1 Ultra-fast Lensless Computational Imaging through 5D Frequeny Analysis of Time-resolved Light Transport The MIT Faulty has made this artile openly available. Please share how this aess benefits you. Your story matters. Citation As Published Publisher Wu, Di, Gordon Wetzstein, Christopher Barsi, Thomas Willwaher, Qionghai Dai, and Ramesh Raskar. Ultra-Fast Lensless Computational Imaging through 5D Frequeny Analysis of Time-Resolved Light Transport. International Journal of Computer Vision 110, no. 2 (Deember 28, 2013): Springer US Version Author's final manusript Aessed Wed Ot 17 19:39:14 EDT 2018 Citable Link Terms of Use Creative Commons Attribution-Nonommerial-Share Alike Detailed Terms

2 Int J Comput Vis (2014) 110: DOI /s Ultra-fast Lensless Computational Imaging through 5D Frequeny Analysis of Time-resolved Light Transport Di Wu Gordon Wetzstein Christopher Barsi Thomas Willwaher Qionghai Dai Ramesh Raskar Reeived: 26 February 2013 / Aepted: 30 November 2013 / Published online: 28 Deember 2013 Springer Siene+Business Media New York 2013 Abstrat Light transport has been analyzed extensively, in both the primal domain and the frequeny domain. Frequeny analyses often provide intuition regarding effets introdued by light propagation and interation with optial elements; suh analyses enourage optimal designs of omputational ameras that effiiently apture tailored visual information. However, previous analyses have relied on instantaneous propagation of light, so that the measurement of the time dynamis of light sene interation, and any resulting information transfer, is preluded. In this paper, we relax the ommon assumption that the speed of light is infinite. We analyze free spae light propagation in the frequeny domain onsidering spatial, temporal, and angular light variation. Using this analysis, we derive analyti expressions for information transfer between these dimensions and show how this transfer an be exploited for designing a new lensless imaging system. With our frequeny analysis, we also derive performane bounds for the proposed omputational amera arhiteture Eletroni supplementary material The online version of this artile (doi: /s ) ontains supplementary material, whih is available to authorized users. D. Wu G. Wetzstein (B) C. Barsi R. Raskar MIT Media Lab, Cambridge, MA, USA gordonw@media.mit.edu D. Wu Q. Dai Department of Automation, Tsinghua University, Beijing, China T. Willwaher Department of Mathematis, Harvard University, Cambridge, MA, USA D. Wu Graduate Shool at Shenzhen, Tsinghua University, Beijing, China and provide a mathematial framework that will also be useful for future ultra-fast omputational imaging systems. Keywords Computational photography Light transport Frequeny analysis Lensless imaging 1 Introdution With the invention of the eletroni strobosope, Harold Do Edgerton aptured the first photograph of a bullet piering an apple about half a entury ago. Today, ultra-fast sensors (e.g., Hamamatsu 2012) an produe visual information at effetively one trillion frames per seond, roughly one million times faster than Edgerton s eletroni strobes. This tehnology allows for the apture of light as it propagates through spae and failitates a wide variety of appliations in omputer vision. Indeed, the ommon assumption that the speed of light is infinite beomes depreated. Imaging systems with suh high time resolution provide a new degree of freedom available to omputer vision appliations, namely, time-resolved ultra-fast imaging. Whereas ultra-fast imaging tehniques, suh as time-offlight apture or light detetion and ranging (LiDAR), have been suessful in measuring the travel time of short light pulses for appliations in depth estimation, measuring the full time profile of sensor pixels over an extended period of time has started only reently to be explored in the omputer vision ommunity. The key insight of this paper is to treat light propagation among emitters, the sene, and reeivers in five dimensions, with spae, angle, and time all oupled. This leads to joint re-examination of fundamental inverse problems and solutions in omputational imaging. Our paper explores ultrafast omputational imaging in four aspets: theory, apture, analysis, and appliations.

3 Int J Comput Vis (2014) 110: Fig. 1 Overview of operators for forward and inverse light field transport. While onventional analysis (e.g., Durand et al. 2005) onsider only time-independent transport (left olumn), we analyze timeresolved forward and inverse light transport in this paper (enter In partiular, we analyze light propagation in free spae in both the primal and frequeny domains, onsidering spae, angle, and time for a finite speed of light. We disuss rossdimensional information transfer and derive upper bounds for ultra-fast streak sensors, whih reord only subsets of the high-dimensional spae. Failitated by this analysis, we propose a new omputational amera arhiteture that requires no optial elements besides a time-resolved sensor and a pulsed light soure. Relevant sene information is reovered via omputed tomographi reonstrution in postproessing. Summarized in Fig. 1, our ontributions are as follows: We analyze free spae light propagation in the frequeny domain onsidering spatial, temporal, and angular light variation. We show that the propagation in frequeny spae an be modeled analytially as a ombination of a shear in the light field and a onvolution along the angular frequenies. We show that propagation in free spae has unique information-preserving properties by oupling information among different dimensions. We derive upper bounds for how muh of the information ontained in one dimension is transferred to those that are measured. Based on this analysis, we introdue a novel, lensless amera. This approah exploits ultra-fast imaging ombined with iterative reonstrution, while removing the need for optial elements, suh as lenses or masks. Using the proposed frequeny analysis, we derive upper bounds on the depth of field of the proposed amera and verify them with simulations. With syntheti senes and an experimental prototype amera, we demonstrate and evaluate the proposed omputational imaging approah. 1.1 Overview This work is organized as follows. In Set. 2, wereview relevant literature. Setion 3 introdues a detailed analysis and right olumns) and show that the operations required to inversepropagate a light field in spae, time, and angle orrespond to omputed tomographi reonstrutions. Based on these insights, a new lensless omputational imaging system is proposed of transient light propagation in free spae in both primal and frequeny domains. This analysis is used in Set. 4 to derive upper bounds for the amount of spatial image information that is preserved after propagation. We introdue a new approah to lensless omputational imaging in Set. 5. The proposed tehnique aptures spatial and temporal image information using a bare sensor and uses omputed tomography for reonstrution. We evaluate the performane of the CT-based reonstrution algorithm with resolution harts in Set. 6 and with simulated and physial experiments in Set. 7. Setion 8 disusses features of the tehnique and suggests future work. 2 Related Work Frequeny analyses of light transport (Durand et al. 2005) have been of ritial importane in omputer vision and graphis for evaluating the performane of imaging systems and identifying novel omputational amera designs (Edward et al. 1995; Levin 2009; Veeraraghavan et al. 2007) or implementing faster (Ng 2005) and better (Levin and Durand 2010) omputational proessing shemes. Time-resolved forward and inverse light transport, however, has begun to be studied only reently (Raskar and Davis 2008). Lensless imagers have been studied and inlude designs that employ attenuating layers with ontrol of transmittane in spae and time (Zomet and Nayar 2006), template-based approahes using miro sensors (Koppal et al. 2011), Fresnel zone plates or photon sieves for imaging using diffrative optis (Andersen 2005), angle-sensitive pixels that require no off-hip optis (Gill et al. 2011), and dynami LED illumination ombined with omputed tomographi image reonstrution (Isikman et al. 2011). We introdue a new omputational amera design that requires only a single image aptured by a time-resolved ultra-fast sensor and does not require additional refrative, attenuating, or diffrative optial elements. Ultra-fast sensors with the ability to measure temporal impulse responses provide a rih soure of imaging

4 130 Int J Comput Vis (2014) 110: information in seismology, sonology, and optis. Refletion seismology and medial ultrasound use suh measurements for performing non-invasive volumetri imaging, whereas sonar, radar, and LiDAR tehnologies determine an objet s range, among other properties, from the ehos of sound, radio waves, and light pulses refleting from a target objet (Saleh 2011). Unlike LiDAR, whih measures the time of arrival of diret refletions of light, the proposed omputational amera employs an ultra-fast sensor that exploits the full timeresolved profile, measuring photons as they arrive at the sensor over an extended period of time, for imaging both albedo and depth. Imaging at speeds fast enough to observe radiometri hanges over time has reently unveiled new appliations for sene analysis that annot be performed with standard ameras. These ultra-fast imaging methods have the ability to reonstrut geometry (though not its albedo) of oluded objets around a orner (Velten et al. 2012), aquire in-thewild BRDFs without using enirling equipment for senes with known geometry (Naik et al. 2011), reover an oluded target s motion using multipath analysis (Pandharkar et al. 2011), and deompose global light transport (Wu et al. 2012). In this paper, we present a 5D frequeny analysis of timeresolved light transport in free spae that produes an intuitive explanation of forward light transport within all of the above works. We hope to provide the fundamental mathematial tools for developing and understanding emerging ultra-fast omputational amera designs and new appliations enabled by this tehnology. 3 Frequeny Analysis of Transient Light Transport In this setion, we analyze time-resolved light transport in free spae. While an analysis of steady-state light transport is intuitive in revealing where information is loalized in the frequeny domain (Durand et al. 2005; Levin 2009), onsidering the time-resolved ase, to the best knowledge of the authors, has not been disussed in the literature. We analyze first the flatland, three-dimensional ase and then generalize the analysis to five dimensions D Frequeny Analysis Starting from first priniples, spae angle light transport is ommonly denoted as (Isaksen et al. 2000; Chai et al. 2000) l (S) (x,ν)= l (S) 0 (x νd,ν), (1) where l (S) (x,ν)is the steady-state light field after transport by a distane d, x is the spatial variable, and ν is the relative oordinate on a plane at unit distane (Chai et al. 2000) (Fig. 2, left). Inluding the speed of light, we reognize that different ray paths arrive at different times. In order to au- Fig. 2 Ray diagram illustrating the two-plane parameterization of light fields. Rays arrive at a plane a distane d from the soure at times that depend on their propagation angle. The left is objet spae, and the right (shaded blue) is the orresponding spae time image (Color figure online) rately model suh systems, the time-resolved forward light transport must onsider the temporal dimension t, inorporating the finite speed of light as l(x,ν,t) = l 0 (x νd,ν,t d 1 + ν 2 ). (2) Physially, rays propagating at large angles arrive later than low-angled ones. The delay an be alulated geometrially (Fig. 2). Note that for a pulsed point soure, the light field l 0 (x,ν,t) = δ(x)δ(t) beomes l(x,ν,t) = δ(x νd)δ(t d 1 + ν 2 ) δ( d x ν)δ(t d 1 + ( d x )2 ) after a propagation distane d. From the seond δ-fator, we see that time-resolved propagation produes a hyperboli spae time urve (Fig. 2, right). The urvature and vertex of the hyperbola vary with the origin of the pulse. For example, a 3 mm hange in path length results in a time shift of 10 ps. A summary of spae time propagation is shown in Fig. 3. In the Fourier domain, propagation in free spae for the steady-state ase is a shear along the angular frequenies (Durand et al. 2005): l (S) ( f x, f ν ) = l (S) 0 ( f x, f ν + f x d), (3) where denotes the Fourier transform of a quantity and f x and f ν are the frequeny variables of spae and angle, respetively. For this ase, the spae angle shear in the primal domain (Eq. 1) intuitively translates to a shear along the opposite dimension in the frequeny domain. As illustrated in Fig. 4, the time-resolved light field spetrum remains sheared in the f x - f ν dimension, but is now blurred in the f t dimension, mainly along two branhes that form an -shape. To quantify this result, we alulate the Fourier transform of the time-resolved light field after propagation in free spae (both Eqs. 4 and 5 are derived in the Supplement): l( f x, f ν, f t ) = e 2iπ f x x 2iπ f ν ν 2iπ f t t ( l 0 x νd,ν,t d ) 1 + ν 2 dx dν dt = l 0 ( f x, f ν + f x d, f t ) F ft d ( f ν ), (4) fν

5 Int J Comput Vis (2014) 110: Fig. 3 Without time resolution, free spae propagation shears the light field in both the primal and frequeny domains (left). Due to relative time delays of different rays, a pulsed point soure produes a spae time hyperbola, whih ats as a onvolution kernel on time-resolved light fields (enter). In frequeny spae, this hyperboli onvolution is transformed to multipliation with its spetrum. Importantly, without a time-resolved soure, only the DC omponent is propagated (right). Please note that the spae angle shear is maintained, though not shown, in the time-resolved ase Fig. 4 Visualization of light field propagation. Top row: primal domain propagation of light pulse (l 0 (x,νt) = δ(x)δ(t)). As the propagation distane d inreases, the x-t hyperbola inreases in urvature, and the x- ν shear beomes steeper. Bottom row: orresponding frequeny domain representations. The steady state ( f t = 0) ross setion is the standard Fouriershear, whih splitsintoan -shape off-plane. (Left): ross setion of spetrum at various off-axis positions. The time range of this figure is 0 50 ps where F κ ( f ν ) := dνe 2iπ f νν 2iπκ 1+ν 2 πκ H (2) = κ 2 fν 2 1 (2π κ 2 fν 2) for κ2 > fν 2 2iκ K f 2 1(2π f 2 ν κ 2 ν κ2 ) for κ 2 < fν 2. (5) Here, H (2) 1 is a Hankel funtion of the seond kind, K 1 is a modified Bessel funtion of seond kind, and κ = f t d/. Cross setions of this funtion are plotted in Fig. 4. Equation 4 mathematially expresses the effets of propagation on the light field spetrum through (1) shearing the spae angle dimension, then (2) onvolving the result with F ft d ( f ν ) along the f ν dimension. Defining the bandwidth of F κ ( f ν ) as the separation distane between the two delta funtions, we see that the spetral width is 2κ and therefore inreases linearly with the temporal frequeny and the propagation distane. At a time resolution of 10 ps and a total propagation distane of 3 mm, for example, the angular bandwidth is 2rad 1. A non-dimensional value of κ = 2 orresponds to f t d = 6 mm/ps. Further, note that in partiular that as κ 0, the kernel approahes the delta funtion for all f ν, so that the final propagated light field approahes the time-independent ase. This simplifiation ours for either d or f t vanishing (orresponding to no propagation or onsidering only the steady-

6 132 Int J Comput Vis (2014) 110: l(r, ν, t) = l 0 (r ν d, ν, t d 1 + ν 2 ), (6) where ν = (ν x,ν y ), r = (x, y), and ν 2 = νx 2 + ν2 y. Here, we will use the same notation, l and l, for the light field and its spetrum as we did in the 3D ase, distinguishing them by their arguments. The 5D Fourier transform is alulated in the same way (see the Supplement). l(f x, f ν, f t ) =... dr dν dte 2πi(f x r +f ν r + f t t) l 0 (r ν d, ν, t d 1 + ν 2 ) = l 0 (f x, f ν + f x d, f t ) ( f ν ), (7) F (2) ft d f νx, f νy Fig. 5 Three-dimensional visualization of the frequeny propagation kernel F κ (also see Fig. 4, lower left). This kernel models the onvolution along the angular light field dimension indued by propagation of a timeresolved light field in free spae. The kernel depends on the propagation distane, the speed of light, and the temporal frequeny, as modeled by the parameter κ = f t d/. The time range of this figure is 0-50ps state omponent f t = 0, respetively), or for the speed of light going to infinity (orresponding to neglet of timeresolution). With any of these assumptions, the system (Eq. 4) redues to the ommonly used steady-state Fourier propagator (Eq. 3) (derived in the Supplement). The onvolution operation for the time-resolved ase, however, generally blurs information in the onvolved dimension. We analyze this behavior in more detail in Set. 4 and demonstrate in Set. 5 that this effet an be diretly exploited to design a novel, lensless omputational imaging system. Here, however, the kernel profile depends on not only the onvolved dimension, but also the parameter f t d/. As shown from Eq. 5 and Fig. 5, there are two regimes to onsider. Whereas the deay of F κ is exponential for κ < f ν (so that information loated there annot be deteted to first order), the falloff is muh slower for κ > f ν (where information an be deteted). At the boundary, κ = f ν, and F κ beomes singular. We all this boundary the light one 1 and use it to distinguish between these two regimes. Light propagation of a point light soure in the primal and frequeny domain is illustrated in Fig D Frequeny Analysis Consider now a fully five-dimensional light field, with lateral variables x and y, respetive angular variables ν x and ν y, and time t. Free-spae propagation is generalized straightforwardly: 1 The onept of light ones is ommonly used in spae time physis; see e.g. html. where the onvolution is two-dimensional, over both angular frequeny dimensions, with f ν = ( f νx, f νy ). Here, F (2) ft d ( f ν ) is defined as (derived in the Supplement) F (2) κ ( f ν ) (2πi + 1 ) κe 2πi κ 2 f ν 2 = 1 (2π + ) iκe 2πi fν 2 κ 2 κ 2 fν 2 2π(κ 2 f ν 2 ) fν 2 κ 2 2π( f ν 2 κ 2 ),for κ 2 > f ν 2,for κ 2 < f ν 2. (8) Note here that F κ (2) ( f ν ) is axially symmetri; however, the onvolution in Eq. 7 annot be redued to a single, axial integral for a general light field unless it is radially symmetri. Further, the 5D kernel is struturally similar to the 3D version: two regions, one deaying exponentially and one algebraially, separated by a singularity along κ 2 = f ν 2. However, beause the 5D funtion is axially symmetri, this singularity is a ring of radius κ in the ( f νx, f νy ) plane. Thus, time-resolved imaging an be plaed naturally in the frequeny analysis of light transport. Similar to steady state analysis (Durand et al. 2005), transient frequeny analysis an be built upon the results here so as to inlude, e.g. diffuse refletions and shading, olusions and interations with BRDFs. 4 Cross-dimensional Information Transfer and Spae Time Angle Bandwidth Analysis The Fourier analysis in the previous setion shows that there is a transfer of information between the different dimensions; spae, time, and angle are all oupled. In this setion, we explore the details of this proess. Beause urrent ultrafast ameras in the pioseond range have only one spatial dimension, we fous on the 3D subspae of the 5D kernel. We exploit this information transfer by proposing an ultrafast lensless sensor that aptures only spae and time. The

7 Int J Comput Vis (2014) 110: Fig. 6 Time-resolved bandwidth analysis of a Lambertian sene of single spatial frequeny f 0. Top: planar impulse illumination. Immediately after interation, the spetrum opies the inident spetrum at ± f 0. After propagation, the signal is attenuated along the f x diretion, but is preserved in time. The green line indiates the spae time bandwidth of the sensor, and the pink line represents the propagation MTF light one (slopes are ±). Here, spatial resolution must be greater than f 0 for detetion. Bottom row: point pulse illumination. After propagation, signifiant information is loated along the f t diretion, so that spatial bandwidth an be relaxed, for instane information an be deteted even if the spatial sensor bandwidth fx max is low. (Please note that we use the side view of the osine patten just to show a perspetive view of the sene plane, and looks like the light is going through it for better understanding of the sene.) reorded data, alled a streak image s(x, t), ontains angular information that an be used to infer image and depth of the sene. Suh streak sensors are urrently ommerially available (e.g., Hamamatsu 2012). We demonstrate and evaluate the proposed omputational amera arhiteture in Set. 5 and derive its limits in Set. 6. We also analyze the bandwidth and reonstrution limits of any streak imaging system in this setion. In partiular, we write s(x, t) (Fig. 2, right) as: s(x, t) = l(x,ν,t) dν = l 0 (x νd,ν,t d 1 + ν 2 ) dν. (9) Equivalently, the streak spetrum is found by setting the DC omponent ( f ν ) to zero. To simplify the analysis, we onsider a Lambertian sene, so that the light field beomes l 0 (x,ν,t) = l0 L (x, t), i.e. it is angle-independent. Using Eq. 4, we an write the measured streak image as (derived in the supplement): s( f x, f t ) = l 0 L ( f x, f t )F ft d ( f x d). (10) Physially, the final spetrum is the original light field spetrum multiplied by a spae time-dependent weighting fator, whose harateristi width is determined by the propagation distane d. Thus, the kernel F κ ats as an optial transfer funtion (OTF). This equation allows us to derive an upper bound on the spatial frequenies f x of a Lambertian plane at distane d that an be reovered from a streak image s(x, t). As an illustration of Eq. 10, we see in Fig. 6 (bottom) the initial transient point light soure l 0 L( f x, f t ) propagating a distane d s in the frequeny domain under Lambertian assumptions. The effets of different time sales ould be understood simply by varying window height in the f t axis. To gain some intuition, onsider the simplest ase of a osine albedo (of spatial frequeny f 0 ) that is illuminated by a planar impulse: l0 L(x, t) = os(2π f 0x)δ(t), asshown in Fig. 6. (Experimentally, this setup ould be effeted by illuminating the objet with a pulsed point soure whose origin is far from the objet so that the objet sees an approximately planar wavefront.) Note that here, and in the theory below, we assume the sene to be planar, but this is not a neessary ondition and an be relaxed. Immediately after the mask, the Fourier transform is a pair of vertial delta funtions, entered at ± f 0. After propagation, the spetrum is multiplied by the OTF above. For f x > f t, F ft d ( f x d) deays exponentially, so that, to first order, information in these regions is lost. For f x < f t, the information is preserved. This behavior is shown in Fig. 6 (top right). In order to detet the signal, its spae time bandwidth must lie within that of the amera. Beause there is no rossdimensional information transfer, i.e. spatial frequenies are

8 134 Int J Comput Vis (2014) 110: not sheared into time, the resolution riteria is the same as for onventional imaging: the detetor s spatial bandwidth fx max must exeed the signal frequeny: f 0 < fx max, regardless of the time bandwidth. In ontrast, onsider the same sene illuminated by a spherial wavefront pulse. For simpliity, we assume the point soure is loated on the z-axis a distane d s from the sene (Fig. 6). The initial light field beomes l0 L (x, t) = os(2π f 0 x)δ(t (d s /) 1 + (x/d s ) 2 ). Now, the light field s Fourier transform is the onvolution of the transforms of the fators. After propagating a distane d r, the resulting streak spetrum (derived in the supplement) is s( f x, f t ) = F ft Z (Z( f x + d s f 0 /Z)) +F ft Z (Z( f x d s f 0 /Z)), (11) where Z d s + d r. When plotted in the f x - f t plane (see Fig. 6, bottom), we see a strong signal in the time domain (i.e. along the vertial axis), as we would expet from a spherial pulse, whih illuminates different sene points at different times. In this ase, the sene frequeny is reovered if ft max is large enough (roughly ft max > d s Z f 0), regardless of the spatial bandwidth. Thus, a suffiiently fast amera an superresolve objet features. Note that in the limiting ase of ft max 0, the detetor no longer detets time-resolution. In this ase, one will measure a signal only if its spae bandwidth is greater than the signal s bandwidth and if the propagation distane is suffiiently small so as to have the signal fall within Z 1 of the propagator singularity. For ft max 0, the argument of the propagator in Eq. 11 is Z( f x d s f 0 /Z), so that spatial frequenies beyond (1 + f 0 d s )/Z annot be resolved. For spatial features of 1 mm, a total propagation distane of 1 m, and a d s value of 0.5 m, this utoff is approximately 0.6 mm 1. Essentially, this is the onfiguration for onventional (steady state) white light imaging of out-of-fous images. With the sensor plaed a short distane away, the features of an image are blurred into the noise. Using these onsiderations, we expet that time-resolved measurements an be used to extrat sene information otherwise unavailable. In partiular, using only a single pixel, whih orresponds to the f x = 0 axis in Fig. 6, we still see information as a funtion of time, so we ould potentially reover the (spatial) sene texture, though a first priniples alulation has not yet been. 5 Lensless Computational Imaging Using Inverse Time-Resolved Light Transport While the previous setions analyzed time-resolved forward light propagation and its ross-dimensional orrelations, inluding upper bounds on preserved information, we now proeed with the inverse problem. The frequeny analysis and bandwidth will help us define a modulation transfer funtion with whih to ompare our reonstrution results to ground truth values. Given a streak image, that is spae time sene information, aptured by a bare sensor with no additional optial elements, suh as lenses or attenuators, we aim at reonstruting image and depth of the sene. This is an ambitious goal; onventional ameras an only resolve a two-dimensional image without any depth information. A light field amera, reording spae and angle x-ν, provides suffiient information to refous the photograph in post-proessing (Isaksen et al. 2000); unfortunately, this usually omes at the ost of a signifiant drop in spatial image resolution due to the employed lenslet arrays. As derived in the previous setion, even a low-resolution bare sensor is theoretially apable of resolving image information with a higher spatial resolution, i.e. with super-resolution. For this appliation, we make the following assumptions for the remainder of this setion: a Lambertian sene (the sene is not neessarily planar), exhibiting no signifiant global illumination effets or olusions, is illuminated with a pulsed point light soure and reorded with a bare sensor that provides a temporal resolving power in the lower pioseond range. Under these onditions, the time-resolved rendering equation (Kajiya 1986; Smith et al. 2008) an be written as s(x, t)=i 0 1 (r s r l ) 2 ρ ( x ) ( osθ s osθ l δ t r ) s + r l dx, (12) where the reorded streak image s is restrited in spae x = (x x, x y, x z ) T to the plane (x x, x y, 0) T and ρ (x) desribes the diffuse albedo of a point. As illustrated in Fig. 7, the distanes between a sene point and a pixel, as well as between the soure and a sene point, are, respetively,r s andr l, both of whih depend impliitly on x. The osine terms model the angle between surfae point normal and inoming or outgoing light diretion, whereas the δ funtion desribes the time-dependeny of the system introdued by the pulsed point light soure. Equation 12 is a tomographi reonstrution problem (Kak and Slaney 2001), as eah pixel in the streak image measures a weighted integral over the unknown surfae albedos in the sene. As opposed to onventional line tomography, the integration surfaes in this appliation are elliptial surfaes as illustrated by the dashed red line in Fig. 7. This is similar to range tomography in seismi imaging (Saleh 2011) with the signifiant differene that no phase information is aptured in our appliation, making the inversion problem more diffiult.

9 Int J Comput Vis (2014) 110: Fig. 7 One-dimensional illustration of tomographi sene reonstrution. A lensless streak sensor (enter) reords the time of arrival of photons emitted by a pulsed point light soure (left). The aptured streak image for two white pathes in the sene is illustrated on the right. For an unknown sene, however, the ontribution at eah sensor pixel (x, t) is ambiguous as it may ontain ontributions from any point in the sene that is loated along an ellipse whose lous of points have idential path lengths from soure to detetor, illustrated by the dashed red line (left) While filtered bakprojetion is one of the most popular tehniques for omputed tomography, the algorithm requires projetions that vary over the full hemisphere. Our appliation is more similar in spirit to limited baseline tomography, where projetions over only a limited set (in our ase the finite area of the sensor) are available. For these problems, algebrai reonstrutions tehniques are ommonly applied. Follow (Herman 1995), we employ iterative algebrai reonstrution methods. For this purpose, we model the sene ontaining an unknown geometry and surfae albedos as a basis expansion: ρ (x) = N ρ k φ k (x), (13) k=1 where ρ k are the oeffiients and the basis funtions are φ k (x), k 1...N, i.e. the basis funtions are disrete voxels. This notation allows us to write Eq. 12 as N 1 s(x, t) = I 0 ρ k (r k=1 s r l ) 2 φ ( k x ) os θ s os θ l ( δ t r ) s + r l dx. (14) In pratie, this formulation is disretized into a linear system s = Pρ, where eah element of s is, s xt = N k=1 ρ k P (k) xt, (15) and we approximate the elements of matrix P as ( ) P xt (k) I 0 = (r xk r lk ) 2 da rxk + r lk k os θ xk os θ lk ζ t. (16) Inspired by surflets (Chandrasekaran et al. 2004), we model the unknown sene geometry as a olletion of planar pathes of size da k, faing the bare sensor. The distane from path k to sensor pixel x is r xk, r lk is the distane from path k to the light soure, and ζ t ( ) maps the travel time of a photon to the nearest time slot in the streak image. Using this formulation, the final optimization problem is formulated as minimize s Pρ 2 ρ 2, subjet to 0 ρ 1, (17) whih an be solved with standard approahes of onstrained linear optimization as disussed in Set Depth of Field Analysis In this setion, we derive the upper bounds of the proposed optial aquisition system and evaluate the performane of the ombined optial and omputational amera design. In onventional optial systems, the performane of a lens or image quality measured by a sensor through some optial assembly is mostly derived as the modulation transfer funtion (MTF). The MTF is defined as the magnitude of the optial transfer funtion and provides a measure for ontrast and resolution of an imaging system. Intuitively, the MTF gives the relative loss of energy of a diffuse target for a speifi spatial frequeny. The desired bound of any optial system is the diffration limit; in pratie, however, this is not always ahieved, and oftentimes energy is optially lost for higher spatial frequenies, resulting in a dereased image ontrast. The proposed optial setup onsists of a bare sensor without any refrative optial elements. Intuitively, we expet a planar target to be perfetly reonstruted when loated

10 136 Int J Comput Vis (2014) 110: diretly on the sensor. As the target moves away from the sensor, the image information is omputationally extrated from the measurements, but possibly at a lower quality. The final image quality depends on both the optial system and the reonstrution algorithm we proeed by evaluating the upper bounds provided by the proposed optial system using a modified MTF, followed by an evaluation of the reonstruted image quality. Motivated by modulation transfer funtions in onventional imaging systems, we define an MTF for the proposed lensless imager as MTF d, f0 = s ( f 0, f t ) l 0 L ( f 0, f t ) df t = F ft d ( f 0 d) df t, (18) where the limits of integration orrespond to the temporal bandwidth of the sensor, 1/(20 ps). Following Eq. 10, this formulation measures the relative loss of energy for a diffuse plane textured with a pattern that exhibits a single spatial frequeny f 0. The Fourier transform (FT) of the light field emitted by the plane is l 0 L, whereas s is the FT of the streak sensor image measuring spatial and temporal light variation of l 0 L after a propagation distane d. As disussed in Set. 4, the speial ase of a planar objet imaged at a distane to the sensor redues the onvolution, in the frequeny domain, with the propagation kernel F κ to a multipliation. This simplifiation allows us to write the MTF in Eq. 18 (right) as the integral over the magnitudes of the kernel s temporal frequenies. For finite limits, Eq. 18 annot be alulated in losed form. Figure 8 plots the MTF for an optial setup with dimensions desribed in Set. 7. We observe that, similar to onven- tional optial systems, higher spatial frequenies of a plane at a fixed distane d exhibit a stronger loss of energy than low frequenies. Similarly, as the distane between the sensor and the plane inreases, the MTF dereases, indiating that the expeted reonstrution quality of objets at larger distanes would derease. In partiular, for a time bandwidth lower than the spatial frequeny under investigation, the MTF falls off strongly, beause the integral in Eq. 18 inludes only the exponentially small region of F κ. Furthermore, the MTF does not take into aount the finite sensor size, whih would at as a low pass filter, or the reonstrution algorithm. Eah point in the sene generates a urved wavefront, but due to the finite sensor size, only a portion of it is deteted. For smaller sensors, the tails of the hyperbolas are lost, so that it beomes hallenging to separate different hyperbolas. We evaluate the quality of the proposed omputational amera design, inluding optial aquisition and tomographi reonstrution, in Fig. 9. For this purpose, we simulate a resolution hart of size pixels at various distanes d, ranging from 0 to 100 m, to the bare sensor. Reonstrution is performed by solving Eq. 17 as desribed in Set For this experiment, the same dimensions are used as for the MTF in Fig. 8 and for the simulations in Set. 7. As predited by the MTF, reonstrution quality measured in peak-signal-to-noise ratio (PSNR) drops with an inreasing distane d but also for inreasing spatial frequenies at a fixed depth. We onlude that the proposed omputational imaging systems exhibits a finite depth of field, similar to onventional ameras, with the foal plane being loated diretly on the sensor. We ontinue in the following setion to evaluate the performane of the proposed system using more omplex senes omposed of textured objets at varying distanes to the sensor. We also show initial results of a prototypial ultrafast omputational amera Fig. 8 Modulation transfer funtion (MTF). Numerially evaluating Eq. 18 allows for an upper bound on the expeted reonstrution quality to be plaed. The MTF measures the relative preserved energy of a diffuse plane at a distane d to a bare image sensor. The plane is textured with a pattern that exhibits a single spatial frequeny f 0. The lower the MTF magnitudes, the more energy is optially lost, making a omputational reonstrution inreasingly diffiult and sensitive to noise. We observe that the MTF magnitudes for a fixed spatial frequeny derease with inreasing distane to the sensor. A similar derease is observed for inreasing spatial frequenies at a fixed distane to the sensor Experimental Results In this setion, we show four different experiments of the proposed lensless imaging system. The results in Figs. 10 and 11 simulate a streak sensor that aptures a threedimensional volume ontaining two spatial dimensions as well as temporal light variation. The experiments show reonstrutions of only the diffuse albedos for a single plane at a distane to the sensor in Fig. 10 and for two planes at different distanes in Fig. 11. In the latter ase, the depth is estimated along with the diffuse albedos. In the experiments shown in Figs. 10 and 11, the laser is simulated to be loated in the enter of the optial axis, whereas the laser is aimed at the enter of the diffuser for the real experiment (Fig. 12, left). Figures 12 and 13 show reonstrutions of simple two-

11 Int J Comput Vis (2014) 110: Fig. 9 Depth of field (DOF). This plot evaluates the ombined quality of optial aquisition and tomographi omputational reonstrution. A syntheti planar resolution hart (enter row, left) is aquired at different distanes to a bare sensor and subsequently reonstruted at the known depth. As predited by the MTF (Fig. 8), reonstrution quality drops with an inreasing distane between resolution hart and sensor Fig. 10 Simulated reonstrutions of a textured plane at a distane to the sensor. A bare sensor plaed at a distane to the sene (left) aptures a streak image, whih is a series of 2D images (bottom row). The sene is reonstruted using omputed tomography for varying levels of additive sensor noise (top row). The point soure wavefront, olored from blue to green to yellow, represents inreasing time (Color figure online) dimensional sene pathes from a streak sensor that only measures one spatial dimension along with temporal variation. The results illustrated in Fig. 10 are aptured with a simulated streak sensor that has a spatial resolution of pixels and a size of mm. This sensor reords 32 time slots, eah integrating over 10 ps (Fig. 10, bottom row). The sene is a diffuse plane loated 60 mm away from the sensor and has the same size as the sensor, but a spatial resolution of pixels. The top row shows reonstrutions with varying levels of additive Gaussian sensor noise. Figure 11 shows simulated results for a sene with a varying depth. As disussed in Set. 5, this requires the tomographi reonstrution to estimate both diffuse albedo and sene depth simultaneously. In this experiment, the spatial resolution of eah of the two depth planes is pixels, and the number of time slots reorded is inreased to 38 (with the same exposure time of 10 ps). The redued quality of these results an be entirely attributed to the olusion

12 138 Int J Comput Vis (2014) 110: Fig. 11 Simulated results for reonstruting image and depth of an unknown sene aptured with a bare sensor. Several time slies and a volumetri rendering of the reorded streak images are shown in the bottom row. While the sene depth an be reonstruted relatively aurately (enter row), even small amounts of olusion in the sene, as exhibited by the two playing ards, result in global noise in the reonstruted diffuse albedos (top row). The point soure wavefront, olored from blue to green to yellow, represents inreasing time Fig. 12 The prototype amera setup shows the streak amera and the laser (left) and the sene(enter left). Due to physial onstraints, we model a bare sensor in spae by imaging the light sattered onto an optial diffuser. A shemati illustration of the setup (enter right). Experimental and simulated streak images agree (right). Suessful reonstrution of a simple, white 2D path from a 1D sensor is shown on the right between the ards in the sene. Even a small amount of olusion, as exhibited in this experiment, results in a global inrease in reonstrution noise. As stated in Eq. 12, the goal of the proposed reonstrution algorithm is to reover a three-dimensional volume with albedos. For senes ontaining opaque objets, this diretly results in the depth of the sene, with the upper limit being the MTF above. Real results aptured with a one-dimensional streak amera (see Set. 7.2) are shown in Fig. 12. The intriate design of this amera prevents us from plaing the bare sensor diretly in the sene; we emulate this setup by reording a sene path behind an optial diffuser instead. Challenges in this experiment are manifold: intensity variations and temporal jittering of the laser along with vignetting and non-linear temporal distortions in the streak amera result in slight variations of the reorded signal as ompared to the predited data using known sene geometry and albedo for this simple sene. Furthermore, a two-dimensional sene at a alibrated distane is measured with a one-dimensional sensor, whih makes the reonstrution partiularly diffiult. Nevertheless, we show suessful reonstrution of a simple sene a diffuse white path in Fig. 12. The white path is a two-dimensional objet imaged with a one-dimensional streak sensor. Although not immediately obvious, this time-

13 Int J Comput Vis (2014) 110: time is approximately 3 min for 5,000 iterations for the real experimental data. 7.2 Hardware Setup For the aptured result shown in Fig. 12, the illumination soure is a femtoseond (fs) Titanium:Sapphire laser, whih produed 50 fs long pulses, entered at 795 nm at a repetition rate of 75 MHz. The beam is foused onto the diffuser wall (ground glass). The streak amera is a Hamamatsu C5680, whih is used to detet one spatial dimension of the diffuser with a time resolution of 15 ps and quantum effiieny of about 10%. The amera position and viewing diretion are fixed. Fig. 13 Experimental results. A one-dimensional barode onsisting of five pathes with different albedos (lower left) is plaed ata distane behind a diffuser and reorded with the streak amera prototype (upper right). The tomographi reonstrution of the albedos (lower right) exhibits slight errors in the darker pathes, whih are due to imperfet alibration of the prototype (differene between simulated and aptured streak images, top row) resolved 1D image aptures suffiient information for the simple 2D sene to be reovered. This is possible beause of the ross-dimensional information transfer, from spatial to spae time domains, introdued by light propagation in free spae. Figure 13 shows another result aptured with the prototype ultra-fast amera. For this experiment, we plaed a 1D barode at a distane behind the diffuser (see setup in Fig. 12). The barode onsists of five pathes with varying albedos (Fig. 13, lower left). Again, we observe slight differenes between the simulated streak image rendered from the known sene geometry and refletane (Fig. 13, upper left) and the data reorded with our prototype amera (Fig. 13, upper right). The albedos of the five pathes are suessfully reonstruted from the aptured data (Fig. 13, lower right), yet with some errors in the two darker pathes that are due to the mismath of expeted and reorded streak image. 7.1 Software Implementation We solve Eq. 17 using the simultaneous algebrai reonstrution tehnique (SART) (Kak and Slaney 2001). This is an iterative approah to solve omputed tomography problems. In our Matlab-based implementation, we run 10,000 iterations for eah of the reonstrutions in Figs. 10, 11, 12, and 13. The large number of iterations is required due to the slow onvergene rate of SART. The omputations are performed with a sparse representation of matrix P in Eq. 17. Computing times are approximately one to two hours for all three olor hannels of eah of the results. The omputation 8 Disussion In summary, we have presented a frequeny analysis and a rigorous derivation of analytial expressions for the most fundamental proess in light transport propagation in free spae, onsidering spae, time, and angle. While prior models for the steady-state ase intuitively desribe light field propagation as a loalized shear in both the primal and frequeny domain, the hyperboli urvature in spae time, introdued by differenes in travel distanes for different angles, deloalizes signal energy in the frequeny domain. However, we demonstrate that it an be intuitively expressed as a ombination of the well known shear in the light field and a onvolution along the angular frequenies. This onvolution optially blurs information between the different dimensions; we demonstrate that this information transfer an be omputationally exploited using a novel, lensless omputational imaging tehnique whih has potential appliations in biomedial imaging, thin mobile devies, remote sensing, and surveillane. While the forward analysis presented is most general in onsidering spae, time, and angle for arbitrary senes, the inverse problem inluding bandwidth analysis and tomographi sene reonstrution restrit the imaged sene to be diffuse and to exhibit negligible amounts of global illumination effets and olusions. The proposed sensing tehnique requires an ultra-short pulsed illumination, as for instane provided by a laser, and a streak amera apable of resolving time variations in the lower pioseond range. Currently available hardware ahieving the required temporal resolution is restrited to sensing one spatial sanline per reorded image. The exposure time of the proposed system is signifiantly shorter than onventional imaging system; this fat in ombination with the removed lens redues the overall amount of light measured at eah sensor pixel. Note that, for the spatial and temporal sales onsidered here, the geometri model is suffiient for modeling and reonstrution,

14 140 Int J Comput Vis (2014) 110: as shown previously (Velten et al. 2012; Wu et al. 2012). For features that approah the wavelength of light, however, diffration must be inorporated through, e.g. transient generalizations of phase spae and wave optis. Furthermore, it is unneessary to resort to the partile (quantum) nature of light at this time sale. (Indeed pioseond optis is treated lassially in e.g. fiber opti ommuniations.) The quantum nature of light would ome into play in low light onditions, for whih photon statistis plays a role. The main hallenges of high-quality image reonstrution are urrently ditated by the employed hardware setup. The streak amera is diffiult to alibrate as nonlinear effets in laser intensity and image projetion, are observed. These hallenges limit experimental reonstrutions to simple senes onsisting of textured planar pathes. Nevertheless, the work presented in this paper provides the fundamental analyti platform on whih to re-examine many omputer vision appliations in light of a new degree of freedom: time-resolved image information. Further, it provides a method for building a onsistent theory of time-resolved light field propagation. Future work will examine effets of optial elements other than propagation in free spae, inluding lenses, diffusers, diffrative elements, and sattering media. Aknowledgments The work of the MIT affiliated oauthors was funded by the Media Lab Consortium Members. Gordon Wetzstein was supported by an NSERC Postdotoral Fellowship. Tsinghua University affiliated oauthors were supported by China National Basi Researh Projet (No. 2010CB731800) and the Key Projet of NSFC (Nos and ). Referenes Andersen, G. (2005). Large optial photon sieve. Optis Letters, 30, Chai, J. X., Tong, X., Chan, S. C., & Shum, H. Y. (2000). Plenopti sampling. ACM Transations on Graphis (SIGGRAPH), 19, Chandrasekaran, V., Wakin, M., Baron, D., & Baraniuk, R. (2004). Surflets: A sparse representation for multidimensional funtions ontaining smooth disontinuities. In: Information Theory, ISIT Proeedings. International Symposium on, IEEE, Chiago (p. 563). Durand, F., Holzshuh, N., Soler, C., Chan, E., & Sillion, F. X. (2005). A frequeny analysis of light transport. ACM Transations on Graphis (SIGGRAPH), 24, Edward, R., Dowski, J., & Cathey, W. T. (1995). Extended depth of field through wave-front oding. Applied Optis, 34, Gill, P. R., Lee, C., Lee, D. G., Wang, A., & Molnar, A. (2011). A mirosale amera using diret fourier-domain sene apture. Optis Letters, 36, Hamamatsu: Hamamatsu Streak Camera. (2012). hamamatsu.om/tutorials/java/streakamera/ Herman, G. T. (1995). Image reonstrution from projetions. Real- Time Imaging, 1, Isaksen, A., MMillan, L., Gortler, S. J. (2000) Dynamially reparameterized light fields. In: Proeedings of the ACM Transations on Graphis (SIGGRAPH), San Diego (pp ). Isikman, S. O., Bishara, W., Mavandadi, S., Yu, F. W., Feng, S., Lau, R., Ozan, A. (2011). Lens-free optial tomographi mirosope with a large imaging volume on a hip. In: Proeedings of the National Aademy of Sienes, Washington. Kajiya, J. T. (1986). The rendering equation. ACM Transations on Graphis (SIGGRAPH), 20, Kak, A. C., & Slaney, M. (2001). Priniples of omputerized tomographi imaging. Tahoe: Soiety for, Industrial Mathematis. Koppal, S. J., Gkioulekas, I., Zikler, T., Barrows, G. L. (2011). Wideangle miro sensors for vision on a tight budget. In: Proeedings of the CVPR, New York (pp ). Levin, A., & Durand, F. (2010). Linear view synthesis using a dimensionality gap light field prior. In: Proeedings of the CVPR, New York (pp. 1 8). Levin, A., Hasinoff, S. W., Green, P., Durand, F., & Freeman, W. T. (2009). 4D frequeny analysis of omputational ameras for depth of field extension. ACM Transations on Graphis (Siggraph), 28, 97:1 97:14. Naik, N., Zhao, S., Velten, A., Raskar, R., & Bala, K. (2011). Single view refletane apture using multiplexed sattering and time-offlight imaging. ACM Transations on Graphis (SIGGRAPH Asia), 30, 171. Ng, R. (2005). Fourier slie photography. ACM Transations on Graphis (SIGGRAPH), 24, Pandharkar, R., Velten, A., Bardagjy, A., Lawson, E., Bawendi, M., & Raskar, R. (2011). Estimating motion and size of moving non-lineof-sight objets in luttered environments. In: Proeedings of the CVPR, New York (pp ). Raskar, R., & Davis, J. (2008). 5D Time-Light Transport Matrix: What Can We Reason about Sene Properties?. Report: MIT Tehnial. Saleh, B. (2011). Introdution to subsurfae imaging. Cambridge: Cambridge University Press. Smith, A., Skorupski, J., & Davis, J. (2008). Transient Rendering. Tehnial Report UCSC-SOE-08-26, Shool of Engineering, University of California, Santa Cruz. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007). Dappled photography: Mask enhaned ameras for heterodyned light fields and oded aperture refousing. ACM Transations on Graphis (SIGGRAPH), 26, 69. Velten, A., Willwaher, T., Gupta, O., Veeraraghavan, A., Bawendi, M., & Raskar, R. (2012). Reovering three-dimensional shape around a orner using ultra-fast time-of-flight imaging. Nature, Communiations, 3, Wu, D., O Toole, M., Velten, A., Agrawal, A., & Raskar, R. (2012). Deomposing global light transport using time of flight imaging. In: Proeedings of the CVPR, New York (pp. 1 8). Zomet, A., & Nayar, S.K. (2006). Lensless imaging with a ontrollable aperture. In: Proeedings of the CVPR, New York (pp ).