Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging

Transcription

1 Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging MOHIT GUPTA and SHREE K. NAYAR Columbia University and MATTHIAS B. HULLIN and JAIME MARTIN University of Bonn In orrelation-based time-of-flight (C-ToF) imaging systems, light s with temporally varying intensities illuminate the. Due to global illumination, the temporally varying reeived at the is a ombination of light reeived along multiple paths. Reovering properties (e.g., depths) from the reeived requires separating these ontributions, whih is hallenging due to the omplexity of global illumination and the additional temporal dimension of the. We propose phasor imaging, a framework for performing fast inverse light transport analysis using C-ToF s. Phasor imaging is based on the idea that by representing light transport quantities as phasors and light transport events as phasor transformations, light transport analysis an be simplified in the temporal frequeny domain. We study the effet of temporal illumination frequenies on light transport, and show that for a broad range of s, global (multi-path interferene) vanishes for frequenies higher than a -dependent threshold. We use this observation for developing two novel reovery tehniques. First, we present Miro ToF imaging, a ToF based shape reovery tehnique that is robust to errors due to global illumination. Seond, we present a tehnique for separating the diret and global omponents of. Both tehniques require apturing as few as 3 4 images and minimal omputations. We demonstrate the validity of the presented tehniques via simulations and experiments performed with our hardware prototype. Categories and Subjet Desriptors: I.4.8 [Image Proessing and Computer Vision]: Sene Analysis Shape Additional Key Words and Phrases: Computational photography, time-offlight imaging, light transport, interrefletions, sattering, orrelation imaging, transient imaging, phasors, multi-path interferene, global illumination, radiosity 1. INTRODUCTION Correlation-based time-of-flight (C-ToF) imaging systems onsist of temporally modulated light s and s with temporally modulated exposures. The brightness measured by the is the orrelation between the temporally varying inident on the and the exposure funtion. This is illustrated in Figure 1. Beause of their ability to measure depths with high preision and speed, these systems are fast beoming the method of hoie for depth sensing in a wide range of appliations. Several low ost and ompat C-ToF systems are available as ommodity devies, inluding the Mirosoft Kinet and the Intel SoftKineti s. Global light transport in C-ToF imaging: Conventional C-ToF imaging systems assume that pixels reeive light only due to diret illumination of points from the. However, due to global illumination, the reeives along several paths, after multiple refletion/sattering events. Reovering properties (e.g., depths) from the reeived requires separation of ontributions from different paths. This is a diffiult task due to the omplexity of global illumination, and is made even more hallenging beause of the additional temporal dimension of the. Phasor representation of : Our goal is to develop a ompat model for generalized C-ToF imaging, i.e., a model of C-ToF imaging that aounts for full global illumination. To this end, we make the following observations. If the is illuminated with sinusoids of a given temporal frequeny, the at any point and diretion is always a sinusoid of the same frequeny, irrespetive of the. Sine all the sinusoids are of the same frequeny, the frequeny an be fatored out and the at any point and diretion an be represented by a single omplex number, or phasor. With phasor representation, light transport at eah temporal frequeny an be analyzed separately, thus signifiantly reduing the omplexity. Also, sine phasor orresponds to a partiular modulation frequeny, it an be aptured by a C-ToF operating at that frequeny with only two measurements. Phasor imaging: Based on these observations, we propose phasor imaging, a framework for analyzing light transport in C-ToF imaging, using phasor representations of and light transport events. In partiular, we analyze the effet of temporal frequeny on light transport and show that for a broad range of s, global dereases with inreasing frequeny, eventually vanishing beyond a threshold frequeny. Using this property, we develop two analysis tehniques: Transport-robust shape reovery, Fast separation of diret and global. Transport-robust shape reovery: An important problem faed by C-ToF based depth reovery systems is the errors aused by global illumination (multi path interferene). These errors are systemati and dependent, and an be orders of magnitude larger than the random errors ourring due to system noise. This problem has reeived a lot of attention reently, with a variety of tehniques having been proposed to mitigate the errors [Godbaz et al. 008; Dorrington et al. 011; Kirmani et al. 013]. These approahes assume global illumination to be a disrete sum of ontributions along a small number ( 3) of light paths. For general s, pixels may reeive light along several, potentially infinite, light paths. Consequently, these approahes are limited to s with only high frequeny light transport (e.g., speular interrefletions). We present Miro ToF imaging, a tehnique for reovering shape that is robust to errors due to global illumination, and is appliable to s with a broad range of light transport effets. It is based on using high temporal frequenies at whih global illumination vanishes, and hene does not introdue errors in the phase of the

2 Mohit Gupta et al. reeived. Although using high frequenies ahieves robustness to global illumination, the unambiguous depth range is small due to phase ambiguities. Miro ToF uses two (or more) high frequenies and standard phase unwrapping tehniques to disambiguate the high frequeny phases, thus ahieving robustness to global illumination as well as a large depth range, with as few as four measurements. Fast separation of diret and global : We present a tehnique for separation of diret and global omponents. One way to separate the two omponents (using temporal light modulation) is to measure the full transient image of the [Heide et al. 013; Velten et al. 013]. These approahes, although theoretially valid, require prohibitively large aquisition times. We show that it is possible to perform the separation by apturing only three measurements at a single high temporal frequeny. The proposed tehnique an be thought of as the temporal ounterpart to the tehnique presented by Nayar et al. [006] whih performed separation using high spatial frequeny illumination. Limitations and impliations: We have demonstrated our analysis tehniques by building a hardware prototype based on a low ost C-ToF. Currently, these s have a limited range of modulation frequenies, whih restrits the appliation of our tehniques to relatively large sale s. However, this is not a theoretial limitation. As devie frequenies inrease [Akbulut et al. 001; Wu et al. 010; Buxbaum et al. 00; Shwarte 004; Busk and Heiselberg 004], it will be possible to apply our tehniques on smaller sale s. Due to their generality, near real time aquisition and omputation times, we believe that the proposed tehniques will be readily integrated into future C-ToF imaging systems for performing a variety of analysis tasks.. RELATED WORK Impulse Time-of-Flight Imaging: Impulse ToF imaging tehniques measure the temporal impulse response of the by illuminating it with very short (pio/nanoseond) laser pulses and reording the refleted light at high temporal resolution. Impulse ToF imaging was the basis of one of the first ToF range imaging systems [Koehner 1968]. While earlier systems assumed only a single diret refletion of light from the, reent tehniques (alled transient imaging) have used the impulse ToF priniple to measure and analyze both diret and indiret light transport for apturing images around a orner [Kirmani et al. 009], measuring 3D shape [Velten et al. 01] and motion of objets [Pandharkar et al. 011] around the orner, performing separation of light transport omponents [Wu et al. 01], measuring BRDF [Naik et al. 011], apturing images with a lens-less [Wu et al. 01], and apturing the propagation of light [Velten et al. 013]. Correlation-Based Time-of-Flight Imaging: These tehniques were introdued as a low ost alternative to impulse ToF imaging. The is illuminated with ontinuous temporally modulated light (e.g., with sinusoids), and the measures the temporal orrelation of the inident light with a referene funtion [Shwarte et al. 1997; Lange and Seitz 001]. Sene depths are omputed by measuring the relative phase-shift between the inident light and the emitted light. While there has been researh on optimizing the modulation waveform [Payne et al. 010; Ferriere et al. 008; Ai et al. 011] for ahieving high preision and for handling interferene among multiple ToF ameras [Buttgen et al. 007], it is mostly assumed that reeives only diret refletion from the. Our work seeks to generalize orrelation-based ToF imaging to inlude a variety of indiret (global) light transport effets. m(t): modulation funtion signal generator R(t): exposure funtion measured brightness Fig. 1. Correlation-based ToF image formation model. The is illuminated by a temporally modulated light, with radiant intensity I(θ, t) along diretion θ. The s exposure is also temporally modulated during the integration time aording to the funtion R(t). The brightness B(p) measured at a pixel p is the orrelation of the inoming L(p, t) and the exposure funtion R(t). Multi Path Interferene in Time-of-Flight Imaging: Reently, there has been a lot of researh towards mitigating the effet of global illumination (multi-path) in ToF ameras. In general, this is a diffiult problem beause global illumination depends on struture, whih is unknown at time of apture. There have been several attempts at solving the problem for speial ases, suh as pieewise planar Lambertian s [Fuhs 010; Fuhs et al. 013; Jimenez et al. 01] or temporally sparse signals [Godbaz et al. 008; 009; 01; Dorrington et al. 011; Jimenez et al. 01; Kadambi et al. 013; Kirmani et al. 013]. These approahes do not generalize to all forms of light transport. Moreover, they often require apturing a large number of images and/or omputationally intensive optimization-based reonstrution algorithms. The approah of Freedman et al. [014] onsiders ompressible signals (instead of sparse signals), and an handle limited amount of diffuse interrefletions. However, sine the signal is assumed to be ompressible, it is limited to s where the dominant amount of global illumination is due to only a small number of light paths. The approah presented in this paper requires taking as few as four measurements and only a few linear operations, and is appliable to s with a wide range of light transport effets. Light Transport Analysis Using Spatial Light Modulation: In the last few years, several tehniques performing light transport analysis using spatially modulated light have been presented. This inludes methods for inverting light transport [Steven M. Seitz 005], performing global-transport-robust shape reovery [Gupta et al. 009; Gupta and Nayar 01; Gupta et al. 013; Couture et al. 014] and separating or seletively enhaning light transport omponents [Nayar et al. 006; Reddy et al. 01; OToole et al. 01]. Reently, O Toole et al. [014] have used a ombination of spatial and temporal light modulation for performing a variety of analysis tasks. While their tehniques rely on high-spatialfrequeny light modulation, our fous is on studying the behavior of light transport as a funtion of temporal frequenies. We develop tehniques that use only high-temporal-frequeny light modulation, and ahieve near real-time apture rates. 3. BACKGROUND AND IMAGING MODEL A C-ToF imaging system onsists of a temporally modulated light, and a whose exposure an be temporally modulated during integration time. This is illustrated in Figure 1. Let the be modulated with a periodi funtion m(t) (normal-

3 Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging 3 ized to be between 0 and 1). Then, the radiant intensity I(θ,t) of the in diretionθ is given as: I(θ,t) = i(θ)m(t). (1) The exposure is temporally modulated aording to the exposure funtion R(t), whih an be realized either by on-hip gain modulation (e.g., photoni mixer devies [Shwarte et al. 1997]) or by external optial shutters [Carnegie et al. 011]. Let the inident at a pixel p be L(p,t). The brightness B(p) measured at pixel p is given by the orrelation between the inoming and the exposure funtion: B(p) = τ 0 R(t)L(p, t)dt, () where τ is the total integration time. Light transport equation for C-ToF imaging: Let L θ (p,t) be the inident at pixel p due to light emitted from the along diretion θ.l θ (p,t) is given as: ( L θ (p,t) = β(p,θ)i θ,t Γ(p,θ) ), (3) where Γ(p,θ) is the length of the path taken (through the ) by the ray emitted in diretion θ and arriving at p. The onstant is the speed of light. β(p,θ) is the light transport oeffiient between diretionθ and pixelp; it is defined as the fration of emitted intensity that reahes the. The total reeived L(p, t) is the integral of ontributions from the set of all outgoing diretions Ω: L(p,t)= L θ (p,t)dθ= β(p, θ)i Ω Ω ( θ,t Γ(p,θ) ) dθ. (4) This is the light transport equation for C-ToF imaging. It expresses the temporal profiles reeived at a pixel in terms of the emitted I(t) and the properties (light transport oeffiients and path lengths). Sine it is dependent, in general, L(p,t) does not have a ompat analyti form (as a funtion of t). L(p,t) is a ombination of light oming along multiple paths, whih annot be easily separated and analyzed for reovering properties. Also, apturing the entire time profile requires long aquisition times. A ompat representation of : L(p, t) is the impulse response of the, i.e., the temporal reeived at the if the is illuminated with an temporal impulse illumination. If, however, the is illuminated with sinusoidally varying illumination at a fixed frequeny, the at every point and every diretion in spae (inluding at the ) will also vary sinusoidally with the same frequeny. This is beause L(p, t) is an integral of shifted and saled emitted funtions I(t) (Eq. 4), and sinusoids are losed under saling, shifting and integration. Sine all the sinusoids are of the same frequeny, we an fator the frequeny out, and represent the at any point x in spae (inluding the ) along any diretion θ by a single omplex number, or phasor L(x,θ) = L(x,θ)e jφ(x,θ), wherelis the amplitude and φ is the phase of the sinusoid 1.j = 1 is the 1 Sine light is non-negative, the sinusoidal modulation funtions have a non-zero offsetl DC. The orresponding phasor representation is a -tuple: [L DC, L ω ], where L DC is the DC and L ω is the osillating omponent. For the intensity to be non-negative, L DC L ω, where. is the modulus operator that returns the magnitude of the omplex number. emitted phasor reeived phasor (a) Single emitted-reeived ray pair emitted phasors reeived phasors (b) All emitted-reeived ray pairs Fig.. Signal proessing view of phasor light transport. (a) Rays emitted from the and reeived at the are represented by single phasors. The transforms every emitted phasor into a reeived phasor. The transformation is linear (multipliation by the light transport oeffiient for the emitted-reeived ray pair). (b) Light transport between all the emitted and reeived rays an be ompatly represented as a matrix multipliation. omplex square-root of unity. We all L the phasor, short for phasor representation of. Phasor light transport: The an be onsidered as a system that transforms the phasor emitted by the (by modulating its phase and magnitude) into phasor reeived by a pixel. The transformation an be expressed as a multipliation of the emitted phasor by another phasor - the light transport oeffiient between the emitted-reeived ray pair. This is illustrated in Figure (a). As has been shown reently [O Toole et al. 014], the light transport between all the emitted and reeived rays an be represented as a matrix multipliation: L = M I, (5) where L is the array of phasor s reeived at pixels and I is the array of phasor s emitted by the along different diretions (we use bold upper-ase letters to denote arrays and matries). This is shown in Figure (b). We all Eq. 5 the phasor light transport equation and M the phasor light transport matrix of the. Phasor and onventional light transport matries are related as: Γ(p,θ) jω M(p,θ) = M(p,θ)e, (6) where M is the light transport matrix for onventional imaging and ω is the modulation frequeny. Note that the phasor light transport matrix is a funtion of the modulation frequeny ω. For DC omponent (ω = 0), phasor transport matrix is the same as the onventional light transport matrix. The phasor light transport equation expresses light transport in C-ToF imaging (Eq. 4) for a given modulation frequeny as a linear, matrix multipliation. This simplifies light transport analysis in C-ToF imaging, espeially the study of how light transport depends on the modulation frequenies. From a pratial standpoint also, the phasor representation naturally lends itself to C-ToF imaging. This is beause the phasor reeived at every pixel has only two unknowns (phase and magnitude), whih an be aptured diretly by C-ToF s operating at a single frequeny with only two measurements. This forms the basis of the tehniques presented in the paper, whih require taking as few as four and three measurements for transport-robust shape reovery and diret-global separation, respetively. 4. PHASOR REPRESENTATION OF LIGHT TRANSPORT EVENTS Light transport events an be ategorized into three basi groups based on the phasor transformations that they indue, as illustrated The matrix representation assumes that the spae of light rays has been disretized along spatial and angular dimensions.

4 4 Mohit Gupta et al. Conventional Imaging Phasor Imaging Propagation Through Spae Initial Final Loal Refletion refleting surfae BRDF refleting surfae BRDF Initial Final Superposition of Multiple Rays + + Initial Final Fig. 3. Phasor representation of light transport events. Using phasors, all the light transport events an be represented by linear operations on omplex numbers. (Top row) Propagation of a light ray through spae hanges the phase of the phasor. The amount of hange is proportional to both the distane traveled and the modulation frequeny. This is represented by multipliation of the initial with a phasor of unit amplitude. (Middle row) Loal refletion and sattering events hange only the amplitude of the. These events are represented by multipliation with phasors having zero phase. This is similar to onventional imaging. (Bottom row) Multiple rays at the same point in spae traveling in the same diretion an be represented by a single ray whose is the omplex sum of the of individual rays. in Figure 3. First, events that hange the phase of the (propagation through spae). Seond, events that hange only the magnitude of the (loal refletion and sattering). Third, the superposition event where multiple phasors are added to give a resultant phasor. In the following, we onsider these individually. Propagation Through Spae: Propagation through free spae hanges the phase of the, while the magnitude is onserved. Let L(x,θ) be the phasor at a point x in spae along the diretionθ. Then, the after propagating through a distane Γ is given as: L(x+Γθ,θ) = L(x,θ) e jω Γ, (7) where ω is the modulation frequeny. Propagation through partiipating media hanges both the magnitude and the phase: L(x+Γθ,θ) = L(x,θ) e (σγ+jωγ ), (8) where σ is the medium s extintion oeffiient. Note that the amount of phase hange φ = ω Γ is proportional to both the modulation frequeny ω and the travel distane Γ. Loal Refletion and Sattering: Loal refletion at a surfae point hanges only the magnitude of the : L(x,θ o ) = L(x,θ i ) b(x;θ i,θ o ), (9) where b(x;θ i,θ o ) is the BRDF term 3 at point x for inoming light diretion θ i and outgoing light diretion θ o. Loal sattering has the same effet as refletion, with the sattering term (produt of sattering albedo and the sattering phase funtion) replaing the BRDF term. Superposition of Multiple Rays: Multiple light rays traveling in the same diretion through the same point an be represented as a single ray whose is the phasor sum of individual s: L(x,θ) = L i (x,θ), (10) i where L i (x,θ) are the individual s, and L(x,θ) is the total. Due to phasor summation, the magnitude of the total may be lesser than the sum of the individual magnitudes, i.e. L(x,θ) i L i (x,θ). The resultant magnitude an be zero as well, even if all the initial s have non-zero magnitudes. This is different from onventional imaging where sum of non-zero s is stritly positive. 3 The foreshortening effet is subsumed within the BRDF term.

5 Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging 5 Ray Diagram Ray Diagram Ray Diagram pixel light path pixel multiple light paths pixel Phasor Representation Phasor Representation Phasor Representation resultant emitted by reeived at pixel emitted by reeived at pixel emitted by reeived at pixel (a) Global light transport (b) Global light transport () Global light transport for a single ray for a loal ray neighborhood with high frequeny illumination Fig. 4. Vanishing global light transport for high modulation frequeny. (a) A single indiret light path P(p, θ) between outgoing diretion θ and pixel p. The phasor reeived at the along P(p,θ) is given by rotating and attenuating the emitted phasor. The angle of rotation φ is proportional to the length of P(p,θ). (b) A set of indiret light paths P(p,C θ ) in a small neighborhood of P(p,θ). All the paths end at p. The phasor s along pathsp(p,c θ ) an be assumed to have onstant amplitudes and linearly varying phases, and thus form a irular setor in the phasor diagram. The angle of the setor φ is proportional to the modulation frequeny ω. The total global is the resultant of the individual phasors. () The light emits high frequeny illumination. The individual phasor s span the omplete irle, and the resultant (total global ) is zero. 5. FREQUENCY DEPENDENCE OF PHASOR LIGHT TRANSPORT Consider the phasor light transport equation (Eq. 5). We an deompose the inident as the sum of the diret omponent L d ω and the global omponent L g ω, where the diret omponent is the light reahing the after single refletion and the global omponent is the light reahing the after multiple refletions (or sattering) events: L ω = L d ω + L g ω = M d ωi ω + M g ωi ω. (11) M d ω and M g ω are the diret and global omponents of the light transport matrix M ω, respetively, for modulation frequeny ω. PROPOSITION 1. Vanishing high-frequeny global light transport: For a broad range of s, if the frequenyωis higher than a threshold ω thresh, the global omponent vanishes: L g ω = M g ω I ω = 0 for ω ω thresh. (1) This is the key observation underlying our work. It is a onsequene of the fat that typially, global is temporally smooth, and an be assumed to be bandlimited. In the following, we provide an intuition behind the above observation by using phasor representations of light transport events. A frequeny-domain proof is given in Setion 5.1. Intuition: Consider a light path P(p, θ) involving multiple interrefletions, starting at the light in diretion θ, and ending at a pixel p. An example light path is shown in Figure 4 (a). The L θ (p) reeived at p along P(p,θ) is given by: L θ (p) = M(p,θ) I, (13) wherei is the emitted 4 and M(p,θ) is the light transport oeffiient for the path P(p, θ). Sine P(p, θ) involves propagation and refletion, L θ (p) is given by rotating and attenuating the emitted phasor (Figure 4 (a)), as desribed in Setion 4. Next, onsider the set of light paths P(p,C θ ) in a loal neighborhood of P(p,θ) that start in a one of diretions C θ around θ, and end at p. This is illustrated in Figure 4 (b). The magnitudes of the light transport oeffiients M(p,θ) = M(p,θ) an be assumed to be approximately onstant in a small light path neighborhood. This assumption forms the basis of methods that use high spatial frequeny illumination for separating light transport omponents [Nayar et al. 006] and performing transport-robust shape reovery [Gu et al. 011; Chen et al. 008; Gupta and ( Nayar) 01; Couture et al. 014]. The phases φ(p,θ) = arg M(p,θ) an be assumed to be linearly varying as a funtion of θ. This an be shown by onsidering the first order Taylor s expansion of the phases φ(p,θ). See the supplementary tehnial report for a proof. Thus, the individual reeived s L g θ (p) = M(p,θ) I have onstant amplitudes and linearly varying phases, and sweep out a irle setor. From Eq. 7, the angle φ of the setor is: φ = ω Γ(p,θ), (14) where Γ(p,θ) is the range of the lengths of paths P(p,C θ ). The total global L g C θ (p) is the resultant phasor of all the individual phasors (Eq. 10). Its magnitude is given by: L g C θ (p) = Q sin( ) φ, (15) φ 4 For ease of exposition, we assume an isotropi, i.e., I(θ) = I.

6 6 Mohit Gupta et al. emitted reeived diret emitted reeived global emitted reeived global reeived diret t Time Domain reeived global emitted t (a) Diret Radiane Time Domain reeived global emitted diret response t t t (b) Global Radiane Time Domain global response t t t emitted global response t reeived diret reeived global Frequeny Domain emitted Frequeny Domain reeived global () Global Radiane for High Frequeny Illumination emitted Frequeny Domain emitted diret response global response global response Fig. 5. Frequeny domain analysis of C-ToF light transport. (a) In time-domain, the diret reeived at the is given by onvolving the emitted signal and the diret response, whih is a dira delta funtion. The Fourier transform of a delta funtion is onstant. Thus the diret has all the frequenies that are present in the emitted. (b) The global is the onvolution of the emitted and the global response. For most real-world s, the global response is temporally smooth, and thus, bandlimited. If the bandlimit of the global response is ω b, the global is also bandlimited by ω b. () If the emitted is a sinusoid with frequeny ω > ω b, the global ontains only the DC omponent. where Q is the sum of magnitudes of the individual phasors. The derivation is given in the supplementary tehnial report. L Cθ (p) is a monotonially dereasing funtion of the setor angle φ for 0 φ π. Sine φ is proportional to the modulation frequeny ω (Eq. 14), as ω inreases, φ inreases, and the resultant magnitude dereases. If ω = π, φ = π, and the magnitude of the global L g C θ Γ(p,θ) (p) = Frequeny Domain Proof Of Vanishing High-Frequeny Global Transport Let the temporally varying light intensity emitted from the be given by I(t). The diret reeived at a pixel p is given by L d (t) = αi(t φ), where α enapsulates the albedo and intensity fall-off. φ = Γd is the temporal shift due to travel of light and Γ d is the length of the diret light path for pixel p. We an writel d (t) as a onvolution: 5 Stritly speaking, L g C (p) 0. This is beause the assumptions (loal θ onstany of light transport magnitudes and loal linearity of light transport π phases) hold approximately. Asω inreases beyond, we an apply Γ(p,θ) the above analysis in smaller light path neighborhoods (narrower onec θ ), whih improves the approximation. ( L d (t) = I(t) αδ t Γ ), (16) where δ() is the dira delta funtion. This is illustrated in Figure 5 (a). We define D(t) = αδ(t Γ ) as the diret response. D(t) is the diret reeived if the is illuminated with a temporal impulse funtion δ(t). Eq. 16 an be expressed in the frequeny domain as: ˆl d (ω) = î(ω) ˆd(ω), (17) where ˆl d (ω), î(ω) and ˆd(ω) are Fourier transforms of L d (t), I(t) and D(t), respetively. Sine D(t) is a dira delta funtion, magnitude of ˆd(ω) is onstant. This is illustrated in Figure 5 (a). The global response G(t) is defined as the global reeived if the is illuminated with a temporal impulse δ(t). Similar to the diret omponent, the global omponent L g (t) is: L g (t) = I(t) G(t). (18) In frequeny domain, the above equation is expressed as: ˆl g (ω) = î(ω) ĝ(ω), (19) where ˆl g (ω) and ĝ(ω) are Fourier transforms of L g (t) and G(t), respetively. This is illustrated in Figure 5 (b).

7 Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging 7 emitted Ray Diagram emitted Ray Diagram Ray Diagram emitted (high frequeny) Plot of Reeived Radiane Versus Time Plot of Reeived Radiane Versus Time Plot of Reeived Radiane Versus Time diret diret interrefletions total diret interrefletions total time time time (a) Diret only (b) Diret and global () Using high frequeny illumination Fig. 6. Effet of global illumination on shape reovery using C-ToF imaging. (a) Sene depths are omputed using C-ToF imaging by measuring the phase of the reeived relative to the emitted. (b) Due to global, a pixel reeives light along multiple light paths. These light paths have different lengths, and hene different phases as ompared to the diret refletion path. The phase of the total is different from the orret phase (diret phase), resulting in inorret depth. () For high frequeny illumination, the phase of the total is the same as that of the diret, and an be used for aurate depth reovery. For most real-world s, G(t) is temporally smooth, and hene an be assumed to be bandlimited, i.e., there exists a frequenyω b, alled the global transport bandlimit, suh thatĝ(ω) = 0 ω > ω b. Thus, if the emitted I(t) is a sinusoid with modulation frequeny ω larger than the global transport bandlimit ω b, the osillating omponent of the global, L g ω, is zero. Then, the global has only a onstant DC term (due to the DC term of the emitted ). This is illustrated in Figure 5 (). Other global illumination effets: While we have used interrefletions for the analysis so far, the results and the proposed tehniques are appliable to s with a broad range of global illumination effets suh as subsurfae sattering, volumetri sattering and diffusion. For eah of them, the global response is typially smooth, and thus, bandlimited. By hoosing a modulation frequeny higher than the bandlimit, the global an be made temporally onstant. 5. How High Is The Frequeny Bandlimit? The global transport bandlimit ω b depends on the geometry and material properties, as well as the global illumination effet. For volumetri sattering and diffuse interrefletions, ω b is typially low. On the other hand, for speular interrefletions,ω b is relatively high. Sene size is also a fator in determiningω b. For large s, the indiret light paths have a large range of path lengths, resulting in a low bandlimit ω b. For smaller s, indiret light paths have a smaller range of path lengths, and thus G(t) has a higher bandlimit. As a rule of thumb, ω b is 1 10 times, where ξ is the geometri sale of the (in meters). For instane, for a room ξ sale of size3.0 meters,ω b is approximately 100 MHz GHz., depending on the material properties (higher bandlimit for more speular s). For table-top s of size approximately 50 entimeters, ω b is approximately 600 MHz. to6ghz. Arbitrary modulation funtions: So far, we have onsidered sinusoidal modulation funtions. In general, any modulation funtion an be deomposed into its Fourier omponents, and the presented analysis applied to eah omponent separately. If the lowest frequeny omponent of the funtion (exept the DC) is higher than the global transport bandlimit, the global at all the non-dc frequenies is zero. Thus, we get the following result: RESULT 1. If the lowest frequeny omponent (exept DC) of the modulation funtion is higher than the global transport bandlimit ω b, the global reeived at the is temporally onstant. For instane, if the emitted I(t) is a square wave with a period more than π, all the frequeny omponents of I(t) are ω b higher thanω b. In this ase, the global reeived at the ontains only the DC omponent, and is temporally onstant. 6. TRANSPORT ROBUST DEPTH RECOVERY Effet of global light transport on depth reovery: C-ToF systems reover depths by measuring the phase φ = ω Γ of the reeived at the, where Γ is the depth. This is illustrated in Figure 6 (a). Depth is omputed from the reovered phase as Γ = φ. Due to global illumination effets suh ω as interrefletions (multi-path interferene) and sattering, a pixel may reeive light along multiple light paths. These paths have different lengths, and hene light reeived along these paths have

8 8 Mohit Gupta et al. different phases as ompared to the diret refletion path. Consequently, the phase of the total (sum of diret and global omponents) is different from the orret phase, as shown in Figure 6 (b). The resulting depth errors are systemati, and an be orders of magnitude larger than the random errors due to noise. 6.1 Miro ToF Imaging We now present our tehnique for mitigating depth errors due to global illumination. The basi idea is simple, and relies on the observation that global transport vanishes at high frequenies (Proposition 1). Let the be illuminated by a light with intensity varying sinusoidally at frequeny ω. In phasor notation, the intensity is given by the -tuple [I DC, I ω ], where I DC is the DC omponent and I ω is the osillating omponent. The diret L d (p) reeived at a pixel p is given by the -tuple: L d (p) = [D(p)I DC, D ω (p) I ω ], (0) where D(p) and D ω (p) are the DC and osillating terms of the diret for a light with unit intensity. Similarly, the global L g (p) is: L g (p) = [G(p)I DC, G ω (p) I ω ]. (1) As shown in the previous setion, ifω > ω b, the osillating term of the global G ω (p) = 0. Then, the total is: L(p) [(D(p)+G(p)) I DC, D ω (p) I ] ω. () Sine the global omponent of the manifests only as a onstant offset, it does not influene the phase. Thus, for high frequeny illumination, the phase of the total is the same as that of the diret, and an be used for aurate depth reovery. This is shown in Figure 6 (). Wrapped phase problem and unambiguous depth range: The phase φ = ω Γ is omputed by using inverse trigonometri funtions (e.g., artan, arsine, arosine) [Payne et al. 010], whih have a range of π. Consequently, the set of depths Γ+n π ω for any integer n will all have the same reovered phase, leading to depth ambiguities. This is alled the wrapped phase problem. It limits the maximum depth range R max in whih depths an be measured unambiguously. R max is inversely proportional to modulation frequeny ω, and is given by R max = π [Lange ω 000]. For example, for ω = π 1500 MHz. 6,R max is only 10 entimeters. While it is possible to unwrap high-frequeny phases using a low-frequeny phase [Jongenelen et al. 010], if there is global illumination, the low frequeny phase is inaurate. This auses unwrapping errors, resulting in erroneous shape. This presents a tradeoff between ahieving a large depth range, and robustness to global illumination. On one hand, higher modulation frequenies are robust to global illumination effets. On the other hand, using high frequenies result in depth ambiguities. How an we measure aurate depths in a large range when only high temporal frequenies are used? Fortunately, it is possible to estimate a low frequeny phase from multiple high frequeny phases. This is a standard problem in interferometry [Gushov and Solodkin 1991; Takeda et al. 1997], strutured light based triangulation [Gupta and Nayar 01] and timeof-flight imaging [Jongenelen et al. 010; Jongenelen et al. 011]. 6 ω is the angular modulation frequeny, whih is π times the modulation frequeny. frequeny phase (wrapped) frequeny phase (wrapped) 0 high frequenies (small periods) depth (unwrapped) Fig. 7. Miro ToF imaging. The proposed Miro ToF imaging tehnique onsists of illuminating the sequentially with multiple high frequeny sinusoids, and omputing phases orresponding to eah of them. Theoretially, two high frequenies are suffiient. If all the frequenies are suffiiently high, global illumination does not introdue errors in the phases. The individual phases have depth ambiguities. Unambiguous depth is reovered by unwrapping the phases, whih an be done either analytially or by building a look-up table. There are both numerial and analytial solutions available whih an be implemented effiiently. Algorithm: The proposed tehnique involves illuminating the sequentially with multiple high frequeny sinusoids, and omputing phases orresponding to eah of them. Sine all the emitted signals have miro (small) periods, the tehnique is alled Miro ToF Imaging. This is illustrated in Figure 7. Let the set of frequenies used be Ω = [ω 1,...,ω F ]. For eah frequeny ω f, 1 f F, the measures the orrelation of the reeived with the exposure funtion R f (t), whih is also a sinusoid of the same frequeny as the emitted light. The phasor representation of R f (t) is R f = R f e jψ f, where R f and ψ f are the magnitude and phase of the exposure funtion. The measured brightness B f (ψ) is a sinusoid as a funtion of ψ f (see Appendix A for derivation): B f (ψ f ) = O f +A f os(φ f ψ f ). (3) B f (ψ f ) is a funtion of three unknowns,φ f,o f anda f. Phases φ f, 1 f F enode the depth, and an be reovered by taking three measurements for eah of the F frequenies, while varying the exposure funtion phase, ψ f = 0, π, 4π. The unambiguous depth Γ an then be omputed from the wrapped phases 3 3 φ f and the frequenies ω f either analytially (using the Gushov- Solodkin (G-S) algorithm [Gushov and Solodkin 1991; Jongenelen et al. 010]) or by building a look-up table: Γ = T (φ 1,ω 1,...,φ F,ω F ). (4) 6. Number of Measurements Sine there are three unknowns (offset, amplitude and phase) for every frequeny (Eq. 3), in general, there are 3F unknowns if F frequenies are used. Thus, Miro ToF imaging with F frequenies requires taking 3F measurements. 0

9 Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging 9 However, if the set of frequenies lie in a narrow band, the offsets and the amplitudes an be assumed to be approximately the same for all the frequenies. In this ase, the number of unknowns is F + ; F phases, one offset and one amplitude. These unknowns an be estimated by taking F + measurements where three measurements are taken for the first frequeny {B 1 (0),B 1 ( π 3 ),B 1( 4π 3 )}, and one measurement is aptured for every subsequent frequeny {B (0),...,B F (0)}. The offset, amplitude and the first phase are omputed from the first three measurements. Using the omputed offset and amplitude, remaining phases are omputed from the remaining measurements. How many frequenies are needed? Theoretially, phases omputed for F = appropriately hosen high frequenies are suffiient for estimating the depths unambiguously in any desired depth range. Thus, sine the number of measurements is F +, we get the following result: RESULT. Four images are theoretially suffiient for transport-robust and unambiguous depth reovery using Miro ToF imaging. In pratie, more frequenies and measurements per frequeny may be needed due to limited dynami range of the, limited frequeny resolution of the light and the and noise. Depending on the brightness, light strength and noise levels, in our simulations and experiments, we use 4 frequenies, and 3 4 images per frequeny. 6.3 Frequeny Seletion for Miro ToF Imaging What frequenies should be used for Miro ToF imaging? For any hoie of F frequenies, a given depth Γ i is enoded with a vetor V i of measured brightness values. The number of elements in V i is equal to the total number of measurements. Ideally, depths and intensity vetors should have a one-to-one mapping and depths an be reovered without error from the measured intensity vetors. However, due to various of noise in the measurements, depth estimations an be erroneous. We define the error funtion E ij = e V i V j between vetors V i and V j. E ij is proportional to the probability of vetor V j being inorretly deoded as Γ i, and vie versa. For a given frequeny set Ω, the mean weighted error funtion E(Ω) is defined as: E(Ω) = Γ i Γ j E ij. (5) Γ i,γ j [0,...,R max] E(Ω) is the average expeted depth error if the frequeny set Ω is used. In order to minimize the depth error, the optimal set of frequenies should minimize the error funtion: Ω = argmin Ω E(Ω), ω f [ω min,ω max ] for 1 f F, (6) where [ω min,ω max ] is the range of values from whih the frequenies are hosen. This is a onstrained F dimensional optimization problem. We used the simplex searh method implemented in MATLAB optimization toolbox for solving this. Note that sine we use a searh based proedure, the omputed frequenies may not be theoretially optimal. In pratie, we have found that the frequeny set omputed by running the method for > 100,000 iterations ahieves stable depth results. 6.4 Simulations In the following, we show depth reovery results using simulations for two different geometries. 4.0m (a) V-groove 3.0m 3.0m (b) Cornell Box Fig. 8. Simulation settings. (a) A v-groove with two planes of size3m 4m eah. The angle between the planes is 70. (b) Cornell box with faes of size 3m 3m. The is at a distane of 4.5 m from the s. Simulation setup: The setups are illustrated in Figure 8. The first is a v-groove, with an apex angle of 70deg. Both faes are retangles of dimension 4 3 meters. The seond is the Cornell box. Eah fae is a square of side 3 meters. The and (o-loated) are 4.5 meters from the s. Simulation of input images: The input images were generated by disretizing the into small pathes and simulating forward phasor light transport. We assumed the s to be Lambertian beause for Lambertian s, the s an be omputed using an analyti losed-form expression. This is similar to onventional imaging where the radiosity equation an be solved in a losed form manner for Lambertian s [O Toole et al. 014]. In all our simulations, the affine noise model [Hasinoff et al. 010] was used for the, with both independent read noise and dependent shot noise added to the aptured images. Simulation parameters and omparisons: For both s, we performed Miro ToF imaging using two frequenies 1063, 1034 MHz., whih were omputed using the frequeny seletion proedure (Setion 6.3). We ompare Miro ToF with two different onventional ToF tehniques. The single frequeny onventional ToF tehnique uses a frequeny of 10 MHz., so that the unambiguous depth range is more than the depths, and no phase unwrapping is required. Sine the depth resolution of ToF tehniques is diretly proportional to the modulation frequeny [Lange 000], this tehnique ahieves low depth resolution. The dual-frequeny onventional ToF tehnique uses one high (1063 MHz.) and one low frequeny (10 MHz.). The high frequeny provides high resolution, and the low frequeny is used for unwrapping. For eah frequeny, four measurements were aptured, orresponding to the exposure funtion phases ψ = 0, π,π, 3π. In order to ompensate for the low SNR ahieved by the single frequeny ToF tehnique, we applied temporal averaging to its input images so that the random depth errors due to noise are approximately the same for all three tehniques. The differene in the results is due to the strutured errors aused by interrefletions. Results: Figures 9 (a-b) show phase maps for the two frequenies used for Miro ToF. Both the phase maps have ambiguities. Figures 9 () show the unwrapped phase map, whih is used to ompute unambiguous depths. Figures 9 (d) show the omparison of depths along horizontal san-lines (shown in ()). In both the onventional tehniques, the low frequeny phase is inaurate, resulting in large depth errors (mean errors of 04 and 07 millimeters

10 10 Mohit Gupta et al. Conventional ToF [1 frequeny] Conventional ToF [ frequenies] Miro ToF Imaging [ frequenies] Ground Truth Conventional ToF [1 frequeny] Conventional ToF [ frequenies] Miro ToF Imaging [ frequenies] Ground Truth 0.5 m 0.93 m (a-b) Reovered phase maps for Miro ToF () Unwrapped phase 3.0 m (d) Reovered Shape Fig. 9. Simulation results for shape reovery using Miro ToF. (a-b) Phase maps for two high frequenies used for Miro ToF. Both the phase maps have ambiguities. () The unwrapped phase map, whih is used to ompute unambiguous depths. (d) Comparison of depths along horizontal san-lines with single frequeny and dual (one high and one low) frequeny onventional ToF tehniques. In both the onventional tehniques, the low frequeny phases are inaurate, resulting in large depth errors (mean errors of > 00 and > 500 millimeters for the v-groove and the Cornell box, respetively). Miro ToF imaging ahieves aurate shape, with mean errors of 6.6 and 3. millimeters for the v-groove and the Cornell box, a two orders of magnitude improvement. Conventional ToF [10 MHz.] Conventional ToF [30 MHz.] Miro ToF [45, 50 MHz.] Miro ToF [160, 170 MHz.] Miro ToF [1034, 1063 MHz.] Ground Truth 6.5 Error Analysis for Depth Computation If ω is less than the global transport bandlimit ωb, the osillating term of the global G~ω (p) may not be zero. This will result in errors in phase reovery (and hene, depth estimation). As derived in Appendix B, the phase error ǫφ is given by (for brevity, we have dropped the argument p): ǫφ = φ aos q D os φ + Grω os φg D + Grω + DGrω os(φ φg ), (7) ~ ω (p)) is the phase and Gr = G ~ ω (p) is the where φg = arg(g ω magnitude of the global at frequeny ω. As ω inreases, the magnitude of the global Grω 0, and thus, ǫφ 0. Figure 10 (a) shows the omparison of shapes reovered for the Cornell-box using different frequenies. Figure 10 (b) shows the mean depth error vs. frequeny. Single frequeny onventional ToF tehnique was used for frequenies less than 30 MHz. Miro ToF tehnique was used for frequenies more than 30 MHz. For eah depth omputation using Miro ToF, two high frequenies were used. Depth errors are plotted for the mean of the two frequenies. As the frequenies are inreased, depth error approahes zero and the reonstruted shape approahes the ground-truth. (a) Reonstrutions for different frequenies 500 depth error (mm) for the v-groove and 534 and 538 millimeters for the Cornell box). Miro ToF imaging ahieves aurate shape, with mean errors of 6.6 and 3. millimeters for the v-groove and the Cornell box, respetively, a two orders of magnitude improvement over onventional tehniques (b) Mean error vs. frequeny Fig. 10. Effet of modulation frequeny on shape reovery. (a) Shapes reovered using different frequenies, for single frequeny onventional and Miro ToF imaging (using two frequenies). As the frequenies are inreased, reonstruted shape approahes the ground-truth. (b) Mean depth errors vs. frequeny. Single frequeny onventional ToF tehnique was used for frequenies less than 30 MHz. Miro ToF tehnique was used for frequenies more than 30 MHz. For Miro ToF, two frequenies were used. Depth errors are plotted for the mean of the two frequenies. 7. FAST SEPARATION OF DIRECT AND GLOBAL IMAGE COMPONENTS In this setion, we present a tehnique for separating the diret and global light transport omponents. The tehnique requires apturing as few as three measurements at a single high frequeny. The measurements needed for the separation algorithm are a subset of the measurements taken for depth reovery (Setion 6). Hene, separation an be ahieved as a by-produt of depth estimation modulation frequeny ሺ ͳͳͳ Ǥ ሻ Separation Algorithm If the is illuminated with sinusoidally varying intensity, the diret and global, as well as the total are all sinu-

11 Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging 11 L time total diret global (a) Sensor Inident Radiane Versus Time pixel brightness B referene funtion phase amplitude offset (b) Pixel Brightness Versus Referene Funtion Phase amplitude diret (a) Sene image (b) Diret omponent () Global omponent aptured images offset () Steps of Diret-Global Separation Algorithm global Fig. 11. Diret-global separation algorithm using high frequeny illumination. (a) If the is illuminated with high-frequeny illumination, the global is temporally onstant. (b) The brightness B measured by the is a sinusoid as a funtion of the phase ψ of the exposure funtion. Sine the global is temporally onstant, the amplitude of the sinusoid depends only on the diret. () The diret and global omponents an be omputed by measuring the offset and the amplitude of the sinusoid B(ψ) at every pixel. soids of the same frequeny. In general, it is diffiult to separate the diret and global omponents from the total due to the inherent ambiguity; a given sinusoid an be expressed as sum of two sinusoids of the same frequeny in infinitely many ways. However, reall from Eq. that for high frequeny illumination, the global manifests only as a DC offset. This forms the basis of our diret-global separation approah, and is illustrated in Figure 11 (a). Our goal is to separately reover D(p) and G(p), whih are the diret and global omponents resulting from a light with temporally onstant, unit intensity. Let the exposure funtion be a sinusoid that is represented by the -tuple R DC, R ω. Let ψ = arg( R ω ) be the phase of the exposure funtion. The orrelation measurement B(ψ) reorded at the (see Appendix A for derivation) is given by: B(ψ) = τ(d+g)r DC I DC }{{} offset O +τ DR ωi ω }{{} amplitude A os(φ ψ), (8) where φ = arg( D ω (p)) is the phase of the diret, and τ is the integration time. B(ψ) is a sinusoid with three parameters, the offset O = τ(d + G)R DC I DC, the amplitude A = τ DRωIω and the phase φ, as shown in Figure 11 (b). The three parameters an be reovered by taking three orrelation measurements. Sine the onstants τ,r ω,i ω,r DC,I DC are known, the diret and global omponents are reovered from the estimated offset and the amplitude, as shown in Figure 11 (): D = A O, G = D. (9) τr ω I ω τr DC I DC Diret Component Global Component Fig. 1. Simulation results for diret-global separation. (a) Sene image. (b) Diret and () global omponents for the v-groove and the Cornellbox s, omputed using the algorithm in Setion 7. Notie the olorbleeding between different planes in the global omponent due to interrefletions, and the diret omponent dereasing with inreasing depth due to intensity fall-off. (a) ω = 10 MHz. (b) ω = 15 MHz. () ω = 5.3 GHz. (d) Ground Truth Fig. 13. Effet of modulation frequeny on diret-global separation. (a) For low frequenies (e.g., 10 MHz.), the diret omponent is overestimated and the global omponent is underestimated. (b) For 15 MHz., the separation is qualitatively orret, but ringing artifats are notieable, speially around the edges of the ube. () If the frequeny is higher than the bandwidth of the global response, aurate separation is ahieved. (d) Ground-truth separation. RESULT 3. Three images aptured at a single high frequeny are theoretially suffiient for separating the diret and global omponents of light transport. In pratie, more measurements may need to be aptured if the measurement noise is high. In our simulations and experiments, we use 3 4 measurements. 7. Simulations Figure 1 shows the diret and global omponents estimated for the two simulated s used in the previous setion. Four measurements were taken for eah of the examples. For the v-groove, notie that the global omponent is high near the orner, and dereases away from the orner. In the global omponent of the Cornell box, notie the olor bleeding around the edges due to interrefletions.

12 1 Mohit Gupta et al. 7.3 Error Analysis for Diret-Global Separation Similar to the error analysis for depth estimation (Setion 6.5), ω being less than the global transport bandlimit ω b may result in erroneous diret-global separation. As derived in Appendix B, the estimation errors ǫ D and ǫ G for diret and global omponents, respetively, are given by: ǫ D = D ǫ G = D +G r ω +DG r ω os(φ φ G), (30) D +G r ω +DG r ω os(φ φ G) D, (31) light (bank of laser diodes) (PMD CamBoard Nano) where φ G = arg( G ω (p)) is the phase and G r ω = G ω (p) is the magnitude of the global at frequeny ω. As ω inreases, G r ω 0, and thus, ǫ D,ǫ G 0. Figure 13 shows the effet of modulation frequeny on the results of diret-global separation. If a low modulation frequeny (10 MHz.) is used, the diret omponent is over-estimated and the global omponent is under-estimated. As frequeny inreases, the global dereases, and the separation auray inreases. For 15 MHz., the estimation errors are lower, but the resulting images have ringing artifats. At5300 MHz., the result is lose to the ground truth. 8. HARDWARE PROTOTYPE AND RESULTS Our hardware prototype is based on the PMDTehnologies Cam- Board Nano, a low-ost ommerially available C-ToF imaging system. It is shown in Figure 14. In order to operate the system at various modulation frequenies, we used an external signal generator to provide the modulation signal instead of the on-board signal generator. Our light is an array of 650 nm laser diodes 7, driven using an ic-haus onstant urrent driver. With this setup, we an ahieve a maximum modulation frequeny of 15 MHz. The modulation signals ahieved by our setup are nearly sinusoids. As disussed in Setion 5 though, this is not a strit requirement. Results of depth reovery using Miro ToF imaging: The onsists of a fixed wall and a movable wall arranged so that they form a onave v-groove, as illustrated in Figure 15 (a). The onave shape produes interrefletions between the two walls. The amount of interrefletions depends on the apex angle Υ, whih an be hanged by moving the right wall. Both walls are made of white, nearly diffuse material. The size of the walls is approximately m m eah, and the is plaed at a distane of 5m from the orner of the groove. For Miro ToF imaging, we use two high frequenies of 15 and 108 MHz, omputed using the frequeny seletion proedure 8. We ompare the results of Miro ToF with the single frequeny onventional ToF tehnique. The frequeny is hosen so that the unambiguous depth range is larger than the depths. In order to ompensate for the low SNR ahieved by the onventional ToF tehnique, we applied averaging to its input images so that the random perturbations due to noise are similar for both tehniques 7 The size of the array is signifiantly smaller than the modulation wavelength. Hene, the array of diodes is assumed to be a single light. 8 For frequeny seletion, we set ω max = 15 MHz., the maximum frequeny ahievable by the imaging hardware, and ω min to be 0.8ω max. Although setting ω min to an even higher value may ahieve more robustness to global illumination, in pratie, the differene ω max ω min needs to be above a threshold due to the limited intensity resolution and dynami range of the and the light. Fig. 14. Image aquisition setup. Our hardware prototype is based on the PMDTehnologies CamBoard Nano, a C-ToF. An array of 650 nm laser diodes ats as the light. (onventional and Miro ToF). The differene in the results is due to the strutured errors aused by interrefletions. Depth omputed using the onventional ToF tehnique has mean errors of87,70 and 57 millimeters, for Υ = 45, Υ = 60 and Υ = 90, respetively. Miro ToF ahieves reonstrutions that have 1 orders of magnitude lower errors (mean errors of.8, 6.7 and 6. millimeters). Figure 16 shows the performane of onventional and Miro ToF tehniques as a funtion of the modulation frequeny. For onventional ToF, we performed reonstrutions using a single frequeny in the range of 1 5 MHz. For Miro ToF, we performed reonstrutions using two frequenies in the range[ω, ω+] MHz. for 5 < ω < 10 MHz. The ground truth was ahieved by measuring the distanes using a measuring tape. We ompute the reonstruted apex angle by fitting two planes to the reonstruted shape, and omputing the angle between them. As the frequeny inreases, the reonstrution error dereases and the apex angle approahes the ground truth. For frequenies higher than 100 MHz., the reonstrution error is less than 5 millimeters and the apex angle is within degrees of the ground truth. Results of diret-global separation: Figure 17 shows the diretglobal separation results for the v-grooves of different apex angles. The modulation frequeny used was 14 MHz. Three images were used for the separation in eah ase. As the apex angle inreases (from top to bottom), the amount of interrefletions redue, and the global omponent dereases. 9. DISCUSSION AND LIMITATIONS In this paper, we proposed phasor imaging, a tool for light transport analysis in C-ToF imaging, whih an inspire novel imaging tehniques in the future. Using this framework, we studied the (temporal) frequeny dependene of light transport, and showed that global transport vanishes at high modulation frequenies. Based on this observation, we present tehniques for transport-robust shape reovery and for separation of diret and global omponents. Sine the presented tehniques require few images and have low omputational ost, we believe they an be inorporated into future ToF imaging systems. In the following, we disuss the limitations of our tehniques. Sope and limitations: Our tehniques assume that the global light transport is temporally smooth. While this assumption holds for a broad range of s, for s with high-frequeny light transport suh as mirror interrefletions, the presented tehniques are prone to errors. For suh s, shape reovery tehniques that

13 fixed wall Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging ȯ.0 meters fixed wall 13 ȯ movable wall (a) Illustration and photo of the imaging setup (b) V-groove (Υ = 45 ) () V-groove (Υ = 60 ) (d) V-groove (Υ = 90 ) ground truth Miro ToF 45o Υ = 45 onventional ToF onave edge (e) High frequeny phase maps used for Miro ToF imaging (f) Reonstruted shape (g) Shape omparison ground truth Υ = 60 Miro ToF 60o onventional ToF onave edge (h) High frequeny phase maps used for Miro ToF imaging (i) Reonstruted shape (j) Shape omparison ground truth Υ = 90 Miro ToF onventional ToF 90o onave edge (k) High frequeny phase maps used for Miro ToF imaging (l) Reonstruted shape (m) Shape omparison Fig. 15. Experimental results of depth estimation on v-groove s. (a) The onsists of a fixed wall and a movable wall arranged at an angle so that they form a v-groove. The apex angle Υ an be hanged by moving the right wall. The size of the walls is approximately m m eah, and the is plaed at a distane of 5 m from the orner. (b-d) Images aptured by the PMD, for the v-groove in three different onfigurations, Υ = 45, Υ = 60 and Υ = 90. For Miro ToF imaging, we use two high frequenies of 15 and 108 MHz. (e, h, k) Reovered phase maps for the two frequenies used in Miro ToF. All the phase maps have ambiguities. (f, i, l) The unambiguous shapes reonstruted by unwrapping the two high frequeny phases. (g, j, m) Comparison of shapes reonstruted using Miro ToF and onventional ToF tehniques. Shape omputed using the onventional ToF tehnique has mean errors of 87, 70 and 57 millimeters, for Υ = 45, Υ = 60 and Υ = 90, respetively. Miro ToF ahieves reonstrutions that have 1 orders of magnitude lower errors (mean errors of.8, 6.7 and 6. millimeters). assume temporally sparse light transport are better suited [Godbaz et al. 008; Dorrington et al. 011; Kirmani et al. 013]. Our diret-global separation tehnique an separate the diret from relatively low-frequeny global, by apturing only three images. Tehniques whih an separate both low and

14 14 Mohit Gupta et al. depth error (mm) modulation frequeny (MHz.) (a) Depth error reonstruted angle (degrees) modulation frequeny (MHz.) (b) Reonstruted apex angle Fig. 16. Auray of shape reovery vs. modulation frequeny for the v-groove s. (a) Mean reonstrution error vs. the modulation frequeny. (b) Reonstruted apex angle vs. the modulation frequeny. The apex angle was omputed by fitting two planes to the reonstruted shape, and omputing the angle between them. For frequenies higher than 100 MHz., the reonstrution error is less than 5 millimeters and the apex angle is within degrees of the ground truth. Diret Component there are s and lasers that an ahieve GHz. frequenies, they are expensive and require large aquisition times [Kirmani et al. 009; Velten et al. 013]. LEDs and PMDs are low ost s and s that an ahieve high SNR in real time, but due to various pratial onsiderations suh as power requirement, urrent devies are limited to approximately 150 MHz. With these devies, our tehniques are restrited to large sale (room-size) s with relatively smooth refletane. Future outlook on hardware devies: Fortunately, high frequeny LEDs have been atively researhed with the goal of ahieving high bandwidth optial ommuniation networks. Reently, several researh groups have demonstrated LEDs that an ahieve modulation frequenies of multiple GHz. [Akbulut et al. 001; Chen et al. 1999; Walter et al. 009; Heinen et al. 1976; Wu et al. 010], with a low power requirement. On the other hand, a new kind of PMD based on MSM-tehnology (metal-semiondutor-metal) has reently been proposed that an potentially ahieve > 10 GHz. modulation frequenies [Buxbaum et al. 00; Shwarte 004]. With these advanes, it would be possible to apply the proposed tehniques to a muh larger lass of s - s at entimeter/millimeter sale and omprising a broad range of refletane properties. An additional motivation for ahieving higher frequenies is that the depth resolution ahieved by ToF s is proportional to the frequeny used [Lange 000]. Thus, higher frequenies an inrease both the depth resolution and auray of ToF based depth sensing systems. Generalization of light s: Although the analysis and derivations were performed for a single point light, the imaging framework proposed in the paper is appliable to extended light s, arrays of light s with eah potentially having a different phase and amplitude, or spatially modulated s for performing spatio-temporal analysis of light transport [O Toole et al. 014]. Global Component Υ = 45 Υ = 60 Υ = 90 Fig. 17. Results of diret-global separation for the v-groove s. The diret and global omponents omputed for the v-grooves. The modulation frequeny used was14 MHz. As the apex angle inreases (from left to right), the global omponent dereases. high frequeny global transport [O Toole et al. 014], albeit by apturing more images, an be used for s with austis and speular interrefletions. It may be possible to develop hybrid dependent algorithms (for both shape reovery and light transport analysis) where harateristis (low or high frequeny light transport) determine the reonstrution tehnique to be used. This forms a promising diretion of future work. From a pratial standpoint, for the proposed tehniques to ahieve aurate results, the modulation frequenies ahieved by the system should be higher than the global transport bandlimit of the. The higher the modulation frequeny, the larger the range of s (in terms of geometri sale and material properties) on whih the proposed tehniques are appliable. Although APPENDIX A. SENSOR CORRELATION MEASUREMENT Let the s exposure funtion be R os(ωt ψ) (phasor representation: R = Re jψ ). Let the inident belos(ωt φ) (phasor representation: L = Le jφ ). The measured brightness B ω is a funtion of ψ, and is given by the orrelation between the exposure funtion and inident : B ω (ψ) = τ 0 (Ros(ωt ψ))(los(ωt φ))dt = τ LR os(φ ψ), (3) where τ is the integration time, and is assumed to be a integral multiple of the modulation period π. ω DC omponent: If both the and the exposure funtion have a DC omponent as well, the measured brightness also has a DC omponent. Let the DC omponents of the exposure funtion and the be R DC and L DC, respetively. The DC omponent of the measured brightness is given as: B DC = τ 0 R DC L DC dt = τl DC R DC. (33) The total brightness is the sum of B ω andb DC :