arxiv: v1 [cs.cv] 20 Jan 2015

Transcription

1 A Light Transport Model for Mitigating Multipath Interferene in TOF Sensors Nikhil Naik 1,2 Ahuta Kadambi 1 Christoph Rhemann 2 Shahram Izadi 2 Ramesh Raskar 1 Sing Bing Kang 2 1 MIT Media Lab, 2 Mirosoft Researh arxiv: v1 [s.cv] 20 Jan 2015 Abstrat Continuous-wave Time-of-flight (TOF) range imaging has beome a ommerially viable tehnology with many appliations in omputer vision and graphis. However, the depth images obtained from TOF ameras ontain sene dependent errors due to multipath interferene (MPI). Speifially, MPI ours when multiple optial refletions return to a single spatial loation on the imaging sensor. Many prior approahes to retifying MPI rely on sparsity in optial refletions, whih is an extreme simplifiation. In this paper, we orret MPI by ombining the standard measurements from a TOF amera with information from diret and global light transport. We report results on both simulated experiments and physial experiments (using the Kinet sensor). Our results, evaluated against ground truth, demonstrate a quantitative improvement in depth auray. 1. Introdution Time-of-flight (TOF) ameras represent an inreasingly popular method to obtain depth maps at full framerates of 30 FPS. The reent release of the TOF-based Kinet sensor [44] will result in the wide-spread adaption of TOF ameras for various traditional and non-traditional (e.g., [23, 31, 24]) appliations in omputer vision. In this paper we restrit our disussion to ontinuous-wave TOF ameras, whih alulate depth by measuring the phase differene between an emitted and reeived optial signal. In Setion 3 we provide an overview of the tehnology and a more extensive overview an be be found in [6, 21]. Even though TOF ameras represent an attrative tehnology for generating depth images in real-time, they suffer from many soures of error [8]. We fous on perhaps the main soure of error: the problem of multi-path interferene (MPI). The MPI problem ours when multiple refletions of light in a sene return to a single pixel on the amera sensor. This problem is partiularly signifiant for TOF sensors, where depth alulations assume a single optial refletion. (a) No MPI b) MPI: 2 boune () MPI: Subsurfae Sattering Figure 1. Multipath interferene in time of flight ameras. In (a) the typial operation of TOF ameras is shown. A single optial path returns to the image sensor. In (b) an indiret boune from point q to p auses two optial paths to return to the image sensor. In (), subsurfae sattering auses many optial paths to return to the sensor (one possible path is shown). Figure 1 illustrates three different measurement senarios for a TOF sensor. In Figure 1a, the standard operation is shown, where a single refletion from point p strikes the image sensor. Figure 1b and Figure 1 illustrate ases of MPI where multiple optial refletions strike the imaging sensor. The former onsiders a simple two-boune refletion and the latter subsurfae sattering, whih annot be desribed by a disrete model. Most prior MPI orretion algorithms assume sparsity in optial refletions. Sine sparsity is a weak approximation for the omplexity of sattering, the spae of sparse solutions has, to date, been restrited to toy senes [2, 9, 10, 7]. Other tehniques rely on optial omputing, whih involves signifiant optial and eletroni modifiations to the amera [38, 32]. In this paper we ast the problem of MPI into the realm of diret-global illumination. Reall that diret illumination represents optial paths from soure, to sene point to amera (Figure 1a). In ontrast, global illumination represents optial paths that reah the sene point through indiret means (Figure 1b and Figure 1). The rux of this paper 1

2 Paper Multipath Type Solution Type Quantitative Analysis Hardware Modifiations Fuhs [11] Continuous Iterative Limited None Dorrington et al. [7] 2-sparse Iterative Limited Frequeny sweep Godbaz et al. [12] 2-sparse Closed-form None Frequeny sweep Kadambi et al. [32] K-sparse Iterative None Custom ode Kirmani et al. [33] K-sparse Iterative Simulations only Frequeny sweep Freedman et al. [10] K-sparse Iterative Simulations only None Jiminez et al. [29] K-sparse Iterative Limited None O Toole et al. [38] Continuous None None Extensive Gupta et al. [19] Continuous Closed-form Yes Extensive This Paper Continuous Closed-form Yes External projetor Table 1. This paper proposes a losed form approah to orreting multipath, by relying on light transport information. is as follows: we develop a losed-form solution that ombines the measurements from standard TOF sensors with diret and global radiane maps (f. Nayar et al. [36]) Contributions Our key ontribution is a novel image formation model for TOF sensors that inorporates diret-global separation into phase and amplitude alulations. We use this model to signifiantly redue errors aused by multipath interferene. Another important ontribution is our quantitative evaluation of the benefit of our tehnique using ground truth data. Our approah uses both the diret and global omponents of radiane to modify the traditional depth alulation in time of flight sensing. Diret and global omponents an be aquired with the use of a projetor [36]. For omparison, O Toole et al. [38] use an involved, optial setup to filter out the sattered radiane. In addition, where most previous methods provide limited or no quantitative analysis of their results [11, 12, 32, 38], we report errors for all senes in the paper by evaluating our tehnique against ground truth results (e.g., from an aurate laser sanner). 2. Related Work 2.1. Multipath Interferene Corretion We summarize the important related publiations on MPI orretion in table 1. Fuhs [11] and Jiminez et al [29] solve for the MPI-free phase using an optimization problem and a forward model for TOF imaging. Dorrington et al. [7], Godbaz et al. [12] and Kirmani et al. [33] model MPI using two boune approximation and propopse iterative or losed form solutions using multiple frequeny measurements (or frequeny sweep ). These solutions are tied to the sparsity assumption, whih leads to signifiant error when sene returns are not sparse (e.g., subsurfae sattering). Kadambi et al. [32] introdue a oded illumination tehnique for TOF imaging and demonstrate results for MPI orretion on senes ontaining transluent objets. This tehnique, while promising, requires modifiations in hardware, and inludes hardware limitations that prelude real-time performane. More reently, Freedman et al. [10] and Bhandari et al. [3, 2] propose iterative solutions for generalized MPI using multiple frequeny measurements. A limitation of The Freedman and Bhandari tehniques is that they are onstrained to K-sparse models, where K is a finite quantity. Suh models are not appliable to many real-world senes (e.g., subsurfae sattering). We now turn to an emerging researh trend, whih inorporates light transport information to orret for MPI. Our work is of this flavor and is losely related to two reent papers by O Toole et al. [38] and Gupta et al. [19]. Building on [39], O Toole et al. [38] propose a hardware solution to blok the global omponent of light transport during the apture proess, thus eliminating MPI. O Toole s optial omputing approah is aurate, but requires extensive optial modifiations. The work by Gupta et al. [19] enters on a theoretial justifiation of temporal light transport. Speifially, Gupta shows that the global light transport vanishes at high frequenies. In related work, Gupta et al. [18, 15] has also takled the problem of strutured light sanning in presene of global illumination using oded illumination patterns. For a more omplete overview, we enourage the reader to review the work by O Toole et al. and Gupta et al Light Transport Aquisition Sine our work ties light transport aquisition to MPI orretion, we provide a brief overview of this work. Seitz et al. [43] introdued the problem of inverse light transport, where the goal is to deompose an image into a sum of n-boune images. Nayar et al. [36] propose a fast method for separating the diret omponent of light transport from the global omponent using high frequeny illumination patterns. Ng et al. [37] introdue a stratified light transport matrix whih allows omputation of the inverse light transport using matrix multipliations. Chandraker et al. [5] treat inverse light transport aquisition as a dual of forward rendering and propose an algorithm that needs

3 only matrix-vetor multipliation. Reddy et al. [42] introdue a method to deompose the light transport in a sene into diret (zero-th boune), near- range (subsurfae sattering and loal inter-refletions) and far- range (diffuse interrefletions) omponents. An exiting approah by O Toole et al. [40] exploits oaxial optial setups to separate distint bounes of optial paths. Light transport has been suessfully applied to a variety of pratial problems, inluding rendering of omplex phenomena [16, 48], imaging in poor visibility environments [17], and reovery of surfae refletane [26, 28, 27]. In addition, light transport tehniques have been very suessful for the hallening problem of 3-D sanning of transluent objets [25, 20] and high quality strutured light sanning [15, 14, 34, 13, 35]. While there are several light transport tehniques to onsider, we inorporate the method by Nayar et al. [36]. We believe it remains the simplest method to aquire and separate the diret and global omponents of light transport; it requires few images (2 in theory) and, importantly, no alibration. 3. Continuous Wave TOF Measurements A ontinuous-wave TOF amera (hereafter, TOF amera) is able to measure the phase delay of optial paths and obtain depth through the following relation: z = ϕ 4πf M, (1) where f M is the modulation frequeny of the amera and is the speed of light. To estimate ϕ with high preision, a TOF amera ontains an ative illumination soure that is strobed aording to a periodi illumination signal. In this paper we onsider standard implementations (e.g. MS Kinet) where the emitted signal takes the form of a sinusoid g(t) = os (f M t). (2) Suppose the light is refleted from a sene point p. Then at the sensor plane, the reeived optial signal an be written as s(t; p) = α p os (f M t + ϕ p ) + β p, (3) where α p is the attenuation in the projeted radiane and β p is the intensity of ambient light. Note the time-invariane of parameters α p, ϕ p, and β p it is assumed that they are onstant within the exposure time. A TOF amera omputes the ross-orrelation of the emitted and reeived signals, whih evaluates to (τ; p) = s(t; p) g(t) = αp 2 os (f Mτ + ϕ p ) + β p. (4) Note that the same information from Equation 3 amplitude, phase, and ambient light is present in the ross-orrelation funtion. However, the ross-orrelation equation is in the domain of τ, whih is easier to sample (in fat, the TOF sensor diretly reads (τ; p) from the sensor). As our tehnique does not inorporate spatial parametrization, hereafter we onsider (τ) as the reeived signal. In order to reover the phase and amplitude from the reeived signal, the four buket priniple is typially used (see [7, 2]). In this method, the TOF amera evenly samples (τ) four times over the length of a period, for instane, τ = [0, π 2, π, 3π 2 ]T. Then the alulated phase an be written as ( ) (τ4 ) (τ 2 ) ϕ = artan, (5) (τ 1 ) (τ 3 ) and alulated amplitude as α = 1 ( (τ 4 ) (τ 2 )) 2 ( (τ 1 ) (τ 3 )) 2. (6) 2 This method is a losed-form tehnique in the absene of fators suh as multipath interferene and sensor noise, the alulated variables ϕ and α are equal to ground truth values ϕ and α. 4. Multipath Interferene To alulate aurate depth it is neessary to obtain the time of flight of the diret light path. However, the light reeived from a TOF sensor is a ombination of diret and global illumination [36]. Multipath interferene refers to the undesirable mixing of global illumination with diret illumination, whih, in a TOF amera, auses a deviation in the measured phase wrt. the diret phase only. We now provide the mathematial intuition behind MPI, starting with the most general ase Generalized MPI In the presene of global illumination, the reeived signal inludes ontributions from illumination refleted from multiple sene points. Speifially, we denote illumination that has refleted from K loations in the sene as a K- boune sinusoid with phase ϕ K and amplitude α K. In this ontext, the reeived signal is a omposite of multi-boune sinusoids and an be written as (τ) = α K 0 2 os (f α i Mτ + ϕ 0 ) + }{{} 2 os (f Mτ + ϕ i ) +β, i=1 }{{} Diret Contribution Global Contribution (7) where the problem is not onstrained to finite-dimensions, i.e., K = is possible. Substituting Equation 7 into Equations 5 and 6 results in the following expressions for the measured phase and amplitude ϕ = artan ( α 0 sin ϕ 0 + ) K i=1 α i sin ϕ i α 0 os ϕ 0 + K i=1 α, (8) i os ϕ i

4 K K K α 2 = α0 2 + αi α i α j os (ϕ i ϕ j ). (9) i=1 i=0 j=0 } {{ } i j Not suprisingly, the measured phase ϕ is not equal to the phase ontribution from the diret boune ϕ 0. As a result, the measured depth will be distorted Approximating Global Illumination In this paper, we approximate global illumination using only the lowest order, indiret boune (i.e., K = 1). This simplifiation has been exploited in previous tehniques for multipath interferene orretion (e.g., [11, 7, 12, 33]). Under this assumption, the reeived signal an be modelled as (τ) = α D 2 os (f Mτ + ϕ D ) + α G }{{} 2 os (f Mτ + ϕ G ) +β, }{{} Diret Approximate Global (10) where subsripts D and G expliitly denote parameters of diret and approximate global illumination. The total projeted radiane α T is equal to α D + α G, where 0 α D, α G, 1 and the total radiane is equal to unity. After substituting the measurement from Equation 10 into Equations 5 and 6, we obtain ( ) αd sin ϕ D + α G sin ϕ G ϕ = artan, (11) α D os ϕ D + α G os ϕ G α 2 = α 2 D + α 2 G + 2α D α G os (ϕ D ϕ G ). (12) 4.3. Diret-Global Separation for MPI Corretion Aurately estimating ϕ D would make it possible to ameliorate MPI and obtain a robust measurement of depth. Reall that the fast method from Nayar et al. [36] an be used to obtain the radiane of diret and global illumination. For instane, in the ase of generalized MPI (Setion 4.1), diret-global separation allows us to obtain radiane values α D and K i=1 α i. However, these two measurements are not helpful in the ontext of general MPI, where Equations 8 and 9 onsist of 2N+2 variables, of whih only α 0 is known. We now turn to the ase of approximate global illumination, where we use the lowest-order indiret boune to approximate ontributions from global illumination. Under this approximation, diret-global separation returns values for α D and α G. Notie that between Equations 16 and 12 there are four variables, two of whih are provided by the Nayar method. Given that there are two equations and only two unknown variables, the diret phase an be omputed in losed-form as: ( ) αd γ + α G (sin (ϕ D ϕ G ) + γ os (ϕ D ϕ G )) ϕ D = artan, α D + α G (os (ϕ D ϕ G ) γ sin (ϕ D ϕ G )) (13) RGB TOF IR Soure Kinet Projetor Kinet Projetor Sene Figure 2. We demonstrate our method with no hardware modifiations to the TOF hardware. Our apture setup (left) onsists of a Kinet One sensor and a DLP Lightommander Infrared projetor. The apture setup is plaed 1 meter away from the sene (right). where γ = tan ( ϕ). In summary, relaxing the problem to the seond boune approximation provides a losed-form solution for ameliorating multipath interferene. In the remainder of the paper, we evaluate this losed-form orretion through simulations and aptured data. 5. Implementation We first desribe our simulation setup, followed by details on the physial setup Simulations To perform simulations, we use the time of flight simulator desribed by Gupta et al. [19]. Given ertain sene parameters, for instane, amera and sene speifiations, the simulator outputs the raw measurements from a TOF amera in the presene of sattering. 1 Following from Setion 3, we obtain four phase samples of the ross-orrelation (f. Equation 4), whih are then used to obtain amplitude and phase images, derived using Equations 5 and 6. Reall that our proposed algorithm requires both the diret and global omponents of light transport. To obtain suh values in simulation we plae onstraints on the number of bounes. Speifially, we follow the onvention from [19], where diret and global simulations onstrain the maximum number of bounes to zero and four, respetively. The ground truth depth map is obtained by applying, in order, Equation 5 and then Equation 1, to the measurements from diret illumination only. To establish the simulated experiments as a reasonable proxy for Kinet measurements, we inorporate the noise model desribed by Hasinoff et al [22]. In partiular, we onsider two types of additive noise: (i) sene-dependent shot noise, and (ii) sene-independent read-out noise Physial Implementation All physial experiments are performed using the Mirosoft Kinet One sensor, whih provides amplitude and 1 Speifially, we use the following amera parameters: a spatial resolution of pixels, a foal length of 368 pixels and an illumination frequeny of 120 MHz.

5 True Shape Measured Shape Correted Shape Error: Measured Error: Correted 25 mm Error = 16.1 mm Error = 3.3 mm 0 mm 100 mm Error = 57.3 mm Error = 16.2 mm 0 mm 20 mm Error = 4.8 mm Error = 1.9 mm 0 mm Figure 3. Our MPI orretion algorithm redues the root mean squared error on a variety of objets (simulated experiments). The first olumn shows the ground truth point loud for a given objet. The seond olumn depits the measured point loud, simulated using rendering software. The output of our algorithm is shown in the third olumn. Finally, error omparisons with ground truth are shown in the fourth and fifth olumns. phase measurements (we use a modulation frequeny of 120 MHz). To separate the diret and global omponents of light transport we follow the Nayar method [36], whih requires high-frequeny oded illumination. Instead of modifying the Kinet One to projet suh patterns, we use an external projetor. Sine the Kinet sensor has an IR-pass, VIS-blok filter, we use the DLP Lightommander projetor, whih an projet patterns at near infrared wavelengths (850 nm). 2 Figure 2 summarizes the experimental setup. We must also remark that the internal light soure of the Kinet One sensor is muh brighter ( 8 times) than the external projetor. Therefore, it is neessary to perform radiometri alibration. To enourage reproduibility, further details on the apture proess (inluding radiometri alibration and details on projeted patterns) an be found in supplemental material. Ground Truth Datasets: A key ontribution of this paper lies in the evaluation of our tehnique in the ontext of ground truth datasets. To apture ground truth for the orner sene in Figure 4, we use the Kinet sensor to apture eah 2 Projetor website: planar omponent of the orner separately. Then, we fit a plane through eah point loud using least-squares, whih are then fused together to obtain the ground truth 3D model. For omplex senes, as in Figure 6, we obtain ground truth by apturing a san of the objet using a multi-stripe, triangulation, laser sanner Experimental Results We divide our results into simulations and physial experiments. For all senes, we evaluate our orreted depth maps using ground truth data. Following standard protool (e.g., [45, 46, 47]), we report errors using the root-meansquared error (RMSE) metri. This metri provides less weight to outliers Simulations To validate the algorithm in a ontrolled setting we perform simulated experiments. Ground-truth 3-D models obtained from an online database are illustrated in the first olumn of Figure 3. In partiular we selet three Lamber- 3 Laser Sanner Website:

6 Figure 4. Correting MPI in senes with orners (physial experiments). Given an input sene ontaining a orner, we use the Nayar method to separate out diret and global omponents (zoom rop shown). Our orretion algorithm shows a redution in root mean squared error for orners of varying angles. For eah angle, we plot a ross-setion of the reonstrution. The dashed line represents ground truth, while the red and magenta lines represent measured and orreted ross-setions. tian objets: a 90 degree orner, a onave bowl, and an eigenfuntion of the wave equation (i.e. the Matlab logo). To obtain the measurements, we use a raytraing approah, as detailed in Setion 5.1. The measurements are shown in the seond olumn of Figure 3, and the error aused by MPI an be observed. In partiular, MPI is most signifiant for the orner and bowl senes. Both senes enourage multipath interferene in the form of interrefletions between the planes (for the orner) and inside the onave region (for the bowl). To summarize the 3-D measurements: the orner is now rounded, the bowl has been drastially flattened, and the features of the Matlab logo are slightly less distintive. The third olumn of Figure 3 shows the point loud after applying the proposed MPI-orretion algorithm. Notably, the orreted orner regains part of its original sharpness, and the bowl returns to its original geometry. Quantitative results are shown in the final two olumns of Figure 3. In all ases the orreted RMSE is lower than the measured RMSE. The simulated results also reveal a drawbak of our proposed tehnique: the MPI-orreted results are noisy, jagged, and overall lak the smoothness of the measurements (ompare olumns two and three of Figure 3). However, we believe the MPI-orreted results are preferable. First, the overall error is redued after MPI orretion. Moreover, errors from additive noise an be orreted by using smoothing filters, already a ommon element of the depth sensing pipeline (see supplement for a smoothed result). For omparison, MPI errors annot be solved by simple filtering (e.g., the bowl in Figure 3). In summary, the simulated results suggest that a proper physial implementation of our tehnique should redue the RMSE on real senes Experiments with Real Senes The results from physial experiments losely mirror the simulations, in the sense that the MPI-orreted results redue the error in depth maps for eah sene we tested. We now desribe results for three separate senes that exhibit different forms of MPI. Speifially, this inludes MPI from interrefletions, subsurfae sattering, and geometri surfae variations. Corner Sene: Figure 4 illustrates a simple orner sene, where interrefletions ontribute to MPI. Following from our method, we first separate the original sene radiane into diret and global omponents. Note that, as expeted, the diret omponent is brightest at the enter of the orner and at the outer edges, whih is where the global omponent is at a minimum. To make the problem a bit more interesting, we apply our orretion algorithm on orners with different angles. As illustrated in Figure 4, at all angles, the measured depth is overestimated and the measured profile laks sharpness. After applying our orretion algorithm, at all angles, we report redued error in the depth map (see Figure 4). For all angles, the orreted shape is most aurate at the enter and the outer edges of the orner, sine the global omponent is muh weaker as ompared to the diret omponent. Our proposed tehnique works best on wide orners (with angles greater than 60 degrees). Note that the orner problem beomes harder as the angle beomes inreasingly aute due to inreasing amounts of global illumination. Subsurfae Sattering Sene: As illustrated in Figure 5, this test sene onsists of a blok of wax plaed in front of a ardboard bakground. The wax material exhibits strong subsurfae sattering, and therefore the expeted MPI is of the form desribed earlier in Figure 1. Note that subsurfae multipath is a ontinuous phenomena it is not possible to provide a finite K for the number of optial refletions [41]. After separation with the Nayar method, we note that (as expeted), the wax is signifiantly brighter than the ardboard in the global image. After applying our orretion tehnique to the data, we observe that the measured depth has an RMSE of about 4.25

7 Wax Sene Diret Global Meas. Depth Corr. Depth Error: 4.25 m Error: 1.34 m 110 Cross Setion 108 Slie Depth (m) 100 Pixel Index Figure 5. MPI-orretion redues error on senes with subsurfae sattering (physial experiment). The original sene (left) onsists of a wax andle. The diret and global omponents are obtained following the Nayar method. The measured depth of the wax andle is overestimated due to the subsurfae sattering. After applying our orretion algorithm, we observe a 70 perent redution in RMSE error. Ground Truth Shape Error: Measured Error = 8.71 mm Error: Correted Error = 5.35 mm 25 mm 0 mm 7. Disussion In summary, we present a light transport tehnique to mitigate MPI. Instead of relying on sparsity an unrealisti assumption we instead use the data from diretglobal separation to reover depth. Our system arhiteture employs off-the-shelf hardware omponents, speifially a Kinet and a projetor. For all tested senes, the orreted point louds demonstrate a redution in RMSE error. Figure 6. MPI-orretion redues error on senes with omplex geometry (physial experiment). The original sene onsists of a plasti faial mannequin. At (left) is a laser san of the objet, used as ground truth. The measured shape overestimates depth in several areas. We observe a 39 perent redution in RMSE error after applying MPI-orretion. m, while the orreted depth has an RMSE of about 1.34 m. Note that the error that remains in the orreted depth map is lose to the depth preision of the TOF sensor (this value is about 0.5 m [30]). Geometri Sene: We evaluate our method on a plasti faial mannequin, whih exhibits relatively omplex geometry. The ground truth point loud obtained with a laser sanner is illustrated in Figure 6. Sine the fae is made of plasti it laks signifiant subsurfae sattering. Therefore, we expet that the MPI should arise from onave setions in the surfae geometry. The measured shape in Figure 6 exhibits spekle like artifats, whih at first glane look like random noise. However, these artifats arise beause the plasti fae has fine pits in the surfae, whih auses similar MPI artifats to those observed in the large bowl from Figure 3. After applying our orretion tehnique, the reovered shape is more representative of the surfae. Quantitatively, the orretion redues the RMSE error by 39 perent. After orretion, the distribution of residual error is uniform over the surfae geometry, and at 5.35 mm, the error is very nearly at the depth preision of the TOF sensor [30]. Approximating Global Transport: The proposed tehnique relies on approximating global light transport with the first indiret boune. This is a ommon simplifiation that has been exploited in prior work in light transport (see Bimber et al. [4]). Intuitively, the ontribution of higher order bounes rapidly diminishes due to a ombination of surfae absorbane, the inverse square law, and Lambert s law (see supplementary material for details). Numerial Stability: Our losed form solution for MPI orretion (Eq. 13) depends on the phase differene between the diret and approximate global boune i.e (ϕ D ϕ G ). The sene harateristis and modulation frequeny determine the amplitude of this term. For low modulation frequenies, this term an get buried in the noise floor in ase of small senes, thus introduing errors in our orretion. In experiments, we observe that our method is robust to noise at the modulation frequeny of 120 MHz. Please see the supplement for the pertubation analysis of this problem. Real-time Performane: Correting MPI at fullframerates (i.e. 30 Hz) is not a ontribution of this paper. The key hallenge lies in the time required to separate the diret-global omponents of sene radiane. Speifially, while the Nayar method requires, in theory, only two photographs, we must take 25 photographs to ompensate for projetor bleed and defous. In a ommerial setting, a more robust implementation ould be ahieved by using improved projetion systems or inorporating real-time diret-global separation as desribed in [1].

8 7.1. Comparisons Our prototype ombines the advantages of a light transport based approah with an implementation realized in ommodity hardware. For omparison, the solution by O Toole et al. [38] is tied to a partiular hardware arhiteture. Speifially, O Toole et al. ombine transient imaging hardware, beamsplitters, along with modified projetive optis. By itself, inorporating suh hardware is not a limitation for a researh paper, but the measurement model for transient imaging preludes real-time performane [32, 23]. In ontrast, our tehnique relies on the Nayar model of diret-global separation [36], whih has been shown in separate work to work in real-time [1]. Therefore, we believe the reent work by Gupta et al. [19] represents a more fair omparison. In their paper they use a slightly modified time of flight sensor to generate dual-frequeny measurements. To resolve MPI, they ontribute novel theory, whih shows that global light transport is negligible at high modulation frequenies. However, for tabletop senes, this vanishing property requires GHz modulation frequenies. As a result, their physial experiments are onstrained to large, meter-size senes. In omparison, our method is not onstrained to large senes or high modulation frequenies. We provide results on a variety of objets, inluding speimens with subsurfae sattering or omplex geometry Limitations Our method redues error on every tested sene, but annot be implemented in the Kinet pipeline without introduing additional limitations. In partiular, to aquire the global and diret omponents, we use a separate projetor, requiring the use of radiometri alibration. Due to the quality of the projetor, the we oversample beyond Nayar s theoretial rule (25 photographs instead of 2). In addition, the FOV of the projetor is narrow, at approximately 43 degrees. Taken together, these three limitations on the projetor restrit the urrent setup to small, stati senes; however we are onfident that real-time implementations are possible with improved projetive apture setups now available for diret-global separation (f. [1]). 8. Conluding Remarks We propose a new omputational photography tehnique to generate higher quality 3-D sans than the standard Kinet (in the ontext of RMSE error). By oupling lightweight optial omplexity with a losed-form, mathematial solution, the proposed tehnique takes a step toward salable MPI-orretion. Referenes [1] S. Ahar, S. T. Nuske, and S. G. Narasimhan. Compensating for motion during diret-global separation. In ICCV, , 8 [2] A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar. Resolving multipath interferene in time-of-flight imaging via modulation frequeny diversity and sparse regularization. Optis Letters, 39(6): , , 2, 3 [3] A. Bhandari, A. Kadambi, R. Whyte, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. Multifrequeny time of flight in the ontext of transient renderings. In ACM SIGGRAPH 2013 Posters, page 46. ACM, [4] O. Bimber, D. Iwai, G. Wetzstein, and A. Grundhöfer. The visual omputing of projetor-amera systems. In Computer Graphis Forum, volume 27, pages , [5] M. Chandraker, J. Bai, T.-T. Ng, and R. Ramamoorthi. On the duality of forward and inverse light transport. IEEE TPAMI, 33(10): , [6] C. Dal Mutto, P. Zanuttigh, and G. M. Cortelazzo. Time-of- Flight Cameras and Mirosoft Kinet. Springer, [7] A. A. Dorrington, J. P. Godbaz, M. J. Cree, A. D. Payne, and L. V. Streeter. Separating true range measurements from multi-path and sattering interferene in ommerial range ameras. In IS&T/SPIE Eletroni Imaging, pages , , 2, 3, 4 [8] S. Foix, G. Alenya, and C. Torras. Lok-in time-of-flight (tof) ameras: a survey. IEEE Sensors Journal, 11(9): , [9] D. Freedman, E. Krupka, Y. Smolin, I. Leihter, and M. Shmidt. Sra: Fast removal of general multipath for tof sensors. arxiv preprint arxiv: , [10] D. Freedman, Y. Smolin, E. Krupka, I. Leihter, and M. Shmidt. SRA: Fast removal of general multipath for ToF sensors. In ECCV, pages , , 2 [11] S. Fuhs. Multipath interferene ompensation in time-offlight amera images. In ICPR, pages , , 4 [12] J. P. Godbaz, M. J. Cree, and A. A. Dorrington. Closed-form inverses for the mixed pixel/multipath interferene problem in amw lidar. In IS&T/SPIE Eletroni Imaging, pages , , 4 [13] J. Gu, T. Kobayashi, M. Gupta, and S. K. Nayar. Multiplexed illumination for sene reovery in the presene of global illumination. In Computer Vision (ICCV), 2011 IEEE International Conferene on, pages IEEE, [14] M. Gupta. Shape from satter. Computer Vision: A Referene Guide, pages , [15] M. Gupta, A. Agrawal, A. Veeraraghavan, and S. G. Narasimhan. A pratial approah to 3d sanning in the presene of interrefletions, subsurfae sattering and defous. International journal of omputer vision, 102(1-3):33 55, , 3 [16] M. Gupta and S. G. Narasimhan. Legendre fluids: a unified framework for analyti redued spae modeling and rendering of partiipating media. In Proeedings of the 2007 ACM

9 SIGGRAPH/Eurographis symposium on Computer animation, pages Eurographis Assoiation, [17] M. Gupta, S. G. Narasimhan, and Y. Y. Shehner. On ontrolling light transport in poor visibility environments. In Computer Vision and Pattern Reognition, CVPR IEEE Conferene on, pages 1 8. IEEE, [18] M. Gupta and S. Nayar. Miro phase shifting. In CVPR, pages , June [19] M. Gupta, S. K. Nayar, M. Hullin, and J. Martin. Phasor imaging: A generalization of orrelation-based time-offlight imaging. Tehnial report, Jun , 4, 8 [20] M. Gupta, Q. Yin, and S. K. Nayar. Strutured light in sunlight. In Computer Vision (ICCV), 2013 IEEE International Conferene on, pages IEEE, [21] M. Hansard, O. Choi, S. Lee, and R. Horaud. Time-of-Flight Cameras. Springer, [22] S. W. Hasinoff, F. Durand, and W. T. Freeman. Noiseoptimal apture for high dynami range photography. In CVPR, pages , [23] F. Heide, M. B. Hullin, J. Gregson, and W. Heidrih. Lowbudget transient imaging using photoni mixer devies. ACM TOG, 32(4):45, , 8 [24] F. Heide, L. Xiao, W. Heidrih, and M. B. Hullin. Diffuse mirrors: 3d reonstrution from diffuse indiret illumination using inexpensive time-of-flight sensors. In CVPR, [25] M. B. Hullin, M. Fuhs, I. Ihrke, H.-P. Seidel, and H. P. Lensh. Fluoresent immersion range sanning. ACM Transations on Graphis-TOG, 27(3):87 87, [26] M. B. Hullin, J. Hanika, B. Ajdin, H.-P. Seidel, J. Kautz, and H. Lensh. Aquisition and analysis of bispetral bidiretional refletane and reradiation distribution funtions. ACM Transations on Graphis (TOG), 29(4):97, [27] M. B. Hullin, I. Ihrke, W. Heidrih, T. Weyrih, G. Damberg, and M. Fuhs. Computational fabriation and display of material appearane. In Eurographis 2013-State of the Art Reports, pages The Eurographis Assoiation, [28] M. B. Hullin, H. Lensh, R. Raskar, H.-P. Seidel, and I. Ihrke. Dynami display of brdfs. In Computer Graphis Forum, volume 30, pages Wiley Online Library, [29] D. Jiménez, D. Pizarro, M. Mazo, and S. Palazuelos. Modeling and orretion of multipath interferene in time of flight ameras. Image and Vision Computing, 32(1):1 13, [30] A. Kadambi, A. Bhandari, and R. Raskar. 3d depth ameras in vision: Benefits and limitations of the hardware. In Computer Vision and Mahine Learning with RGB-D Sensors, pages Springer, , 11 [31] A. Kadambi, A. Bhandari, R. Whyte, A. Dorrington, and R. Raskar. Demultiplexing illumination via low ost sensing and nanoseond oding. In IEEE ICCP, [32] A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. Coded time of flight ameras: sparse deonvolution to address multipath interferene and reover time profiles. ACM TOG, 32(6):167, , 2, 8 [33] A. Kirmani, A. Benedetti, and P. A. Chou. Spumi: Simultaneous phase unwrapping and multipath interferene anellation in time-of-flight ameras using spetral methods. In IEEE International Conferene on Multimedia and Expo, pages 1 6, , 4 [34] S. G. Narasimhan, S. K. Nayar, B. Sun, and S. J. Koppal. Strutured light in sattering media. In Computer Vision, ICCV Tenth IEEE International Conferene on, volume 1, pages IEEE, [35] S. K. Nayar and M. Gupta. Diffuse strutured light. In Computational Photography (ICCP), 2012 IEEE International Conferene on, pages IEEE, [36] S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar. Fast separation of diret and global omponents of a sene using high frequeny illumination. In ACM TOG, volume 25, pages ACM, , 3, 4, 5, 8, 10 [37] T.-T. Ng, R. S. Pahwa, J. Bai, T. Q. Quek, and K.-H. Tan. Radiometri ompensation using stratified inverses. In ICCV, [38] M. O Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrih, and K. N. Kutulakos. Temporal frequeny probing for 5D transient analysis of global light transport. ACM TOG, 33(4), , 2, 8 [39] M. O Toole, J. Mather, and K. N. Kutulakos. 3d shape and indiret appearane by strutured light transport. In Pro. CVPR, [40] M. O Toole, R. Raskar, and K. N. Kutulakos. Primaldual oding to probe light transport. ACM Trans. Graph., 31(4):39, [41] R. Raskar and J. Davis. 5D time-light transport matrix: What an we reason about sene properties. Int. Memo07, 2, [42] D. Reddy, R. Ramamoorthi, and B. Curless. Frequenyspae deomposition and aquisition of light transport under spatially varying illumination. In ECCV, [43] S. M. Seitz, Y. Matsushita, and K. N. Kutulakos. A theory of inverse light transport. In ICCV, volume 2, pages , [44] J. Sell and P. O Connor. The xbox one system on a hip and kinet sensor. IEEE Miro, 34(2):44 53, Mar [45] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A benhmark for the evaluation of rgb-d slam systems. In IROS, pages , [46] T. Weise, H. Li, L. Van Gool, and M. Pauly. Fae/off: Live faial puppetry. In ACM SIGGRAPH/Eurographis Symposium on Computer Animation, pages ACM, [47] T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. MDonald. Robust real-time visual odometry for dense rgb-d mapping. In ICRA, pages , [48] L.-Q. Yan, M. Hašan, W. Jakob, J. Lawrene, S. Marshner, and R. Ramamoorthi. Rendering glints on high-resolution normal-mapped speular surfaes. ACM Transations on Graphis (TOG), 33(4):116, Supplemental Contents This supplement is organized as follows: Setion 1 desribes the apture proess and radiometri alibration (see line 478 of main doument).

10 Setion 2 shows results of MPI orretion after postfiltering (see line 583 of main doument). Setion 3 expands on the approximate global illumination model used to orret MPI (see line 731 of main doument). Setion 4 analyzes numerial stability (see line 743 of main doument). 10. Capture Proess We perform separation of diret and global omponents of light transport using the method introdued by Nayar et al. [36]. We illuminate the sene using multiple high frequeny patterns and the diret and global omponents of illumination are obtained from the aquired images using the Nayar method, whih is a losed-form, per-pixel solution. We use the DLP Lightommander projetor in near-infrared mode to illuminate the sene using a pixel hekerboard pattern. The pattern is shifted 5 times by 7 pixels eah time in X and Y dimensions to apture a total of 25 images. We alulate two images α + and α from these patterns, by omputing the maximum and minimum intensity values observed at eah pixel aross all 25 images. We also apture α T, the total radiane, by projeting an all-white image on the sene. Speifially, the following equations are used to obtain the diret (α D ) and global(α G ) omponents: α G = 2 1 b 2 (α b α + ), (14) α D = α T α G, (15) where b is the ratio between the brightness of the deativated and ativated state of the projetor pixel. We set b to Radiometri Calibration In our experimental setup, the internal light soure of the Kinet One sensor is approximately 8 times brighter than the external projetor. Sine both soures projet different amounts of radiane on the sene, the diret and global omponents obtained using the external projetor (Equations 14, 15) need to be radiometrially orreted before they an be used for obtaining the orreted phase ϕ D in Equation 13 of the main text. To perform radiometri alibration, we first preompute a radiometri alibration mask and multiply images by this alibration mask before projetion. The aptured images are saled with a onstant to omplete the radiometri orretion proess. We now provide details on omputing the alibration mask. Calibration Mask Computation To ompute the alibration mask, we first obtain an image of a white plane using the Kinet TOF sensor with the internal soure illuminating the sene (I INT ). Next we turn the internal soure off and aquire another image (I EXT ) with the external projetor illuminating the sene with an all-white image. We then establish pixel orrespondene between the internal and external projetor using gray odes. We projet 10-bit gray odes and their omplements, both in vertial and horizontal dimensions, apturing 40 images in total. Using the 20 images along X and 20 images along Y, we determine the pixel orrespondenes between the projetor pixels and the muh smaller image aptured by the amera, whose size is determined by the distane of the sene from the apture hardware. Using these orrespondenes, we obtain the radiometri alibration mask I R = I INT /I EXT. We repeat this exerise at three depth values (95, 100, and 105 m) that approximately span the sene volume we set I R to the mean value aross the three depths. Next, we onvert I R to an 8-bit image as I R8 = 255 ( max(i R ) I R) for projetion. We multiply every hekerboard pattern by I R8 before projetion. The image aquired by the Kinet sensor is post-multiplied by a salar fator max(i R ) 255 to obtain the final, radiometrially orreted image I C. We validate the radiometri orretion by measuring the perent intensity error between I INT and I C for the three white planes used to ompute I R and find that the mean absolute error is 2.3%. After radiometri orretion of all the projeted hekerboard images, the diret (α D ) and global (α G ) omponents are used in Equation 13 of the main text. It is neessary to synhronize the amera exposure and projetor projetion time to apture an 8-bit image. Sine the Kinet One sensor does not allow synhronization with an external devie, we projet the 8 bit planes of I R8 separately and perform a weighted sum of these 8 images in post-apture. Note that the radiometri alibration mask needs to be aquired only one for a given amera-projetor setup. In a ommerial implementation, the same light soure an be used to projet both the modulated illumination and highfrequeny oded illumination needed for diret-global separation, thus eliminating the need for radiometri alibration. 11. Post-filtering for Smoothing We pointed out in the main doument that our MPIorreted results were noisy and jagged as ompared to the measured shape (see Figure 3 of the main doument). In line 583 of the main doument, we made the argument that our noisy results were preferred to the MPI-orrupted results. This is beause the former an be enhaned with standard smoothing filters. We now substantiate our laim. As illustrated in Figure 8, the measured shape distorts the geometry of the original model, while the MPI-orreted model imparts noise. The

11 Projeted Image without radiometri orretion (I EXT ) Projeted Image with radiometri orretion (I RC ) Image with internal light soure (I INT ) (a) (b) Perent Error between I INT and I RC = 2.3% () Figure 7. Intermediate images from radiometri orretion. (a) A sene illuminated with the external projetor without any radiometri orretion. In (b) the radiometri orreted projetion. Finally, in () we show the same sene with the Kinet light soure. Note that the projeted image in (a) is muh dimmer than the observed image under Kinet lighting shown in (). latter artifats an be orreted. In Figure 8d, the orreted shape after bilateral filtering resembles the bowl. 4 We must mention that suh smoothing is a ommon step in today s depth sensing pipeline [30]. It would serve as an interesting follow-up to inorporate joint omputation, where we ombine our losed-form approah with spatial proessing to obtain a higher quality result. 12. Approximating Global Light Transport Reall from the main doument that our model approximates the amplitude of all bounes of global light transport. Our results empirially substantiate the validity of this approximation. We now provide further intuition on this point. For generalized MPI, the measured phase takes the form ( α0 sin ϕ 0 + α 1 sin ϕ 1 + α 2 sin ϕ ϕ = artan α 0 os ϕ 0 + α 1 os ϕ 1 + +α 2 os ϕ ), (16) where a i and ϕ i orrespond to the amplitude and phase from the i-th optial boune. The amplitude of seond and higher indiret bounes (α 1, α 2,...) diminishes rapidly due to a ombination of surfae absorbane, the inverse square law, and Lambert s law. As a result, the ontribution of higher bounes to the measured phase (Equation 16) redues rapidly for most senes. We substantiate this laim by simulating MPI in presene of different number of global bounes for a 90 degree orner sene. Figure 9 illustrates error plots from four indiret bounes from the orner sene with assoiated errors. Here, the listed error orresponds to the error in the measured shape introdued by the assoiated boune. For instane, the first indiret boune indues an error of 8.27 mm to the measured shape. In ontrast, the fourth indiret boune indues an error of only 0.11 mm to the measured 4 Bilateral filter parameters: half-window size 2, spatial domain standard deviation of 3 and intensity domain standard deviation of 0.1 shape. Note that the ontribution to RMSE of the fourth indiret boune is just 1.3 perent of that of the first indiret boune. In summary, this explains why our model, whih aggregates the amplitude of all indiret bounes, allows us to reover a lose approximation of the true diret phase for a wide variety of natural senes. Our method is not restrited to senes with sparse refletions (see main doument for results from subsurfae sattering). 13. Numerial Stability In line 743 of the main doument, we deferred analysis of numerial stability to the supplemental material. Here, we use perturbation theory to analyze the onditioning of our losed-form solution. Reall from the main doument that the losed-form solution to reover the diret phase is written as ( ) αd γ + α G (sin (ϕ D ϕ G ) + γ os (ϕ D ϕ G )) ϕ D = artan, α D + α G (os (ϕ D ϕ G ) γ sin (ϕ D ϕ G )) (17) where γ = tan (( ϕ). We note ) that α G = (1 α D ) and (ϕ D ϕ G ) = 4πfM dz, where dz is the differene in depth between the sene point ontributing the diret boune and the sene point ontributing the indiret boune. Therefore, this system is sensitive to hanges in α D and dz. We perform perturbation analysis of the ondition number of the funtion using these two arguments. The ondition number of a multivariate funtion f(x) wrt. x is given by κ = J(x) f(x) / x. (18) We evaluate this formula for ϕ D and ompute the norm of the Jaobian. We replae (1 α D ) by α G for brevity and simplify further to obtain the ondition number κ, whih takes the form of

12 (a) True Shape (b) Measured Shape () Correted Shape (d) Correted Shape with Bilateral Filter Figure 8. From left: (a) the true shape, (b) the measured, MPI-orrupted shape, () the result after our MPI-orretion algorithm, and (d) bilateral filtering of our orreted shape in (). The take-home point: after MPI-orretion, noise an be smoothed. But without MPI-orretion, a filter annot reover the original geometry. Indiret Boune 1 Indiret Boune 2 Indiret Boune 3 Indiret Boune 4 11 mm Error = 8.27 mm Error = 1.99 mm Error = 0.48 mm Error = 0.11 mm 0 mm Figure 9. Here we use the raytraing software to simulate a 90 degree orner, and ompute the error indued by different indiret bounes. Note that the ontribution to RMSE of the fourth indiret boune is just 1.3% of that of the first indiret boune. κ = α 2 D + dz2 ( artan ( 4πf M 4πfM )α G (α D os( dz ) ) )+α G 2α 2 D +2α G(os( 4πf M dz)+1) 1 ( ( ) )) 4πfM 4πfM α D γ+α G sin( dz +γ os( dz (os( ) )) 4πfM 4πfM α D +α G dz γ sin( dz ). (19) Although this is the simplified form of the ondition number, it is too omplex to ast immediate intuition. To obtain a sense of empirial intuition we plug in reasonable, physial values for the parameters and ompute κ. In partiular, we vary values of α D between its physial limits of 0 and 1, and dz between 1 and 1 meters. We assume the following imaging senario: a sene point one meter away, aptured with Kinet amera parameters (e.g. f M = 120 MHz). The results are shown in Figure 10. On the left hand side we plot κ as a 2-D matrix for different entries of dz and α D. Note that the ondition number is low throughout the entire plot, exept for the loop regions. These loops our, beause the denominator of the Jaobian norm reahes extremely low values, whih auses the ondition number to inflate. On the right hand side of Figure 10, we plot a slie through the matrix plot, for dz = 0.2, whih is a physially reasonable value. Note that the ondition number is less than 1 throughout this region, whih indiates a wellonditioned problem. In summary, our losed-form solution is wellonditioned, whih is refleted in the quality of our results.

13 1 Condition Number ( ) Condition Number Slie dz Slie Condition Number ( ) dz = D D Figure 10. Analyzing numerial stability. We ompute the ondition number (see Equation 19) and plot its value for various values of α D and dz. Exept for the two symmetri bands with high ondition number at large values of dz, the system is stable. At right, a horizontal slie of the ondition number matrix, where dz = 0.2.