DIRECT AND PROGRESSIVE RECONSTRUCTION OF DUAL PHOTOGRAPHY IMAGES

Transcription

1 DIRECT AND PROGRESSIVE RECONSTRUCTION OF DUAL PHOTOGRAPHY IMAGES Binh-Son Ha 1 Imari Sato 2 Kok-Lim Low 1 1 National University of Singapore 2 National Institte of Informatics, Tokyo, Japan ABSTRACT Dal photography is a well-known application of light transport acqired by a projector-camera system. By applying compressive sensing, compressive dal photography [1] is a fast approach to acqire the light transport for dal photography. However, the reconstrction step in compressive dal photography can still take several hors before dal images can be synthesized becase the entire light transport needs to be reconstrcted from measred data. In this paper, we present a novel reconstrction approach that can directly and progressively synthesize dal images from measred data withot the need of first reconstrcting the light transport. We show that or approach can prodce high-qality dal images in the order of mintes sing only a thosand of samples. Or approach is most sefl for previewing a few dal images, e.g., dring light transport acqisition. As a by-prodct, or method can also perform low-resoltion relighting of dal images. We also hypothesize that or method is applicable to reconstrcting dal images in a single projector - mltiple cameras system. Index Terms dal photography, compressive sensing (a) (b) SSIM: 0.46, RMSE: INTRODUCTION Light transport [2] is a mathematical operator that captres how light bonces among srface points in a scene. In compter graphics and compter vision, several applications have been proposed that make se of light transport sch as relighting [2], dal photography [3], and radiometric compensation. Among those, dal photography is an interesting and well-known application of light transport thanks to its simplicity and seflness. Given a light transport of a scene lit by a controlled light sorce and captred by a camera, dal photography can virtally swap the roles of the light sorce and the camera to prodce dal images. The dal images can be perceived as if the scene is lit by the camera and captred by the light sorce. Dal photography is also sefl in captring 6D light transport [3]. Traditionally, to obtain dal images of a scene, it is necessary to first acqire and reconstrct the entire light transport matrix of the scene. Several approaches have been proposed to efficiently acqire and reconstrct light transport sch as mltiplexed illmination [4], compressive sensing of light transport [1, 5], or optical compting of light transport [6]. However, reconstrcting the entire light transport matrix from the acqired data can still be very costly since the nmber of rows and colmns of the light transport matrix can be tens or hndreds of thosands. This means a hge amont of comptational time is reqired before the first dal image can be synthesized and ready for display. In this paper, we present a novel approach to efficiently compte dal images from measred data withot reconstrcting the light transport. We bild or method pon compressive sensing of light transport [1, 5] and propose an approach to directly and progressively reconstrct high-qality dal images sing L1-norm optimiza- (c) SSIM: 0.47, RMSE: (d) SSIM: 0.55, RMSE: Fig. 1. Dal photography. (a) Camera view. (b) Dal image directly reconstrcted from samples, which is not practical. (c) Dal image progressively reconstrcted from only 1000 samples sing or method with 64 basis dal images. (d) Dal image reconstrcted with settings as in (c) bt from 1500 samples. Haar wavelet is sed for the reconstrction. tion. The nmber of measrement samples needed is comparable to that sed for light transport reconstrction in compressive dal photography. Sch direct reconstrction allows s to qickly synthesize dal images as soon as the acqisition data is enogh. Or method can also generate progressive reslts while the dal image is being reconstrcted. Therefore, or method can be beneficial for previewing a few dal images. Besides, we also demonstrate that or method can be sed for low-resoltion relighting of dal images. We also hypothesize that or approach is extendable to synthesize dal images in setps that have a single light sorce and mltiple cameras. 2. RELATED WORKS Recently, several approaches to efficiently acqire and reconstrct light transport have been proposed [4, 3, 1, 5, 6]. In the seminal work abot dal photography [3], Sen et al. proposed a hierarchical approach to detect projector pixels that can be trned on simltane-

2 osly in a single light pattern. This greedy-like approach can redce the nmber of light patterns in the acqisition to the order of thosands. In the worst case when most of projector pixels conflict to each other and can only be schedled to be trned on seqentially, this approach can be as slow as brte-force acqisition. In compressive dal photography [5, 1], the athors proposed to se rows of measrement matrices in compressive sensing as light patterns, ths trns light transport acqisition into a compressive sensing problem that allows the light transport to be reconstrcted sing the well-known L1-norm optimization. This approach works well for high-rank and sparse light transport matrix which is often seen in a projector-camera system. In this work, we also bild or approach based pon compressive sensing. We provide a simple reformlation of compressive dal photography that allows s to directly and progressively reconstrct dal images sing L1-norm optimization. Recently, O Toole and Ktlakos [6] proposed to se Arnoldi iterations to determine eigenvectors of a light transport sing optical compting. While their method only reqires less than a hndred of images, it is more sitable for dense and low-rank light transport where the light sorce is diffse. In this work, we target sparse and high-rank light transport. While compressive sensing of light transport is designed to minimize the nmber of images to acqire, it often reslts in long comptation time needed to reconstrct the light transport in the postprocessing step. This is an isse for dal photography, especially when we only need to see a handfl nmber of dal images. Therefore, it is necessary to have an approach that can compte dal images from measred data as fast as possible. In this work, we fill in this gap by proposing sch an approach based on compressive sensing and L1-norm optimization. Sen et al. [1] also discssed abot single pixel imaging and how it is related to compressive dal photography. This is probably most closely related to direct reconstrction of dal images which we proposed in this work. The athors noticed that directly recovering the reflectance fnction of this single pixel, which is eqivalent to directly compting the dal image nder floodlit lighting, is rather troblesome becase the dal image is more complicated and therefore a lot more samples are needed. In this work, we solve this problem by presenting a simple basis so that dal images can be progressively reconstrcted from a small amont of measrement samples. Finally, while it is not closely related to dal photography, we note that the idea of direct reconstrction sing compressive sensing was also exploited to obtain the inverse light transport [7]. 3. COMPRESSIVE DUAL PHOTOGRAPHY Let T be the light transport matrix of a scene captred by a projectorcamera system. Sppose that the light sorce emits pattern l. The image c of the scene captred by the camera can be represented by the light transport eqation: c = Tl. (1) In dal photography, by tilizing Helmholtz reciprocity, the dal image can be compted as c = T l, (2) where l is the dal light pattern virtally emitted by the camera and c is the dal image virtally captred by the light sorce. By projecting a set of N light patterns L = [l 1... l N ] and captring images of the scene C = [c 1 c 2... c N ] lit by this set of patterns, we can rewrite the light transport eqation as C = L T, (3) which sggests an elegant way to measre light transport T sing compressive sensing. Each row of T can be measred by letting L be a measrement matrix sch as Bernolli or Gassian matrix that satisfies the restricted isometry property [8]. Each row t i of light transport matrix T can be independently reconstrcted by minimizing t i = arg min c i L λ W 1 (4) where c i denotes colmn i of C, i [1... T ], T the nmber of rows of matrix T, W the basis of the space where each row of the transport matrix can be sparse. However, since T can be tens of thosands, e.g., T = which represents a rather low-resoltion camera view, the reconstrction of T can take several hors to complete [1]. To speed p, it is possible to frther exploit coherency among pixels in each colmn of matrix T by sing another compression basis P as in [5]. We get: P C = (P TW)(W L). (5) We captre images as before bt transform them into basis P in the post-processing. As before, compressive sensing can be applied to reconstrct each row of the compressed matrix P TW independently, bt this time the nmber of rows needed to reconstrct can be less. However, in or observation, the nmber of non-zero rows of P C is still in the order of thosands becase the captred images C lit by measrement patterns L can contain a lot of complex blocky patterns that are difficlt to compress by basis P. 4. DIRECT AND PROGRESSIVE RECONSTRUCTION 4.1. Direct reconstrction We are now ready to present or approach to directly reconstrct dal images, which we bild on top of compressive dal photography [1]. We start by showing that dal image can be directly compted from the acqired images and light patterns. By mltiplying the dal light pattern l to both sides of Eqation 3, it is easy to get: C l = L c. (6) By letting L be a measrement matrix and pre-compting the left part C l, we can view dal image synthesis as a compressive sensing problem. Therefore, the dal image can be directly reconstrcted by L1-norm optimization: c = arg min C l L λ W 1. (7) Theoretically, this approach shold be able to reconstrct the dal image c. Unfortnately, in practice, in order to obtain a high-qality dal image, almost tens of thosands nmber of measrement samples, or camera images and light patterns, are necessary. This is becase dal image is not as sparse as reflectance fnctions stored in rows of light transport T, ths it reqires more samples in the reconstrction.

3 scene (a) 4000 samples. SSIM: 0.43 RMSE: (b) 8000 samples. SSIM: 0.44 RMSE: (c) samples. SSIM: 0.44 RMSE: camera projector Fig. 3. Progressive reslts of the dal image in Figre 1(d) by accmlating those reconstrcted basis dal images. Or projectorcamera setp to acqire light transport is shown in the diagram. (d) 1000 samples. SSIM: 0.41 RMSE: (e) 2000 samples. SSIM: 0.57 RMSE: (f) Grond trth. Fig. 2. Comparison between direct and progressive reconstrction. Dal image (a), (b), and (c) are from direct reconstrction. Dal image (d) and (e) are from progressive reconstrction with 64 basis dal images. (f) Grond trth is generated from light transport from samples by inverting the circlant measrement matrix. Dabechies-8 wavelet is sed for the reconstrction Progressive reconstrction We propose a simple approach in order to overcome the above isse. Sppose that we can project the dal light pattern l into a basis Q = [q 1 q 2... q Q ]: l = Qw = i w iq i, (8) where Q is the nmber of basis vectors in Q, i [1... Q ], w the coefficient vector of l in basis Q. Therefore, the dal image can be compted by c = w ic i (9) i where c i is the basis dal image which satisfies C q i = L c i. (10) Each basis dal image can be fond independently by optimizing c i = arg min C q i L λ W 1. (11) The intition behind this formlation is that we can split the reconstrction of the dal image into several passes, and reconstrct each basis dal image that forms a part of the dal image in each pass. It is significant to garantee that each basis dal image shold be sfficiently sparse so that it can be sccessflly reconstrcted sing Eqation 11 withot sing too many measrement samples. As shown in Figre 1 and 2, the nmber of samples needed to reconstrct basis dal images is comparable to that reqired to reconstrct the entire light transport in traditional compressive dal photography, which is more practical than direct reconstrction. Figre 3 shows a few examples of the progressive reconstrction. We choose basis Q based on two following criteria. First, the dimension of space Q shold be as low as possible. It is best to choose Q of which the dimension is abot tens or hndreds. Second, the basis dal images c i obtained by setting dal lighting to basis vectors of Q shold be sparse so that high qality reconstrction can be achieved. Based on sch criteria, we propose a simple and easy to implement basis Q as follows. We sbdivide the dal lighting pattern l into a grid and let each patch in the grid be a basis vector q i. Therefore, the weight w i is simply set to one. It is easy to see that smaller patch size tends to prodce sparser coefficients of basis dal images in the wavelet domain. This can yield higher accracy in the reconstrction bt reslt in longer comptational time. An advantage of choosing basis Q as above is that we can display progressive reslts of the dal image by accmlating existing basis dal images while other remaining basis dal images are pending for reconstrction, which is sefl for previewing applications. 5. IMPLEMENTATION We se a projector-camera system to acqire the light transport. The projector is a Sony VPL-DX11. The camera is a Sony DXC-9000 of which the response crve is linear. The light patterns to compressively acqire the light transport are obtained from a circlant matrix of which the first row is an i.i.d Bernolli distribtion with vale 1 and 1 [9]. An advantage of sing a circlant measrement matrix is that its mltiplication with a vector can be qickly compted sing fast Forier transform. Also, circlant matrix reqires very little memory storage as only the first row needs to stored. Since or patterns contain both positive and negative vales, we project positive and negative patterns separately and combine the corresponding camera images in the post-processing by the formla c = T(l + l ) = c + c, where sperscript + and denote positive and negative patterns and images, respectively. For simplicity, we also crop and downsample camera images to the same size as the light patterns so the light transport is a sqare matrix. We implement or system in MATLAB. We implement split Bregman iterations [10] for L1-norm optimization in Eqation 11. We let λ = for all progressive reconstrction. We let λ = 0.05 for direct reconstrction to frther sppress noise. We test the reconstrction with Haar wavelet and Dabechies-8 wavelet provided by the Rice Wavelet Toolbox [11].

4 Dring progressive reconstrction, we discard basis dal images of which the absolte maximm vale of their corresponding lefthand side vector C q i is less than In fact, this corresponds to regions that can be lit by the projector bt are ot of field of view of the camera so zero soltions for basis dal images are appropriate. 6. EXPERIMENTS The reslts of or method are shown in Figre 1. The resoltion of the dal image is As can be seen, or method is able to reconstrct a good-qality dal image withot first obtaining the light transport. We provide qantitative comparisons between or reslts of direct and progressive reconstrction and the grond trth shown in Figre 2 sing both strctral similarity index (SSIM) [12] and root-mean-sqare error (RMSE). In Figre 1, by sing basis Q with patch size set to 16 pixels, only 1000 samples are needed to reconstrct total 64 basis dal images and a high-qality final dal image. Figre 3 shows some of the progressive reslts of the dal image dring reconstrction. In contrast, directly reconstrcting the dal image withot basis Q reqires samples in order to reach similar image qality, which is far less practical. In fact, given samples, it is often more preferable to reconstrct the entire light transport in the post-processing by inverting the circlant measrement matrix sing FFT, which is fast. Here we se this approach to generate the grond trth dal image as shown in Figre 2(f). We do not opt to reconstrct the light transport from only 1000 measrement samples since it takes tens of thosands of L1-norm optimization which is too time consming to perform. Figre 2 frther demonstrates how or method works with different nmber of samples for both direct and progressive reconstrction sing Dabechies-8 wavelet for compression. As expected, more samples allows more details of the dal image to be revealed. As a by-prodct, we demonstrate a relighting application by linearly combining basis dal images by setting the weight vector to a low-resoltion lighting pattern. Figre 4 shows or relit images. The new lighting has resoltion 8 8 since or basis vectors are derived from 8 8 grid patches. We measred the rnning time of or progressive reconstrction on an Intel Core 2 Qad processor clocked at 2.8 GHz with 8 GB of RAM. Or MATLAB implementation otpt the direct reslt (b) of Figre 1 in 10 mintes and the progressive reslt (c) in 40 mintes. While progressive reconstrction is a few times slower, it saves a large amont of acqisition time as it reqires far less nmber of samples to reach similar image qality. With the same nmber of samples, progressive reconstrction is also faster than reconstrcting the entire light transport when only a few images are needed Rnning time analysis We provide a simple analysis to estimate how mch and when progressive reconstrction is better than traditional light transport reconstrction in terms of rnning time as follows. We assme the following model to predict the rnning time of progressive reconstrction: t = 2αN + kρ Q, (12) where t is the rnning time in seconds, N the nmber of samples acqired, α the time to acqire a single image, ρ the time to reconstrct a basis dal image, k the nmber of dal images we are interested in in total. The constant 2 represents the need to captre two images per sample de to positive and negative entries of the measrement Fig. 4. Relighting of the dal image in Figre 2(e). matrix. Similarly, the rnning time of traditional light transport reconstrction can be predicted by: t = 2αN + ρ T, (13) where N is the nmber of samples needed to acqire for light transport reconstrction, ρ the time to reconstrct a row of T. Empirically, we set α = 1 second, N = 1000 samples, ρ = 75 seconds, according to the examples in the previos figres. Since the reflectance fnction stored in each row of light transport T can be more sparse than dal images, we pessimistically assme that N = 500 which means or progressive reconstrction reqires twice the nmber of samples. We also set ρ = 2 to assme that each row of T can be reconstrcted mch faster. We also have Q = 64 and T = As a reslt, in order to garantee t < t, we need to bond k 6. This indicates the maximm nmber of dal images we can reconstrct before or method cannot offer any time savings. When only a dal image is needed, or k = 1, the speed p is abot DISCUSSION Conventionally, in order to compte a dal image of light transports of a scene captred by a single projector and mltiple cameras, the light transport matrix between each pair of projector-camera needs to be reconstrcted. In sch case, for qick reconstrction, or method is still applicable. In the case of two cameras, we have: [ ] [ ] C 1 C l 1 2 l = L c. (14) 2 It is natral to extend the formlation to the case of mltiple cameras. We leave the implementation of sch a system for ftre works. 8. CONCLUSIONS In this paper, we presented an approach based on compressive sensing to directly and progressively reconstrct dal photography images withot the need of reconstrcting the entire light transport. Or method can be sefl for previewing of dal images. We are also able to perform low-resoltion relighting of dal images. There are a few limitations in or approach. First, or reconstrcted dal images tend to be noisier than those prodced by the fll light transport. This can be explained by the dot prodct between the camera images and the dal lighting pattern, which sms p the variance of each camera pixel. Second, or method may fail when the basis dal images are not sparse enogh. It is interesting to extend this work frther in the ftre. First, it can be sefl to have a carefl noise analysis of dal images obtained by or method. Second, it can be exciting to seek a more optimal basis than or grid basis in order to reconstrct dal images in higher qality.

5 9. REFERENCES [1] Pradeep Sen and Soheil Darabi, Compressive dal photography, Compt. Graph. Form, vol. 28, no. 2, pp , [2] Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan, Triple prodct wavelet integrals for all-freqency relighting, ACM Trans. Graph., vol. 23, pp , Agst [3] Pradeep Sen, Billy Chen, Garav Garg, Stephen R. Marschner, Mark Horowitz, Marc Levoy, and Hendrik P. A. Lensch, Dal photography, in ACM SIGGRAPH, 2005, pp [4] Yoav Y. Schechner, Shree K. Nayar, and Peter N. Belhmer, A Theory of Mltiplexed Illmination, in IEEE International Conference on Compter Vision (ICCV), 2003, vol. 2, pp [5] Pieter Peers, Dhrv K. Mahajan, Brce Lamond, Abhijeet Ghosh, Wojciech Matsik, Ravi Ramamoorthi, and Pal Debevec, Compressive light transport sensing, ACM Trans. Graph., vol. 28, no. 1, pp. 1 18, [6] Matthew OToole and Kiriakos N. Ktlakos, Optical compting for fast light transport analysis, in SIGGRAPH Asia, [7] Xinqi Ch, Tian-Tsong Ng, Ramanpreet Pahwa, Tony Q. Qek, and Thomas Hang, Compressive inverse light transport, Proceedings of the British Machine Vision Conference, [8] Richard Baranik, Compressive sensing, IEEE Signal Processing Mag, pp , [9] Wotao Yin, Simon Morgan, Jnfeng Yang, and Yin Zhang, Practical compressive sensing with toeplitz and circlant matrices, Proceedings of Visal Commnications and Image Processing (VCIP), [10] Tom Goldstein and Stanley Osher, The split bregman method for l1-reglarized problems, SIAM J. Img. Sci., vol. 2, pp , April [11] Richard Baranik et al., Rice wavelet toolbox, [12] Zho Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli, Image qality assessment: from error visibility to strctral similarity, IEEE Transactions on Image Processing, vol. 13, no. 4, pp , april 2004.