All in-focus View Synthesis from Under-Sampled Light Fields

Transcription

1 ICAT 2003 December 3-5, Tokyo, Japan All in-focu View Synthei from Under-Sampled Light Field Keita Takahahi,AkiraKubota and Takehi Naemura TheUniverityofTokyo Carnegie Mellon Univerity 7-3-1, Hongo, Bunkyo-ku, Tokyo 5000 Forbe Ave, Pittburgh, PA, , Japan 15213, USA Abtract Light field rendering (LFR) i a fundamental method for generating new view from a et of pre-acquired image. We ue denely-aligned camera for the proce of acquiring the et of image. In mot practical cae, the denity of the aligned camera i not high enough to yntheize appropriate view. Thi under-ampling condition caue focu-like effect in the yntheized view. Thi paper propoe a new method for olving thi problem. Firt, a et of differently focued view i yntheized from the underampled et of pre-acquired image. Then, an all in-focu view i generated from the et of differently focued view. Thi i baed on a new focu meaurement algorithm pecialized for light field rendering and plenoptic ampling theory. Experimental reult how the effectivene of our approach. Key word: Light Field Rendering, Focal Plane, All-in Focu, Focu Meaure, Plenoptic Sampling 1. Introduction In virtual reality and telecommunication ytem, interactive viewing of photo-realitic 3D cene i one of the mot important technical iue. A a fundamental method of image-baed rendering, light field rendering (LFR) [1] offer a promiing olution for thi iue. By thi method, we can yntheize free-viewpoint image (ynthetic view) from a et of pre-acquired image (input image) with little or no geometric model of the cene. It i poible to yntheize highly photo-realitic image in real time with graphic hardware acceleration. The algorithm of LFR i imple. Each pixel value of input image i parameterized a a light-ray data, and tored in a light-ray databae. We can yntheize a view of a given viewpoint from the databae by picking up the date of uch light-ray thoe pa through the viewpoint. In thi proce, light-ray data are interpolated on the aumption that object in the cene are ituated on a plane, which i called a focal plane. In mot practical cae, the et of input image cannot not be acquired denely enough to produce free-viewpoint image with adequate quality. In other word, light-ray data i under-ampled. Thi under-ampling condition caue focu-like effect 1 on the ynthetic view [2]. Objectituatednearthefocal plane are yntheized clearly and harply (in focu), while object apart from the focal plane are with blurring and ghoting (out of focu). Thi i a eriou problem in the ene of ynthei quality. Therefore, our concern in thi paper i how to yntheize all in-focu image at arbitrary viewpoint without increaing the number of input image. A main approach to that problem i utilization of geometric information of the cene in addition to the lightray databae [2, 3, 4, 5, 6]. In thi approach, the tructure of the cene i approximated by a focal urface, aet of depth layer or a 3D meh model which repreent the cene tructure more preicely than a imple focal plane. In general, more precie geometry can generate more favorable ynthetic view. However, thee method need a pre-etimation proce to recover the cene tructure before the image ynthei proce. It conume much time and computation power, and in ome cae it need ome manual procedure. Even the mot ophiticated computer viion technologie might fail to etimate the hape of object with adequate quality, ince their reult are ubject to the material of the object and lighting condition. Defectivene of geometric model would caue highly viible hole and cum in the ynthetic view. In thi paper, we propoe a different approach from the one mentioned above for yntheizing all in-focu image. Firt, for a given viewpoint, we yntheize ome equence of differently focued image by changing the depth of the focal plane of the LFR method. Then, we apply a new focu meaure to detect in-focu region in the differently focued image by comparing the equence. Finally, we integrate them into an all in-focu image. Thi mean that a view-dependent depth map i generated for each viewpoint. Thu, the propoed method doe not ue a global depth map that require pre-etimation proce. The merit of the view-dependent depth map i that it i good enough for the correponding viewpoint. Our approach ha a cloe connection to the conventional depth from focu method [7, 8] or image fuion method [9], which utilize multiple differently focued im- 1 In LFR, input image are aumed to have no defocu blurring. Notice that what i called focu here i an unique effect in LFR.

2 f z min Ω u + Ω = 0 Ω u O Recontruction Filter K Ω u f z max 2 2 Ω +Ω = 0 u Ω Fig. 1: Signal analyi of light field in the frequency domain. age taken by phyical camera. However, the focu-like effect generated by the LFR method have different nature from the one by phyical camera. For example, in the cae of LFR, not only the blurring but alo the ghoting effect appear in defocued region. Therefore, we need a new focu meaure that i pecialized for LFR. The remainder of thi paper i a follow. In Section 2, related work are ummarized to explain the phenomenon of focu in LFR from the theoretical viewpoint. In Section 3, we propoe a new focu meaure that i pecialized for a et of image yntheized by the LFR method. We alo decribe a pecific method for yntheizing an all infocu image from the et of image uing focu meaurement. In Section 4, our implementation and experimental reult are decribed. Finally we conclude our paper in Section Background In thi ection, we introduce the theory of plenoptic ampling [10] that analye light field ignal in the frequency domain. Thi theory give an inightful view to clarify the reaon why the focu-like effect appear on the ynthetic view by the LFR method [2, 11]. In thi paper, we adopt the parameterization of light field decribed in [2]. 2.1 Spectrum of Light Field Signal Input image are captured at 2D grid point on a plane which i called a camera plane. We conider that the optical axi of each camera i orthogonal to the camera plane. The poition of a camera and a pixel on the camera are denoted by (, t) and(u, v). Then, each light-ray captured by input camera i uniquely determined by a et of 4 parameter a (, t, u, v). For implicity, we dicu the 2D ubpace (, u) of the original 4D light field (, t, u, v). In the plenoptic ampling theory, light field ignal are analyzed in the frequency domain. (Ω, Ω u)denote the frequency pace correponding to the (, u) pace. A decribed in [10], the ignal pectrum appear only in the hadowed area in Figure 1. That i to ay: (1) When the range of depth (ditance from the camera plane) of the cene i repreented a z min Z z max, the pectrum i bounded by the following two line: f f Ω u +Ω =0, Ω u +Ω =0, (1) z min z max where f denote the focal length of the input camera. (2) the maximum frequency in the Ω u axi i denoted a K Ωu (= 2K fu ), which i determined by the complexity of the cene texture, the reolution of the input camera or the reolution of the yntheized image. Since the light field i ampled dicretely, replication of the original pectrum appear in a contant interval a hown in Figure 1. When the ditance between input camera i, the interval in the frequency domain i repreented by 2/. The original pectrum would not overlap the neighboring replication, when the following relation i atified: z min z max K fu f. (2) 2.2 Focu in Synthetic View In the image ynthei proce, the light-ray data are interpolated and re-ampled. To undertand focu in yntheized image, we conider how to recontruct a continuou ignal from the dicretely ampled ignal. In uch cae that Equation (2) i atified, the ideal box filter hown in Figure 1 recontruct the continuou ignal without aliaing artifact. The filter let through the entire pectrum component inide the box, while it completely cut off them outide the box. A decribed in [10], the lope of the box correpond to the depth of the focal plane. Actually, the gradient of the idewie line of the box i z 0/f,wherez 0 denote the depth of the focal plane. The filter i optimized when z 0 i equal to z opt that i defined a 1 = 1 z opt 2 ( ). (3) z max z min Shown in Figure 2 i the relationhip between z 0 and the hape of the recontruction filter. Even if Equation (2) i atified, when z 0 i much larger or maller than z opt, the continuou ignal i not recontructed properly a hown in Figure 2(a) and (c). The leakage of high frequency component and the interfuion of the neighboring replication caue blurring and ghoting artifact in the ynthetic view. In mot practical cae, Equation (2) i not atified for the whole cene ince the ampling denity (1/) i not high enough. In thi paper, we conider uch a mall region of the cene where the depth variation i mall enough to atify Equation (2). When the value of z 0 i optimized for the region, the light field ignal of

3 (a) z 0 <z opt (b) z 0 = z opt (c) z 0 >z opt Fig. 2: Relation between the depth of the focal plane z 0 and the hape of the recontruction filter. the region i recontructed clearly and harply (in focu) without aliaing. In other word, when the value of z 0 i not optimized, the region i recontructed out of focu. 3. All in-focu View Synthei from Under- Sampled Light Field In thi ection, we propoe a new algorithm for yntheizing all in-focu view from under-ampled light field. In our method, we firt yntheize equence of view by the conventional LFR method. What i important here i that their focal plane are different from each other. Then, we integrate their in-focu region to generate an all in-focu view. Thi i baed on our propoal of focu meaure pecialized for LFR. 3.1 Focu Meaure in LFR In the cae of phyical camera, defocu blurring can be modeled a the degradation of high-frequency component. Therefore, the high-frequency energy i ued a a good meaure of focu. Given a equence of differently focued image, we could eaily detect the in-focu region by earching uch region that ha the highet energy in the high-frequency domain among them. In the cae of image yntheized by LFR, however, not only blurring but alo ghoting artifact that contain ome high-frequency component arie in the defocued region. Therefore, we need a novel focu meaure pecialized for LFR. Our propoal for thi focu meaurement i to utilize the difference among image yntheized by different recontruction filter which are to interpolate dicretelyampled light field. Conider a et of image generated by different filter at an identical focued depth. The focued region would be almot the ame in all the image regardle which filter i ued. The defocued region, however, would how the difference of the characteritic of the filter. Therefore, ubtraction of image generated by different filter can be ued a the focu meaure for LFR. That i to ay, if a region i in focu, pixel value of the region are zero or very mall in the ubtraction image. Now, we give a theoretical explanation on the principle mentioned above in frequency domain analyi. Conider uch a mall region that the depth variation i mall enough to atify the following equation z min z max 2 1 (4) K fu f The condition of thi equation i more trict than that of Equation (2), ince we need to ue ome other recontruction filter in addition to the one decribed in [10]. In thi paper, we ue three different recontruction filter hown a dahed parallelogram in Figure 3, 4 and 5. But, the number of filter i not limited to three in our method. (1) normal filter (Figure 3): the mot fundamental boxhaped filter. The width i 2/. Whenz 0 i z opt, it i the optimal filter decribed in [10]. (2) wide-aperture filter (Figure 4) [2]: The width of thi filter i maller than 2/. Thi lead to that the yntheized image would look like image taken by a wide-aperture camera. In thi paper, the width i et to / a hown in Figure 4. (3) camera-kipped filter (Figure 5): The input image ued for the recontruction are kipped alternately. In thi cae, The repeating interval of the pectrum become /. We apply the normal filter, the widthofwidthi/, for the reduced light-ray data. When the region i in focu (z 0 = z opt), all the filter would produce the ame reult, a hown in Figure 3(a), 4(a) and 5(a). When the region i out of focu (z 0 z opt), however, different filter would produce different reult. Thi i becaue the frequency component paed through by the filter are obviouly different from each other; the leakage of the original pectrum and the interfuion of the replication pectrum are caued differently a illutrated in Figure 3(b), 4(b) and 5(b). Now, conider a ubtraction image between image yntheized by different kind of filter for the ame depth value of z 0. According to the Pareval identity, the total energy of a ignal i contant in the patial domain and in the frequency domain. Therefore, the following concluion are drawn from the frequency domain analyi. In a region that i in focu, every pixel in the region mut be 0 theoretically. In a region that i out of focu, ome pixel might have non-zero value.

4 2 2 Input image 2 2 Step (1),(2) h1 h2 h3 (a) z 0 = z opt (b) z 0 <z opt Sequence of multiple focued image Fig. 3: Normal filter. 2 2 Step (3) Subtraction (a) z 0 = z opt (b) z 0 <z opt Smoothing Sequence of evaluation image Fig. 4: Wide-aperture filter. Step (4) Minimum earch Step (5) Depth index map Pixel read (a) z 0 <z opt (b) z 0 <z opt Fig. 5: Camera-kipped filter. All in-focu image Fig. 6: Diagram of the propoed method. 3.2 All in-fouc View Synthei Firt of all, we determine depth candidate z n (n = 0, 1..., N 1) where the focal plane i placed, in uch a waythateachobjectwouldbein focu at leat one image in the equence. There are ome ueful theorie [12, 13] for thi configuration that dicu how much range of depth i in focu in front and rear of the focal plane. The outline of our method i hown in Figure 6. The following tep are iterated for each novel view. (1) The viewpoint for novel view ynthei i given. (2) Three different et of differently focued image, I <n> k (x, y) (n = 0, 1..., N 1, k = 1, 2, 3), are generated by the LFR method correponding to the filter h k and the focued depth z n. (x, y) denote poition of pixel on the yntheized image. (3) For each depth z n, an evaluation image for the focu evaluation, E <n> (x, y), i calculated a follow: E 0 <n> (x, y) = a k,l I <n> k (x, y) I <n> l (x, y) (5) k<l E <n> (x, y) = M<i,j<M E <n> 0 (x + i, y + j). (6) By taking combination of more than two kind of filter and umming E 0 <n> (x, y) in a window region, the ize of which i (2M +1) (2M +1), we aim to enhance the tability of the focu meaurement. a k,l denote weighting coefficient. (4) For each pixel (x, y), the index of the optimal depth where the pixel i motly in focu can be given by n(x, y) that yield the minimum of E <n> (x, y): n(x, y) =arg min 0 n N 1 E<n> (x, y) (7) (5) An all in-focu image I(x, y) i generated a follow. I(x, y) =I <n(x,y)> k (x, y) (8) In thi way, each pixel value i read from the image yntheizedinstep(2)wherethepixelimotlyin focu, and an all in-focu image i generated. 4. Implementation and Experiment 4.1 Implementation of LFR Firt of all, thi ection explain our implementation of the LFR method, ince our method conit of iteration of image ynthei by LFR. We adopt an implementation with texture mapping [4, 5], which can be accelerated by graphic hardware. In thi method, each input image i projected onto the focal plane a a texture data weighted

5 t w(x,y)= (1- x/ )(1- y/ t ) (a) Normal filter. t w(x,y)= 4 1 (1- x/(2) )(1- y/(2 t) ) (b) Wide-aperture filter. t w(x,y)=(1- x/(2) )(1- y/(2 t) ) (c) Camera-kipped filter. Fig. 8: Reult of free-viewpoint image ynthei by the LFR method. From the top to the bottom, the viewpoint move from the right to the left while gazing at the nearet building. Thi lead to that the background move from the right to the left. Fig. 7: Weighting pattern for the filter. by uch a pattern that decribe the contribution of each input image for the interpolation of light-ray data. Shown in Figure 7 are weighting pattern adopted in our implementation and their formula for each recontruction filter. The weighting value i a function of poition (x, y), having a peak at the center and decreaing to zero at the edge. Circle repreent the grid point of the camera plane which denote the poition where input image were captured. For each texture data, the weighting pattern i aligned in uch a way that it center i on the grid point where the correponding input image wa captured. Then, the weighting pattern and the texture data are multiplied and projected onto the focal plane. The above proce i iterated for all the input image to yntheize an image. Figure 7(a), (b) and (c) correpond to a normal filter, a wide-aperture filter and a camera-kipped filter, repec- tively. Note that thee are approximation of the low-pa filter hown in Figure 3, 4 and 5. Thoe approximation are uitable for real-time implementation and ufficient to yntheize image with acceptable quality. The above decription correpond to the tep (1) and (2) in Subection 3.2 which are executed on graphic hardware fat enough for interactive frame rate. Then, yntheized image are tored in main memory, and a CPU execute the remaining procee of Step (3), (4) and (5). 4.2 Experiment For our experiment, we ue a Pentium GHz, Linux baed PC with 2.0 GB main memory. The video board load a nvidia Geforce4 Ti 4200 GPU and 128 MB video memory. Our method i implemented with OpenGL enhanced by GLUT librarie. A et of multi-viewpoint tatic image i ued for input image, which i from The multiview image databae courtey of Univerity of Tukuba, Japan. Thee image are captured at 81 (9 by 9, in the horizontal/vertical direction) viewpoint. The interval of viewpoint i 8 mm.

6 (a) z 0 = 300 (around the nearet building). (a) Normal filter. (b) z 0 = 600 (around the furthet building). (b) Wide-aperture filter. (c) z 0 = 1800 (around the background wall). (c) Camera-kipped filter. Fig. 9: Fouc effect appear according to the depth of the focal plane. Region near the focal plane are yntheized clearly and harply, while region apart from it eem blurry and noiy. Fig. 10: Syntheized image by different recontruction filter. In-focu region (further building) look imilar regardle of the filter, while defocued region have different artifact from each other.

7 The horizontal field of view i 27, and the reolution i reduced from pixel to pixel for the convenience of our implementation. Ditance from the camera plane to the nearet object, the farthet object and the background wall are 33 cm, 66 cm and 184 cm, repectively. Shown in Figure 8 are yntheized image by the LFR method. Photo-realitic view are yntheized depending on the uer viewpoint interactively. But thi conventional method uffer from the focu, which i noticeable in Figure 9. In Figure 9(a), (b) and (c), the focal plane i placed at 30 cm (around the nearet building), 60 cm (around the farthet building), and at 180 cm (around the background wall), repectively. Object near the focal plane are yntheized clearly and harply (in focu), while object apart from the focal plane are with blurring and ghoting (out of focu). To make everything in focu, the conventional method require that the interval of input camera hould be maller than 1.34 mm according to the plenoptic ampling theory [10]. It mean that the number of input image needed for thi cene amount to 2916, which i 36 time (6 by 6) larger than that of the original input image. However, our method generate all in-focu image without requiring any increae in the number of input image. All image mentioned above are produced by the normal filter. Shown in Figure 10 are yntheized image by the normal filter, the wide-aperture filter and the camerakipped filter with the focal plane placed at 60 cm. They how the important fact that the focued region are yntheized imilarly regardle of the filter, while the defocued region are yntheized differently according to the filter. Thi i the nature we utilize for focu meaure a decribed in Section 3.. Now we decribe the reult of our propoed method. Firt, the number of depth candidate i et a N =13, where in principle, all the object in the cene would be in focu at leat in one image of the equence of differently focued image. The recontruction filter h 1, h 2 and h 3 are aigned to the normal filter, the wide-aperture filter and the camera-kipped filter, repectively. We et that a 0,1 = a 0,2 =1,anda 1,2 = 0 in Equation (5), M =7in Equation (6), and k = 1 in Equation (8), are determined empirically. Their optimization i our future work. Shown in Figure 11(a) i a depth map, each pixel of which repreent the index of the optimal depth (n(x, y) in Equation (7)). The brighter pixel indicate an object cloer to the camera ytem. The tructure of the cene i etimated approximately. Thi reult i not o accurate for the ue of the 3D recontruction, but ufficient for the image ynthei. Figure 11(b) i a final image at the ame viewpoint, it can be een that all object are recontructed clearly and harply in comparion with thoe in Figure 9. Note that we can yntheize image like thi one at arbitrary viewpoint. It take econd to yntheize an image with pixel. There could be overhead cot in data tranfer from video memory to main mem- (a) Depth map. (b) All in-focu image. Fig. 11: The propoed method produce (a) depth map and (b) all in-fouc image. ory, and the procee on CPU are not well-optimized yet. Our future work include optimization and implementation technique to achieve interactive frame rate. 5. Concluion In thi paper, we have propoed a free-viewpoint image ynthei method that overcome the limitation of the conventional LFR method with no pre-etimated depth/ hape information. The propoed method generate all infocu image from differently focued image yntheized by LFR. Our approach i reaonable ince it utilize the focu of LFR to etimate the depth, which i directly related to the appearance of yntheized image. We have alo given a theoretical explanation on how to meaure focu in the yntheized image by LFR. Experimental reult how the effectivene of the propoed method.

8 Acknowledgement Thank to Prof. Hirohi Harahima of the Univerity of Tokyo for hi helpful dicuion. Reference 1. Marc Levoy and Pat Hanrahan. Light Field Rendering, Proc. ACM Conference on Computer Graphic (SIGGRAPH 96), pp (1996) 2. Aaron Iaken, Leonard McMillan and Steven J. Gortler. Dynamically Reparameterized Light Field, Proc. ACM Conference on Computer Graphic (SIGGRAPH 2000), pp (2000) 3. Daniel N. Wood, Daniel I. Azuma, Ken Aldinger, Brian Curle, Tom Duchamp, David H. Salein and Werner Stuetzle. Surface Light Field for 3D Photography, Proc. ACM Conference on Computer Graphic (SIGGRAPH 2000), pp (2000) 4. Steven J. Gortler, Radek Grzezczuk, Richard Szeliki and Michael F. Cohen. The Lumigraph, Proc. ACM Conference on Computer Graphic (SIGGRAPH 96), pp (1996) 5. Chri Buehler, Michael Boe, Leonard McMillan, Steven J. Gortler and Michael F. Cohen. Untructured Lumigraph Rendering, Proc. ACM Conference on Computer Graphic (SIGGRAPH 2001), pp (2001) 6. Takehi Naemura, Junji Tago and Hirohi Harahima. Real-Time Video-Baed Modeling and Rendering of 3D cene, IEEE Computer Graphic and Application, Vol. 22, No. 2, pp (2002) 7. Eric Krotkov. Focuing, International Journal of Computer Viion, Vol. 1, No. 3, pp (1987) 8. Shree K. Nayer. Shape from Focu Sytem, Proc. IEEE Computer Viion and Pattern Recognition (CVPR), pp (1992) 9. Peter J. Burt and Raymond J. Kolczynki, Enhanced Image Capture through Fuion, Proc. IEEE International Conference on Computer Viion (ICCV), pp (1993) 10. Jin-Xiang Chai, Shing-Chow Chan, Heung-Yeung Shum and Xin Tong. Plenoptic Sampling, Proc. ACM Conference on Computer Graphic (SIG- GRAPH 2000), pp (2000) 11. Jaon Stewart, Jingyi Yu, Steven J. Gortler and Leonard McMillan. A New Recontruction Filter for Underampled Light Field, Proc. Eurographic Sympoium on Rendering 2003, pp (2003) 12. Yutaka Kunita, Maahiko Inami, Taro Maeda and Suumu Tachi. Real-Time Rendering Sytem of Moving Object, Proc. IEEE Workhop on Multi-View Modeling & Analyi of Viual Scene (MVIEW 99), pp (1999) 13. Keita Takahahi, Takehi Naemura and Hirohi Harahima. Depth of Field in Light Field Rendering, Proc. IEEE International Conference on Image Proceing (ICIP 2003), Vol. 1, pp (2003)