Precomputed Radiance Transfer with Spatially-Varying Lighting Effects

Precomputed Radiance Transfer with Spatially-Varying Lighting Effects Masahiro Fujita Keio University SFC Graduate School of Media and Governance syoyo@sfc.keio.ac.jp Takashi Kanai Keio University SFC Faculty of Environmental Information kanai@sfc.keio.ac.jp Abstract We propose a new method to render physicallybased globally illuminated scenes at interactive speed. Our method is based on Precomputed Radiance Transfer (PRT) [0]. Although the shape of model does not change and its position is fixed, we can move any light source with arbitrary shape dynamically during rendering process. This enables a spatially-varying lighting effect which has not been considered in the previous approach [0]. We also describe the details of rendering the scene with dynamic lights using cube mapping. Our implementation demonstrates that our method is suited for interactive applications such as 3D games and lighting simulations. Keywords: interactive graphics, global illumination, PRT, spherical harmonics. Introduction In this paper, we propose a new interactive rendering technique based on Precomputed Radiance Transfer [0]. Our major contribution in this paper is to handle more general scenes with PRT, especially for indoor scenes which has middle scale lighting variation. Our implementation demonstrates that our method is suited for interactive applications such as 3D games and lighting simulations... Precomputed Radiance Transfer Precomputed Radiance Transfer (PRT) was first proposed by Sloan et al. in [0]. In PRT, spherical harmonic function is employed for the decomposition of incident lighting and geometric scattering. In [0], the light source is placed at infinity. A rendering equation is expanded using spherical harmonic functions. It replaces the integral of the rendering equation by a simple set of dot products using orthogonality of spherical harmonics. This enables dynamic movements of lights, using environment mapping and a rigid object, at an interactive speed..2. Related Work An application of spherical harmonic functions to computer graphics as the deviation of isotropic BRDF was first introduced by Cabral et al. []. They represented a light integral on a hemisphere as several dot products of its coefficient vector and then reduced its computational cost. Westin et al. [4] extended their approach to the case of anisotropic BRDF. Sillion et al. [8] used spherical harmonic functions for approximating the formulation of a radiance launched from a point in the radiosity method. Ramamoorthi et al. [7] represented a light integral of diffuse reflectance as several dot products of spherical harmonic coefficients in the scene of environment mapping. Kautz et al [4] proposed a method to combine Sloan s original PRT to an arbitrary BRDF. Lehtinen et al. [5] generalized their combined formulations in a matrix form, and improved the rendering speed by compression of the matrices using principal component analysis (PCA). Sloan et al. [] decomposed the original PRT into a macro-scale PRT with the accuracy of ordinary surface level and a meso-scale PRT with the accuracy of surface texture level. They were used in combination with bidirectional texture functions to establish more sophisticated rendering results with high accuracy. They also improved rendering speed by compressing the matrix of spherical harmonic coefficients using clustered principal component analysis (CPCA) in [9]..3. Our Contribution Our contributions presented in this paper are as follows: Spatially-varying lighting effect. In the previous approach by Sloan et al. [0], light sources are placed at infinity. This assumption causes constant color shading. For example, when applying the light to a plane, any point of the plane will be shaded with a constant color (spatially-invariant

(a) Figure. (a) Spatially-invariant lighting effect in [0]. Note that the ground plane is flat-shaded. Our spatially-varying lighting effect. Note that the ground plane is now shaded gradually. lighting effect). PRT is by nature suitable for rendering outdoor scenes. But it is difficult to render indoor scenes in which the scale of variation in lighting is small with this approach. In Sloan et al. [0], vector quantization is used to represent local lighting effects. They propose an approach for sampling points in a part of a model. Then lighting environments are re-sampled from these points. Our approach considers the attenuation term by the distance from a light position to a point of an object. It enables a spatially-varying lighting effect even when applied to a plane. Figure illustrates the comparison between spatiallyconstant (left) and spatially-varying (right) lighting effects. It can be seen that in our approach the ground plane is shaded gradually compared to the previous one which is flat-shaded. PRT can be used in more general scenes including indoor scenes thanks to our spatially-varying lighting effects. Application to the dynamic lighting environments. In the previous approach, lighting environments are given by environment mapping, dynamic variation of lighting is done by rigid rotations thanks to rotation-invariant properties of spherical harmonics. Thus the environment of lighting is fixed. In our approach, We use a cube mapping to render the scene with moving lights or its background scene. The usage of cube mapping has already been pointed out by Sloan et al. [0]. However, we show here the details of its implementation, and demonstrate its use for more general lighting environments. 2. Interactive Global Illumination Using PRT 2.. Pseudo Code We show in this subsection a pseudo code of our approach. Our approach is divided into two steps. One is the simulation of irradiance transformation as pre-processing, and the other is an interactive rendering process. First, the transfer simulation of radiance is described as follows: Generate sampling points using [2]; For each vertex x Compute coefficients for shadow pass M 0 x; For b =, 2,... Compute coefficients for interreflection pass M b x; end M x = M 0 x + M x +...; end We store M calculated by transfer simulation. In the interactive rendering step, the following procedure is executed by using M. Compute lighting coefficients L by cube map rendering; For each vertex x; compute matrix-vector multiplication of M x and L; end

x (a) q q 2 q 3 Figure 2. (a) Shadow pass. Rays are shot towards the hemisphere and a flag (bit) is set where a ray hits the hemisphere. Interreflection pass. Transfer coefficients Mq b i are computed from each q i to point x. 2.2. Transfer simulation In the pre-process, transfer simulation is done by the similar procedure as radiosity method. Each transferred radiance is expanded by using a set of spherical harmonic functions, and transfer coefficients are computed for each vertex of an object. This process can be done by the following two steps. We use here a quasi-monte Carlo method [2] for generating sampling rays on a hemisphere. The low-discrepancy sequence we have choosen is the Hammersley point sets. In the first shadow pass, a shadow of a ray directly entering each vertex is computed. In the following interreflection pass, inter-reflective effects are computed. We use transfer coefficients computed by the shadow pass. This computation is repeated until the whole energy integral is less than a threshold. Diffuse reflection The transfer simulation of diffuse reflection (Lambert BRDF) is processed as follows: Shadow pass. The following integral is computed for each vertex (See Figure 2(a)). (M x ) 0 i = ρ x π Ω x M b q q M b q2 M b q3 V (x, ω ) max(cos θ x, 0)y i (ω )dω, where (M x ) 0 denotes the transfer vector in the shadow pass for a vertex x. A subscript i is defined as i = l(l +)+ m +where l, m denote parameters of spherical harmonic functions. ρ x (0 ρ x ) denotes the diffuse reflectance at a point x. V represents a two-valued visibility function which is equal to if no interruption exists between x and ω, otherwise it is equal to 0. This equation is computed for each RGB value. q 2 q 3 A binary flag is set to for a sampling direction with an interruption. It is used to reduce the computational cost of ray tracing in the following inter-reflection pass. Inter-reflection pass. The following integral is computed based on the transfer vector (M x ) 0 calculated in the previous pass (See Figure 2). (M x) b i = ρx π Ω ( V (x,ω )) max(cos θ x,0)(m q(ω ) ) b i dω, where b ( ) denotes the number of inter-reflection pass. This integral is computed only for sampling directions which have a binary flag. q(ω ) is a point hit at first on a direction of ω. The computation of transfer simulation is done for each vertex. Then it is rare that q(ω ) is exactly on a vertex. Thus, M q is computed by linear interpolation of M between vertices neighbor to q. We record the whole transfer energy for each interreflection pass. We stop the computation if it is less than a threshold. In practice, only three or four iterations are sufficient due to the rapid decrease of the energy. A transfer vector at a point x is represented by the sum of those computed in shadow pass and in inter-reflection pass: (M x ) i =(M x ) 0 i +(M x ) i +(M x ) 2 i + = N (M x ) k i, k=0 where N is the number of passes. Figure 3 demonstrates the results of three different settings (without interruption of diffuse reflectance transfer, shadow pass only, and shadow pass + inter-reflection pass). The case without interruption (Figure 3(a)) is just the same as the irradiance environment mapping described in [7]. It is brighter than the other two settings, because the plane below the model does not stop the light from illuminating the model. The difference between a shadow pass (Figure 3) and both a shadow pass and an inter-reflection pass (Figure 3(c)) can not be easily distinguished. However, in the latter case, the shadowed regions are slightly brighter due to inter-reflective effects. Glossy reflection Glossy BRDF can also be represented by PRT. This has an obtuse reflectance property in between Lambert BRDF and ideal specular reflectance BRDF. Here we treat only an isotropic case described in [0]. It should be noted that the transfer coefficient in glossy BRDF is represented by a matrix and not by a vector [0]. Shadow pass. A transfer coefficient (M x ) 0 ij computed in the shadow pass is as follows: (M x ) 0 ij = V (x, ω )y i (ω )y j (ω )dω. Ω

(a) (c) Figure 3. (a): No occlusion. : After shadow pass. (c): After shadow pass + inter-reflection pass. scene is surrounded with a virtual sphere. The lighting environments projected by cube mapping (described later) correspond to such a virtual sphere. We also assume that the light source is outside the sphere. In the first incidence of a light, we use the formulation which is measured by their surface, that is, the area A(x) for a point x on a surface is integrated by da(x) as intervals of the whole surface A R2. The rendering equation re-formulated by its surface measure is as follows: (a) Figure 4. Different BRDF models. (a): Using diffuse reflection. : Using glossy reflection. Inter-reflection pass. A transfer coefficient (Mx )bij computed in the following inter-reflection pass is as follows: b (Mx )ij = ( V (x, ω )) Ω n αk (G xq(ω ) )k (Mq(ω ) )b kj yk R( ω, Nq(ω ) ) yi (ω )dω, L(x, x ) = L e (x, x ) L(x, x )f (x, x )V (x, x ) + A cos θx cos θx da(x ), x x 2 () Since a point in the lighting environment can be projected to that on a surface of a sphere by cube mapping, we get the following transfer coefficients by treating an interval of a scene surface A as that of a sphere with radius r: ρx 0 (Mx )i = V (x, x )G(x, x )yi (x )dx, π S cos θ where α is a constant and G x is a coefficient of projected isotropic glossy BRDF at a point x by spherical harmonic functions. where G = r(x xx ) 2. A term of cos θx in Equation () can be omitted because it is included in cube mapping. In the second inter-reflection pass or later, we use an original formulation based on a solid angle. Consequently, we can represent a PRT including the attenuation by the distance from a light source. 3. PRT with Spatially-varying and Dynamic Lights 3.2. Generation of the distribution of lights by cube map rendering 3.. Spatially-varying formulation of rendering equation We get a distribution of lights incident to a point by using cube mapping. This can be done using a cube map constructed by rendering each face of a cube with a camera put on its origin. We set fov of the camera to 45. The camera direction and the up vector for each face are set as de- k=0 Here we describe our novel spatially-varying formulation. In spatially-varying lighting, we assume the whole

Face direction view direction up vector +x +x y x x y +y +y +z y y z +z +z y z z y Table. Coordinates of each face in cube map rendering. Z y Figure 5. The derivation of pixel weight. scribed in Table (which are based on a coordinate system used in OpenGL). By this, we can get a lighting environment of a light with arbitrary shape. For each pixel of textures from cube map rendering, we can quickly compute an expansion of spherical harmonic functions by multiplying spherical harmonic functions which are computed in advance. Correction for a pixel Distortion arises at the corner of a cube by projection. This happens because different pixels in each face of a cube map have no longer the same solid angle. To solve this issue, a correction is applied for a pixel as in [2]. In Figure 5, a weight correction W of (x, y) for upper and lower faces is as follows: θ r = x 2 + y 2 + cos θ = x2 + y 2 + W (ω(x, y, )) = x2 + y 2 + A Y x X where A =4π/N. N denotes the total number of pixels for six faces of a cube. We can also compute the correction W (ω(,y,z)) = A for other four side y2 +z 2 + faces. Finally, a spherical harmonic function for each pixel is computed by SH k (s, t) i,j = y i,j (ω(k, s, t))w (ω(k, s, t)), where k denotes a face id, (s, t) denotes the uv-coordinate of a texture, and ω(k, s, t) denotes the unique direction clearly defined by (k, s, t). 3.3. Interactive rendering Using transfer coefficients as pre-computation and lighting coefficients by dynamic cube map rendering, we can render the scene with dynamically moving lights by GPU. It is easy to implement the computation of this stage by using the vertex program on GPU. Diffuse reflection In the case of diffuse reflection, radiance L x for each vertex can be computed simply by the following sum of dot products: L x = N (M x ) i L i, (2) i= where N denotes the number of coefficients of spherical harmonic functions, and L denotes the lighting coefficients computed by dynamic cube map rendering. Glossy reflection In the case of glossy reflection, the multiplication of a matrix and a vector is needed. L x = N α i G i ((M x ) ij L i ) y i (R), (3) i= where R denotes an inverse direction of a viewing vector. α and G can be computed the same fashion as described in Section 2.2. 4. Results and Discussion All of our programs are implemented in C language. We also use OpenGL for rendering and Cg [6] for the vertex program functionality on GPU. We use a software renderer lucille [3] to pre-compute transfer coefficients. For the expansion of spherical harmonic functions, we set l =4, e.g. coefficients of N =25. This is because it is the maximum value that can be processed with a current CPU and GPU. In our experiments, the case of N =25presents a good approximation for the light distribution of low frequency. If we treat only the ideal diffuse reflection and the complexity of occlusion is adequately small, N =9also gives a good approximation [7]. In the case of diffuse reflection, Equation (2) can be processed by a simple combination of the

Mobility Radeon 9600 GeForce FX 5900 Ultra diffuse glossy diffuse glossy stage seconds (%) seconds (%) seconds (%) seconds (%) Scene rendering 0.500 (83%) 0.0400 (6%) 0.0260 (70%) 0.0260 (0%) Cube map rendering 0.002 (0.7%) 0.000 (0.%) 0.005 (4%) 0.005 (0.6%) glreadpixels 6 0.0028 (.5%) 0.0028 (0.4%) 0.006 (6%) 0.006 (2.4%) Lighting coefficients computation 0.080 (0%) 0.080 (3%) 0.0032 (9%) 0.0032 (%) Glossy coefficient matrix manipulation - 0.6000 (90%) - 0.200 (84%) Others 0.0080 (4%) 0.0030 (4%) 0.0002 (%) 0.0032 (%) Total 0.8 sec. 0.66 sec. 0.037 sec. 0.25 sec. Frame rate 5.6 FPS.5 FPS 27 FPS 4 FPS Table 2. Computation time for each frame. summation and multiplication, and then a vertex program on GPU can be used. In the case of glossy reflection, coefficients of glossy transfer is a 25 25 matrix when N =25. In the matrix computation, however, the number of elements is much more than the number of instructions which a vertex program can be processed at a time. We use CPU computation only for this case. Table 2 shows the detail of computation time in each frame. These timings are measured on two machines (PowerPC G4.25 GHz / ATI Mobility Radeon 9600 / Mac OS X, and Pentium 4 3.2 GHz / NVIDIA GeForce FX 5900 Ultra / Windows XP). The scene used for measuring computation time is Stanford dragon scene (Figure 4) consisting of about 50k polygons. Dragon diffuse is shown in Figure 4(a) and Dragon glossy in Figure 4. Scene rendering is the rendering time of polygons using vertex program on GPU in final rendering stage. Cube map rendering is the rendering time based on cube map rendering in the scene with dynamic lights. glreadpixels counts the time consumed by transferring cube map pixel memory on VRAM to CPU memory. lighting coefficients computation calculates lighting coefficients convolving read-backed cube map pixel values and pre-computed spherical harmonic functions. Coefficients for the diffuse surface expanded by spherical harmonics and lighting coefficients are multiplied in vertex program. For the glossy surface, we calculate shading for each vertex on CPU (this is listed in Table 2 as Glossy coefficient matrix manipulation) because the data associated to each vertex can not fit into the hardware limit of vertex program. Interactively rendered images using our method are shown in Figure 6. We used area light sources to illuminate these scenes. The number of lights can be freely chosen and no performance drops occurs even if hundreds of lights are placed in the scene. Each of these scenes consists of 20-50 k polygons which can be rendered interactively. 5. Conclusion and Future Work We proposed a new interactive rendering method by an extended PRT method. It can be implemented on GPU almost entirely if the scene consists of diffuse surface. Our method is suited for scenes with middle scale lighting variation such as indoor environment, which is difficult to represent in the previous work. Our future work includes the support for more general BDRFs, data compression and animated objects. Acknowledgements The dragon and Buddha models are courtesy of Stanford University, Computer Graphics Laboratory, other models we used is from 3D cafe. We would like to thank Mr. Alexandre Gouaillard for proofreading this paper. We would also like to thank Prof. Yoshinori Dobashi, Hokkaido University for useful discussions and Prof. Tomoshi Miyamura, Nihon University for granting us the usage of 6 CPU parallel PC cluster. A. Spherical Harmonic A.. Real Spherical Harmonic Spherical harmonic is a complex function, but in computer graphics we need only real part of it. So we use real spherical harmonic. There is a correspondence between spherical harmonic Y l,m and real spherical harmonic y l,m. y l,m = 2 (Y l,m + Y l, m ) m>0 i (Y 2 l,m Y l, m ) m<0 Y l,m m =0 Rewriting it in terms of associated Legendre polynomial P l,m, we get

(a) (c) (d) Figure 6. Interactively rendered images using our method. yl,m m>0 2Kl,m cos(mφ)pl,m (cos θ) = 2Kl,m sin( mφ)pl, m (cos θ) m < 0 m = 0, Kl,m Pl,m (cos θ) where Kl,m = 2l+ (l m)! 4 (l+m)!. A.2. Spherical Harmonics in Cartesian Coordinates Solid harmonic rl Yl,m (θ, φ) in spherical coordinates (r, θ, φ) can also be described in Cartesian coordinates [3]. r Yl,m (θ, φ) l = 2l + (l + m)!(l m)! 4π x + iy p x iy q s ( ) ( ) z p!q!s! 2 2 p,q,s p, q, s are positive integer numbers. Summation is taken over all combination of p, q, r that fulfills p + q + s = l, p q = m. If we think a unit sphere(r = ), real spherical harmonic in Cartesian coordinates is represented by simple polynomial up to order l. Table 3 lists real spherical harmonic in Cartesian coordinates up to l = 4.

l=0,,2 l=3 l=4 y 0,0 = 4π 3 y, = 3 y,0 = 3 y, = y 4π z 4π x 4π 5 y 2, 2 = xy 4π 5 y 2, = yz 4π 5 y 2,0 = 6π (3z2 ) 5 y 2, = xz 4π 5 y 2,2 = 6π x2 y 2 35 y 3, 3 = 32π (3x2 y y 3 ) 05 y 3, 2 = xyz 4π 2 y 3, = 32π y(5z2 ) 7 y 3,0 = 6π (5z3 3z) 2 y 3, = 32π x(5z2 ) 05 y 3,2 = 6π z(x2 y 2 ) 35 y 3,3 = 32π (x3 3xy 2 ) 35 y 4, 4 = 6π (x3 y xy 3 ) 35 y 4, 3 = 32π z(3x2 y y 3 ) 45 y 4, 2 = 6π xy(7z2 ) 45 y 4, = 32π (7yz3 3yz) 9 y 4,0 = 256π (35z4 30z 2 +3) 45 y 4, = 32π (7xz3 3xz) 45 y 4,2 = 64π (x2 y 2 )(7z 2 ) 35 y 4,3 = 32π z(x3 3xy 2 ) 35 y 4,4 = 256π (x4 6x 2 y 2 + y 4 ) Table 3. Real spherical harmonics up to l 4 with normalized vector (x, y, z) representation in Cartesian coordinates. References [] B. Cabral, N. Max, and R. Springmeyer. Bidirectional reflection functions from surface bump maps. In Proceedings of ACM SIGGRAPH 987, pages 273 28, 987. [2] M. F. Cohen and D. P. Greenberg. The hemi-cube: A radiosity solution for complex environments. Proceedings of ACM SIGGRAPH 985, 9(3):3 40, 985. [3] M. Fujita and T. Kanai. lucille: Open source global illumination renderer. In The 7th IASTED International Conference on Computer Graphics and Imaging (CGIM 2004), 2004. to appear, http://lucille.sourceforge.net/. [4] J. Kautz, P.-P. Sloan, and J. Snyder. Fast, arbitrary BRDF shading for low-frequency lighting using spherical harmonics. In Proceedings of Thirteenth Eurographics Workshop on Rendering, pages 29 296, 2002. [5] J. Lehtinen and J. Kautz. Matrix radiance transfer. In Proceedings of the 2003 symposium on Interative 3D graphics, pages 59 64, 2003. [6] W. R. Mark, R. S. Glanville, K. Akeley, and M. J. Kilgard. Cg: a system for programming graphics hardware in a c-like language. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2003), 22(3):896 907, 2003. [7] R. Ramamoorthi and P. Hanrahan. An efficient representation for irradiance environment maps. In Proceedings of ACM SIGGRAPH 200, pages 497 500, 200. [8] F. X. Sillion, J. R. Arvo, S. H. Westin, and D. P. Greenberg. A global illumination solution for general reflectance distributions. In Proceedings of ACM SIGGRAPH 99, pages 87 96, 99. [9] P.-P. Sloan, J. Hall, J. Hart, and J. Snyder. Clustered principal components for precomputed radiance transfer. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2003), 22(3):382 39, 2003. [0] P.-P. Sloan, J. Kautz, and J. Snyder. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2002), 2(3):527 536, 2002. [] P.-P. Sloan, X. Liu, H.-Y. Shum, and J. Snyder. Bi-scale radiance transfer. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2003), 22(3):370 375, 2003. [2] W.-S. L. Tien-Tsin Wong and P.-A. Heng. Sampling with hammersley and halton points. Journal of Graphics Tools, 2(2):9 24, 997. [3] D. A. Varshalovich, A. N. Moksalev, and V. K. Khersonskii. Quantum Theory of Angular Momentum. World Scientific Publishing Co., Singapore, 988. [4] S. H. Westin, J. R. Arvo, and K. E. Torrance. Predicting reflectance functions from complex surfaces. In Proceedings of ACM SIGGRAPH 992, pages 255 264, 992.