Real-time High Dynamic Range Image-based Lighting

Transcription

1 Real-time High Dynamic Range Image-based Lighting César Palomo Department of Computer Science PUC-Rio, Rio de Janeiro, Brazil ABSTRACT In this work we present a real time method of lighting virtual objects using measured scene radiance. To compute the illumination of CG objects inserted in a virtual environment, the method uses previously generated high dynamic range (HDR) image-based lighting information of the scene, and performs all calculations in graphics hardware to achieve real-time performance. This work is not intended to be a breakthrough in real-time processing or in HDR image-based lighting (IBL) of CG objects, but still we expect to provide the reader with the overall IBL technique knowledge upto-date, an in-depth understanding of the HDR rendering (HDRR) pipeline in current graphics cards hardware, and interesting images post-effects which deliver higher realism for the scene final viewer. Keywords Image-based lighting, High Dynamic Range, Real-time Rendering, Shading, Graphics Hardware Programming 1. INTRODUCTION We describes the use of a previously captured HDR map with lighting information of a scene to illuminate computergenerated virtual objects as if they were seamlessly placed in the captured scene. All computations are performed using graphics hardware, by means of extensive use of vertex and fragment shaders in order to enable for real-time interaction and display. We provide effects like reflection, refraction, fresnell effect and chromatic dispersion [29] as options for the lighting equation. Furthermore, post-effects such as bloom [12] are produced to give the final scene a more realistic appearance. We make the simplification of not performing inter-reflection among virtually computer-generated objects to achieve a high frame-rate. Specially video games can benefit from this technique as it creates more realistic scenes than with standard lighting, which allowed developers and artists to produce incredible effects as in games like Half-Life 2:Lost Coast (Valve Software R ), Oblivion (Bethesda Softworks R ) and Far Cry (Ubisoft R ), to make the list short. The rest of this paper is organized as follows. In the next section we discuss related work relevant to this paper. Section 3 provides background about the image-based lighting technique and how lighting calculations are done, since this is the core of our method of lighting the virtual objects. Section 4 presents the concepts involved in HDR rendering: acquiring HDR images, storage, rendering, tone mapping and commonly used post-effects. Section 5 describes in depth the general method used in this work, joining the use of graphics hardware and techniques IBL and HDRR described in previous sections to produce real-time and realistic scenes. Section 6 presents results obtained, and we conclude in section RELATED WORK Debevec [5] uses high-dynamic range images of incident illumination to render synthetic objects into real-world scenes. However, that work employed non-interactive global illumination rendering to perform the lighting calculations. Significant work has been developed to approximate these full illumination calculations in real-time by using graphics hardware. As a particular example, [17] and [18] use multipass rendering methods to simulate arbitrary surface reflectance properties. In this work we use a multi-pass rendering method to perfom all the illumination and post-effects calculations to render the final scene at a highly interactive frame rate, similar to Kawase s 2003 work [19]. 3. IMAGE-BASED LIGHTING TECHNIQUE Image-based lighting can be summarized as the use of realworld images of a scene to make up a model representing a surrounding surface, and the later use of this model s lighting characteristics to correctly illuminate added subjects to the 3D scene. From the explanation above, we can identify two main decisions which need to be made when using IBL: how to represent the surrounding scene into a model and how to perform the illumination calculations of added subjects into the scene. Subsection 3.1 discusses the main kinds of environment maps commonly used with IBL, and also makes a brief comparison among them, while subsection 3.2 lists the lighting effects used in this work. 3.1 Environment Mapping Techniques

2 In short, environment mapping (EM ) simulates objects reflecting its surroundings. This technique assumes that an object s environment is infinitely far from the object, and that there is no self-reflection. If that assumptions hold, the environment surrounding the subject can be encoded and modeled in an omnidirectional image known as an environment map. The method was introduced by Blinn and Newell [3]. All EM methods start with a ray frow the viewer to a point in the reflector. This ray, then, is reflected or refracted with respect to the normal at that point. This resulting direction is used as an index to an image containing the environment to determine the color for the point on the surface Spherical Environment Mapping The early method described in 1976 by Blinn and Newell [3] is known as Spherical Environment Mapping. For each environment-mapped pixel, the reflection vector is transformed into spherical coordinates, which in turn are transformed to the range [0, 1] and used as (u, v) coordinates to access the environment texture. Despite of being easy to implement, this technique has several disadvantages as described in [1], such as the limitations of view-point dependency, distortions in the environment map s poles and the required computational time Cubic Environment Mapping Ten years later, in 1986, Greene [8] introduced the EM technique which is by far the most popular method implemented in modern graphics hardware, due to its speed and flexibility. The cubic environment map is obtained taking 6 projected faces of the scene that surrounds the object. This cube map shape allows for linear mapping in all directions to six planar texture maps. For that reason, the resulting reflection does not undergo the warping nor damaging singularities associated with a sphere map, particularly at the edges of the reflection. For its characteristics of being view-independent, not presenting singularities and common implementation in current graphics hardware, cube maps have been our choice in this work. 3.2 Lighting calculations Having made the decision of how to represent the model of the surrounding scene, the illumination model for the IBL technique still needs to be chosen. Below we briefly present the physical basis for the selected illumination models available in this work. A deeper explanation can be found in [29] Reflection R is used to access the cube map texture in the correct face. If we assume that the object is a perfect reflector such as a mirror, the vector R can be computed in terms of the vectors I and N with Equation Refraction R = I 2N(N.I) (1) When light passes through a boundary between two materials of different density (air and glass, for instance), the light s direction changes, since light travels more slowly in denser materials. Snell s Law describes what happens in this boundary with Equation 2. η 1 and η 2 are the refraction index for media 1 and 2, respectively. η 1 sin θ I = η 2 sin θ T (2) Basically, we use the built-in GLSL function refract to compute vector T to be used to lookup the environment map. Vector T is calculated in terms of the vectors I, N and the ratio of the index of refraction η 1/η Fresnell effect In real scenes, when light hit a boundary between two materials, some light reflects off the surface and some refracts through the surface. The Fresnell equation describes this phenomenon precisely, but since it is a complex equation, it is common to use the simplified version depicted in Equation 3. In this case, both reflection and refraction vectors are calculated and used to look up the environment map, but the resulting color in the incident point is calculated as shown in Equation 4 reflcoef = max(0, min(1, bias + scale (1 + I N) power )) (3) finalcolor = reflcoef reflcol+(1 reflcoef) refraccol (4) Chromatic dispersion When an incident vector I from the viewer s position reaches the object s surface in a point P, the reflection vector R is calculated taking into account the normal N at point P and the incident angle θ I, as depicted in figure This vector

3 The assumption that the refraction depends on the surface normal, incident angle and ratio of indices of refraction is in fact a simplification for what happens in reality. In addition to the mentioned factors, also the wavelength of the incident light affects the refraction. This phenomenon is known as chromatic dispersion, and it is what happens when white light enters a prism and emerges as a rainbow. Figure illustrates chromatic dispersion conceptually. The incident illumination (assumed to be white) is split into several refracted rays. That effect can be simulated in computers by making the simplification that the incident vector I is split into 3 refracted vectors, one for each RGB channel, namely T Red, T Green and T Blue. So, to calculate the resulting illumination, 3 ratios of indices of refraction are provided and used, one for each channel. 4. HIGH DYNAMIC RANGE IMAGING AND RENDERING Usual digital images are created in order to be displayed on monitors which support up to 16.7 million colors (24bits). For that reason, it is logical to store numeric images which match the color range of the display. For instance, image formats like bmp and jpeg traditionally use 16, 24 or 32 bits for each pixel. These image formats are known to have a low dynamic range, or LDR. HDR images basically use a wider dynamic range, which allows pixels to represent a larger contrast and a larger dynamic range. When a picture is taken with a conventional camera, the exposure time chosen by the photographer indicates which areas should be taken into account and therefore be captured by the picture, while the other areas are ignored. For instance, if a photographer chooses a long exposure time, bright areas will not be correctly captured by the picture, while details in dark areas will be visible in the final image. In constrast, if a short exposure time is set, only bright areas will reach the sensors at the camera, and so they will be the only areas with depicted details in the final image. In short, HDR rendering avoids any clipping of the values and everything is calculated more accurately. The result is an image that resembles reality more closely because it can contain both extremely dark and extremely bright areas. It allows the user to set a specific exposure time and simulate real photographs as if taken of the real scene. 4.1 Acquiring and Storage Format An early source of high dynamic range images in computer graphics were the renderings created by global illumination and radiosity algorithms. For instance, RADIANCE synthetic imaging system by Greg Ward [27] outputs its rendering in Ward s Real Pixels format [26]. Later, Debevec would present in his seminal paper [4] a method for acquiring real-world radiance using standard digital cameras photography. We use HDR images acquired with this technique in this paper, stored in RGBE format [28],[10]. 4.2 Tone Mapping Since conventional current monitors have limitations of contrast and dynamic range, with common contrast ratios between 500:1 and 3000:1, and HDR images have pixel values which by far exceed those limitations, we need to use a method called Tone Mapping to display those images in these conventional monitors. These tone mapping operators are simply functions which map values from [0; ) to [0; 1). Several operators have been developed and give different results, as can be seen in [7], [20], [22], [23] and [2]. Some of them can be simple and efficient to be used in real-time, while others can be very complex and customizable for image editing. The tone mapping operator used in this work is described in Equation 5, where Exp is the exposure level for the scene and Bright Level is a maximum luminance value for the scene. This operator allows the user to control exposure: with small exposure, the image is dark and some hidden details in bright areas appear, and conversely, with a high exposure, the image is very bright and contains details in dark areas. Y = Exp Exp Bright Level + 1 Exp Blooming A very common effect applied to HDR images is called bloom [12]. It enhances bright areas and highlights, giving a vivid impression to an image. To achieve a good blooming result as explained in [15], successive gaussian blur median filters [13] must be applied, with different kernel sizes. However, applying filters with big kernels does not allow for real-time performance, so an alternative approach is usually taken. With bilinear interpolation on, a gaussian filter is applied to the original image. Then this image is downsized (usually by powers of 2), and the gaussian filter is applied again to the resulting image. This process continues until a tiny image is obtained and no further downsizing improves the effect. Then, all these images are blended altogether, delivering a great blooming result. The trick of the technique is that, due to the image downsampling, different gaussian filter s kernels can be achieved with little expense. In fact, if a original texture of 128x128 is downsized to 64x64, then to 32x32 and finally to 16x16, and if a 3x3 kernel is used to blur all images, the tiniest texture, with 16x16 dimensions, blurs as if a 24x24 kernel were applied to the original texture, but using much less computations, and for that reason allowing for better performance. 4.4 Rendering Pipeline As already stated, HDR images have values beyond the [0, 1] range. For that reason, the normal 3D Graphics Pipeline can not be used to render those images, otherwise the image pixels values would be clamped to [0, 1] range. With the support for floating-point textures, arithmetic and render targets in graphics hardware, HDRR became possible, and no workarounds would be necessary such as those (5)

4 previously suggested by Debevec et al [6]. There are many methods for implementing HDR Rendering in current graphics hardware, but the following pipeline is very commonly used in OpenGL 2.0 API [16], which includes the application of bloom effect [12]: The HDR image is sent to the graphics card using the GL RGBA16F internal storage format, so that pixels values do not get clamped to [0, 1] range. To apply the bloom effect [12], first of all very bright areas of the image are extracted in a HDR Frame Buffer Object (FBO). This texture is then downsampled, with bilinear filtering active, into other textures. These down-sampled textures are blurred with a median filter shader [13]. The original HDR texture is composited by adding all down-sampled blurred textures with additive blending. In this phase we obtain a HDR Frame Buffer Object with the desired bloom effect, but its associated color texture still contains high-dynamic range values. Tone mapping is applied to the resulting HDR framebuffer to convert it to an LDR image, allowing for display by the conventional framebuffer. The operator described in Equation 5 was implemented in this work using the GLSL [24] shading language. Finally, a quad [16] is drawn with the LDR texture generated in the previous step active, so that the final image can be displayed for the viewer. 5. GENERAL METHOD To achieve realtime HDR IBL in this work, we apply a rendering pipeline consisting of the steps depicted below. For the sake of clarity, some of them are described in following subsections. A HDR cross image RGBE file [28], representing the environment scene, is loaded into memory. The 6 faces of this cube map image are extracted into 6 different images, corresponding to positive and negative directions for x, y and z axis [8]. A cube map texture is created using OpenGL s 2.0 GL RGBA16F ARB internal format [16], and each extrated face is sent to graphics hardware into its corresponding face of the cube map texture. Useful shaders for performing the lighting, applying the bloom effect and tone mapping the final HDR image are created. More details are provided in subsection 5.1. With one framebuffer object [16] FBO1 active, the cube map texture is applied to a big cube, which works as a skybox for the scene [14], representing the environment around the virtual objects. The virtual objects are positioned appropriately into the scene and their rendering is done still with FBO1 active and with the selected lighting effect (reflection, refraction, fresnell effect or chromatic dispersion) shader active, and with the cube map texture active and bound. At the end of this phase, FBO1 contains the rendered HDR scene in its color texture [16], with the virtual objects already illuminated accordingly to the environment. Another framebuffer object FBO2 and the shader responsible for extracting bright areas from a HDR image are activated, and with FBO1 s color texture active, a quad is [16] drawn. At this point FBO2 s color texture contains an HDR image with only bright areas. To apply the gaussian blur median filter, FBO2 s color texture is down-sampled into some other framebuffer objects FBO DS to allow for better blooming results [15]. At the end of this phase the collection of framebuffer objects FBO DS contains blurred HDR images, downsized from the original one. FBO DS color textures are composited with active blending into framebuffer object FBO2. That results in the bloom effect which can be added to the final image. Finally, FBO2 is composited with FBO1 into the very same FBO1, using the tone mapping shader to perform the HDR to LDR conversion. To display the final rendered scene, FBO1 s color texture is activated and a quad is drawn. 5.1 Shaders In this work, seven shaders (seven pairs of fragment and vertex shaders) perform all the illumination calculations, apply the bloom effect and tone map the final HDR image. All of them have been developed using the GLSL shading language [24]. Four shaders perform the illumination calculations of reflection, refraction, fresnell effect and chromatic dispersion [29] each. Two shaders are part of the bloom effect [12]: one extracts bright areas of an HDR image and the other applies the gaussian blur filter to an HDR image, outputing a blurred HDR image. To perform the tone mapping operator presented in Equation 5, there is one last shader whose output is an LDR image which can be displayed in the conventional OpenGL s framebuffer [16]. 6. RESULTS In a Intel Duo Core R 1.83MHz with a NVIDIA GeForce R Go 7600 graphics card, the technique implemented as presented here performed a frame-rate of 60 FPS at 640x480 screen resolution, lighting a model with 4500 triangles faces. Even at 1400x900 screen resolution, it can run at 20 FPS illuminating the same subject, with either available illumination effect, clearly mantaining a highly interactive rate.

5 This work provides a higher frame-rate than Kawase s one [19], which runs at 45 FPS at 640x480 screen resolution in the same machine, but with other effects like depth of field [11] and glare [25]. At 1400x900, its performance drops to 15 FPS, clearly showing that the addition of further effects really impairs the final performance of rendered scenes with HDR. The main dificulties in the course of this work have been finding good OpenGL FBO implementation examples and problems to debug the written GLSL shaders. After extense research, good code samples have been found for FBO implementation on OpenGL. To solve the issues on shaders, GLIntercept function call interceptor has been used. It could show textures status at each frame, allowing for debugging the results for each phase of the proposed pipeline for HDRR, easing the development process. In the end of this article we show some snapshots for results of our work. They illustrate how the bloom effect provides a more realistic impression of the scene, and how each one of the illumination effect applied to the virtual subject contributes differently to the resulting scene. 7. CONCLUSION In this work we presented a consolidated method for illuminating virtual objects inserted in a scene whose lighting characteristics are captured in a HDR image, using this environment lighting information to correctly calculate the shading of the CG subjects in real-time using graphics hardware computation power. We achieved a real-time performance as intentioned, and the final images presented to the viewer accomplished the aim of compositing virtual objects with a real-scene environment without viewer s noticing what is virtual and what is real, due to good illumination calculations made possible with the HDR values from the environment. The assumption that there is no inter-reflection among virtual subjects or with the surrounding scene is a necessary limitation in our work to enable for real-time performance. As a future work, this assumption could be mitigated by using the proposed method in Hakura s and Snyder s work [9], where they use a combination of minimal ray tracing for local objects and layered environment maps to produce reflections and refractions tha closely match fully ray traced solutions. Using their technique associated with graphics hardware programming through shaders programming would probably result in good visual results and improved performance if compared to their approach. Some improvements to this work, not done yet due to lack of time, could be the inclusion of other effects beside blooming, like depth of field [11] and glare [25], which can imitate some cameras lenses s physical characteristics in response to light, and can bring even more reality to the final composited scene.

6 8. REFERENCES [1] AKENINE-MOLLER, T., AND HAINES, E. Real-time rendering. (2002), 2nd edition, pp ISBN [2] ASHIKMIN, M. A Tone Mapping Algorithm for High Contrast Images. In 13th Eurographics Workshop on Rendering (2002) [3] BLINN, J.F., And M.E NEWELL. Texture and reflection in computer generated images. Communications of the ACM (October 1997), vol. 19, no. 10, pp [4] DEBEVEC, P.E., And MALIK, J. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 (August 1997), pp [5] DEBEVEC, P. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH 98 (July 1998). [6] COHEN, J., TCHOU, C., HAWKINS, T., And DEBEVEC P. Real-time high dynamic range texture mapping. In Proceedings of the Eurographics Rendering Workshop (2001). [7] DRAGO, F., MYSZKOWSKI, K., ANNEN, T. And CHIBA, N. Adaptive Logarithmic Mapping for Displaying High Contrast Scenes. In Eurographics 2003 [8] GREENE, NED. Environment Mapping and Other Applications of World Projections. IEEE Computer Graphics and Applications (November 1986), vol. 6, no. 11, pp [9] HAKURA, ZIYAD S. And SNYDER, J. M. Realistic Reflections and Refractions on Graphics Hardware With Hybrid Rendering and Layered Environment Maps. In 12th Eurographics Workshop on Rendering (2001), pp [10] HOLZER, B. High Dynamic Range Image Formats. (2006). [11] [12] (shader effect) [13] blur [14] (video games) [15] [16] [17] KAUTZ, J., AND MCCOOL, M. D. Approximation of glossy reflection with prefiltered environment maps. Graphics Interface (2000), pp ISBN [18] KAUTZ, J., AND MCCOOL, M. D. Interactive rendering with arbitrary BRDFs using separable approximations. Eurographics Rendering Workshop 1999 (June 1999). [19] KAWASE, M. Real-Time High Dynamic Range Image-Based Lighting. masa/rthdribl/ [20] PATTANAIK, S.N., TUMBLIN, J., YEE, H. And GREENBERG D.P. Time-Dependent Visual Adaptation for Realistic Image Display. In Proceedings of ACM SIGGRAPH (2000) [21] RANDIMA FERNANDO And MARK J. KILGARD The Cg Tutorial: The Definitive Guide to Programming Real-Time Graphics. ISBN-10: ISBN-13: [22] REINHARD, E. And DEVLIN, K. Dynamic Range Reduction Inspired by Photoreceptor Physiology. In IEEE Transactions on Visualization and Computer Graphics (2004) [23] REINHARD, E., STARK, M., SHIRLEY, P. And FERWERDA J. Photographic Tone Reproduction for Digital Images. In ACM Transactions on Graphics (2002) [24] ROST, RANDI J. OpenGL Shading Language. 2nd Edition. ISBN-10: ISBN-13: [25] SPENCER, G., SHIRLEY, P., ZIMMERMAN K., And GREENBERG D. P. Physically-Based Glare Effects for Digital Images. In SIGGRAPH (1995) [26] WARD, G. Real Pixels. Graphics Gems II (1991), [27] WARD, G. J. The RADIANCE lighting simulation and rendering system. In SIGGRAPH 94 (July 1994), pp [28] WARD, G. J. High Dynamic Range Image Encodings. [29] WESLEY, ADDISON Environment Mapping Techniques. /

7 Figure 1: Effects illustrations of this work: reflection (top-most, left), refraction (top-most, right), fresnell effect (two images in the middle), and chromatic dispersion (two bottom-most images)