Spotlight Shadow Mapping in OpenGL

Transcription

1 SCHOOL OF ELECTRONICS AND COMPUTER SCIENCE FACULTY OF ENGINEERING, SCIENCE AND MATHEMATICS UNIVERSITY OF SOUTHAMPTON Spotlight Shadow Mapping in OpenGL Advanced Computer Graphics ELEC6025 Tony Ambrus

2 Abstract The report details a project submitted for Advanced Computer Graphics ELEC6025. The project focuses on the issue of real-time shadowing though an image-space technique called shadow mapping. The field is examined and discussed; the steps and issues with implementation found over the course of the project explained. Realtime shadow mapping is achieved and various optimisation and extensions made, including: large mesh sub-division, multiple light sources, and texture-masked shadows. i

3 Contents Chapter 1 Introduction... 1 Chapter 2 Analysis Background Research Algorithms Projective Texturing Shadow Mapping Project Goals... 7 Chapter 3 Design Mesh Material Model & Entity Sector Visible Light Set Light Shader... 9 Chapter 4 Implementation Shadow Mapping Complex Meshes Terrain Optimisation Case Study Mesh Multi-Texturing Multiple Light Sources Visible Light Sets Clipping Omni-Directional Lights Texture-Masked Shadowing Chapter 5 Testing Chapter 6 Evaluation Reflection Future Work More Effective Light-Camera Object Culling Soft Edged Shadows Chapter 7 Bibliography Appendix CD Contents ii

4 Chapter 1 Introduction Shadows are an important part of our perception of the world. They provide important depth cues such as information on the surface on which a shadow is cast, the shape of the object, and the relative position of the light. For graphics applications, they provide a much more realistic scene. Shadow Mapping is an image-based method of producing real-time shadows, by partitioning the scene into two regions; that in direct light and that occluded by other objects. The report examines the steps involved in implementing Shadow Mapping, as well as optimisations and extensions that can be made to increase performance and aesthetics. 1

5 Chapter 2 Analysis 2.1 Background Research Much of the research on shadowing in the past has focused on non-real-time implementations: shadow mapping was first seen publicly in Pixar s first feature film Luxo Jr. in It has only been relatively recently that hardware has advanced sufficiently to make real-time usage an option. Now that applications can handle shadowing in real-time, shadow mapping, by itself or as a hybrid, is taking over as the preferred method. Shadow mapping was developed by Lance Williams in Casting Curved Shadows on Curved Surfaces [1], pointing out that the use of a z-buffer for visible surface determination could also be used as a binary visibility depth map. Cass Everitt of NVIDIA wrote Hardware Shadow Mapping [2] in 2001, becoming the de facto paper on standard shadow mapping. The paper contains detailed explanations on the algorithm and provides implementation details for both OpenGL and DirectX (see section 2.2.2). Shadow mapping is based on the use of projective textures; a method of generating texture coordinates such that the size of the texture is linearly proportional to the distance from the projection origin, by the use of homogeneous coordinates. Andrei Gradinari presents a method in [7c], Everitt s paper on Projective Texture Mapping [3], explains in greater depth the mathematical transformations and issues involved. 2

6 2.2 Algorithms Projective Texturing Projective texturing is a method of mapping a texture onto an object such that the texture is projected onto the scene, as shown in Figure 1. This is accomplished by the use of inbuilt texture co-ordinate generation facilities. Figure 1 Two views of a projective texture, courtesy of NVIDIA. OpenGL provides access to the eye coordinates stage of the transformation pipeline (see Figure 2) for texture coordinate generation, so incoming fragments are specified in eye space as opposed to object space. Eye coordinates are subsequently transformed by the texgen planes 1 and the texture matrix, both of which are user specified. 1 Four plane equations that for the purposes of the project can be viewed as a 4x4 matrix. 3

7 Figure 2 The OpenGL Transformation Pipeline, courtesy of NVIDIA. To produce projective texturing, coordinates in object space (x e,y e,z e,w e ) must be transformed into homogeneous texture coordinates (s,t,r,q) such that (s/q, t/q) are coordinates in real space and r/q is the z-distance from the origin in light space. This can be calculated as follows: [ T 1 T s, t, r, q] = SPlight L V[ xe, ye, ze, we ] Equation 1 Here, the inverse of the view matrix, V, shifts the point from eye space to world space, as shown in Figure 3. Note the convention of defining the forward transforms as going to world space. Figure 3 - Basic transformations involved in shadow mapping, courtesy of NVIDIA Applying matrix L -1 moves the system to light space and P light sets the projective matrix for the light frustum. However, the resultant coordinate system has the range [- 1, 1], so to convert to a valid texture space [0, 1] a scale-bias matrix must be applied: 4

8 Equation Looking at Figure 2, the two transformations applied to eye coordinates produce (s,t,r,q). The SP light L -1 V transformation can thus be spread between the texgen planes E po and texture matrix T: T T [ s, t, r, q] = TE [ x, y, z, w ] TE po = SP light L 1 po V e Equation 3 For eye linear texgen planes, OpenGL will multiply the eye coordinates by the inverse of the modelview matrix in effect at the time the planes were set. Setting the modelview matrix to V -1 will thus result in the elimination of V from Equation 3. As there are multiple methods of splitting the remaining SP light L -1 transformation between the texture matrix and the texgen planes, the texture matrix was chosen to hold the full transformation due to ease of implementation; the texgen planes were subsequently set to the identity matrix. e e e 5

9 2.2.2 Shadow Mapping Shadow mapping utilises the fact that the depth buffer, from the light point of view, is a pre-computed light visibility test for the light s view frustum, partitioning the volume into lit and shaded regions. The depth test from eye s point of view can be written as: pz shadowmap( px, p y ) Where p is a point in light space and shadowmap is the depth buffer texture. The test fails if the z value of the incoming point is further from the light source than the value stored in the shadow map texture. In Figure 1, only points lying on the green line are in the lit region. Figure 4 Objects in world space (left) transformed into light space (right). Note the green line denoting the minimum distance from the light source, representing the values in the depth buffer. The steps involved in creating the shadow map can be summarised as follows: 1. Transform the camera to view the scene from the light s perspective, setting the projection matrix to reflection the type of the light (either parallel or perspective). 2. Set the viewport to match the size of the depth texture being used. An offscreen framebuffer object can be used to allow depth textures with greater resolution than the application window. 3. Disable all rendering states such as texturing and lighting to increase performance as only the depth values are required. 4. Copy the depth buffer into a depth texture. The shadow map only needs to be updated when the light moves or an object intersecting the light frustum moves. However, this is usually every frame for dynamic scenes. Once the shadow map has been generated, shadows can be added to the scene by doing the following: 6

10 1. Render the scene without any rendering states to obtain a black silhouette, but with the correct depth values for the scene. 2. Project the shadow map onto the scene using projective texturing. 3. Render the scene with full lighting; at each fragment, test the distance of the current fragment from the light against the value in the shadow map. If it is larger, the fragment is occluded from the light by a nearer object and should remain unlit. If the value is equal, the fragment is the closest to the light and should thus be lit. The last step can be performed by the use of OpenGL extensions, resulting in a white or black texture colour to represent passing and failing respectively. The texture can be modulated with the object s texture to give the illusion of per-pixel shadowing. 2.3 Project Goals This section outlines the goals of the project as derived from research and the project proposals. The goals defined were as follows: 1. Implement a framework displaying basic shadow mapping 2. Add the ability to apply shadow mapping to arbitrary meshes 3. Add multi-texturing to allow both shadow texturing and user-defined textures on models. 4. Formulate a method of tracking what lights are intersecting a model, increasing performance by only drawing a silhouette of the model if not lit. 5. Allow multiple light sources, additively blending the results. 6. Allow omni-directional lights 7. Add the ability to perform texture-based masking for shadows, increasing shadow detail without affecting computation time. 7

11 Chapter 3 Design The system has a very modular object-orientated design, abstracting the lighting system from the physical meshes, which are maintained in a scene-graph. The engine used to demonstrate shadow mapping is very complex. However, only a subset of the modules implemented the actual shadow mapping functionality, described below. 3.1 Mesh A mesh is the most basic geometric object, represents a number of polygons that share the same material. As this module constituted the bulk of the fill rate, it was important that optimisations were made to reduce the potential of it being a bottleneck. 3.2 Material Materials are comprised of the textures to apply to the mesh, the multi-texturing combination parameters and the light reflectivity proportions (ambient, diffuse and specular). 3.3 Model & Entity A model is one or more meshes rendered in sequence in the same co-ordinate system, allowing multi-material objects to be displayed. An Entity is a physical object that can interact with other Entities in its Sector. Entities do not own Models, but borrow a handle from the Model manager. In this way a single Model can be used by multiple Entities without incurring additional memory overhead. 3.4 Sector This is a container to package Entities and Lights. The system renders only the Sector the camera is viewing, so multiple scenes can be maintained in parallel and switched between by simply changing the viewport Sector. This feature will be used to show the various stages of development in the final demonstration program. 3.5 Visible Light Set This module is used by all Entities to maintain a record of which lights are relevant to the Entity, minimising the rendering iterations as each shadow requires a render pass. Completely unlit meshes can be rendered without texturing, maintaining the silhouette of the mesh against lit backgrounds with a lower overhead. 8

12 3.6 Light The Light class is the heart of the lighting model, consisting of two stages: the first transforms the scene to the light s projection and sets up the system to render the depth buffer to a texture. The second stage sets up the projective texture to map the depth texture onto the scene. 3.7 Shader Shaders are programs that can be pushed on the graphics card at runtime. Vertex shaders affect the graphics pipeline on a per-vertex basis, whereas pixel shaders determine the surface properties of a polygon on a per-pixel basis. For the purposes of the project, only pixel shaders were utilised to implement per-pixel lighting. The Shader module facilitates the use of shader scripts by exposing an API to compile, bind and to manipulate shader parameters. 9

13 Chapter 4 Implementation The implementation of the framework and algorithm proceeded quite smoothly. However, the first version had a moiré pattern (see Figure 5), arising from selfshadowing errors. This is due to the variation in sampling frequencies of the shadow map in light space compared to the framebuffer in eye space as demonstrated by Figure 6. Figure 5 Without a polygon offset, shadow mapping causes a moiré pattern Figure 6 The duelling frusta problem shows the variation in sampling frequency between eye and light space; the blue area near the camera (left) requires high frequency sampling, but is only a small portion of the shadow map from the light s view (right). Image courtesy of NVIDIA. Polygon offset was used to remove this effect, by slightly increasing the z-values of polygons when rendered to the depth map. This offset can be a constant bias or a value linearly proportional to the z-gradient of the polygon as seen from light space. The effect is not just a lack of resolution or precision; sampling frequency 10

14 considerations must also be made. As a polygon s z-gradient increases and it becomes parallel to the view direction, the number of texels it occupies in the shadow map decreases. Figure 7 Effect of polygon offset on self-shadowing error for differing z-gradient polygons. After the application of suitable values for the polygon offset, the moiré pattern disappeared, resulting in a correctly shadowed scene. Figure 8 shows the light frustum from which the shadow map is calculated, and in the bottom-left corner the depth map can be seen showing the z-values of the scene in light space. Figure 8 Basic shadow mapping with polygon offset applied. 11

15 The lighting system at this stage can be reduced to pseudo-code as follows: void Sector::render() { if(camera.isvisible(light.frustum)) { setupviewprojection(light); drawentities(); } light.depthmap = getdepthbuffer(); setupprojectivetexture(light.depthmap); } setupviewprojection(camera); drawentities(); Code 1 Basic shadow mapping. 4.1 Shadow Mapping Complex Meshes The next stage of development was to handle arbitrary meshes in lieu of the simple geometric objects provided by the GLU and GLUT libraries. This required refactoring the shadow mapping algorithm to use a modular approach which was object agnostic. The initial version had used a single monolithic render function developed from tutorials on the internet, most usefully [6], although their approach used the texgen planes to hold the projective transformation matrix. The subsequent version implemented the system as described in the design moving the entities and model into a scene-graph Sector, abstracting the physical world from the lighting model. Figure 9 Two complex meshes (both the bear and plane are imported models) showing correct shadow mapping. Due to the mesh being the unit of granularity in the system, other steps must be taken to maintain reasonable frame-rates for highly detailed objects. For the project, terrain was taken as an example of a large scale mesh that must be optimised for shadow mapping purposes. 12

16 4.1.1 Terrain Optimisation Case Study The basic tenet of optimisation is to only render a subset of the entire scene. The most effective method is the use of frustum-culling to determine if an object s boundingbox intersects the camera s view frustum. There are many methods of subdividing a large mesh, but due the regularity of the heightmap for terrain a quadtree is usually utilised. This is mainly due its ability to hierarchically dismissal of sub-nodes; if a parent is not visible none of the descendants will be. Additionally, determining if an arbitrary node is visible only takes O(log n), where n is the level of the node in the tree. Figure 10 Showing three levels of subdivision of the quadtree, and the associated levels of detail. Performance can further be increased by reducing both the number of triangles rendered to screen and the number of OpenGL render calls. A chunked level-of-detail (CLOD) algorithm is used for the terrain [7a] in which each node in the quadtree contains an appropriately detailed portion of the heightmap, not just the leaf nodes. The root node contains a lower level sampling of the heightmap 2 ; its four children nodes each hold a quarter of the heightmap at a greater detail; their children hold a 16 th and so on, demonstrated in Figure 10. A disadvantage to this method is that popping occurs when the rendered quadtree level changes, due to the variation in sampling level between nodes. However, this is not an issue in the case of the project as unlit nodes only provide a silhouette for the system; lit nodes are rendered down to their leaf nodes, as the increase in granularity minimises the number of triangles rendered. During development of the terrain, a problem was exposed with culling objects too early during shadowing. In certain situations, an object can sit outside the camera frustum but within the light s frustum, casting a shadow though the object itself can t be seen. Due to culling happening on objects outside the camera s view, this has the effect of eliminating an object s shadow when the object itself is not at least partially visible. 2 The method of sampling the heightmap is a recursive subdivision method [7b] that concentrates the triangle distribution around high curvature areas, rather than a linear interval distribution. 13

17 Figure 11 Frustum culling during depth map generation: by camera frustum (left) and light frustum (right). A simple fix was made by recalculating the frustum used for visibility tests during the depth map generation phase, utilising the light view-projection as opposed to the camera. The relatively small volume of a light frustum in comparison to the camera s view also gives a higher probability of object culling, increasing performance. For large lights, only culling by the light frustum has the possibility of a high overhead of rendering objects that will not have any effect on the shadow map. A method of reducing this is discussed in section

18 4.2 Mesh Multi-Texturing Up to this point, the projected shadow map was the only texture used in the system; the modularity of the code, with separation into models and materials facilitated the addition of multi-texturing. User-specified textures are only used when a mesh is lit during the projection stage of shadow mapping. Since each light correlates to one render pass of the lit geometry, one texture unit will always be in use by the shadow map texture. Using this knowledge, the maximum number of user-specified texture was set as the hardware maximum less one. The light class was extended to allow an optionally specified overlay image, such as a smiley face in Figure 1 or a light mask as in Figure 12. Figure 12 Mesh multi-texturing. On the terrain 4 textures are in effect: the large-scale terrain texture (the green effect), a detail texture, the shadow map projective texture and a light mask to give soft edges to the scene. 4.3 Multiple Light Sources Scenes usually consist of multiple light sources, so the next step was to add the ability to render multiple shadow maps. OpenGL has a maximum of eight hardware lights, but by additively blending multiple passes, more can be displayed. Due to limitations on the number of texture units (usually four on modern hardware), only one light is used per pass, allowing three texture units for objects, or two with a projective texture overlay. 15

19 4.3.1 Visible Light Sets To reduce the amount of geometry drawn and increase performance, each entity keeps track of which lights are currently lighting it as a set of bits. During the shadow map generation stage, each light renders the scene in turn from their viewpoint. Before an entity renders its mesh, a bound box intersection test is made against the light s frustum, the result of which is saved in a bit-set 3, using the light id as the index. During the projection stage, each entity is asked to render its mesh. The entity uses the stored bit-set to determine if any lights are currently lighting it. A silhouette of the mesh is rendered without textures or shading, irrespective of the number of lights intersecting the mesh. For each intersecting light, the mesh is re-rendered with the shadow map. The extra passes are performed with additive blending to give a composite effect the same as if all the lights had been active at the same time. Little was changed in the sector render step; the depth map generation phase was simply repeated for each light visible in the camera frustum. void Sector::render() { foreach(light in alllights) if(camera.isvisible(light.frustum)) { setupviewprojection(light); drawentities(light); } light.depthmap = getdepthbuffer(); } setupviewprojection(camera); drawentities(null); void Sector::drawEntities(light) { foreach(entity in allentities) entity.draw(light); } 3 An array of Boolean values backed by a vector of integers, using bitwise operations to access individual bits. 16

20 void Entity::draw(light) { if(light) { if(!light.isvisible(this.boundingbox)) { bitset.set(light.id, false); return; } bitset.set(light.id, true); mesh.render(no_material); }else{ if(!camera.isvisible(this.boundingbox)) return; mesh.render(no_material); // render silhouette glenable(gl_blend); for(i = 0; i < bitset.size(); i++) if(bit-set[i]) { setuplight(alllights[i].gllightparams); setupprojectivetexture(alllights[i].depthmap); mesh.render(use_material); } glenable(gl_blend); } } Code 2 - The Entity render step collates a list of lights intersecting its bounding box; this is then used to determine what extra passes to render with texturing. Figure 13 Multiple light sources are brighter at their intersection (left); debugging lines used to show light volumes (right). 17

21 4.3.2 Clipping As mentioned in [2], a side-effect of the texture generation functions is a phantom negative projection. Due to OpenGL s texture clamping, side projections are also visible as shown in Figure 14. Instead of using an extra texture unit to eliminate the back and sides, further limiting the number of concurrent user textures for the object, extra clip planes can be specified to bound the light frustum. There is a slight drop in frame-rate due to the addition processing by OpenGL, so when an overlay texture is set for light, clip planes are not specified to increase performance. It is left to the user to set a texture with black borders of at least one pixel to eliminate the back and side projections. Figure 14 Shadow projection without clipping (left) and with extra clip planes (right) Figure 15 When using the projective overlay, the texture must have a 1px black border to eliminate back and side projections. 18

22 4.4 Omni-Directional Lights Omni-directional lights, were implemented as 6 axis-aligned frusta. The resulting frame-rate was found to be drastically reduced due to the five extra scene-renders needed in comparison with a standard spotlight. Many solutions have been found to overcome the need for multiple shadow maps when rendering lights with large fieldsof-view, but they are outside the scope of the project. Figure 16 An omni-directional light (left); the 6 frusta are displayed emanating from a lantern (right) 4.5 Texture-Masked Shadowing Textures add detail without extra geometry, complexity or computing time. As the colour channels modulate the surface colour during polygon rendering, so the texture s alpha channel can modulate the surface shape during shadowing, at a pertexel resolution a significant increase over geometric resolution. Alpha testing is a binary fragment test performed after depth and scissor testing, allowing the alpha channel to discard both colour and depth information of a fragment if the alpha value is past a user-specified threshold. This is different to blending, which modifies the framebuffer colour value based on the fragment colour and alpha channel, but still updates the depth buffer. Figure 17 Leaf and grass textures with alpha showing texture-masked shadows 19

23 The shadowing algorithm had to be modified to allow objects to request alpha-testing, as texturing must remain enabled to for the texture alpha channel to be tested during the depth map generation phase. Rendering the silhouette was disabled for transparent materials, as it caused borders around opaque texels due to variation between the alpha test threshold and interpolated alpha values on the polygon. Figure 18 Self-shadowing fur using alpha-transparent texture shadowing (left); rendering the mesh silhouette adds borders around opaque texels (right). 20

24 Chapter 5 Testing Testing was performed throughout the development of the project; an evolutionary lifecycle was followed to ensure requirements we completed on a stage-by-stage basis. Formal testing not an option due to time constraints and the inherently subjective nature of formalising tests for a graphical project. However, some issues arose that had subtle effects unnoticed until nearing project completion. The first was the assumption that the scale bias was acting on 2d texture coordinates and thus did not need to scale-bias the z-axis, resulting in a discrepancy between objects and their shadows; camera movement caused the shadows to swim as if the camera s movement also moved the objects. Using the scale-bias matrix presented in Equation 2 resolved the problem. The second issue was noticed after the implementation of clip planes to bound the light s frustum. The clip planes were being set after the objects transformation from world coordinates to local coordinates, causing clipping to occur as if the model were located at the world origin. This was rectified by setting the clip planes at the same time as the texgen planes. The last concern was with the rendering of omni-directional lights; overlap between the frusta due to precision errors caused a ring of brighter pixels at the intersections. As the 6 frusta are mutually exclusive volumes blending can be disabled and the light group rendered to a texture. The texture can then be additively blended on top of the framebuffer (disabling depth writes) to remove the artefacts. This was not implemented due to time constraints and effort required was not deemed appropriate to the visibility of the results. If omni-directional lights were implemented, an alternate method would be used such as dual-paraboloid mapping that requires only two depth maps to achieve a 360 o light view. 21

25 Chapter 6 Evaluation 6.1 Reflection The implementation of spotlight shadowing mapping in OpenGL fulfilled all the requirements specified during the analysis phase. A number of optimisations and extensions have been made to the technique explored in [2], including large mesh subdivision, multiple simultaneous light sources and texture-masked shadows. An additional non-functional requirement implicitly defined, that the application operates in real-time was also accomplished, running from fps. 6.2 Future Work Shadowing techniques have come on apace since the commencement of the project and as such shadow mapping is no longer at the forefront of shadowing technology. Current systems are now turning to hybrid method to retain the simplicity of shadow mapping with the precision of shadow volumes, utilising pen-umbra wedges to assist in the creation of true soft shadows. However, whilst maintaining a focus on spotlight shadow mapping, a number of extensions can be implemented to increase the performance and aesthetics More Effective Light-Camera Object Culling Further culling can be performed on objects in the light frustum, for the case of the object and its shadow both not being visible in the camera view. This can be implemented by performing shadow volume extrusion on the object s bounding box. Transform the bounding box into light space, and derive a light space-aligned bounding rectangle. Unproject the rectangle and extrude the quad away from the light source; the resulting geometry can then be tested against the camera frustum to determine whether any part of the object and its shadow intersects the camera view. The solution is essentially the method used for stencil shadowing [7c] Soft Edged Shadows Standard shadow mapping inherently has aliasing problem, demonstrated in the duelling frusta problem (Figure 6). Soft-edged shadowing [8] is an image-space solution, whereby the shadows are rendered to an off-screen buffer, Gaussian blurred and then modulated with the unshadowed screen. This filtering achieves higher quality, anti-aliasing shadows, however, it does not use penumbra/umbra wedges, thus only giving the illusion of soft shadows. 22

26 Bibliography [1] Lance Williams. Casting curved shadows on curved surfaces. In Proceedings of SIGGRAPH 78, pages , [2] C. Everitt, A. Rege, and C. Cebenoyan. Hardware Shadow Mapping. White paper, NVIDIA Corporation, [3] C. Everitt. Projective Texture Mapping White paper, NVIDIA Corporation, [4] C. Everitt. Shadow Mapping. White paper, NVIDIA Corporation, [5] William T. Reeves, David H. Salesin, and Robert L. Cook. Rendering antialiased shadows with depth maps. In Computer Graphics (SIGGRAPH 87 Proceedings), volume 21, pages , July [6] P. Baker. Shadow Mapping Tutorial, [7] D. Astle, et al. More OpenGL Game Programming, ISBN a: Chunked Level-of-Detail (LOD), pp b: View Independent Progressive Meshes, pp c: Projective Textures, pp [8] A. Shastry. Soft-Edged Shadows,

27 Appendix CD Contents Compile using Microsoft Visual Studio ShadowMapping.pdf bin/ data/ ShadowMapping.exe *.dll src/ include/ lib/ ShadowMapping.sln *.cpp *.h Controls mouse look around escape w,s,a,d c space r y quit forward, back, left, right down up toggle debug lines toggle wireframe mode 1-8 scenes showing aspects of shadow mapping 24