Shadows & Programmable Shaders. Dr. Alexander G. Gee. Texture Mapping. Buffers Texture Mapping OpenGL Texture Mapping Compositing and Blending

Transcription

1 Shadows & Programmable Shaders Dr. Alexander G. Gee Texture Mapping Buffers Texture Mapping OpenGL Texture Mapping Compositing and Blending page 2 1

2 Where does mapping take place? Mapping techniques are implemented at the end of the rendering pipeline Very efficient because few polygons make it past the clipper Vertex Geometric Processing Rasterization Display Pixels Pixel Processing page 3 The Pixel Pipeline OpenGL has a separate pipeline for pixels Writing pixels involves Moving pixels from processor memory to the frame buffer Format conversions Mapping, Lookups, Tests Reading pixels Format conversion Pack Processor Memory Unpack Pixel Map Lookup Table Pixel Test Frame Buffer page 4 2

3 Buffer Define a buffer by its spatial resolution (n x m) and its depth (or precision) k, the number of bits/pixel pixel m n k Stencil buffer Accumulation buffer Overlay buffer Auxiliary buffer Color buffer Depth buffer Back buffer Front buffer page 5 Texture Mapping Parametric coordinates Window coordinates Angel 2006 Interactive Computer Graphics 4 th Ed. World coordinates page 6 3

4 Two-part mapping One solution to the mapping problem is to first map the texture to a simple intermediate surface Example: map to cylinder Angel 2006 Interactive Computer Graphics 4 th Ed. page 7 Second Mapping Map from intermediate object to actual object Normals from intermediate to actual Normals from actual to intermediate Vectors from center of intermediate actual intermediate Angel 2006 Interactive Computer Graphics 4 th Ed. page 8 4

5 Basic Strategy Three steps to applying a texture 1. Specify the texture Read or generate image Assign to texture Enable texturing 2. Assign texture coordinates to vertices Proper mapping function is left to application 3. Specify texture parameters Wrapping, filtering page 9 Wrapping Mode Clamping: if s, t > 1 use 1, if s, t < 0 use 0 Wrapping: use s, t modulo 1 gltexparameteri(gl_texture_2d, GL_TEXTURE_WRAP_S, GL_CLAMP) gltexparameteri(gl_texture_2d, GL_TEXTURE_WRAP_T, GL_REPEAT) t texture s GL_REPEAT wrapping GL_CLAMP wrapping Angel 2006 Interactive Computer Graphics 4 th Ed. page 10 5

6 Magnification and Minification More than one texel can cover a pixel (minification) or more than one pixel can cover a texel (magnification) Can use point sampling (nearest texel) or linear filtering (2 x 2 filter) to obtain texture values Texture Polygon Magnification Texture Polygon Minification Angel 2006 Interactive Computer Graphics 4 th Ed. page 11 Mipmapped Textures Point Sampling Linear Filtering Mipmapped Point Sampling Mipmapped Linear Filtering Angel 2006 Interactive Computer Graphics 4 th Ed. page 12 6

7 Opacity and Transparency Opaque surfaces permit no light to pass through Transparent surfaces permit all light to pass Translucent surfaces pass some light translucency = 1 opacity (α) opaque surface α = 1 Angel 2006 Interactive Computer Graphics 4 th Ed. page 13 Shadows & Programmable Shaders Shadows Shadow Mapping Using OpenGL Hardware Soft Shadow Volumes Programmable Pipelines OpenGL Shading Language (GLSL) Coupling GLSL to Applications page 14 7

8 Shadows What have we displayed so far What are shadows Determining shadows Representing shadows page 15 Conceptual Model for Computer Graphics Real Light Real Object Synthetic Light Synthetic Camera Human Eye Synthetic Model Graphics System Display Device page 16 8

9 Defining a Scene of Objects page 17 Adding Lighting to the Scene page 18 9

10 Defining a Camera to Capture the Scene page 19 Projecting the Scene page 20 10

11 Rendering the Scene to the Display page 21 Standard OpenGL Projected Scene page 22 11

12 OpenGL Scene Lacking Shadows page 23 The Scene of Objects with Shadows page 24 12

13 Global Effects multiple reflection translucent surface shadow Angel 2006 Interactive Computer Graphics 4 th Ed. page 25 Why Shadows? Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page 26 13

14 Why Shadows? Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page 27 Phong Model Light Equation For each light source and each color component, the Phong model can be written as I = ( 1 / (a + bd + cd 2 ) )( k d I d l n + k s I s (v r) α ) + k a I a distance diffuse specular ambient For each color component we add contributions from all sources Surface is solely determined as a combination of the light sources and the object s material l n r θ i θ r φ v page 28 14

15 Shadows Shadows are the result of one or more surfaces blocking one or more light sources from the surfaces being displayed Shadows are independent of the camera s point of projection and solely determined by the position of objects and light sources within a scene Shadows from individual point light sources is equivalent to the inverse problem of rasterization If the view of objects A, B, and C is from the position and direction of the light source, then objects B and C produce shadows on A C A B page 29 Simple Shadows Consider a light source at infinity, then all the light rays within a scene are define to be parallel across the entire scene page 30 15

16 Simple Shadows The shape of the shadow is dependent on The position of the light source The shape of the object The form of the surface on which the plane of the shadow falls shade lines parallel light rays shading based on surface normal shadow of shade lines surface in shade cast shadows page 31 Different Views of the Same Scene View from light source Top down Camera view page 32 16

17 Shadows in Perspective Frank Ching 1985 Architectural Graphics 2 nd Ed. page 33 Interior Shadows in Perspective Frank Ching 1985 Architectural Graphics 2 nd Ed. page 34 17

18 Non-Parallel Point Light Source Shadows Standard light sources are generally those defined at an infinite distance from the objects, as depicted previously The other standard light source is one that represents a light bulb or candle that occupies no space These light sources project rays of light in all directions from the lights spatial position page 35 Geometry Pipeline Model Coordinates Modeling Transformation World Coordinates Viewing Transformation View Coordinates Shadows 4D Projection Transformation Projection Coordinates 3D Normalization Transformation and Clipping against default cube Normalized Coordinates 3D Viewport Transformation Display Coordinates 2D page 36 18

19 Light Sources with Significant Area Solar eclipse on the surface of the earth Umbra total shadow Penumbra partial shadow Hearn & Baker 2004 Computer Graphics with OpenGL 3 rd Ed. page 37 Two Options for Defining Shadows 1. Define additional objects to represent shadows Define additional object patches for shadows Define new objects for each shadow segment Only works with point light sources 2. Define or modify object textures to include shadows Able to handle varying light source shapes page 38 19

20 Shadows from a Single Polygon Consider the shadow generated by the point source at (x i, y i, z i ) of object A on the x-z plane, y = 0 Not only is the shadow a flat polygon S, called a shadow polygon, but it is also the projection of the original polygon A onto the plane, with the center of projection at the light source (x i,y i,z i ) We can use a projection matrix y and OpenGL to compute the vertices of the shadow polygon M 1 0 = y l z A S x page 39 Shadow Polygon Calculation GLfloat m[16] // shadow projection matrix for(i=0; i<15; i++) m[i]=0.0 m[0] = m[5] = m[10] = 1.0 m[7] = -1.0/y glcolor3fv(polygon_color) /* draw the polygon normally */ glmatrixmode(gl_modelview) glpushmatrix() // save state gltraslatef(x,y,z) // translate back glmultmatrix(m) // polygon projection gltraslatef(-x,-y,-z) // move light to origin glcolor3fv(shadow_color) /* draw the polygon again as shadow */ glpopmatrix() // restore state page 40 20

21 Shadow Mapping Using OpenGL Hardware Shadow Mappings Visualizing the Shadow Mapping Technique Construct Light View Depth Map Depth Map Bias Projective Texturing Shadow Mapping Hardware Issues with Shadow Mapping page 41 Common Shadow Technique Problems Various limitations Projected planar shadows Well works only on flat surfaces Stenciled shadow volumes Determining the shadow volume is hard work Light maps Totally unsuited for dynamic shadows In general, hard to get everything shadowing everything page 42 21

22 Shadow Mapping: Another Technique Image-space shadow determination Lance Williams published the basic idea in 1978 By coincidence, same year Jim Blinn invented bump mapping (a great vintage year for graphics) Completely image-space algorithm Means no knowledge of scene s geometry is required Must deal with aliasing artifacts Well known software rendering technique Pixar s RenderMan uses the algorithm Basic shadowing technique for toy story, etc. page 43 Shadow Mapping References Important SIGGRAPH papers Lance Williams, Casting Curved Shadows on Curved Surfaces, SIGGRAPH 78 William Reeves, David Salesin, and Robert Cook (Pixar), Rendering antialiased shadows with depth maps, SIGGRAPH 87 Mark Segal, et. al. (SGI), Fast Shadows and Lighting Effects Using Texture Mapping, SIGGRAPH 92 page 44 22

23 The Shadow Mapping Concept Depth testing from the light s point-of-view Two pass algorithm First, render depth buffer from the light s point-of-view The result is a depth map or shadow map Essentially a 2D function indicating the depth of the closest pixels to the light This depth map is used in the second pass page 45 The Shadow Mapping Concept Shadow determination with the depth map Second, render scene from the eye s point-of-view For each rasterized fragment Determine fragment s XYZ position relative to the light This light position should be setup to match the frustum used to create the depth map Compare the depth value at light position XY in the depth map to fragment s light position Z page 46 23

24 The Shadow Mapping Concept The Shadow Map Comparison Two values A = Z value from depth map at fragment s light XY position B = Z value of fragment s XYZ light position If B is greater than A, then there must be something closer to the light than the fragment Then the fragment is shadowed If A and B are approximately equal, the fragment is lit page 47 Shadow Mapping with a Picture in 2D The A < B shadowed fragment case depth map image plane light source depth map Z = A eye position eye view image plane, a.k.a. the frame buffer fragment s light Z = B page 48 24

25 Shadow Mapping with a Picture in 2D The A B unshadowed fragment case depth map image plane light source depth map Z = A eye position eye view image plane, a.k.a. the frame buffer fragment s light Z = B page 49 Shadow Mapping with a Picture in 2D The depth map could be at a different resolution from the frame buffer, this mismatch can lead to artifacts page 50 25

26 Visualizing the Shadow Mapping Technique A fairly complex scene with shadows Point light source page 51 Visualizing the Shadow Mapping Technique Compare with and without shadows with shadows without shadows page 52 26

27 Visualizing the Shadow Mapping Technique The scene from the light s point-of-view Camera s point-of-view page 53 Visualizing the Shadow Mapping Technique The depth buffer from the light s point-of-view page 54 27

28 Visualizing the Shadow Mapping Technique Projecting the depth map onto the camera s view Light s point-of-view page 55 Visualizing the Shadow Mapping Technique Projecting light s planar distance onto eye s view page 56 28

29 Visualizing the Shadow Mapping Technique Comparing light distance to light depth map Green is where The light planar distance and the light depth map are approximately equal Non-green is where shadows should be page 57 Visualizing the Shadow Mapping Technique Complete scene with shadows Notice how the specular highlights never appear in shadows Notice how the curved surfaces cast shadows on each other page 58 29

30 Construct Light View Depth Map Realizing the theory in practice Constructing the depth map Use existing hardware depth buffer Use glpolygonoffset to offset depth value back Read back the depth buffer contents Depth map can be copied to a 2D texture Unfortunately, depth values tend to require more precision than 8- bit typical for textures Depth precision typically 16-bit or 24-bit page 59 Why glpolygonoffset Depth buffer contains window space depth values Post-perspective divide means non-linear distribution glpolygonoffset is guaranteed to be a window space offset Doing a clip space gltranslatef is not sufficient Common shadow mapping implementation mistake Actual bias in depth buffer units will vary over the frustum No way to account for slope of polygon page 60 30

31 Sampling Polygon s Depth at Pixel Centers Consider a polygon covering pixels in 2D Polygon X Z Pixel centers page 61 Sampling Polygon s Depth at Pixel Centers Consider a 2 nd grid for the polygon covering pixels in 2D X Z X Z page 62 31

32 Sampling Polygon s Depth at Pixel Centers How Z changes with respace to X slope X Z z/ x page 63 Why You Need glpolygonoffset Slope Consider pixel center on another grid For example, the shadow map texture s grid! The re-sampled depth could be off by +/-0.5 z/ x and +/-0.5 z/ y The maximum absolute error would be 0.5 z/ x z/ y max( z/ x, z/ y ) This assumes the two grids have pixel footprint area ratios of 1.0 Otherwise, we might need to scale by the ratio Same as polygon offset s slope bias page 64 32

33 Depth Map Bias Issues How much polygon offset bias depends Too little bias, everything begins to shadow Too much bias, shadow starts too far back Just right page 65 Selecting the Depth Map Bias Not that hard Usually the following works well glpolygonoffset(scale = 1.1, bias = 4.0) Usually better to error on the side of too much bias adjust to suit the shadow issues in your scene Depends somewhat on shadow map precision more precision requires less of a bias When the shadow map is being magnified, a larger scale is often required page 66 33

34 Render Scene & Access the Depth Texture Realizing the theory in practice Fragment s light position can be generated using eye-linear texture coordinate generation specifically OpenGL s GL_EYE_LINEAR texgen generate homogenous (s, t, r, q) texture coordinates as lightspace (x, y, z, w) T&L engines such as GeForce accelerate texgen! relies on projective texturing page 67 What is Projective Texturing? An intuition for projective texturing The slide projector analogy Source: Wolfgang Heidrich [99] page 68 34

35 About Projective Texturing First, what is perspective-correct texturing? Normal 2D texture mapping uses (s, t) coordinates 2D perspective-correct texture mapping Means (s, t) should be interpolated linearly in eye-space So compute per-vertex s/w, t/w, and 1/w Linearly interpolated these three parameters over polygon Per-fragment compute s = (s/w) / (1/w) and t = (t/w) / (1/w) Results in per-fragment perspective correct (s, t ) page 69 About Projective Texturing So what is projective texturing? Now consider homogeneous texture coordinates (s, t, r, q) (s/q, t/q, r/q) Similar to homogeneous clip coordinates where (x, y, z, w) = (x/w, y/w, z/w) Idea is to have (s/q, t/q, r/q) be projected per-fragment This requires a per-fragment divider Yikes, dividers in hardware are fairly expensive page 70 35

36 About Projective Texturing Hardware designer s view of texturing Perspective-correct texturing is a practical requirement Otherwise, textures swim Perspective-correct texturing already requires the hardware expense of a per-fragment divider Clever idea [Segal, et.al. 92] Interpolate q/w instead of simply 1/w So projective texturing is practically free if you already do perspective-correct texturing! page 71 About Projective Texturing Tricking hardware into doing projective textures By interpolating q/w, hardware computes per-fragment (s/w) / (q/w) = s/q (t/w) / (q/w) = t/q Net result: projective texturing OpenGL specifies projective texturing Only overhead is multiplying 1/w by q But this is per-vertex page 72 36

37 Back to the Shadow Mapping Discussion Assign light-space texture coordinates via texgen Transform eye-space (x, y, z, w) coordinates to the light s view frustum (match how the light s depth map is generated) Further transform these coordinates to map directly into the light view s depth map Expressible as a projective transform Load this transform into the 4 eye linear plane equations for S, T, and Q coordinates (s/q, t/q) will map to light s depth map texture page 73 OpenGL s Standard Geometry Pipeline Model Coordinates Modeling Transformation World Coordinates Viewing Transformation View Coordinates 4D Projection Transformation Projection Coordinates 3D Normalization Transformation and Clipping against default cube Normalized Coordinates 3D Viewport Transformation + Depth Range Display Coordinates 2D page 74 37

38 Eye Linear Texture Coordinate Generation Model Coordinates Modeling Transformation World Coordinates Viewing Transformation View Coordinates Eye-Linear (s, t, r, q) 4D Projection Transformation Projection Coordinates 3D Normalization Transformation and Clipping against default cube Normalized Coordinates 3D Viewport Transformation + Depth Range Display Coordinates 2D page 75 Setting Up Eye Linear Texgen With OpenGL GLfloat Splane[4],Tplane[4],Rplane[4],Qplane[4] gltexgenfv(gl_s, GL_EYE_PLANE, Splane) gltexgenfv(gl_t, GL_EYE_PLANE, Tplane) gltexgenfv(gl_r, GL_EYE_PLANE, Rplane) gltexgenfv(gl_q, GL_EYE_PLANE, Qplane) glenable(gl_texture_gen_s) glenable(gl_texture_gen_t) glenable(gl_texture_gen_r) glenable(gl_texture_gen_q) Each eye plane equation is transformed by current inverse modelview matrix Very handy thing for us; otherwise, a pitfall Note: texgen object planes are not transformed by the inverse modelview page 76 38

39 Shadow Map Operation Automatic depth map lookups After the eye linear texgen with the proper transform loaded (s/q, t/q) is the fragment s corresponding location within the light s depth texture r/q is the Z planar distance of the fragment relative to the light s frustum, scaled and biased to [0,1] range Next compare texture value at (s/q, t/q) to value r/q If texture[s/q, t/q] r/q then not shadowed If texture[s/q, t/q] < r/q then shadowed page 77 Shadow Mapping Hardware Support OpenGL now has official ARB-standard shadow mapping extensions Approved February 2002! ARB_depth_texture adds depth texture formats ARB_shadow adds percentage closer filtering for depth textures The two extensions are used together Based on prior proven SGI proprietary extensions SGIX_depth_texture SGIX_shadow page 78 39

40 Shadow Mapping Hardware Support SGIX_depth_texture & SGIX_shadow support SGI s RealityEngine & InfiniteReality Brian Paul s Mesa3D OpenGL work-alike NVIDIA s GeForce3, GeForce4 Ti, and Quadro 4 XGL Software emulation for GeForce1 & 2 ARB extensions now implemented Latest NVIDIA drivers and Mesa 4.0 page 79 ARB_shadow Filtering Mode Performs the shadow test as a texture filtering operation Looks up texel at (s/q, t/q) in a 2D texture Compares lookup value to r/q If texel is greater than or equal to r/q, then generate 1.0 If texel is less than r/q, then generate 0.0 Modulate color with result Zero if fragment is shadowed or unchanged color if not page 80 40

41 ARB_shadow API Usage Request shadow map filtering with gltexparameter calls gltexparameteri(gl_texture_2d, GL_TEXTURE_COMPARE_MODE_ARB, GL_COMPARE_R_TO_TEXTURE) Default is GL_NONE for normal filtering Only applies to depth textures Also select the comparison function Either GL_LEQUAL (default) or GL_GEQUAL gltexparameteri(gl_texture_2d, GL_TEXTURE_COMPARE_FUNC_ARB, GL_LEQUAL) page 81 ARB_shadow API Usage Request shadow map filtering with gltexparameter calls gltexparameteri(gl_texture_2d, GL_TEXTURE_COMPARE_MODE_ARB, GL_COMPARE_R_TO_TEXTURE); Default is GL_NONE for normal filtering Only applies to depth textures Also select the comparison function Either GL_LEQUAL (default) or GL_GEQUAL gltexparameteri(gl_texture_2d, GL_TEXTURE_COMPARE_FUNC_ARB, GL_LEQUAL); page 82 41

42 New Depth Texture ARB_depth_texture supports textures containing depth values for shadow mapping Three new internal formats GL_DEPTH_COMPONENT16_ARB GL_DEPTH_COMPONENT24_ARB GL_DEPTH_COMPONENT32_ARB (same as 24-bit on GeForce3/4/Xbox) Use GL_DEPTH_COMPONENT for your external format Work with glcopysubteximage2d for fast copies from depth buffer to texture NVIDIA optimizes these copy texture paths page 83 Depth Texture Details Usage example: glcopyteximage2d(gl_texture_2d, level=0, internalfmt=gl_depth_component24_arb, x=0, y=0, w=256, h=256, border=0); Then use glcopysubteximage2d for faster updates once texture internal format initially defined Hint: use GL_DEPTH_COMPONENT for your texture internal format Leaving off the n_arb precision specifier tells the driver to match your depth buffer s precision Copy texture performance is optimum when depth buffer precision matches the depth texture precision page 84 42

43 Depth Texture Copy Performance The more depth values you copy, the slower the performance 512x512 takes 4 times longer to copy than 256x256 Tradeoff: better defined shadows require higher resolution shadow maps, but slows copying 16-bit depth values copy twice as fast as 24-bit depth values (which are contained in 32-bit words) Requesting a 16-bit depth buffer (even with 32-bit color buffer) and copying to a 16-bit depth texture is faster than using a 24- bit depth buffer Note that using 16-bit depth buffer usually requires giving up stencil page 85 Hardware Shadow Map Filtering Percentage Closer filtering Normal texture filtering just averages color components Averaging depth values does NOT work Solution [Reeves, SIGGARPH 87] Hardware performs comparison for each sample Then, averages results of comparisons Provides anti-aliasing at shadow map edges Not soft shadows in the umbra/penumbra sense page 86 43

44 Hardware Shadow Map Filtering Example GL_NEAREST: blocky GL_LINEAR: antialiased edges Low shadow map resolution used to heightens filtering artifacts page 87 Advice for Shadowed Illumination Model Typical illumination model with decal texture: ( ambient + diffuse ) * decal + specular The shadow map supplies a shadowing term Assume shadow map supplies a shadowing term, shade Percentage shadowed 100% = fully visible, 0% = fully shadowed Obvious updated illumination model for shadowing: ( ambient + shade * diffuse ) * decal + shade * specular Problem is real-world lights don t 100% block diffuse shading on shadowed surfaces Light scatters; real-world lights are not ideal points page 88 44

45 Advice for Shadowed Illumination Model Illumination model with dimming: ( ambient + diffuseshade * diffuse ) * decal + specular * shade where diffuseshade is diffuseshade = dimming + ( 1.0 dimming ) * shade Easy to implement with NV_register_combiners & OpenGL 1.2 separate specular color support Separate specular keeps the diffuse & specular pervertex lighting results distinct NV_register_combiners can combine the primary (diffuse) and secondary (specular) colors per-pixel with the above math page 89 Other OpenGL Extensions ARB_pbuffer create off-screen rendering surfaces for rendering shadow map depth buffers Normally, you can construct shadow maps in your back buffer and copy them to texture But if the shadow map resolution is larger than your window resolution, use pbuffers NV_texture_rectangle new 2D texture target that does not require texture width and height to be powers of two Limitations No mipmaps or mipmap filtering supported No wrap clamp mode Texture coords in [0..w]x[0..h] rather than [0..1]x[0..1] range. Quite acceptable for for shadow mapping page 90 45

46 Combining Shadow Mapping & Techniques Good in combination with techniques Use stencil to tag pixels as inside or outside of shadow Use other rendering techniques in extra passes bump mapping texture decals, etc. Shadow mapping can be integrated into more complex multipass rendering algorithms Shadow mapping algorithm does not require access to vertex-level data Easy to mix with vertex programs and such page 91 Alternative to Shadow Mapping Hardware Consumer 3D hardware solution Proposed by Wolfgang Heidrich in his 1999 Ph.D. thesis Leverages today s consumer multi-texture hardware 1 st texture unit accesses 2D depth map texture 2 nd texture unit accesses 1D Z range texture Extended texture environment subtracts 2nd texture from 1st shadowed if greater than zero, unshadowed otherwise use alpha test to discard shadowed fragments page 92 46

47 Issues with Shadow Mapping Not without its problems Prone to aliasing artifacts percentage closer filtering helps this normal color filtering does not work well Depth bias is not completely foolproof Requires extra shadow map rendering pass and texture loading Higher resolution shadow map reduces blockiness but also increase texture copying expense page 93 Issues with Shadow Mapping Not without its problems Shadows are limited to view frustums could use six view frustums for omni-directional light Objects outside or crossing the near and far clip planes are not properly accounted for by shadowing move near plane in as close as possible but too close throws away valuable depth map precision when using a projective frustum page 94 47

48 Determining Shadow Map Resolution Requires knowing how pixels (samples) in the light s view compare to the size of pixels (samples) in the eye s view A re-sampling problem When light source frustum is reasonably well aligned with the eye s view frustum, the ratio of sample sizes is close to 1.0 Great match if eye and light frustum s are nearly identical But that implies very few viewable shadows Consider a miner s lamp (i.e., a light attached to your helmet) The chief reason for such a lamp is you don t see shadows from the lamp while wearing it page 95 Determining Shadow Map Resolution So best case is miner s lamp Worst case is shadows from light shining at the viewer that deer in the headlights problem definitely worst case for the deer Also known as the dueling frusta problem (frusta, plural of frustum) Let s attempt to visualize what s happens page 96 48

49 Combining Projective Texturing & Spotlights Use a spotlight-style projected texture to give shadow maps a spotlight falloff page 97 Shadow Mapping Source Code Find it on the NVIDIA web site The source code shadowcast in OpenGL example code Works on TNT, GeForce, Quadro, & GeForce3 using best available shadow mapping support And vendors that support EXT_texture_env_combine NVIDIA OpenGL Extension Specifications documents EXT_texture_env_combine, NV_register_combiners, SGIX_depth_texture, & SGIX_shadow page 98 49

50 Conclusions Shadow mapping offers real-time shadowing effects Independent of scene complexity Very compatible with multi-texturing Does not mandate multi-pass as stenciled shadow volumes do Ideal for shadows from spotlights Consumer hardware shadow map support here today GeForce3, GeForce4 Ti, Xbox Dual-texturing technique supports legacy hardware Same basic technique used by Pixar to generate shadows in their computer-generated movies page 99 Soft Shadow Volumes Area Light Sources Simple Volumetric Light Sources Textures and Short Video Textures as Lights Real-Time Performance Approximate Soft Shadows Speed versus Accuracy Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

51 Real Soft Shadows Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 101 Our Goal is Soft Shadows in Real-Time Soft shadow generation is a fundamental CG problem Many algorithms exist Not a single one that can be used in, e.g., a game We (all) want Good performance (use hardware) Arbitrary receivers and casters Plausible, or even better, physically-correct shadows Our paper is one small step in the right direction Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page

52 Previous Work on Real-Time Shadows Lots Most important algorithms for shadows on arbitrary surfaces: Shadow Mapping [Williams] See Perspective Shadow Maps, SIGGRAPH 2002 Heidrich et al, EGRW2000, soft version Shadow Volumes [Crow,Heidmann] See Everitt and Kilgard s report on how to robustly render shadow volumes Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page 103 Previous Work on Real-Time Soft Shadows Heckbert/Herf [HH]: Average a high number of images from point samples on light source, use as a texture on a plane Uses many passes: (or fewer, and banding artifacts) Not practical for dynamic scenes Soler/Sillion [SS]: For planar configurations, convolve hard shadow with shape of light source Fast Hierarchical error-driven algorithm Image-based can use any shadow casting geometry and light source [same for HH] Haines plateaus: Planar as well Overemphasize the umbra Few passes Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page

53 Area / Volumetric Lights Give Soft Shadows Real lights have area or volume Thus, soft shadows are more realistic HARD SOFT Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 105 Scene Rendered Using Hard Shadows Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

54 Scene Rendered Using Soft Shadows Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 107 Hard vs. Soft Shadows Two different light source types point source area source umbra penumbra umbra Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

55 Soft Shadows Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page 109 General Idea Extend the shadow volume algorithm In the shadow volume algorithm Each silhouette edge shadow quad (polygon) For soft shadows Each silhouette edge wedge-like shape, penumbra wedge Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page

56 Real-Time Soft Shadow Volume Algorithm Same Silhouette Edge Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 111 Wedges with Simplified Assumptions Each silhouette edge has a corresponding wedge Provides a piece of penumbra contribution Rasterized by pixel shader Simplifications Spherical light sources Silhouettes as seen from center of light source Bound penumbra volume with 4 planes Using a stencil Buffer Light Intensity Buffer Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

57 A Wedge for Each Silhouette Edge Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 113 A Wedge for Each Silhouette Edge Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

58 A Wedge for Each Silhouette Edge Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 115 A Wedge for Each Silhouette Edge Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

59 A Wedge for Each Silhouette Edge Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 117 How to Rasterize a Wedge /* initialize light intensity (LI) buffer to 255 */ /* 255=full light, 0=no light, 0<x<155 penumbra */ rasterizewedge() foreach visible fragement(x,y) pf=computeentrypointonwedge(x,y) pb=computeexitpointonwedge(x,y) p=point(x,y,z) // -z Z-buffer value at (x,y) pi=choosepointclosesttoeye(p,pb) sf=computelightintensity(pf) si=computelightintensity(pi) addtolibuffer(round(255*(si-sf))) end Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page

60 Rasterizing the Wedges Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 119 Rasterizing the Wedges Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

61 Rasterizing the Wedges Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 121 Rasterizing the Wedges Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

62 Rasterizing the Wedges Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 123 Overview Computation of visibility mask 1 st pass: render hard shadow 2 nd pass: compensate for overstated umbra Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

63 Implementation Need pretty large vertex & pixel shaders in order to implement using graphics hw So currently: Render scene using hw-accelerated OpenGL Read back Z-buffer Software wedge rasterization into LI-buffer (some optimizations not covered here see paper) For each pixel (x, y) get z and form the world-space coordinates Point inside wedge compute light intensity When all wedges rasterized Clamp to [0,255] and modulate specular + diffuse image, and then add ambient image Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page 125 Example Soft Shadows Varying the size of the light source Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page

64 Example Soft Shadows Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page 127 Soft Shadows for Two Light Sources Assarsson & Akenine-Möller th Eurographics Workshop, Chalmers University of Technology page

65 Single silhouette artifact 1 st approximation of shadows Assumes constant silhouette Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 129 Better Approximation Using Samples Single Area Light Source 2 x 2 Area Light Source 1024 Point Light Sources Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page

66 Better Approximation Using Samples Single Area Light Source 2 x 2 Area Light Sourcs 1024 Point Light Sources Single Area 2 x 2 Area 3 x 3 Area 1024 Points Assarsson & Akenine-Möller 2003 Graphics Hardware, Chalmers University of Technology page 131 Programmable Pipelines Introduce Programmable Pipelines Vertex shaders Fragment shaders Introduce Shading Languages Needed to describe shaders RenderMan page

67 Introduction Recent major advance in real time graphics is programmable pipeline First introduced by NVIDIA GForce 3 Supported by high-end commodity cards NVIDIA, ATI, 3D Labs Software Support DirectX 8, 9, 10 OpenGL Extensions OpenGL Shading Language (GLSL) Cg page 133 Background Two components Vertex programs (shaders) Fragment programs (shaders) Requires detailed understanding of two seemingly contradictory approaches OpenGL pipeline Real time RenderMan ideas offline page

68 Black Box View Vertices Vertices Fragments Fragments CPU Geometry Processor Rasterizer Fragment Processor Frame Buffer page 135 Geometric Calculations Geometric data: set of vertices + type Can come from program, evaluator, display list Type: point, line, polygon Vertex data can be (x, y, z, w) coordinates of a vertex (glvertex) Normal vector Texture Coordinates RGBA color Other data: color indices, edge flags Additional user-defined data in GLSL page

69 Per-Vertex Operations Vertex locations are transformed by the model-view matrix into eye coordinates Normals must be transformed with the inverse transpose of the model-view matrix so that v n = v n in both spaces Assumes there is no scaling May have to use autonormalization Textures coordinates are generated if autotexture enabled and the texture matrix is applied page 137 Lighting Calculations Done on a per-vertex basis Phong model I = k d I d l n +k s I s (v r) α + k a I a Phong model requires computation of r and v at every vertex page

70 Calculating the Reflection Term Angle of incidence = angle of reflection cos θ i = cos θ r or r n = l n r, n, and l are coplanar r = α l + β n Normalize 1 = r r = n n = l l solving: r = 2(l n) n - l l n θ i θ r r page 139 OpenGL Lighting Modified Phong model Halfway vector Global ambient term Specified in standard Supported by hardware page

71 Halfway Vector Blinn proposed replacing v r by n h where h = (l + v) / l + v (l + v)/2 is halfway between l and v If n, l, and v are coplanar: ψ = φ/2 Must then adjust exponent so that (n h) e (r v) e l θ n ψ h φ r v page 141 Primitive Assembly Vertices are next assembled into objects Polygons Line Segements Points Transformation by projection matrix Clipping Against user defined planes View volume: x = ±w, y = ±w, z = ±w Polygon clipping can create new vertices Perspective Division Viewport mapping page

72 Rasterization Geometric entities are rasterized into fragments Each fragment corresponds to a point on an integer grid: a displayed pixel Hence each fragment is a potential pixel Each fragment has A color Possibly a depth value Texture coordinates page 143 Fragment Operations Texture generation Fog Antialiasing Scissoring Alpha test Blending Dithering Logical Operation Masking page

73 Vertex Processor Takes in vertices Position attribute Possibly color OpenGL state Produces Position in clip coordinates Vertex color page 145 Fragment Processor Takes in output of rasterizer (fragments) Vertex values have been interpolated over primitive by rasterizer Outputs a fragment Color Texture Fragments still go through fragment tests Hidden-surface removal alpha page

74 Programmable Shaders Replace fixed function vertex and fragment processing by programmable processors called shaders Can replace either or both If we use a programmable shader we must do all the required functions of the fixed function processor page 147 Development RenderMan Shading Language Offline rendering Hardware Shading Languages UNC, Stanford NVIDIA OpenGL Vertex Program Extension OpenGL Shading Language (GLSL) Cg OpenGL Microsoft HLSL page

75 RenderMan Developed by Pixar S. Upstill, The RenderMan Companion, Addison-Wesley, Model interface file (RIB) Modeler Renderer page 149 Modeling vs Rendering Modeler outputs geometric model plus information for the renderer Specifications of camera Materials Lights May have different kinds of renderers Ray tracer Radiosity How do we specify a shader? page

76 Shading Trees Shaders such as the Phong model can be written as algebraic expressions I = k d I d l n +k s I s (v r) α + k a I a But expressions can be described by trees Need new operators such as dot and cross products and new data types such as matrices and vectors Environmental variables are part of state page 151 Phong Model Phong model can be written as algebraic expressions I = k d I d l n +k s I s (v r) α + k a I a + + k a I a * * () k d I d n l k s I s α r v page

77 GLSL I Shader Applications Vertex Shaders Fragment Shaders Programming Shaders Cg GLSL page 153 Vertex Shader Applications Moving vertices Morphing Wave motion Fractals Lighting More realistic models Cartoon shaders page

78 Fragment Shader Applications Per fragment lighting calculations per vertex lighting per fragment lighting page 155 Fragment Shader Applications Texture mapping smooth shading environment mapping bump mapping page

79 Writing Shaders First programmable shaders were programmed in an assembly-like manner OpenGL extensions added for vertex and fragment shaders Cg (C for graphics) C-like language for programming shaders Works with both OpenGL and DirectX Interface to OpenGL complex OpenGL Shading Language (GLSL) page 157 OpenGL Shading Language (GLSL) OpenGL Shading Language Part of OpenGL 2.0 specification High level C-like language New data types Matrices Vectors Samplers OpenGL state available through built-in variables page

80 Simple Vertex Shader const vec4 red = vec4(1.0, 0.0, 0.0, 1.0) void main(void) { gl_position = gl_projectionmatrix * gl_modelviewmatrix * gl_vertex gl_frontcolor = red } page 159 Execution Model glcolor glortho glloadmatrix Application Program OpenGL State Vertex Shader Primitive Assembly gl_vertex gl_modelviewmatrix gl_projectionmatrix gl_frontcolor gl_position page

81 Simple Fragment Program Void main(void) { gl_fragcolor = gl_frontcolor } page 161 Execution Model OpenGL State Application Program vertices Rasterizer fragments Fragment Shader pixels Frame Buffer gl_position gl_frontcolor (per vertex) gl_frontcolor (interpolated) gl_fragmentcolor page

82 Data Types C types: int, float, bool Vectors: float vec2, vec 3, vec4 Also int (ivec) and boolean (bvec) Matrices: mat2, mat3, mat4 Stored by columns Standard referencing m[row][column] C++ style constructors vec3 a = vec3(1.0, 2.0, 3.0) vec2 b = vec2(a) page 163 Pointers There are no pointers in GLSL We can use C structs which can be copied back from functions Because matrices and vectors are basic types they can be passed into and output from GLSL functions, e.g. matrix3 func(matrix3 a) page

83 Qualifiers GLSL has many of the same qualifiers such as const as C/C++ Need others due to the nature of the execution model Variables can change Once per primitive Once per vertex Once per fragment At any time in the application Vertex attributes are interpolated by the rasterizer into fragment attributes page 165 Attribute Qualifier Attribute-qualified variables can change at most once per vertex Cannot be used in fragment shaders Built in (OpenGL state variables) gl_color gl_modelviewmatrix User defined (in application program) attribute float temperature attribute vec3 velocity page

84 Uniform Qualified Variables that are constant for an entire primitive Can be changed in application outside scope of glbegin and glend Cannot be changed in shader Used to pass information to shader such as the bounding box of a primitive page 167 Varying Qualified Variables that are passed from vertex shader to fragment shader Automatically interpolated by the rasterizer Built in Vertex colors Texture coordinates User defined Requires a user defined fragment shader page

85 Example: Vertex Shader const vec4 red = vec4(1.0, 0.0, 0.0, 1.0) varying vec3 color_out void main(void) { gl_position = gl_modelviewprojectionmatrix * gl_vertex color_out = red } page 169 Required Fragment Shader varying vec3 color_out void main(void) { gl_fragcolor = color_out } page

86 Passing Values Call by value-return Variables are copied in Returned values are copied back Three possibilities in out inout page 171 Operators and Functions Standard C functions Trigonometric Arithmetic Normalize, reflect, length Overloading of vector and matrix types mat4 a vec4 b, c, d c = b*a // column vector d = a*b // row vector page

87 Swizzling and Selection Can refer to array elements by element using [] or selection (.) operator with x, y, z, w r, g, b, a s, t, p, q a[2], a.b, a.z, a.p are the same Swizzling operator lets us manipulate components vec4 a a.yz = vec2(1.0, 2.0) page 173 GLSL II Coupling GLSL to Applications Example applications page

88 Linking Shaders to OpenGL OpenGL Extensions ARB_shader_objects ARB_vertex_shader ARB_fragment_shader OpenGL 2.0 Almost identical to using extensions Avoids extension suffixes on function names page 175 Program Object Container for shaders Can contain multiple shaders Other GLSL functions GLuint myprogobj myprogobj = glcreateprogram() /* define shader objects here */ gluseprogram(myprogobj) gllinkprogram(myprogobj) page

89 Reading a Shader Shader are added to the program object and compiled Usual method of passing a shader is as a nullterminated string using the function glshadersource If the shader is in a file, we can write a reader to convert the file to a string page 177 Shader Reader Char* readshadersource(const char* shaderfile) { struct stat statbuf FILE* fp = fopen(shaderfile, r ) char* buf } stat(shaderfile, &statbuf) buf = (char*)malloc(statbuf.st_size + 1 * sizeof(char)) fread(buf, 1, statbuf.st_size, fp) buf[statbuf.st_size] = \0 fclose(fp) return buf page

90 Adding a Vertex Shader GLint vshader GLunit myvertexobj GLchar vshaderfile[] = my_vertex_shader GLchar* vsource = readshadersource(vshaderfile) glshadersource(myvertexobj,1,&vertexshaderfile,null) myvertexobj = glcreateshader(gl_vertex_shader) glcompileshader(myvertexobj) glattachobject(myprogobj, myvertexobj) page 179 Vertex Attributes Vertex attributes are named in the shaders Linker forms a table Application can get index from table and tie it to an application variable Similar process for uniform variables page

91 Vertex Attribute Example GLint colorattr colorattr = glgetattriblocation(myprogobj, "mycolor") /* mycolor is name in shader */ GLfloat color[4] glvertexattrib4fv(colorattrib, color) /* color is variable in application */ page 181 Uniform Variable Example GLint angleparam angleparam = glgetuniformlocation(myprogobj, "angle") /* angle defined in shader */ GLfloat my_angle my_angle = 5.0 /* my_angle set in application */ gluniform1f(myprogobj, angleparam, my_angle) page

92 Vertex Shader Applications Moving vertices Morphing Wave motion Fractals Lighting More realistic models Cartoon shaders page 183 Wave Motion Vertex Shader uniform float time uniform float xs, zs void main() { float s s = * sin(xs*time) * sin(zs*time) gl_vertex.y = s * gl_vertex.y gl_position = gl_modelviewprojectionmatrix * gl_vertex } page

93 Particle System uniform vec3 init_vel uniform float g, m, t void main() { vec3 object_pos object_pos.x = gl_vertex.x + vel.x*t object_pos.y = gl_vertex.y + vel.y*t + g/(2.0*m)*t*t object_pos.z = gl_vertex.z + vel.z*t gl_position = gl_modelviewprojectionmatrix * vec4(object_pos, 1) } page 185 Modified Phong Vertex Shader I void main(void) /* modified Phong (without distance term) */ { float f vec3 norm = normalize(gl_normalmatrix * gl_normal) vec3 light = normalize(gl_lightsource[0].position gl_modelviewmatrix * gl_vertex) vec3 view = -normalize(gl_modelviewmatrix * gl_vertex) vec3 half = normalize(light + norm) if(0.0 < dot(light, norm)) f = 1.0 else f = 0.0 page

94 Modified Phong Vertex Shader II /* compute diffuse,ambient,specular contributions */ vec4 diffuse = max(0, dot(light, norm)) * gl_frontmaterial.diffuse * LightSource[0].diffuse vec4 ambient = gl_frontmaterial.ambient * LightSource[0].ambient vec4 specular = f * gl_frontmaterial.specular * gl_lightsource[0].specular * pow(max(0, dot(norm, half)), gl_frontmaterial.shininess) vec3 color = vec3(ambient + diffuse + specular) gl_frontcolor = vec4(color, 1) gl_position = gl_modelviewprojectionmatrix * gl_vertex } page 187 Pass Through Fragment Shader /* pass-through fragment shader */ void main(void) { gl_fragcolor = gl_frontcolor } page

95 Vertex Shader for per Fragment Lighting /* vertex shader for per-fragment Phong shading */ varying vec3 normal varying vec4 position void main() { normal = gl_normalmatrixmatrix * gl_normal position = gl_modelviewmatrix * gl_vertex gl_position = gl_modelviewprojectionmatrix * gl_vertex } page 189 Modified Phong Lighting I varying vec3 normal varying vec4 position void main() { vec3 normal = normalize(normal) vec3 light = normalize(gl_lightsource[0].position - position.xyz) vec3 view = normalize(position) vec3 half = normalize(light + view) vec4 diffuse = max(0, dot(light, view)) * gl_frontmaterial.diffuse * gl_lightsource[0].diffuse vec4 ambient = gl_frontmaterial.ambient * gl_lightsource[0].ambient page

96 Modified Phong Lighting II } int f if(0.0 < dot(light, view)) f = 1.0 else f = 0.0 vec3 specular = f * pow(max(0, dot(normal, half), gl_frontmaterial.shininess) * gl_frontmaterial.specular * gl_lightsource[0].specular) vec3 color = vec3(ambient + diffuse + specular) gl_fragcolor = vec4(color, 1.0) page 191 Vertex vs Fragment Shader per vertex lighting per fragment lighting page

97 Samplers Provides access to a texture object Defined for 1, 2, and 3 dimensional textures and for cube maps In shader: uniform sampler2d mytexture vec2 texcoord vec4 texcolor = texture2d(mytexture, texcoord) In application: texmaplocation = glgetuniformlocation(myprog, mytexture ) gluniform1i(texmaplocation, 0) /* assigns to texture unit 0 */ page 193 Fragment Shader Applications Texture mapping smooth shading environment mapping bump mapping page

98 Cube Maps We can form a cube map texture by defining six 2D texture maps that correspond to the sides of a box Supported by OpenGL Also supported in GLSL through cubemap sampler vec4 texcolor = texturecube(mycube, texcoord) Texture coordinates must be 3D page 195 Environment Map Use reflection vector to locate texture in cube map page

99 Environment Maps with Shaders Environment map usually computed in world coordinates which can differ from object coordinates because of modeling matrix May have to keep track of modeling matrix and pass it shader as a uniform variable Can also use reflection map or refraction map (for example to simulate water) page 197 Environment Map Vertex Shader uniform mat4 modelmat uniform mat3 invmodelmat uniform vec4 eyew void main(void) { vec4 positionw = modelmat * gl_vertex vec3 normw = normalize(gl_normal*invmodelmat.xyz) vec3 lightw = normalize(eyew.xyz-positionw.xyz) eyew = reflect(normw, eyew) gl_position = gl_modelviewprojectionmatrix * gl_vertex } page

100 Environment Map Fragment Shader /* fragment shader for reflection map */ varying vec3 reflectw uniform samplercube MyMap void main(void) { gl_fragcolor = texturecube(mymap, reflectw) } page 199 Bump Mapping Perturb normal for each fragment Store perturbation as textures page

101 Normalization Maps Cube maps can be viewed as lookup tables 1-4 dimensional variables Vector from origin is pointer into table Example: store normalized value of vector in the map Same for all points on that vector Use normalization map instead of normalization function Lookup replaces sqrt, mults and adds page