Graphics The Rasterization Pipeline Projection, Visibility, & Shading

Graphics 2014 The Rasterization Pipeline Projection, Visibility, & Shading

Practicals this week Tuesday (today) Announcements Tue 9-11 (was held) Tue 13-15 canceled (programming contest) Wednesday (tomorrow) Wed 15-17: additional practical slot We 17-19: additional practical slot Thursday: no practicals

Addendum: Matrix Algebra CORE core topics important

Transposition T = Matrix Transposition Swap rows and columns In other words: Flip around diagonal Formally: a i,j T = a j,i

Othogonal Matrices Orthogonal Matrices (i.e., column vectors orthonormal) M T = M 1 Proof: next three slides

Matrix Multiplication General matrix products: k B A: possible if #Row(A) = #Columns(B) B m A m n n k R = B A A = B = R = a 1,1 a 1,n a m,1 a m,n b 1,1 b 1,m b k,1 b k,m r 1,1 r 1,n r k,1 r k,n r i,j = m q=1 a q,j b i,q

Matrix Multiplication Matrix Multiplication = a 1 a d = A B a i, b j b 1 b d Scalar products of rows and columns

Othogonal matrices: = = Matrix Multiplication a 1 a d a i, a j A T A = a 1 a d 1 0 0 1 a 1 = I a 2

Transposition Rules Transposition Multiplication: Inversion: A B T = B T A T A B 1 = B 1 A 1 Inverse-transp.: A T 1 = A 1 T Othogonality: A T = A 1 A is orthogonal

Homogeneous Coordinates (short version) CORE core topics important

Problem Translations are not linear x Mx cannot encode translations Proof: Origin cannot be moved: m 11 m 12 m 13 0 M 0 = m 21 m 22 m 23 0 m 31 m 32 m 33 0 = 0 0 0

Homogeneous Coordinates Solution: Just add a constant one Increase dimension R d R d+1 Last entry = 1 in vectors Cheap Trick, Evil Hack M x = = m 11 m 12 m 13 t 1 m 21 m 22 m 23 t 2 m 31 m 32 m 33 t 3 0 0 0 1 M t 0 0 0 1 x 1 = x y z 1 Mx + t 1

Homogeneous Coordinates General case M x = m 11 m 12 m 13 m 14 m 21 m 22 m 23 m 24 m 31 m 32 m 33 m 34 m 41 m 42 m 43 m 44 x y z 1 = x y z w w might be different from 1 Convention: Divide by w-coord. before using Result: x /w y /w z /w 1

Homogeneous Coordinates General case M x = m 11 m 12 m 13 m 14 m 21 m 22 m 23 m 24 m 31 m 32 m 33 m 34 m 41 m 42 m 43 m 44 x 1 x 2 x 3 1 = y 1 y 2 y 3 y 4 y 1 /y 4 y 2 /y 4 y 3 /y 4 1 Can express divisions by common denominator Rules: y 4 = m 41 x 1 + m 42 x 2 + m 43 x 3 + m 44 x 4 Before using as 3D point, divide by last (4th) entry No normalization required during subsequent transformations (matrix-mult.)

Projective Geometry The Full Story? Not just an evil hack Deep & interesting theoretical background More on this later For simplicity We ll treat it as a computational trick for now Focus on the graphics application Remember for now: We can build 4D Translation matrices for 3D+1 points We can divide by a common linear factor

Now for 3D Rendering

3D Rendering Overview BASIC basic topics study completely

3D Computer Graphics Three main aspects Modeling Describe 3D geometry mathematically From machine parts (e.g., CAD) To natural phenomena (e.g., fractals) Animation Set scenes into motion Simple: Camera fly-through Complex: Fluid simulation, human motion Rendering Convert geometry into images Our Focus right now

3D Rendering Assumption 3D Model is given Triangle mesh (for simplicity) How do we get it to the screen?

Upcoming Topics Agenda Modeling: mesh representation Physics: Perspective projection Rendering: Two main rendering methods Rasterization Perspective projection Rasterization Visibility Shading Programmable shaders / GPUs Raytracing

3D Rendering Steps Geometric Model Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination

Modeling Mesh Representation BASIC basic topics study completely

Modeling Shapes Primitives Elementary geometric building blocks Easy to handle directly Complex models Sets of primitives Approximate shapes with primitives yep, triangles! Most-frequently-used Triangles!

Simple Triangle List Vertex list Vector3D vertices[n]; (1) p 1 = (x 1, y 1, z 1 ) (2) p 2 = (x 2, y 2, z 2 ) (3) p 3 = (x 3, y 3, z 3 ) (4) p 4 = (x 4, y 4, z 4 ) Triangle list (int[3]) triangles[m]; (1) t 1 = (i 1, j 1, k 1 ) (2) t 2 = (i 2, j 2, k 2 ) (3) t 3 = (i 3, j 3, k 3 ) (m) t m = (i m, j m, k m ) (n) p n = (x n, y n, z n )

Modeling a Triangle Triangle p 2 p 1 t 1 t 2 p 3 Parametric Plane Equation (with constraints) 0 Triangles: x λ, μ = p 1 + λt 1 + μt 2 = p 1 + λ p 2 p 1 + μ p 3 p 1 0 λ 1, 0 μ 1, μ + λ 1

Attributes How to define a triangle? We need three points in R 3 But we can have more: per-vertex normal per-vertex color texture per-vertex texture coordinates (etc...)

Complete Data Structures Multiple Arrays: Vertices, Triangles, Edges v 1 : (posx posy posz), attrib 1,..., attrib n... v N : (posx posy posz), attrib 1,..., attrib n e 1 : (index 1 index 2 ), attrib 1,..., attrib k... e K : (index 1 index 2 ), attrib 1,..., attrib k edges: optional t 1 : (idx 1 idx 2 idx 3 ), attrib 1,..., attrib m... t M : (idx 1 idx 2 idx 3 ), attrib 1,..., attrib m

3D Rendering Steps Geometric Model Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination

Physics Ray Optics & Color CORE core topics important

Ray Optics Geometric ray model Light travels along rays

Ray Optics Geometric ray model Rays have intensity and color

Ray Optics reddish gray with a tint of green bluish 700nm 390nm wavelength λ Color spectrum Continuous spectrum Intensity for each wavelength

Human Vision greenish reddish low light (monochrome) bluish Color spectrum 700nm 390nm wavelength λ (curves: schematic, not accurate) Two types of receptive cells (color/low-light) Three types of color cells

y-coord. Bitmap (Pixel Display) RGB Model Screen: w h discrete pixels Origin: usually upper left Varying color per pixel RGB Model Every pixel can emit red, green, blue light Intensity range: Usually: bytes 0...255 000 = dark 255 = maximum brightness h 1 0 x-coord. w 1 h w

Human Vision (ideal) monitor: emitted spectra Response curves: human eye 700nm 390nm wavelength λ Create color impressions Basis for three-dimensional color space Wide spacing, narrow bands: purer colors Otherwise: washed out colors (curves: schematic, not accurate)

Physics Perspective Projection BASIC basic topics study completely

Pinhole Camera Pinhole camera Create image by selecting rays of specific angles Low efficiency (small holes for sharp images)

Central Projection Pinhole Camera

Pinhole Camera y y f z Central projection x = f x z Proof: Intercept theorem! y = f y z

(Actual Camera) Camera with Lens Higher efficiency (bundles many rays) Finite Depth of field We will consider pinhole cameras only.

Pinhole Camera y y x = f x z f z y = f y z Undetermined degree of freedom Focal length vs. image size Source of a lot of confusion!

Pinhole Camera α f h Parameters h - size of the screen (pixels, cm, ±1.0,...) f focal length (classical photography) Meaningful parameter: α viewing angle

Pinhole Camera α f h Relation: tan α 2 = h 2f

Pinhole Camera α h h h f f f Invariance tan α 2 = h 2f = h 2f = h 2f Scaling h and f by a common factor: no change

Pinhole Camera α f h Typical choices (vertical angles) Normal perspective: α 30 ( 50mm lens: 27 ) Tele photography: Wide angle lens: α 5 20 (275 70mm) α 45 90 (28 12mm)

General Camera y y x = f x z f z y = f y z Our camera so far: Focus point: origin View direction: z-axis General position/orientation?

Homogeneous Coordinates x y z w = f 0 0 0 0 f 0 0 0 0 1 1 0 0 1 0 Projection Matrix P x y z 1 x = f x z y = f y z z = z 1 z w = z Write in homogeneous coordinates Third row is arbitrary (for now), not used.

View transform α h h f Reminder: tan α 2 = h 2f

To Screen Coordinates 1 tan α 2 0 0 0 0 1 tan α 2 0 0 0 0 1 0 0 0 0 1 Scale to unit screen coordinates We set f to 1 in previous matrix Third row is arbitrary (for now), not used. 1 1 1 + 1 0 1 0 + 1 + 1 + 1 normalized screen coordinates

Aspect Ratio 1 w h tan α 2 0 0 0 0 1 tan α 2 0 0 0 0 1 0 0 0 0 1 Non-square screens? Screen: w h pixels Aspect ratio w h Different horizontal angle! h 1 1 w 1 non-square screen + 1 0 1 + 1 normalized screen coordinates

y-coord. To Screen Coordinates + 1 + 1 w/2 0 0 w/2 0 h/2 0 h/2 0 0 1 0 0 0 0 1 1 0 + 1 1 1 0 x-coord. w 1 0 0 1 + 1 Scale to pixels Third row is arbitrary (for now), not used. h 1 0 h 1 w 1

To Screen Coordinates h/2 w/2 tan α 0 0 tan α 2 2 0 h/2 h/2 tan α 0 tan α 2 2 0 0 a 1 b 0 0 0 1 0 Overall Multiple both a = z far + z near z near z far b = 2 z near z far z near z far Additionally: Also scale + shift such that z = z 1 z are in value [0..1] for inputs z [z near, z far ]

Summary Projection matrix P = f 0 0 0 0 f 0 0 0 0 1 1 0 0 1 0 Projection & conversion to screen coords P s = w/2 0 0 w/2 0 h/2 0 h/2 0 0 1 0 0 0 0 1 1 w h tan α 2 0 0 0 0 1 tan α 2 0 0 0 0 1 0 0 0 0 1 (f = 1) 1 0 0 0 0 1 0 0 0 0 1 1 0 0 1 0 scaling to pixels, upper left origin normalized screen coord s projection matrix

Alternative (1) Alternative formulation: Only two steps P s = w/2 0 0 w/2 0 h/2 0 h/2 0 0 1 0 0 0 0 1 1 w h tan α 2 0 0 0 0 1 tan α 2 0 0 0 0 1 1 0 0 1 0 scaling to pixels, upper left origin normalized screen coord s & projection matrix Different scale factors (not a focal length) Use two different scale factors f x = 1 w h tan α 2, f y = 1 tan α 2

Alternative (2) Another Alternative Formulation w h + 1 0 1 P s = + w h h/2 0 0 w/2 0 h/2 0 h/2 0 0 1 0 0 0 0 1 scaling to pixels, upper left origin 1 tan α 2 0 0 0 0 1 tan α 2 0 0 0 0 1 1 0 0 1 0 vertically normalized screen coord s Constant focal length f = 1 tan α 2 Intermediate result not normalized to 1,1 2 & projection matrix

Alternatives All three derivations lead to the same result Intermediate results not used all equivalent Product of the 2/3 matrices is the same Intermediate results being used: Some graphics APIs (e.g., OpenGL) do use normalized device coordinates as intermediate OpenGL for pixels to appear on screen: x [ 1,1] y [ 1,1] z [0,1) w z near, z far coupled, so this is the same criterion

general camera General Camera object of interest y x z camera in origin, view in z-direction

general camera v General Camera c u w object of interest y x z camera in origin, view in z-direction

general camera v General Camera Camera coordinate system u, v, w u Origin: c w c y x z Standard coordinates x, y, z = 1 0 0, 0 1 0, 0 0 1 camera in origin, view: z-direction

c v u w Derivation y x z

Derivation y x Same effect: Transform the world with inverse camera transform z c v u w u v w = u v w 1

Derivation y x Transform: p u v w 1 p c z c v u w u v w = u v w 1

Derivation y x Transform: u p v w u, v, w orthogonal! p c z c v u w u v w = u v w 1

General Camera general camera u y c x r z v Camera coordinate system u, r, v Origin: c Transform: u p v w (p c) Standard coordinates x, y, z = 1 0 0, 0 1 0, 0 0 1 camera in origin, view: z-direction

general camera y c x u r z General Camera v Camera coordinate system u, r, v Origin: c Homogeneous: p u v c w 0 0 0 1 c = p u v w c

Summary Projection (screen coord s) P s = h/2 0 0 w/2 0 h/2 0 h/2 0 0 1 0 0 0 0 1 Add View Matrix 1 tan α 2 0 0 0 0 1 tan α 2 0 0 0 0 1 0 0 0 0 1 (f = 1) 1 0 0 0 0 1 0 0 0 0 1 1 0 0 1 0 Benefit: Still only one overall 4 4 matrix to multiply with! P s u v c w 0 0 0 1

3D Rendering Steps Geometric Model Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination

Visibility Algorithms BASIC CORE basic topics study completely core topics important

Rasterization Two Rendering Pipelines Project all triangles to the screen Rasterize them (convert to pixels) Determine visibility Apply shading (compute color) Raytracing Iterate over all pixels Determine visible triangle Compute shading, color pixel

Triangle / Polygon Rasterization Observations Straight lines remain straight! Triangles mapped to triangles After Perspective Projection Polygons to polyogns

3D Scene Rasterization Visiblity preprocessing or during rasterization Projection Visibility Rasterization

Raytracing 3D Scene

Comparison Rasterization FOR (each triangle) { } compute pixels covered ( fragments ) FOR (all fragments) { } fragment visible? IF (visible) { } shade fragment write color Raytracing FOR (each pixel) { compute visible triangle IF (found) { shade fragment write color } }

Focus for now: Rasterization Rasterization (Raytracing covered later) Two main algorithms Painter s algorithm (old) Simple version Correct version z-buffer algorithm Dominant real-time method today

Painter s Algorithm CORE core topics important

Painters Algorithm Painter s Algorithm Sort primitives back-to-front Draw with overwrite Drawbacks Slowish O(n log n) for n primitives Millions per second Wrong Not guaranteed to always work

Counter Example Correct Algorithm Need to cut primitives Several strategies Notable: BSP Algorithm in Quake Old graphics textbooks list many variants No need for us to go deeper

z-buffer Algorithm BASIC basic topics study completely

Algorithm Store depth value for each pixel z-buffer Algorithm Initialize to MAX_FLOAT Rasterize all primitives Compute fragment depth & color Do not overwrite if fragment is farer away than the one stored the one in the buffer color depth

Advantages Extremely simple Discussion: z-buffer Versatile only primitive rasterization required Very fast GeForce 2 Ultra: 2GPixel /sec (release year: 2000) GeForce 700 GTX Titan: 35 GPixel / sec (release year: 2013)

Disadvantages Discussion: z-buffer Extra memory required This was a serious in obstacle back then... Invented 39 years ago (1974; Catmull / Straßer) Only pixel resolution Need painter s algorithm for certain vector graphics computations No transparency This is a real problem for 3D games / interactive media Often fall-back to sorting Solution: A-Buffer, but no hardware support

3D Rendering Steps Geometric Model Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination

Shading Models CORE core topics important

Reflectance Models mirror glossy surface diffuse surface

Interaction with Surfaces Viewer v n Light l object surface Local Shading Model Single point light source Shading model / material model Formalization: BRDF Input: light vector l = pos light pos object Input: view vector v = pos camera pos object Input: surface normal n (orthogonal to surface) Output: color (RGB)

Interaction with Surfaces Viewer v n Light l object surface General scenario Multiple light sources? Light is linear Multiple light sources: add up contributions Double light strength double light output Formalization: BRDF

Simplify notation Remark Define component-wise vector product x 1 y 1 x 1 y 1 x y = x 2 x 3 y 2 y 3 x 2 y 2 x 3 y 3 No fixed convention in literature The symbol only used in these lecture slides!

Lighting Calculations Remark Need to perform calculations for r, g, b-channels Often: output r = light r material r function v, l, n output g = light g material g function(v, l, n) output b = light b material b function(v, l, n) Shorter output = light_strength material function v, l, n

Area Light Sources Area Light Sources Integrate over area In practice often: Sample with many point-light sources Add-up contributions n light sources 1 n intensity v n l

Shading effects Diffuse reflection Ambient reflection Perfect mirrors Glossy reflection Phong / Blinn-Phong (Cook Torrance) Shading Effects Transparency & refraction

Diffuse ( Lambertian ) Surfaces n surface normal n θ n θ Equation (set to zero if negative) c ~ cos θ Less light received at flat angles c = c r c l cos θ light color surface color c intensity (scalar) c color (RGB, R 3 ) c r surface color (RGB) c l light color (RGB)

Diffuse ( Lambertian ) Surfaces n surface normal n θ n θ Equation c ~ cos θ Less light received at flat angles Attenuation: (point lights) 1 dist 2 c = c r c l cos θ 1 dist 2 light color surface color c intensity (scalar) c color (RGB, R 3 ) c r surface color (RGB) c l light color (RGB) 1 dist 2

Diffuse Reflection Diffuse Reflection Very rough surface microstructure Incoming light is scattered in all directions uniformly Diffuse surface (material) Lambertian surface (material)

Surface Normal? What is a surface normal? Tangent space: Plane approximation at a point x S Normal vector: Perpendicular to that plane Oriented surfaces: Pointing outwards (by convention) Orientation defined only for closed solids S tangent space surface normal n x R 3 point x

Triangles p 2 n p 3 Single Triangle Parametric equation p 1 p 1 + λ p 2 p 1 + μ p 3 p 1 λ, μ R Tangent space: the plane itself Normal vector p 2 p 1 p 3 p 1 Orientation convention: p 1, p 2, p 3 oriented counter-clockwise Length: Any positive multiple works (often n = 1)

Triangle Meshes Smooth Triangle Meshes Store three different vertex normals E.g., from original surface (if known) Heuristic: Average neighboring triangle normals

Lambertian Surfaces n surface normal n θ n θ Equation c = c r c l cos θ = c r c l n, l n light direction normal vector (assuming: n = l = 1) l

Lambertian Bunny Face Normals Interpolated Normals

Shading effects Diffuse reflection Ambient reflection Perfect mirrors Glossy reflection Phong / Blinn-Phong (Cook Torrance) Shading Effects Transparency & refraction

Problem Ambient Reflection Shadows are pure black Realistically, they should be gray Some light should bounce around... Solution: Add constant c = c a c a Not very realistic Need global light transport simulation for realistic results ambient light color surface color

Ambient Bunny Pure Lambertian Mixed with Ambient Light

Shading effects Diffuse reflection Ambient reflection Perfect mirrors Glossy reflection Phong / Blinn-Phong (Cook Torrance) Shading Effects Transparency & refraction

Perfect Reflection Perfect Reflection Rays are perfectly reflected on surface Reflection about surface normal r = 2 n, l n l + l, n = 1 r n l l arbitrary

Perfect Reflection Difficult to compute Need to match camera and light emitter More later: Recursive raytracing Right image: Environment mapping Silver Bunny Reflective Bunny (Interpolated Normals)

Shading effects Diffuse reflection Ambient reflection Perfect mirrors Glossy reflection Phong / Blinn-Phong (Cook Torrance) Shading Effects Transparency & refraction

Glossy Reflection Glossy Reflection Imperfect mirror Semi-rough surface Various models

Phong Illumination Model Traditional Model: Phong Model Physically incorrect (e.g.: energy conservation not guaranteed) But looks ok Always looks like plastic On the other hand, our world is full of plastic...

How does it work? Phong Model: Specular (glossy) part: (high-) light color c = c p c l r r, v v cos r,v Ambient part: c = c r c a Diffuse part: c = c r c l n, l Add all terms together p 1,2 1 0,8 0,6 0,4 0,2 0 Phong Exponents -90-60 -30 0 30 60 90 p=1 p=2 p=5 p=10 p=50 p=100 v r n l

Blinn-Phong Blinn-Phong Model: Specular (glossy) part: c = c p c l h h, n n p v h n l cos h,n Half-angle direction h = 1 l + v 2 l v In the plane: h, n h Approximation in 3D n = 1 2 r r, v v

Phong+Diffuse+Ambient Bunny Blinn-Phong Bunny Interpolated Normals

Cook-Torrance Model Physically-Motivated Model D Infinitesimal micro-facets Characterize by distribution Expected reflection (density) Gaussian, Beckmann, Approximate occlusion term (G) F Fresnel term Model: wave-optics c spec = D G F 4 v, n n, l Interaction of wave with surface under different angles Percentage reflection/refraction F θ = R 0 + 1 R 0 1 cos θ 5 cos θ = h, v R 0 ="ratio of refractive indices"

Artistic Fresnel Reflection Exponent 4 Approx. Fresnel-Reflection F θ 1 cos θ p unweighted reflection Exponent 5

Better Models Phong Bunny Cook-Torrance Model

Shading effects Diffuse reflection Ambient reflection Perfect mirrors Glossy reflection Phong / Blinn-Phong (Cook Torrance) Shading Effects Transparency & refraction

Transparency Transparency Alpha-blending α = opacity Color + opacity: RGBα Blending Mix in α of front color, keep 1 α of back color 0.0 1.0 0.0 0.5 50% red, 50% green 1.0 0.0 0.0 0.5 front back c = α c front + 1 α c back Not commutative! (order matters) unless monochrome

Refraction Refraction: Snell s Law Materials of different index of refraction Light rays change direction at interfaces Snell s Law sin θ 1 sin θ 2 = n 2 n 1 n 1, n 2 : indices of refraction vacuum: 1.0, air: 1.000293 water: 1.33, glass: 1.45-1.6 θ 2 n θ 1 n n 1 n 2

Refraction Reflection Refraction Implementation (raytraced) Not a local shading model Global algorithms: mostly raytracing Various fake approximations for local shading

3D Rendering Steps Geometric Model Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination

Shading Algorithms CORE core topics important

Flat Shading Flat Shading constant color per triangle

Flat Shading Gouraud Shading Algorithm compute color at vertices, interpolate color for pixels

Flat Shading Phong Shading Algorithm interpolate normals for each pixel

3D Rendering Steps Geometric Model Next: Advanced Rasterization Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination

3D Rendering Steps Geometric Model Global Illum: Keep this for later Perspective Visibility Local Illumination Smooth Shading Simple Shadows Global Illumination