D2 State of the art of Material Rendering Techniques

SIXTH FRAMEWORK PROGRAMME HORIZONTAL RESEARCH ACTIVITIES INVOLVING SMES CO- OPERATIVE RESEARCH Contract for: CO-OPERATIVE RESEARCH PROJECT Project acronym: CADPIPE Project full title: Cad Production Pipeline Proposal no.: 51897 Contract no.: COOP-CT-004-51897 D State of the art of Material Rendering Techniques Version 1.1 1.6.005 G. Papagiannakis, H. Kim, S. Garchery, N. Magnenat- Thalmann CADPIPE

Version history Version Date Author(s) Reviewer Description 0.1 6.05.005 G. Papagiannakis, S. Garchery First draft 1.0 31.05.005 G. Papagiannakis, S. Garchery, H. Kim Release 1.1 1.06.005 H.Kim Conclusion updated Contact information Prof. Dr. Nadia Magnenat-Thalmann Director of MIRALab / C.U.I. University of Geneva 4, rue General Dufour CH111, Geneve-4 Tel : +41 379 77 69, Fax : +41 379 77 80 EMail : thalmann@miralab.unige.ch Web : www.miralab.ch/nthalmann End of editable fields Last modified on June, 005 Y:\Projektit\Hanki0013\CRAFT\CADPIPE\DELIVERABLES\WP1\UNIGE\D_SOA_Material rendering techniques_0605.doc COOP-CT-004-CADPIPE - 51897

UDK Key Words COOP-CT-004-CADPIPE - 51897 i Last modified on.06.05

Preface This work aims to provide a general classification of real physical material properties and their simulation using 3D computer graphics algorithms. Their physical properties are analyzed as well as the general equation that describes the simulation of the reflectance of light when interacts with their surface. Finally various analytical models are illustrated that are currently employed in real-time lighting and shading of such materials. COOP-CT-004-CADPIPE - 51897 ii Last modified on.06.05

Contents Preface... ii Contents... iii List of symbols... iv 1 Introduction... 1 Classification of Materials....1 Opaque.... Translucent... 3.3 Transparent... 4 3 Classification of Reflectance Models [Losasso05]... 5 3.1 The lighting equation... 7 3.1.1 The Lambertian Lighting Models... 7 3.1. The Phong Lighting Model... 8 3.1.3 The Cook-Torrance Lighting Model... 9 3.1.4 The Blinn Lighting Model... 11 3.1.5 The Oren-Nayar Lighting Model... 1 3.1.6 The Minnaert Lighting Model... 13 3.1.7 Wards Anisotropic Lighting Model... 14 3. Bidirectional Reflectance Distribution Function [Kautz04, Wynn05]... 16 4 Rendering techniques... 19 4.1 The rendering equation... 19 4. Computing the rendering equation... 0 5 Light Sources and Illumination... 6 Conclusion and Recommendations... 3 References to key articles... Error! Bookmark not Appendix A: Hardware GPU based Shaders (Lighting and shading programs)... 5 COOP-CT-004-CADPIPE - 51897 iii Last modified on.06.05

List of symbols Symbol Name Explanation I Intensity The output intensity from the lighting equation. This is the final intensity of the fragment. kd Diffuse Reflection The fraction of light that is reflected through diffuse reflection ks Coefficient Specular Reflection Coefficient The fraction of light that is reflected through specular reflection (note that the diffuse and specular coefficients should add up to one) N Surface Normal Normalized direction vector that is perpendicular to the surface L Light Vector Normalized direction vector that points from the surface point to the light location E Eye Vector Normalized direction vector that points from the surface to the viewer (camera location) H Half Vector Normalized direction vector that is the average of the light and viewing vector (normalize (L+V)) R Reflection Vector Normalized vector that is in the direction of the Light Vector, rotated around the Normal Vector 180 degrees. Calculated as follows: R = ( L N) N L n Specular Exponent The specular exponent determines how sharp the specular highlight is (higher is sharper ) c Ellipsoid Eccentricity The eccentricity of the micro facets (0 for very shiny, 1 for very dull) c1,c,c3 Attenuation Coefficients The coefficients for constant, linear and quadratic attenuation of the light source, respectively. d Light Distance The distance to the light G Geometric Attenuation The attenuation factor due to self-shadowing and masking of micro facets on the surface. Calculated as follows: F Fresnel Conductance Term ( N H )( N E) ( N H )( N L) G = min 1,, E H V H Fresnel determines the amount of reflection of the surface (increases as the zenith angle becomes larger). Calculated as follows: ( g c) ( c( g + c) 1) 1 F = 1 ( ) + g + c ( c( g c) + 1) c = cos( ) = L H θ i g = n + c 1 For simplification, the Fresnel conductance term can also be a approximated by the following (much cheaper) equation: F = ( 1 ( 1 ( N E) ) ) + n( 1 ( N E ) ) 5 fatt Attenuation Term The fraction of light that reaches the surface as an effect of light attenuation. Calculated as follows: COOP-CT-004-CADPIPE - 51897 iv Last modified on.06.05

f 1 = min(,1) att c1 + cd + c3d Note that is real life, light is attenuated with the square of the distance, but often, the heuristic (hack) above works better. COOP-CT-004-CADPIPE - 51897 v Last modified on.06.05

1 Introduction The process of the material rendering is processing surface material properties in order to generate illuminated image. The illumination process is usually modeled as reflected lights from the surface. This discourse in real-time material rendering techniques is based on the intuitive separation of the representation of physical properties of materials and their respective reflectance models. The definitions of reflection as well as of the basic equations (BRDF, Rendering) that are employed to simulate physically correct material properties are illustrated. To limit the focus of this study, we emphasize the definition and implementation of real-time material rendering. The most amenable to real-time hardware acceleration real-time reflection models are thus presented in detail together with graphical illustrations of their results. An overview of current hardware lighting and shading techniques (as manifested through shaders) is included in the appendix. In the chapter, existing model for the surface properties are briefly introduced. In the chapter 3, various computational models for reflectance are described. In the chapter 5, as the last parameter, common lighting model is briefly illustrated. The real-time rendering techniques are described in chapter 4, algorithmic approach and in the appendix, the hardware implementation. For further information on the hardware rendering, CADPIPE deliverable D3 will cover thoroughly. VTT Information Technology 1

Classification of Materials First, we would like to show the different kinds of materials that exist in reality. We take a fairly practical approach, and do not classify the materials based on their exact physical properties but rather qualitatively. We then introduce the bidirectional reflectance distribution function, which describes how light is reflected from a surface. It is the most important function in order for rendering realistic materials. We will be classifying material into three large categories in order to study further their reflectance properties and how to render (light and shade) them in 3D. A common property-characteristic of all materials is the one of isotropy-anisotropy. A material is called anisotropic, if its reflection changes when the surface is rotated about the normal. Otherwise it is called isotropic..1 Opaque In Opaque materials, light does not penetrate them and it is only reflected off their surface. This includes for example stone, wood, metals, etc. Opaque materials can be either homogeneous (optical properties are constant) or heterogeneous (optical properties vary across the surface). A good example of a heterogeneous material is wood, as it exhibits a typical ring structure. There are two main types of opaque materials: dielectrics and metals. The highlight of a dielectric has the color of the light source (e.g. plastic, which consists of a white substrate with color pigments that do not contribute to a highlight), whereas metals change the color of the highlight. The reflection properties of all opaque materials can be described by a six-dimensional function, which depends on the position on the surface, the incident light direction, and the viewing direction, as described in section 3. Bidirectional Reflectance Distribution Function [Kautz04, Wynn05]. A general list of material classification with concrete examples of such materials is given below: Dielectrics VTT Information Technology

Plastics Plastics are most popular material type. It reflects light near uniform way and gives highlights affected by color of light source. The optical properties such as color and reflectance level are also uniform. Matte Matte is special kind of opaque material as it does not have highlight. Wood Wood could be conceived as a variation of the plastics except the optical properties are varying on the surface depending on the wood pattern, etc. Leaves and Vegetation These are another variation of the plastics except the variation on the optical properties are much complex. Concrete and Stone The optical properties on these material is not uniform but could be modeled as a uniformed distributed noise. Hair and Fur Each element of hair and fur could be modeled as a plastic material, but it shows very complex appearance as a combination of hundreds of thousands elements. Paint By putting the paint on the existing material, its optical behavior will look like plastics. Worn materials As the paint, worn material effect is an additional layer added to the underlying materials. Optical properties of the worn material differ by type of effects, and underlying material. In general, it could be modeled as a special distribution pattern on the optical properties. Metals. Translucent In contrast to opaque materials, light enters a translucent material, is scattered inside the material and finally leaves the material again. In a translucent material a photon can enter and leave the material at very different positions (called subsurface scattering). Examples of materials that can be categorized as translucent are given in the list below: Skin Marble Milk Alabaster Those exemplar materials have difference in scattering patterns and translucent levels. VTT Information Technology 3

.3 Transparent Transparent materials, such as glass, form a special case of translucent materials. Light enters transparent materials but will generally not be scattered inside the material. It will pass through the material and simply exit again. Examples of materials that can be categorized as transparent are given in the list below: Glass When a photon enters to glass, the simplest model is a part of it reflects on the surface and part of it is refracted inside. The same will be happen when the photon exits from the glass. Water Water is more complex than glass because those simple reflect-refract model is no longer valid. Instead the reflection-refraction phenomenon is continuously occurred inside the material. Figure 1 Different types of materials [RenderMan] VTT Information Technology 4

3 Classification of Reflectance Models [Losasso05] Traditionally, the most used method of finding the color of a pixel has been through Gouraud interpolation of Lambert's Cosine Law at the vertices. This lighting model is extremely cheap to compute and is amenable to fixed function pipelines. With the advent of GPUs, Phong interpolation has become a viable option. This model has allowed for innovations like bump mapping giving more realism to modern virtual worlds. The underlying assumptions however are still the same; the surfaces are Lambertian. Unfortunately, this has the side effect of making most of the computer generated images look very much alike, and very different from the real world. The Lambertian surface assumption works great for some materials (like plastic), which means that different models must be used if other materials are to be represented faithfully. The classification can be made on how the properties of the surface are treated in the reflectance process. Reflection is the process by which light incident on a surface interacts with the surface such that it leaves on the incident side without change in frequency [Wynn05]. In the following figure, the different types of reflection from a single point light source are illustrated: Figure Different kinds of reflections as illustrated by Kautz04 A complete list of various types of reflection models is given in the list below: Type of Reflection Models: VTT Information Technology 5

Ideal Diffuse Lambert's Law Matte Oren-Nayar Minnaert Ideal Specular Reflection Law Glossy Phong Blinn-Phong Ward Schlick Cook-Torrance Some variations are made on the previous fundamental models to represent complex behavior. The extensive list of those methods are, Lambert,Phong, Blinn-Phong Shading models Lafortune described the diffuse cosine lobe (r=cos n x) which is the normal cosine function defined on a sphere, controlled by the sininess parameter n,(phong exponent). That cosine gives a lobe shape which approximates the distribution of energy about a reflected direction. Cook-Torrance Shading Model Torrance-Sparrow microfacets Fresnel's Formulas Anisotropy Models Kajiya Model Poulin-Fournier Model HTSG Model (He,Torrance,Sillion,Greenberg) Empirical Models Straus Model Ward Model programmable Model (RenderMan) Precomputed BRDF Sampled Hemispheres Spherical Harmonics Volume Shading VTT Information Technology 6

Atmospheric Modeling Ocean Modeling Kubelka-Munk Model (paint-pigments) Hanrahan-Krueger Model A complete description of most dominant real-time reflection models is given in the following section together with examples of their application. 3.1 The lighting equation 3.1.1 The Lambertian Lighting Models Lambert created this model over a century ago, and as mentioned previously, it is still the most used model (by a huge margin) in real time graphics today. Lambert assumed that when light strikes a surface, the light is reflected equally in all directions. The light that strikes any given part of a surface is proportional to the cosine of the incident angle 1. The Lambertian Diffuse Lighting Equation: I = k a + k d f att ( N L) The cost of this equation is extremely small, and the results are usable, but it does give the appearance that the lit object is made of rough plastic (no specular component). Screenshot of Lambertian lighting 1 Please see the List Of Symbols for explanation of terms and symbols VTT Information Technology 7

Diagram showing the amount of light emitted in all directions given a light direction (purple line) The bottom two screenshots give a pictorial description of the behavior of the function. The white wireframe contour indicates how much light is given off in each direction, and the purple line indicates the light vector (the direction from which the light is coming). Notice that when the light is coming from an angle, the amount of light reflected in all directions is less that when the incident angle is close to zero (the light vector is perpendicular to the surface) 3.1. The Phong Lighting Model Phong lighting is the other major lighting model that is used in real time rendering, especially after the advent of programmable pipelines. Phong did not change any of Lamberts assumptions, and hence the cosine of the angle between the incident light vector and the surface normal is still used to calculate the diffuse component of the surface. Phong did however create Phong interpolation that involves the interpolation of the vectors across the faces of the polygons as opposed to colors. The largest benefit of interpolating vectors across the polygon is that accurate specular highlights can be reproduced. The major drawback of calculating the lighting equation at every pixel using these interpolated vectors is of course that a lot more computational power is used. The Phong Lighting Equation: a d att n ( N L) + k f ( E R I = k + k f ) s att Two other advantages of Phong interpolation are that it is easy to add bump mapping to surfaces, and the visual appearance of the surface is significantly better than per-vertex interpolated lighting. Objects that are lit by Phong lighting tend to look like various types of plastic (depending on the specular exponent). VTT Information Technology 8

Screenshot of Phong lighting Figures clearly showing that the specular highlight is in the direction of the reflection vector. The diffuse component is the same as the Lambertian diffuse component. 3.1.3 The Cook-Torrance Lighting Model The model presented by Cook and Torrance, strives for as much physical realism as possible, as opposed to the models above that are made entirely ad-hoc for computer graphics only, the Cook-Torrance model uses physical properties, and has all the nice properties one would expect a BRDF to have (such as energy conservation, complex substructures, and a predictive result). BRDFs like Cook-Torrance also has the advantage of using parameters with physical analogies, meaning that artists can tweak the properties of the surface easily to create realistic looking objects. The Cook-Torrance model assumes that the surface is made up of microscopic perfect Lambertian reflectors called microfacets. Several of the terms in the lighting equation deal with how these microfacets are oriented, masked and shadowed. The Cook-Torrance Lighting Equation VTT Information Technology 9

I = k a ρ = k f s e D = 4m + k ρ( L N) + k att d DGFλ ( θi) π ( L N)( E N) tan( β ) m 4 cos ( β ) s DGFλ ( θi) π ( E N) The D term in the equation is the distribution function of the microfacets based on the Beckman distribution function. The m parameter is the root-mean-square (RMS) of the slope of the microfacets. This means that a large m makes the reflections spread out (since the average slope of the microfacets is larger). The G term is the geometric attenuation term, which deals with how the individual microfacets shadow and mask each other. The F term is the Fresnel Conductance term that is wavelength dependent (notice the λ), but for computational simplicity, we will assume that it is wavelength independent (and hence only one calculation is needed). The Fresnel Conductance term deals with the amount of light that is reflected versus absorbed as the incident angle changes (an example of this is often seen when driving on a straight road, and the road appearing mirror-like when viewed from grazing angles). Screenshot of Cook-Torrance lighting From these diagrams, it is clear that this BRDF is clearly different. Notice the complete change in behavior when the incident angle is large VTT Information Technology 10

3.1.4 The Blinn Lighting Model The Jim Blinn model for specular reflection is built on top of the work done in 1967 by Torrance and Sparrow who worked on a model to explain the fact that the specular intensity varies with both the direction of the light source and the direction of the viewer, whereas previous models had ignored the direction of the viewer. The Blinn Model is a modification of the Torrance-Sparrow model that yields similar results, but is significantly cheaper to compute. The Blinn Lighting Equation: I = k f s att DGFλ ( θi) ( E N) c D = ( N H ) ( c 1) 1 + The lighting equation has the same form as the Cook-Torrance lighting equation above, with a geometric attenuation factor and a Fresnel conductance term. The distribution function (D) is now significantly easier to compute. Screenshot of Blinn lighting Diagrams depicting the Blinn BRDF clearly show that Blinn lighting is similar to Cook- Torrance Lighting VTT Information Technology 11

It is evident from the screenshots that it is now possible to make surfaces that look more like metal than plastic using this equation, but without the huge cost of the Cook-Torrance or the Torrance-Sparrow models. Just like all the other per pixel based methods, the per pixel version of this lighting model is also amenable to add-ons such as bump mapping. 3.1.5 The Oren-Nayar Lighting Model Oren and Nayar created a new BRDF in the hope to generalize the Lambertian diffuse lighting model. This BRDF can reproduce several rough surfaces very well, including wall plaster, sand, sand paper, clay, and others. It is however very computationally expensive, and it requires the calculation of azimuth and zenith angles. The need for a better diffuse lighting model seems very real, and in their original paper, Oren and Nayar presented a clay vase that was rendered using the Lambertian and their proposed models compared to a real image. It is glaringly obvious from that demonstration that the Lambertian model was not suitable for representing certain materials, whereas in this case their model was much better. This lighting equation can be calculated either per-pixel or per-vertex, and a per-pixel implementation is amenable to bump mapping, although the pixel program may get prohibitively expensive. The Oren-Nayar Lighting Equation (the simplified qualitative model): ρ I = kd fatt E0 cos( θi ) π σ A = 1.0 0.5 σ + 0.33 σ B = 0.45 σ + 0.09 ( A + B( max[ 0,cos( φ φ )] sin( α) tan( β ))) The lighting equation above is simplified model that Oren and Nayar presented in their paper. It ignores terms like inter-reflections, in an effort to make the model cheaper to calculate. Since hardware like nv30 can calculate the above equation in hardware, at the fragment shader level, no preprocessing is necessary, which is helpful if the light configuration or the geometry is modified. r i VTT Information Technology 1

Screenshot of Oren-Nayar lighting At low angles of light incidence, the Oren-Nayar approximation is similar to the Lambertian model, but at high angles of incidence, the result is flatter, making the edges brighter. As the screenshots above show, the benefit to using the Oren-Nayar lighting model over the Lambertian model is probably not worth the increase computational requirements. 3.1.6 The Minnaert Lighting Model Minnaert added darkening limbs to the lighting equations to make the surface seem darker from certain viewing/lighting directions. This effect is seen in some types of clothing (such as velvet). The computational power required to implement Minnaert style lighting is not very high, and can easily be optimized through the use of a texture lookup. The Minnaert Lighting Equation: I = k d f att ( N L) *( N L) k (1 N E) 1 k Note that the first dot product can be combined with the second making the exponent k+1. The first part of the equation is simply Lambertian lighting, which is then modulated by the darkening factor. VTT Information Technology 13

This lighting model is well suited for bump mapping in it s per-pixel variant. Screenshot of Minnaert lighting It is clear from the diagrams that the darkening limbs bring the amount of light reflected towards zero when the viewer looks onto the surface at perpendicular angles 3.1.7 Wards Anisotropic Lighting Model An isotropic surface has the property that for any given point on the surface, the light reflected does not change when the surface is rotated about the normal. This is the case for many materials, but some materials such as brushed metal or hair this is not the case. The reason for these anisotropic surfaces is that the micro facets that make up the surface have a preferred direction in the form of parallel grooves or scratches. There are several ad-hoc models for lighting anisotropic surfaces that have been developed for use in real time graphics. Other nvidia demos for anisotropic lighting use a texture lookup based on the cosine of the angle between the surface normal and the light vector one axis, and the cosine of the angle between the surface normal and the half angle vector on the other axis. If the texture map has a bright line down the diagonal, then the surface will be bright when those two values are approximately the same. The approach presented here does not use any textures, but is instead based on the BRDF introduced by Greg Ward Larson in 199. The Ward Anisotropic Reflection Model: VTT Information Technology 14

VTT Information Technology 15 ) ( 1 4 ) ( ) )( ( 1 ) ( N H Y H X H y x att s att d y x e L N E N L N f k L N f k I + + + = α α α α The X and Y terms are two perpendicular tangent directions on the surface. They give represent the direction of the grooves in the surface. The α terms are the standard deviations of the slope in the X and Y direction (given by their respective subscripts). Proper tessellation is essential when per vertex calculations are used; otherwise per pixel calculations should be used. Screenshot of Wards Anisotropic lighting (The grooves in the surface were made to look like circular brushed metal patterns) The diagrams of the function clearly show that the lobe of reflected light is oriented perpendicularly to the groove in the surface (the grove is oriented with the wireframe box in the left-right direction)

3. Bidirectional Reflectance Distribution Function [Kautz04, Wynn05] To understand the concept of a BRDF and how BRDFs can be used to improve realism in interactive computer graphics, we begin by discussing what we know about light and how light interacts with matter. In general, when light interacts with matter, a complicated light-matter dynamic occurs. This interaction depends on the physical characteristics of the light as well as the physical composition and characteristics of the matter. For example, a rough opaque surface such as sandpaper will reflect light differently than a smooth reflective surface such as a mirror. Figure 1 shows a typical light-matter interaction scenario. Reflected Light Incoming Light Internal Reflection Transmitted Light Absorption Scattering and Emission Figure 3. Light interactions From this figure, we make a couple of observations about light. First, when light makes contact with a material, three types of interactions may occur: light reflection, light absorption, and light transmittance. That is, some of the incident light is reflected, some of the light is transmitted, and another portion of the light is absorbed by the medium itself. Because light is a form of energy, conservation of energy tells us that light incident at surface = light reflected + light absorbed + light transmitted For opaque materials, the majority of incident light is transformed into reflected light and absorbed light. As a result, when an observer views an illuminated surface, what is seen is reflected light, i.e. the light that is reflected towards the observer from all visible surface regions. A BRDF describes how much light is reflected when light makes contact with a certain material. Similarly, a BTDF (Bi-directional Transmission Distribution Function) describes how much light is transmitted when light makes contact with a certain material. In general, the degree to which light is reflected (or transmitted) depends on the viewer and light position relative to the surface normal and tangent. Consider, for example, a shiny plastic teapot illuminated by a white point light source. Since the teapot is made of plastic, some surface regions will show a shiny highlight when viewed by an observer. If the observer moves (i.e. changes view direction), the position of the highlight shifts. Similarly, if the observer and teapot both remain fixed, but the light source is moved, the VTT Information Technology 16

highlight shifts. Since a BRDF measures how light is reflected, it must capture this viewand light- dependent nature of reflected light. Consequently, a BRDF is a function of incoming (light) direction and outgoing (view) direction relative to a local orientation at the light interaction point. Additionally, when light interacts with a surface, different wavelengths (colors) of light may be absorbed, reflected, and transmitted to varying degrees depending upon the physical properties of the material itself. This means that a BRDF is also a function of wavelength. Finally, light interacts differently with different regions of a surface. This property, known as positional variance, is most noticeably observed in materials such as wood that reflect light in a manner that produces surface detail. Both the ringing and striping patterns often found in wood are indications that the BRDF for wood varies with the surface spatial position. Many materials exhibit this positional variance because they are not entirely composed of a single material. Instead, most real world materials are heterogeneous and have unique material composition properties which vary with the density and stochastic characteristics of the sub-materials from which they are comprised. Considering the dependence of a BRDF on the incoming and outgoing directions, the wavelength of light under consideration, and the positional variance, a general BRDF in functional notation can be written as BRDF λ ( θ, φ, θ, φ, u v) i i o o, where λ is used to indicate that the BRDF depends on the wavelength under consideration, the parameters θ i, φ i, represent the incoming light direction in spherical coordinates, the parameters θ o, φ o represent the outgoing reflected direction in spherical coordinates, and u and v represent the surface position parameterized in texture space. If you are unfamiliar with spherical coordinates, they are explained in the next section. Though a BRDF is truly a function of position, sometimes the positional variance is not included in a BRDF description. Instead, it is common to see a BRDF written as a function of incoming and outgoing directions and wavelength only (i.e. BRDF λ ( θ i, φi, θ o, φo )). Such BRDFs are often called position-invariant or shiftinvariant BRDFs. When the spatial position is not included as a parameter to the function an assumption is made that the reflectance properties of a material do not vary with spatial position. In general this is only valid for homogenous materials. One way to introduce the positional variance is through the use of a detail texture. By adding or modulating the result of a BRDF lookup with a texture, it is possibly to reasonably approximate a spatially variant BRDF. For the remainder of this tutorial, we will denote a position-invariant BRDF in functional notation as BRDF ( θ, φ, θ φ ) λ i i o, o where λ, θ i, φ i, θ o, and φ o have the same meaning as before. VTT Information Technology 17

When describing a BRDF in this functional notation, it is sometimes convenient to omit the λ subscript for the sake of notation simplicity. When this is done, keep in mind that the values produced by a BRDF do depend on the wavelength or color channel under consideration. In practice what this means is that in terms of the RGB color convention, the value of the BRDF function must be determined separately for each color channel (i.e. R, G, and B separately). For convenience, it s usually preferred not to specify a particular color channel in the subscript. The implicit assumption is that the programmer knows that a BRDF value must be determined for each color channel of interest separately. Given this slightly abbreviated form, the position-invariant BRDF associated with a single color channel can be considered to be a function of 4 variables. When the RGB color components are considered as a group, the BRDF is a three-component vector function. VTT Information Technology 18

4 Rendering techniques 4.1 The rendering equation Light reflectance models have always been of great interest to the computer graphics community. The most commonly used real-time illumination model (Blinn-Phong) was derived approximately twenty-five years ago at the University of Utah [Blinn77] but although widely used, it is not energy consistent or represents physically-accurate reflection behavior of models. The general rendering equation represents physicallyaccurate reflection behavior of materials has been well known since the seminal unification performed by [Kajiya86]. However, only recently [Kautz04] with the processing GPU power and physically based reflection models it was possible to perform accurate simulations of dynamic scenes in real-time based on the rendering equation. The rendering equation describes how much radiance is present at each point and at each direction in a scene. Radiance (L) is the most important quantity in radiometry and is expressed as radiant power (Φ) per unit projected area per unit solid angle (Watt/sr.m ). Thus radiance is expressed in the following equation: = d φ L Φ = L( x, ω ) cosθ dω Α dω da cosθ d From Radiometry we also know that radiant power (radiant flux) Φ is radiant energy per dq unit time Φ = and that radiant flux density (Irradiance) Ε is the radiant flux per unit dt dφ area arriving from any direction above the surface: Ε =. Having these basic d Α definitions we proceed to the definition of the bidirectional reflection distribution function (BRDF) which is the most general expression of reflectance of a material at the level of detail we wish to consider. The BRDF is defined as the ratio between differential radiance reflected in an excitant direction and incident irradiance through a differential solid angle: dl ( ) ( x Θ r ) dl( x Θ r ) f Θi Θ r = = de x Θ L x Θ cosθ dϖ ( ) i ΑΩ ( i ) i i Finally the equation that describes all light transport in a scene is the general rendering equation [Kajiya86] that expresses the radiance leaving a surface as a sum of its selfemitted radiance (in case the surface is a light source) and its reflected radiance, as given by the equation below: L x Θ = L x Θ + f x, Θ Θ L x Θ cosθ dω ( ) ( ) ( ) ( ) r r Ω The problem with the rendering equation, is that it is difficult to compute and definitely not a real-time friendly operation. It is an integral over a hemisphere of directions where r VTT Information Technology 19

L, the radiance, appears on both sides of the equation. In order to compute a photorealistic image and to achieve illumination registration in MR, we need to solve the above equation for each visible point through every pixel and compute each radiance value. Using ray tracing algorithms we could assure the following: We can query all possible attributes in a surface point (vertex position, normal, BRDF, etc.) Query light sources (emitted radiance for a given point and direction) Trace rays to find nearest visible point in some direction Check whether two points are mutually visible 4. Computing the rendering equation As many different light paths contribute to a single radiance value at a point visible through a pixel, certain selections have to be made about which paths to check, light sources etc. Thus the evaluation of the above rendering equation in real-time is prohibitive, thus various heuristic-based, or fast-evaluated analytical BRDF models have been proposed in order to be able to approximate its solution in real-time. Finally on the following list we present an overview of different configurations that the rendering equation can be applied in order to solve different combinations of lights and materials. For a complete description of these, authors are encouraged to look further in Kautz04. Direct (local) illumination In this model, photons are arrived in the surface is only directly coming from the light source. No interaction with other surface parts or object is considered. Thus any shadows or other reflected lights are not appeared. This is basic implementation of most graphics APIs including OpenGL. Indirect (global) illumination This model processes interaction between other objects including light sources. In this model, the effect of shadows appears as the result. Depending on the type of light source and the reflected lights, the shadow could appear as a hard shadow or a soft shadow. The hard shadow effect is easier to model because the behavior of photon for it is simpler than a soft shadow. The hard shadow is usually happens when a point light is illuminating and no complex reflection is applied. For further description on the lighting model, next chapter is briefly describes the type of those models. In interaction between surface parts (or objects) are modeled as interreflections and it could be categorized as diffuse interreflections (radiosity) and glossy interreflections (caustics) based on the behavior and pattern of reflected lights. For efficient computation, those interreflection behaviors are pre-computed among surface pairs (or sets). Accurate Solutions of the Rendering Equation To directly model the behavior of the photon, ray-tracing methods are widely adopted. The method simulates actual trajectory of sampled photon to determine the amount of photons illuminating the target surface and reflected from the target surface. The main drawback is that to give sufficient accuracy, the computational complexity is excessively high. To compensate this, a approximated model called radiosity is proposed to quantize the space into a set of polygonal surface parts and pre-calculate optical parameters for each combination of interacting polygons. Approximate Solutions of the Rendering Equation There are main paths in approximating the rendering equation, based on the type of lights that are to be simulated. In the case of point lights, no integration over the VTT Information Technology 0

hemisphere of possible directions is perfromed and the integral is transformed to Summation over the contribution of all lights sources, as depicted so far with the various heuristics or physically plausible lighting models in section 3.1. Thus the only complicated term left is the estimation of the BRDF. In order to approximate the rendering equation for application in real-time environments, another path is the creation of an environment map of the light emitted from scene enclosing the object. This environment map stores incident radiance from the scene towards the simulated object and traditionally has been exhibited in the form of a reflection texture map. When this reflection texture map is pre-filtered with a kernel representing the BRDF then it stores the exit radiance from the object towards the environment. Thus the rendering equation can be greatly simplified as the integration over L (the incoming radiance term) can be substituted by the environment map leaving as the only complicated term the BRDF and the Geometrical relationship term (self shadowing). This solution of light representation is often termed Global Illumination for Real-Time or Area Lights. Distant Illumination (Area Lights) With Shadowing This involves latest work with self-shadowing low-frequency and allfrequency transfer functions, as described by Sloan et al and Ng et al included in Kautz04. Without Shadowing This involves mainly the work of Ramamoorthi et al with irradiance maps as described in Kautz04. Point Lights Ambient Illumination This involves the simple constant term that is employed to compensate for indirect lighting in the simulated scene, as described with the Ka term in the Phong equation in section 3.1. With Shadowing There are currently two prominent methods for simulating real-time shadows with point lights Without Shadowing This involves the lighting models as described in section 3.1 VTT Information Technology 1

5 Light Sources and Illumination Although no the focus of this work, for completeness we illustrate the various types of Light sources that affect the different solutions of the rendering equation and its interaction with the various BRDF models. The type of lights affects the amount of incoming photons in the illumination process. For a more complete description, authors are encouraged to look in [Kautz04]. Local Illumination Light models used in the local illumination model are categorized as point lights, directional lights, spot lights, and ambient lights. Point light is a model of light source such that it is a point that emits the photon uniformly to every direction. The spot light is a variation of the point light as the emission direction is limited as a corn or similar structures. Directional light is a simplified model of sun-lights as photons are emitted in parallel directional way from the distant light source. The ambient light is a extremely simplified model of uniform reflection as it illuminates the environment truly uniform way. Global illumination Following rapid development of the hardware capability, it is possible to implement the complex illumination models for real-time rendering. Hard shadows and special kind of reflected materials (mirrors) were the first thing that was implemented in common APIs (although, the performance is still quite an issue for efficient rendering, as it requires a couple of more times on computation cost than local only illumination.) Instead of directly modeling the behavior of photons like ray-tracing, the current trend on the real-time rendering is to categorize the environment according to illuminating behaviors. Radiosity is a method to quantize the photon behavior as the interaction between piecewise planar (polygonal surface) parts. It gives comparably high level of rendering results with reduced computation time. The main drawback of the raiosity method is that its computational efficiencies can be achieved only for the static environment. Furthermore, even though it is efficient than the ray-tracing, still it is not efficient enough to be used in the real-time rendering. Environment map captures complex behavior of lights into a set of parameters and provides correct parameters to the one point in the space to be illuminated. It can be implemented with a simple look-up method and gives quite a good rendering result. The main problem is that the captured/modeled parameter is only valid at the one point in the environment and only works on the static environment. Strategic focus is also made on the extending simple illumination model with area lights. As most of light sources in the real environment are area or volume lights, it will give more accurate illumination between objects. The current solution is mainly sampling light points from the area source and modeling it as a set of point lights. Due to the innate complexity of the model, technology is not well matured to be applied in the real-time rendering yet. VTT Information Technology

6 Conclusion and Recommendations To represent a surface material, the illumination parameters are represented and simulated by selected reflectance/illumination model. The type of material is categorized by its behaviour to the light source according to the general illumination model. Graphics APIs like OpenGL and DirectX have historically consisted of a set of fixed functions for fundamental rendering operations including illumination/material rendering. By calling a function and passing a set of parameters, a developer could cause geometrical surfaces to be drawn on the screen in a certain way. That fixed, configurable 3D graphics pipeline consists of two main stages: the geometry processing stage and the rendering stage. Current consumer graphics hardware (Geforce6, Radeon 9500 and onwards) offer the possibility of replacing this fixed-function rendering pipeline with user-developed programs, usually referred to as shaders. There are two kinds of shaders according to the main stages: Vertex Shaders which involve mainly the shape and position of objects and Fragment Shaders which are focused to control the material properties and textures of surfaces. Utilizing these shaders function will give a possibility to use complex illumination models such as BRDF function. As indicated in the survey sections, the implementation of pre-existing rendering functions in realtime has been focused in research and development. There have been a few approaches to accommodate perceptual parameters in accelerating renderings and optimizing representations. Although those approaches showed some possibilities in devising optimal representation according to human perceptions, it still requires more thorough investigations on computational models of psychophysics and its corresponding representations. The rendering equations and BRDF, for example, could be further simplified or modified according to those finding once we get results from it. Although the main focus of CADPIPE does not lie on the opening new researches on this area and the advance in this area would require thorough research on coming decades, it is noted that the choice of general function and methodology would open a possibility to adopt new techniques. As the main material category, which will be used in the CADPIPE project, is metal and plastics, it is sufficient to use an API which supports fundamental reflectance model and material properties such as uniform diffuse and highlighted specula. Also, a possibility to have a shader is necessary to provide enhanced rendering for static scene. As the result of this observation and research described in this section we recommend that the OpenGL would be supported in the CADPIPE project as an open standard for real-time rendering. References to key articles [Blin77] J. Blinn. Models of Light Reflection For Computer Synthesized Pictures. In Proceedings SIGGRAPH, July 1977 [Cg05] Cg Language Toolkit User s Manual, Release 1.3, NVIDIA, http://developer.nvidia.com [Directx05] Microsoft, The DirectX 9 API, http://www.microsoft.com/windows/directx/, accessed 10th May 005 [Kajiya86] J.T. Kajiya, The rendering equation, Proc. of SIGGRAPH 1986 The term vertex-fragment shader is preferred from the term vertex-fragment program, according to the OpenGL.0, DirectX and Renderman nomenclature. VTT Information Technology 3

[Kautz04] J. Kautz, Hardware Lighting and Shading: A Survey, Computers Graphics Forum 3(1), March 004, pages 85-11 [Losasso05] F. Losasso, Surface Reflection Models, http://developer.nvidia.com, accessed 10th May 005 [Mccool01] McCool, M., SMASH: A Next-Generation API for Programmable Graphics Accelerators, SIGGRAPH 000 Course on Real-Time Programmable Shading, New Orleans, 000 [Peercy00] Peercy, M., Olano, M., Airey, J., Ungar, J., Interactive Multi-Pass Programmable Shading, (Proc. SIGGRAPH 00) [Proudfoot01] Proudfoot, K., Mark, W., Tzvetkov, S., Hanrahan, P., A Real-Time Procedural Shading system for Programmable Graphics Hardware, (Proc. of SIGGRAPH 01) [Wynn05] C. Wynn, An introduction to BRDF-Based Lighting, http://developer.nvidia.com, accessed 10th May 005 VTT Information Technology 4

Appendix A: Hardware GPU based Shaders (Lighting and shading programs) Graphics APIs like OpenGL and DirectX have historically consisted of a set of fixed functions. By calling a function and passing a set of parameters, a developer could cause geometrical surfaces to be drawn on the screen in a certain way. That fixed, configurable 3D graphics pipeline consists of two main stages: the geometry processing stage and the rendering stage. Current consumer graphics hardware (Geforce6, Radeon 9500 and onwards) offer the possibility of replacing this fixed-function rendering pipeline with user-developed programs, usually referred to as shaders 3. There are two kinds of shaders according to the main stages: Vertex Shaders which involve mainly the shape and position of objects and Fragment Shaders which are focused to control the material properties and textures of surfaces. Vertex Shader Thus the vertex shader is a set of instructions that operate on incoming vertex values and their associated data. The vertex shader is replacing the traditional graphics operations such as: Vertex Transformation Normal Transformation and normalisation Texture Coordinate generation and transformation Lighting Colour material application Clamping of colours Such vertex shaders are programs written either in assembly or in a High Level shading language. As vhdrenderer is rendering API independent, it is open to support both DirectX9 and OpenGL.0 HLSL APIs. However, for the time being and for fast prototyping, as we are implementing the OpenGL profile first and OpenGL.0 HLSL is not finalised yet, we choose to utilise the Cg HLSL by NVIDIA with only the ARB_OPENGL profiles. Vertex shaders that intend to perform computations similar to the fixed functionality of OpenGL are responsible for writing the code for all the functionality in the list above. For instance, it is not possible to use the existing fixed functionality to perform vertex and normal transformation but have a vertex shader perform a specialised lighting function. The vertex shader must be written to perform all three functions. Furthermore, any OpenGL state used by the shader can be tracked and made available to the shader. It operates on one vertex at a time and the output is sent through subsequent stages of processing as they are defined in OpenGL1.4: primitive assembly, users clipping, frustum 3 The term vertex-fragment shader is preferred from the term vertex-fragment program, according to the OpenGL.0, DirectX and Renderman nomenclature. VTT Information Technology 5

culling, perspective projection, viewport mapping, polygon offset, polygon mode, shade mode and culling. Fragment Shader The fragment shader is a program that operates on fragment values and their associated data. The fragment shader is intended to perform traditional graphics operations such as: Operation on interpolated values Texture access Texture application Fog Colour sum Pixel zoom Scale and bias Colour table lookup Convolution Colour matrix Fragment shaders do not replace the fixed functionality graphics operations that occur in the back end of OpenGL pixel processing pipeline, such as: Shading model, Coverage, Pixel ownership test, Scissor, Stipple, Alpha, Depth, Stencil tests, Alpha blending, Logical ops, Dithering, Plane masking For each fragment, the fragment shader can compute colour, depth, stencil or one or more arbitrary data values, but not its x/y position. Sometimes, the notion of a fragment is mistaken for the notion of a pixel. However, a pixel is only the final colour value written to the frame buffer, and each pixel in the frame buffer usually corresponds to multiple fragments. A fragment is a point in window coordinates produced by the rasterizer with associated attributes, such as interpolated colour values, a depth value and possibly one or more texture coordinates. A fragment modifies the pixel in the frame buffer at the same window space location based on a number of parameters and conditions defined by the pipeline stages following the rasterizer. In the following diagram, the complete 3D graphics pipeline with the addition of vertex and fragment shaders is illustrated, from the Cg [CG05] point of view and an example of shader application is illustrated in Figure 5. VTT Information Technology 6