GREMM: a real-time graphics engine for forest rendering

Transcription

1 Umeå University Department of Computing Science MASTER THESIS GREMM: a real-time graphics engine for forest rendering 11th February 2003 Morgan Johansson neomind@criosphinx.com Supervisor: Berit Kvernes beritk@cs.umu.se c Copyright 2003 Morgan Johansson

2 GREMM: A REAL-TIME GRAPHICS ENGINE 2

3 Abstract One of the hardest problems within real-time computer graphics is the realistic rendering of detailed outdoor scenes. Outdoor scenes with plants, grass and trees are difficult to render in real-time mainly because of the extremely high number of graphical primitives needed for realistic visualization of them. This thesis describes the theory behind an OpenGL-accellerated graphics engine, which was created specifically for the purpose of rendering forest scenes with a very high polygon count. It is based on dynamically generated impostors, which where first described in 1995 but then used in a slightly different way, as the engine among other things introduces a planning stage where the rendering of the scene is treated as an optimization problem. The test results show that rendering engines based on dynamically generated impostors may give significant gains in frame-rate compared to normal rendering, at the cost of a slightly reduced visual quality. The frame-rate was under certain conditions increased by 500% while the visual impact was low. Currently, there is still room for improvement of the techniques as the visual quality is not always satisfactory. It is expected that the future development will yield both improved quality and performance. 3

4 GREMM: A REAL-TIME GRAPHICS ENGINE 4

5 Acknowledgements I would like to thank my supervisor Berit Kvernes for her help and for never saying she was too busy to help me whenever I came knocking at her door. I also wish to express my gratitude to my girlfriend Maria Pääjärvi for her support whenever I needed it. Finally, I would like to thank my parents for the encouragement and support they have always given me. 5

6 GREMM: A REAL-TIME GRAPHICS ENGINE 6

7 Contents 1 Introduction Background Objective Description of the problem domain Structure of the thesis Current techniques Billboards Common techniques in entertainment software Previous work Real-time display of forested virtual scenes Dynamically generated impostors Other relevant work Generation of trees L-systems A modular tree generation system Generation results Overview of GREMM Calculating the importance of objects Approximating rendering cost Measuring visual quality Impostor generation The rendering cache Planning the rendering Object clustering Implementation details Performance analysis of GREMM The visual quality Test setup Test results

8 GREMM: A REAL-TIME GRAPHICS ENGINE 6.4 Analysis Drawbacks with GREMM 61 8 Future development of the engine Alpha blended impostors Shadows Cluster-based impostors Efficient cache lookup Improved texture management Animation support Dynamic K-means clustering Concluding remarks 69 8

9 List of Figures 2.1 Billboards used for distant trees The foliage consists of several billboards The branches are represented with partially transparent textures A very simple L-system example Plants generated using simple L-systems A possible setup of component types Trees generated using the modular tree generator Cylindrical bounding box The impact of apparent size on importance The impact of foreground/background on importance The impact of horizontal centering on importance The impact of camera translation/rotation on importance Error due to changing direction Error caused by perspective view Impostor resolution An impostor of the tree will be rendered The tree is centered on the screen but does not fit the texture Changing the FOV The object and its bounding box fit inside the texture The impostor will use this area as a texture Fake shading by tilting the normals The basic role of the rendering cache Clustering of trees Impostor using a texture of the wrong resolution A virtual fly-through A screenshot of the performance testing program A forest scene with a total of 1280 trees Time-based alpha blending for transition Alpha blending of sampled views A rendering cache tree structure

10 GREMM: A REAL-TIME GRAPHICS ENGINE 8.4 More efficient texture allocation

11 Chapter 1 Introduction The field of real-time 1 computer graphics is one of the fastest evolving fields within computer science and engineering. This has more recently been caused mainly by the rapid developments within hardware acceleration of 3D computer graphics. In fact, the rate at which graphics hardware has become faster has been far exceeding Moore s law 2 during the few years. Even with this rapid development there are challenges within computer graphics that probably will remain very challenging for many years to come. One of those is the the realistic display of outdoor scenes that are rich in plants. The amount of detail needed to realistically display a dense forest is staggering. That is not to say that attempts are not made to display forested scenes as the forest is, for obvious reasons, one of the most important settings for virtual environments. This thesis will describe the theory behind the the Graphical Rendering Engine for Massive Multiplicity (GREMM), an OpenGL 3 -accelerated real-time rendering engine created specifically for the purpose of rendering forests. 1.1 Background Nearly all early representations of trees and plantlife in real-time 3D graphics have relied on billboards (described in closer detail in section 2.1). Billboards can be highly detailed and require very little resources, but generally they look flat and unrealistic. The transition from billboards to the more realistic actual 3D geometry has been slow but is constantly ongoing as the hardware constantly becomes faster. 1 Within computer graphics, the term real-time usually refers to an animation which is created on-the-fly. Usually this means that what is seen on the screen is less than 2/F P S seconds old. 2 More than 25 years ago Gordon Moore predicted that the number of transistors on a microprocessor would double approximately every 18 months. This generally translates to a doubled computing speed every 18 months. 3 An application programming interface for the development of 2D and 3D graphics applications. 11

12 GREMM: A REAL-TIME GRAPHICS ENGINE Today, most entertainment software relies on 3D geometry for representation of trees. While these representations are still simple and not really realistic they still provide a higher degree of realism than billboards. The use of real 3D models for plantlife poses another problem - the geometric models. There are three different ways to create new models. The first is to three-dimensionally scan existing plants (which is expensive), the second is to have an artist create the models (which is time-consuming and often expensive as well) and the third, which is the one used in this thesis, is to automatically generate the models. The most well known, and widely used, technique for generation of plants is by using L-systems introduced in 1968 by Astrid Lindenmayer [Lin68]. These provide highly detailed and realistic models of plants. The only major drawback is that the models are often so natural (and therefore - complex) that forests of such plants are unrealistic to render in real-time, using the usual rendering algorithms. 1.2 Objective The objective of this thesis is to describe how a virtual forest with a very high polygon count can be displayed using a combination of techniques. This is done by describing the techniques used in GREMM. GREMM will also be used to evaluate the effectiveness of these techniques. 1.3 Description of the problem domain Though the objective states that the thesis will describe how a virtual forest with a very high polygon count can be displayed, the techniques behind GREMM are more general than so. In fact, it is expected that the engine will perform even slightly better on some types of animated objects since popping 4 effects are less noticeable then. Generally the problem domain covers all cases where a 3D engine should display identical objects multiple times in the same scene within a short timeframe. To exemplify this, assume a scene where a thousand marching soldiers are displayed. The soldiers are all identical and are in almost identical poses, but are seen in different locations. Unless the number of poses in the scene is huge, the engine will quite often be able to reuse previously rendered soldiers to reduce the time needed to render the scene. A good example of a scene which fits GREMM badly is a scene displaying a simulated lava lamp. The lamp constantly morphs and it is for that reason not possible to reuse previously rendered representations of the lava lamp. 4 When the steps in an animation are so big that the object appears to pop into its new position/pose. 12

13 CHAPTER 1. INTRODUCTION 1.4 Structure of the thesis The thesis begins with a survey of the current techniques in use to solve the problem of rendering trees. It then describes the methods currently in use and the previous scientific work done in mainly the field of dynamically generated impostors but also in some other relevant fields. Following that, L-systems are briefly described. A basic understanding of L-systems is required for the following parts where the specifics of the tree-generator used in the test application is described. The main part of the thesis describes the theory behind GREMM and this is followed by an experimental analysis of the performance of the engine. Finally, the thesis concludes by discussing the drawbacks and possible future development of GREMM. It is assumed that the reader has a basic understanding of concepts and techniques in 3D computer graphics. 13

14 GREMM: A REAL-TIME GRAPHICS ENGINE 14

15 Chapter 2 Current techniques Compared to the rapid developments within computer graphics, the methods used for real-time visualization of trees have evolved very slowly. The improved realism of real-time-rendered plants today is to a large extent an effect of the faster hardware and the better methods used for creation of the 3D geometry. There are several proposed methods to increase the realism of rendered trees, but these have, so far, only sparsely been used outside of the academic domain, and when so - usually in specialized software. 2.1 Billboards Billboarding is one of the most common techniques to reduce the complexity of a graphical scene. The basic concept is very simple - represent an object by using the image of the object instead of using a real-time rendered representation based on its geometric properties. Billboards are usually implemented as rectangular polygons that are textured 1 with an image and which always rotate to face the viewer (hence the name billboard). Often these display different images when viewed from different directions, but only in discrete steps as each of these directions is statically included as a prerendered texture. Billboards have several serious drawbacks. They look flat, especially when the viewer is close, and rarely have a good enough resolution to handle the situation when the viewer is very close to the tree, resulting in clearly visible pixelation. A problem with billboards that display different images for different viewing directions is popping since it is difficult to have sufficiently many viewing directions covered, especially when the viewer moves in three dimensions (in relation to the viewed object). 1 An image is used to drape a polygon. The texture assigns colour information to the polygon. 15

16 GREMM: A REAL-TIME GRAPHICS ENGINE 2.2 Common techniques in entertainment software Figure 2.1: Billboards used for distant trees Modern entertainment software (also known as computer games) generally display trees as simplified 3D geometry where the most complex structure of the tree, the foliage, is made out of partially transparent textures (which often are billboards). This is the case in the game Battlefield developed by Digital Illusions CE, which is fairly representative for entertainment software in general. Figures display screenshots from Battlefield 1942 where the different levels of detail involved the tree rendering are clearly displayed. Figure 2.1 shows a tree as seen from a great distance, represented by a single billboard. As the viewer moves closer, the stem of the tree is replaced by 3D geometry, and the foliage is displayed using several textured, partially transparent polygons, as shown in figures 2.2 and Battlefield 1942 tm is a registered trademark by Digital Illusions CE 16

17 CHAPTER 2. CURRENT TECHNIQUES Figure 2.2: The foliage consists of several billboards Figure 2.3: The branches are represented with partially transparent textures 17

18 GREMM: A REAL-TIME GRAPHICS ENGINE 18

19 Chapter 3 Previous work 3.1 Real-time display of forested virtual scenes Forest rendering is a very complex field, and hence there are many research fields within computer graphics from which methods can be applied to forest rendering. Those mentioned here have either been directly applied to forest rendering or are quite closely related to the work described in this thesis. Marshall et. al. [MWW01] wrote about automated generation of forested scenes and real-time display of them. Their approach to the real-time display of forested scenes was similiar to the one used in this thesis, using dynamically generated impostors and level of detail (LOD) 1 techniques. They showed that the performance of the system was greatly improved by these techniques but they also stated that there are problems associated with these techniques resulting in a lowered visual quality. Meyer et. al. [MNP01] described a method for rendering trees based on blending of precomputed hierarchical impostors. The method supports both shading 2 and shadowing and the main drawback is that it requires a lot of texture memory. Another precomputed approach was described by Jakulin [Jak00] - a simple and fast approach based on slicing and alpha blending 3 of the slices which probably works best on fairly dense vegetation. A technique for simplification of the foliage was described by Remolar et. al. [RCB + 02]. They also described how the technique could be used to reduce the complexity of the trees, not only through common LOD, but also by reducing the complexity at the far side of the foliage. Another technique for simplification was described by Marshall et. 1 A technique to reduce the complexity of the scene. Objects are represented in several levels of detail (usually referring to the number of polygons and resolution of the textures). 2 Colouring a polygon according to its incident light. The colour depends on the position, orientation and attributes of both the surface and the sources of the light. 3 The blending of images by transparency. al. 19

20 GREMM: A REAL-TIME GRAPHICS ENGINE [MFAC97]. It is an interesting approach based on subdivision of the scene into tetrahedrons that are used as approximations of their contents. Deussen et. al. [DCSD02] recently showed how rendering with points and lines (similar to the approach of Reeves and Blau [RB85]) could be used to render the plants in fairly complex outdoor scenes in real time. Weber and Penn [WP95] have also used points and lines for drawing trees. 3.2 Dynamically generated impostors An impostor is a replacement object used to mimic another object of higher complexity. Maciel and Shirley [MS95] defined an impostor as an entity that is faster to draw than the true object, but retains the important visual characteristics of the true object. Billboards (section 2.1) and traditional LOD are examples of commonly used impostors. In [Sch95], Schaufler described dynamically generated impostors and how these could be used to reduce the rendering complexity of the scene. The algorithm used for updating the impostors was simple: DrawObjectWithImpostor() if (maximum resolution of impostor) > (screen resolution) if error for existing impostor is large enough newimposter() drawimpostor() else drawtrueobject() The impostor is created by using the rendered image of the true object as a texture on a billboard. Schaufler et. al. [SS96] later extended the technique by introducing a hierarchy of impostors, where impostors at a lower level are utilized when rendering the higher level impostors. This thesis will not use this extended method mainly because it fits badly with the suggested extensions, since it uses more texture memory and increases the total number of impostors in the scene. There are several differences between Schaufler s impostors and those used in GREMM. The most important ones concern the management of the impostors, and the actual impostor is basically the same. The main reason for the differences between Schaufler s work and the methods used for impostors in this thesis is the slightly different problem domain. While Schaufler focuses on reducing the complexity of the scene using a specified minimum quality (in the case described in [Sch95] - getting the highest possible frame rate without loosing visual quality), forest rendering is often a problem where rendering without visual degradation is not possible so the goal is to get the highest possible visual quality at a specified frame rate. 20

21 CHAPTER 3. PREVIOUS WORK 3.3 Other relevant work Particle systems First described in 1983 [Ree83], particle systems today belong to the most important techniques within both real-time and prerendered graphics. Particle systems stochastically create minute particles in controlled ways. These particles then follow some defined set of rules (often based on the laws of physics) until their lifespans are over and they disappear. In 1985, Reeves [RB85] showed that particle systems could be used (back then in non-real-time) to render realistic trees, by displaying this in the animated short movie The Adventures of Andre and Wally B. [Luc84]. So far this method has only been used in experimental or specialized software, and entertainment software has not yet used particle systems for forest rendering, though particle systems are commonly used for other purposes. Ground vegetation Rendering of grass is one field that has received much attention, much of it because of its close resemblance with the rendering of fur. Reeves [Ree83] showed how particle systems could be used to create realistic grass, however, rendering whole fields of grass in real-time using this approach is not possible yet. Currently, the most widely used way to render fields of grass (in ways more realistic than just texturing the ground) is probably by using volumetric textures [MN98][BB02]. Volumetric textures do not work very well when viewing grass from the side, since the gaps between the layers of textures become visible. Perbet et. al. [PC01] proposed using three different representations to overcome this problem. Their work shows that by combining several techniques it is possible to display fields of grass quite realistically even in real-time. Occlusion culling for forested scenes An efficient and fast algorithm for occlusion culling 4 of forested scenes would greatly enhance the performance when displaying dense scenes. Unfortunately, most trees are badly suited as occluders since the foliage rarely is dense enough to completely occlude everything behind it and also because trees do not resemble any simple geometric shapes. That does not mean there are no available algorithms, but it limits the choices to algorithms which do not utilize the geometric properties of the objects, as such algorithms would be too slow. The Hierarchical Z-Buffer Visibility [GKM93] algorithm is one suitable algorithm, which makes use of the Z-buffer 5 (in practice - the Z-buffer of the 4 The removal of objects that are hidden behind other objects. 5 A buffer containing depth information. 21

22 GREMM: A REAL-TIME GRAPHICS ENGINE previous frame) to find which objects are visible. Aside from using the Z- buffer, the algorithm is completely software-based on most platforms. Bartz et. al. [BMH99] proposed another Z-buffer-based algorithm which made more use of the graphics hardware, and they had very promising results. Their method had the drawback that it could possibly remove objects that were not completely occluded. Further, as it was hardware assisted, it put more strain on the graphics hardware than the hierarchical Z-buffer visibility algorithm. 22

23 Chapter 4 Generation of trees As trees are complex structures, one might think that automatic generation of trees is difficult. That is not the case, as structures resembling trees can be described and generated using very simple rules. The most common way to generate models of trees is by using L-systems but there have been other approaches [Hon71][RB85][Opp86][dREF + 88]. These other approaches will not be described as the generator described in section 4.2, based on L-systems, worked sufficiently well in its current design. 4.1 L-systems Astrid Lindenmeyer [Lin68] introduced L-systems in 1968, originally as a formalism to model and simulate the development of simple multicellular organisms. Later, it became apparent that L-systems worked well also for generating models of plants for virtual environments. A simple L-system is defined by its variables, constants, starting point and rules. An example: Variables: A B Constants: C Start: A Rules: A AB B CA This L-system produces the following output: Stage 0: A Stage 1: AB (since A AB) Stage 2: ABCA (using A AB + B CA) Stage 3: ABCACAB 23

24 GREMM: A REAL-TIME GRAPHICS ENGINE Stage 4: ABCACABCABCA And so on... Methods for plant generation using L-systems have been thoroughly researched, starting as early as 1974 in [FL74]. One of those who have done the most extensive research in this field, Prusinkiewicz, was one of the authors behind Visual Models of Plant Development[PHHM97], the paper describing the techniques on which the tree generation in the testing program was based. Using L-systems, plants are described as a spatial configuration of discrete constructional units called modules, which develop over time. Typically, these modules represent the basic components of the plant, such as flowers and leaves. A simple example of how L-systems can generate models of plants is shown in figure 4.1. Figure 4.1: A very simple L-system example Although this method is simple, it is still powerful enough for the creation of highly realistic models of plants. Figure 4.2 displays some three-dimensional models of plants created created using simple L-systems. L-systems do not actually provide any kind of visual output. Visual output comes from the mapping of a string into a visual domain (which is done implicitly in figure 4.1). The most common way to do this mapping is by using turtle interpretation, a interpretation where L-systems provide movement and orientation instructions for an imaginary turtle [PHHM97]. As the turtle moves through space it draws lines (or cylinders). While the notation is very powerful, it is generally far more flexible than is required in entertainment software. As turtle drawing is also fairly slow, another solution was used. There are several extensions to the basic L-systems that are relevant to this thesis. One is parametric L-systems, which admits the passing of arbitrary numerical information from parents to children. Another highly relevant extension is stochastic L-systems which introduces randomness into the otherwise completely deterministic L-systems. Finally, environmentally-sensitive L-systems 24

25 CHAPTER 4. GENERATION OF TREES Figure 4.2: Plants generated using simple L-systems are relevant as they provide an easy way to affect the growth of the tree based on position and orientation of the individual modules. These extensions (and some other) are very well described in [PHHM97], and will not be described any further in this thesis. 4.2 A modular tree generation system The tree generation is not really a part of GREMM but it was an essential part of the testing system, and was also created with other future applications in mind. The tree generation system is loosely based on L-systems as described by Prusinkiewicz et. al. in [PHHM97]. The tree generation system was created with the use in entertainment software as a potential application. Therefore it does not use any kind of descriptive formal language for the description of the trees. Instead, the focus was on making the tree generation engine highly modular so that some kind of plantdescription plugins could be used. While the drawback is less flexibility in the description of the plants, the advantages are increased speed and better modularity. The generation process is designed to be fast enough for load-time use in entertainment applications. The trees are created by use of components. These components are the basic building blocks for the plants. The component types essentially determine the physical properties of components of that component type, how the component 25

26 GREMM: A REAL-TIME GRAPHICS ENGINE is connected to its parent and where children are connected to the component, including the default orientation of each connection point. For example, the component types displayed in figure 4.3 may be used. Figure 4.3: A possible setup of component types As mentioned above, the goal was to make the tree generation process modular. The generation was divided into three separate modules, all of which may be replaced almost independently of the others. The only restriction is that the number of component types should always remain fixed. The generator module - contains all the rules that define the position and orientation of each component. The component module - specifies the component types and their individual geometric properties such as length/radius-ratio and the relative size of the connected children. The drawing module - specifies how to draw each component, based on its component type and physical properties. Currently there are no more than two different drawing modes. All component types except leaves are represented with cylinders of the same colour. Leaves are represented with green triangles. The algorithm used to grow the tree, starting with the connection point for the root, is the following: function grow(connection point,size,parent): newtype=selectcomponenttype(parent) newpart=create new component of newtype and specified size connected to connection point 26

27 CHAPTER 4. GENERATION OF TREES for every child connection point in newpart perform a minor random translation of the child connection point along its parent orient(child connection point) if child connection point is on the side grow(child connection point,size*k1,newpart) else grow(child connection point,size*k2,newpart) The orient step is a very complex operation. The orientation and position of the connection point are first set by the default values included in the component description of its parent. Orient performs yaw-, pitch- and roll-operations by a set of rules defined in the generation module. These rules may depend on local data such as the orientation and type of the component or its parent, or global data such as the direction up or the direction towards the sun. The rules may also depend on random numbers (actually pseudo-random numbers, so that each generated plant can be reproduced given that the same random seed is used). As the geometric properties of the tree and its graphical representation are separate, it is easy to implement different level-of-detail optimizations in the drawing module. The drawing module accepts a detail value (in the range 0 to 1), which modifies the number of triangles used in the implementation (the size of the part also affects the number of triangles used in its representation). 4.3 Generation results Figure 4.4 displays a few models of trees rendered using this generator. The trees consist of 8000 to triangles. All four trees were generated using almost the same rule set, with only some minor variations to the parameters included. While models with a substantially higher degree of realism have been generated, these usually required ten times more triangles making them unsuitable for real-time applications. 27

28 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 4.4: Trees generated using the modular tree generator 28

29 Chapter 5 Overview of GREMM Schaufler [Sch95] used a fairly simple algorithm for updating the dynamically generated impostors (section 3.2). That algorithm works very well for minimizing the visual impact of using the impostors but as previously stated, this thesis assumes a situation where real-time rendering without a decrease in visual quality is not possible. The main improvement over the early work of Schaufler is the resource management in GREMM. The resource GREMM manages is available rendering time, and it is used to increase the visual quality. GREMM uses the following algorithm: DrawSceneWithImpostors(): calculate importance of the objects in the scene calculate free rendering time (including previously unused time) mark all critically important objects and approximate lost time calculate current scene quality Q old distribute free rendering time to maximize Q new Q old generate the new impostors draw all non-critical objects using best available impostor draw all critically important objects using true rendering All objects are contained in cylindrical bounding boxes 1 (which will sometimes be referred to as bounding cylinders). The cylinder is always aligned with the global upward direction (figure 5.1 displays bounding cylinder of an upright standing tree). Each 3D model is instantiated any number of times (and then referred to as instances or objects). Objects share both the geometric model and impostors which saves resources when rendering forests where the same tree model is placed on many locations. 1 A box bounding an object on all sides. 29

30 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 5.1: Cylindrical bounding box 5.1 Calculating the importance of objects Determining which parts of the scene that are truly important requires more information than is actually available. The assumption that objects in the center of the screen are the most important for the user seems generally true but the reality could be that the user is actually looking at some object in one of the corners of the screen. However, since some kind of assumptions must be made, one can only strive to make them good enough to be true in most cases. The method for importance calculation is based on these four assumptions: Objects with a greater visible size (screen area) on the screen are more important than smaller objects even when displayed at the same position of the screen. This is displayed in figure 5.2 (darker=more important). Objects in the foreground are more important than objects in the background (figure 5.3). Objects in the horizontal center of the screen are more important far from the horizontal center (figure 5.4). When the viewer is translated or rotated the areas that came into view recently are more important than those that have been in view for a long time (figure 5.5, viewer is rotating in direction of arrow). The mathematical function used to calculate the importance is based on these assumptions, however, some approximations are made since the calcu- 30

31 CHAPTER 5. OVERVIEW OF GREMM Figure 5.2: The impact of apparent size on importance lation must be performed very rapidly. The function τ which is used for the importance calculation is defined in equation 5.1. τ(object, view, secview) = (w p g(object, view) + w s g(object, secview) + w r f RND (0, 1))/(w p + w s + w r ) (5.1) The function g, which describes the importance of an object inside a single view is defined in equation 5.2 for all cases where the object is inside view. g(object, view) = (k s α + k f d n d + where α is: α = hwc r 4d 2 tan 2 (F OV/2) k c (F OV 2 cos 1 (v f u (u v u)v u ))/F OV )/ u (u v u )v u (k s + k f + k c ) (5.2) w < d tan(f OV/2) and h < (d/c r ) tan(f OV/2) 1 w d tan(f OV/2) or h (d/c r ) tan(f OV/2) 0 if object is outside view If the object is outside view, then g is defined as: g(object, view) = 0 (5.3) The symbols used in equation 5.2 and 5.3 are defined in the following table 31

32 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 5.3: The impact of foreground/background on importance symbol description f RND (0, 1) Random function giving a value between 0 and 1 w p Weight of the primary view w s Weight of the secondary view k s Weight of size for importance k f Weight of foreground/background for importance k c Weight of horizontal centering d f distance to the back clipping plane d n distance to the front clipping plane h Height of object c r Screen x/y ratio w Width of object(diameter of bounding cylinder) m s Maximum h*w in scene d Distance to object v f Normalized vector in viewing direction v u Normalized vector pointing up in viewers frame of reference u Normalized vector towards object F OV Field-of-view The function g has three different parts. The first part (with k s ) gives the importance due to apparent size. The second part (with k f ) is a very rough approximation of foreground/background importance only based on the distance to the object. The last part (with k c ) models the importance due to horizontal centering. The function g does not model the effects of translation and rotation 32

33 CHAPTER 5. OVERVIEW OF GREMM Figure 5.4: The impact of horizontal centering on importance of the viewer. Instead, that is reflected by the secondary view. The secondary view is a probable view within a fairly short time. It is calculated, as shown in equation using linear dead reckoning 2 on both rotation and translation over a time span of a few frames (which is reflected by the constant k). If Ψ is such that then view current = view previous + Ψ (5.4) view secondary = view current + k Ψ (5.5) The advantage of using this secondary view is not only that it reflects importance based on viewer rotation and translation. It also introduces the possibility of rendering impostors of the off-screen objects that will likely be visible soon. To exemplify the need for this, assume a case where the viewer is looking around in a scene with complex objects and that the system is fast enough to update one object each frame while keeping an adequate frame-rate. Using Schaufler s [Sch95] update algorithm, every time a cluster of objects enters the view, the system will slow down noticeably. If, instead, the system can predict the need for objects before they enter the view a few frames ahead, there will be more time to render the needed impostors and the frame-rate can be kept more stable. 2 Originally a term used by navigators. Blind calculation is used to approximate (guess) the position and heading. 33

34 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 5.5: The impact of camera translation/rotation on importance 5.2 Approximating rendering cost There are several possible approaches to approximating the rendering cost of an object. Funkhouser et. al. [FS93] describes one approach, where the rendering time needed to render an object is calculated based on the number of pixels 3, vertices and triangles in the object and an automatic analysis of the system s rendering speed. The approach has one major drawback, it must analyze each rendering technique used. In GREMM a simpler technique is used. The system dynamically learns the time needed to render an object. Every time a model is rendered, a timer times the clock time needed (in more than µs resolution) to render the model. The new approximation (ϕ new ) for the time needed to render the model is calculated as displayed in equation 5.6. ϕ new = (1 k) ϕ old + k t k [0, 1] (5.6) where t is the time measured the last time and k is the learning rate. The impact of distance (the number of pixels required) was ignored since it had little affect on the testing system. The alternative would have been to store the timings of the last n times and find a least-square fit of a function to that data (which could then have been mixed with the previous fit as above, still using a learning rate). Different LODs are handled as separate models if they are discrete. Should the 3 Modern display devices (monitors etc.) are based on a grid of coloured dots. Each such dot is a pixel. 34

35 CHAPTER 5. OVERVIEW OF GREMM LOD be continuous, then it could be handled by fitting a function as described above, or by introducing discrete steps for the approximation. Compared to the method described in [FS93], this method is generally slower (but not noticeably so when such complex objects as trees are rendered), but it has the advantage that it dynamically reflects the current speed of the rendering system. This is a significant advantage when there are other threads executing in parallel to the graphics system, as is common in entertainment software. 5.3 Measuring visual quality A useful definition is, for the purpose of measuring visual quality, that visual quality is the absence of errors. The use of impostors introduces errors in the system. In GREMM, using dynamically generated impostors, the following errors are relevant: View direction errors. When the viewer translates sideways in relation to the viewed object, old impostors display the objects from the wrong direction. Perspective errors. When the viewer translates towards or away from the object, old impostors display the object with the wrong perspective. LOD errors. If the object can be displayed using several levels of detail, lower levels of detail introduce errors. Resolution errors. Errors introduced when the impostor is displayed at a resolution higher than it was rendered at. Lighting errors could also have been relevant but as GREMM uses a method to fake the lighting (section 5.4) that is not a problem. The quality of the rendered image is expressed by the scene quality function Q(objects, representations). The quality of each object is determined by the quality function q(object, representation) which is composed by quality aspect functions γ aspect (object, representation). Let O be the set of all objects, o j is the object j, r(o j ) the representation of object j (that which is drawn to the screen), and i(o j ) is the importance of object j. The scene quality function, for n objects and m quality aspect functions, is defined in equation 5.7. n j=0 c min q min + c max q max + c mean i(o j) q(o j,r(o j )) n j=0 Q(O) = i(o j) c min + c max + c mean q min = q(x, r(x)) ( x O, y O : q(x, r(x)) q(y, r(y))) q max = q(x, r(x)) ( x O, y O : q(x, r(x)) q(y, r(y))) (5.7) 35

36 GREMM: A REAL-TIME GRAPHICS ENGINE q, the quality function for each object is defined in equation 5.8. In equation 5.8, w j is the weight of the j:th quality aspect. q(o, r) = mj=0 w j γ j (o, r) mj=0 w j (5.8) The quality aspect functions The quality aspect functions describe the quality of the different aspects of the rendering like perspective correctness and resolution. Many of these only apply to impostors, and true rendered objects are always of full quality. Each function ranges from 0.0 to 1.0 with 1.0 being the best possible quality. The functions should preferably be as close to linear to the perceived quality as possible. Ideally, 0.9 should be perceived as 90% of the quality of 1.0 and 0.5 should be half the quality of 1.0. As this is highly subjective care should be taken in the creation of these functions. Also, these functions are called thousands of times each frame, so speed of computation is essential. The quality aspect functions used in GREMM are the following: Quality aspect: View direction Figure 5.6: Error due to changing direction The view direction is always expressed using the the coordinate system of the model as that is the only coordinate system that all instances share. That way, the viewing directions can be compared between instances without the need for further transformations. A suitable quality aspect function for directional errors is defined in equation cos 1 (v o v r) k if v o v r > 0 and γ viewdir (object, representation) = cos 1 (V o V r ) < k 0 otherwise (5.9) 36

37 CHAPTER 5. OVERVIEW OF GREMM symbol v o v r k description Current view direction (normalized) The view direction (normalized) when the impostor was rendered A specified maximum error angle In figure 5.6, the angle marked a is the angle used as a measure of the error. Another possibility would have been to use the length of the section marked error as a measure. Though in many ways a better measure of the error, it is harder to create a good quality aspect function from it. Quality aspect: View perspective The cause for errors due to the way the perspective view is affected by distance is displayed in figure 5.7. As the impostor is flat, any point that is neither Figure 5.7: Error caused by perspective view in the impostor plane nor on the centerline (a line from the viewer through the center of the impostor) moves around on the impostor as the distance is changed. Based on the situation in figure 5.7, the error may by calculated as displayed in equation error = d 1a d 1 b d 2a d 2 b d 1 d 2 (5.10) However, on a cylindrical bounding box, neither a nor b is constant and although they can be calculated for each distance, this thesis will instead use an approximation to save calculations. One can safely assume that b < d 1 and b < d 2, since the impostors are not intended to be used when the viewer is inside the bounding box. A constant c is defined as the maximum distance from the center of the bounding box to any point inside the box so that c a and c b for all values of a and b. Then the equation 5.11 can be derived from the 37

38 GREMM: A REAL-TIME GRAPHICS ENGINE equation 5.10: error d 1c d 1 c d 2c d 1 d 2 (5.11) d 2 c Based on equation 5.11 a suitable quality aspect function for the perspective error may then be the one defined in equation γ perspective (object, representation) = symbol d o d r k { 1 d oc d o c drc description Distance to the impostor Distance when the impostor was rendered A specified maximum error d r c /k doc d o c drc 0 doc d o c drc d < k r c d k r c (5.12) Quality aspect: Levels of detail The use of several levels of detail is simple to handle. Each LOD must be assigned a quality value (from 0.0 to 1.0) with the best LOD having a quality value of 1.0. This may be done manually, or the quality value (q) could be based on the resolution (usually the number of triangles) of the model compared with the most detailed model (which has a quality value of 1.0), as in equation q = res (5.13) res max Based on equation 5.13 the quality aspect function is as defined in γ LOD (object, representation) =< quality value used in impostor > (5.14) Quality aspect: Impostor resolution The basics for the resolution of the impostor are simple. A texel 4 should not be larger than the pixels of the screen. In one dimension the problem is as displayed in figure 5.8. Equation 5.15 defines w s as the full width of the scene at the position of the texel. w s = 2d tan(f OV/2) (5.15) The relation in equation 5.16 (where w i is the width of the impostor, r i is the resolution of the impostor, r s is the resolution of the screen and r max is the maximum available texture resolution) should apply when the resolution of the impostor is optimal. r i = r s w i w s (5.16) 4 A pixel of a texture. 38

39 CHAPTER 5. OVERVIEW OF GREMM Figure 5.8: Impostor resolution A good measure of quality in one dimension is defined in equation γ res1 (object, representation) = { ri / rs w i w s r i < rs w i w s and r i < r max 1 r i rs w (5.17) i w s or r i r max Since the screen is two-dimensional, the square of equation 5.17 is a better measure of the quality which results in the quality aspect function as defined in equation That is under the condition that the x/y resolution ratio of the impostor is the same as the x/y resolution ratio of the screen (which is the case in GREMM). γ res (object, representation) = { (ri / rs w i w s ) 2 r i < rs w i w s and r i < r max 1 r i rs w i w s or r i r max (5.18) 5.4 Impostor generation The rendering of the impostor may be a bit tricky to implement but the actual theory is quite simple. Once there is a situation where an impostor should be rendered (figure 5.9), the camera 5 settings are stored safely. The settings are recovered once the impostor has been created. Then the camera is rotated towards the object so that the object is centered (figure 5.10). OpenGL only allows for texture sizes that are powers of two (like 256x128 or 512x64). For that reason the object probably needs to be either enlarged or 5 A virtual eye, which defines the view showed on the screen. Also referred to as a viewer. 39

40 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 5.9: An impostor of the tree will be rendered made smaller. While it would be possible to always use a larger texture, and if that is not possible, fall back to true rendering (as in [Sch95]). That would be a waste of texture memory and would also restrict the use of impostors. The size of the object should only be changed by increasing or decreasing the field of view (FOV) used in the rendering, while the distance to the object is kept constant. The reason is that changing the FOV only affects the resolution and the perspective to the object is constant (figure 5.11). Equation 5.19 defines the function GREMM uses to calculate a suitable new FOV. FOV new = k F OV sc s w (5.19) symbol s c s w k description Current screen size of the object Wanted screen size A constant (slightly over 1) used as a safety margin Once the FOV is set the object and its bounding box should fit inside the texture (figure 5.12). The only thing remaining is to render the object and to transfer the rendered image to a texture. There are several possible methods for transferring the rendered image to a texture. The best is by using pixel-buffers [NVI] which admit rendering directly into texture memory. Unfortunately, pixel-buffers are currently only available as an extension to OpenGL and are not part of the OpenGL 1.3 standard. For that reason GREMM currently uses the somewhat slower method of rendering into the framebuffer 6 and copying the required area directly into the texture 6 A buffer in which the image displayed on the screen is kept. 40

41 CHAPTER 5. OVERVIEW OF GREMM Figure 5.10: The tree is centered on the screen but does not fit the texture memory. Not the fastest solution, but still faster than some alternatives. There are two major drawbacks. As impostors require alpha channels, the framebuffer must include them as well. This usually means that a 32 bit screen mode must be used. Another drawback is that the screen resolution limits the largest possible impostor size as the screen must fit the entire texture. Using 800 times 600 pixels limits the impostors to 512 times 512 texels. Lighting conditions To avoid the problems related to changing lighting conditions all impostors are generated using a directional light source pointed downward (which is not such a bad approximation for most outdoor scenes). This ensures that all impostors are created with equal lighting. To give some illusion of different lighting directions the normals of the impostor are slightly tilted towards the sides of the impostor, as if the surface had been rounded (figure 5.14 displays the impostor as seen from above). 5.5 The rendering cache All rendered impostors are inserted into the rendering cache of the model they belong to. Before rendering each frame the best available impostor is found for each object for potential use in the rendering of the frame. Figure 5.15 illustrates this basic role of the rendering cache. Impostors are stored with information on the rendering conditions (such as viewing direction). When the most fitting impostor is required, that is the impostor which maximizes the object quality function (section 5.8). 41

42 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 5.11: Changing the FOV Limited either by the available texture memory or by the complexity of searching the cache, the cache must have an upper limit for the amount of impostors it can store. An upper limit implies that impostors must also be discarded. This is the reason why the cache must maintain information on the usage of each impostor. That information is maintained in a usage table: this frame last frame history n this n last n history The usage table contains information on the number of times that each impostor has been drawn on the screen. Every time the impostor is drawn, n this is incremented by one. Once the frame is completed, the table is updated in the following way (where k is a constant controlling the rate of decline): UpdateUsageTable(): n history = k n history + n last n last = n current n current = 0 The usage value of each impostor is calculated as in equation 5.20, where c 1, c 2 and c 3 are constants. usage = c 1 n current + c 2 n last + c 3 n history (5.20) 42

43 CHAPTER 5. OVERVIEW OF GREMM Figure 5.12: The object and its bounding box fit inside the texture In a full rendering cache, the impostor with the lowest usage value is replaced if a new impostor is added. Impostors that may potentially be rendered (impostors that fit some visible object best) are never replaced. If the cache is full and no impostor can be removed, then some alternative method must be used. In GREMM, in such a case, the object is represented with the best available impostor. Recently added impostors are not created with an empty usage table. It will contain some default values high enough to avoid that the impostor is removed from the cache before it has built up some usage value. 5.6 Planning the rendering The idea behind GREMM is that the rendering time should be spent where it benefits the visual quality the most. This optimization problem requires that the rendering of the scene is planned. In GREMM, the first step in that process is to find critically important objects. These are objects which are too important (their importance is above a threshold) for impostored rendering. The next step is the building of a selection table where all the choices available to the rendering engine are presented. The table below displays a possible selection table. Each row represents an impostor which may potentially be used in the scene. 43

44 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 5.13: The impostor will use this area as a texture Figure 5.14: Fake shading by tilting the normals Impostor benefit 1 time 1 benefit 2 time 2 benefit 3 time For each impostor, there are a few benefit-time options. These represent the rendering of a new impostor for the object with the greatest need for improvement (utilizing the impostor of the row). The different option-levels are based on the different LODs the impostor may be generated at. The greatest need for improvement is determined by comparing (1 quality) importance. The objects with the highest (1 quality) importance of each impostor are the objects which may get new impostors rendered to fit them. The other objects using the impostor are all forced to use either the newly rendered impostor or the currently best one. The benefit of each option is calculated as displayed in equation benefit n = (q old q new ) importance (5.21) 44

45 CHAPTER 5. OVERVIEW OF GREMM Figure 5.15: The basic role of the rendering cache symbol q old q new description Current quality of the object with the worst fit using the impostor Quality of the object should it get a new impostor using LOD n Paired with the benefit of a level is time. That is the approximate rendering time needed to render the impostor at that LOD. The selection table is sorted by the best benef it/time of each line. The distribution of rendering time is done with a greedy algorithm by always selecting the line and the level with the best benef it/time. Rendering time is distributed until the best (regarding benefit/time) of some line cannot be used as there is not enough free rendering time left. Then the remaining rendering time is saved for the next frame and the scene is created using the selected choices (as described in the algorithm at the beginning of this section). 5.7 Object clustering Figure 5.16: Clustering of trees GREMM supports the automatic clustering of objects by using the K-means clustering algorithm: 45

46 GREMM: A REAL-TIME GRAPHICS ENGINE For each of the clusters in C: randomly set the center. Loop until termination condition is met: For each data element: assign that element to a cluster such that the Euclidean distance from this element to the center of that cluster is minimized. For each cluster: recalculate the means of the cluster based on the elements that belong to that cluster. end loop; The clusters are currently only used to speed up viewport culling, however, they were intended for faster occlusion culling and cluster-level importance calculation for scenes with a higher number of objects. 5.8 Implementation details GREMM was implemented in C/C++, totalling at about 8000 lines of source code not counting comments in the code. It uses OpenGL 1.3 and Simple DirectMedia Layer [SDL], making it nearly completely platform independent at a source code level (meaning that it can be compiled for most platforms). GREMM has been tested under both Windows 2000 and Linux 2.2, and it was equally stable under both (though there where some - so far - unexplained performance problems under Linux, possibly due to driver issues). 46

47 Chapter 6 Performance analysis of GREMM The big question is, does GREMM really provide such a significant performance advantage that it is worth all the overhead caused by the advanced rendering techniques? 6.1 The visual quality Impostors are not expected to look as good as the real thing. There is always some visual impact caused by the use of impostors. The dynamically generated impostors in GREMM are no exception. In GREMM, the main impact on the visual quality is caused by either using impostors that do not show the right view of the object or by using textures that do not fit the resolution of the screen. There are some other errors with a minor impact on the visual quality. One is caused by the fake shading. While the shading of the impostor is adequately realistic, the transition from impostor to true rendering causes a noticeable change in colour. The other minor errors are caused by the flatness of the impostors. Even if the quality of the impostor is perfect, the boundary between the impostor and the environment might be wrong, for instance causing an impostored tree to appear from the ground in a slightly different place than the true rendering. Finally, the perspective in the impostors is only 100% correct for objects that are centered on the screen. The perspective error is generally not noticeable but it may be if the field of view is very wide. Using impostors displaying the wrong view of the object mainly causes popping, since the illusion of smooth transitions between the different views of the object is lost. When the texture resolution does not fit the resolution of the screen the results can be severe pixelation as shown in figure 6.1. In GREMM, aliasing errors are always present but they are usually hardly noticeable. The most 47

48 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 6.1: Impostor using a texture of the wrong resolution common way to avoid aliasing is by using mip-mapping, but using that solution would make the impostor generation substantially slower. Currently, there are no plans to include any kind of anti-aliasing in GREMM, however, the extreme error in figure 6.1 could be avoided by simply rendering the impostor at a higher resolution. Testing the performance of GREMM is not completely simple. The engine is designed to reduce the visual quality to reach a requested frame-rate. The problem is that the visual quality must be measured as well as the frame-rate. In section 5.3, a function was defined to measure the visual quality of the scene. However, for testing purposes it is too complex to be useful and therefore another simpler measure of the quality was introduced. Whenever some object was represented in two consecutive frames by impostors with an directional error (as described in section 5.3) of at least 18 degrees (1/20 of a complete circle) it was defined as an impostor failure. In all tests the impostor failure rate (equation 6.1) was measured. impostor failure rate = impostor failures/number of impostors displayed (6.1) Though subjectively, the impostor failure rate was found to be a fairly good measure of the quality. The error rate is fairly constant at about 0,0-0,1 % as long as the engine updates the scene quickly enough. As soon as it starts to rise, visual errors start to appear. Below 0,5 %, the errors are hardly noticable. Between 0,5 and 1,0 % the errors are noticeable and mainly in the form of popping. At above 1 % failure rate, the errors are often severe and quite disturbing. This should be seen as a rule of thumb, as there are irregularities. Sometimes, severe errors may occur even if the failure rate is low (but it is quite rare). 48

49 CHAPTER 6. PERFORMANCE ANALYSIS OF GREMM 6.2 Test setup Figure 6.2: A virtual fly-through The testing system was a 1.33 GHz Athlon Thunderbird, with 256 megabytes of DDR SDRAM, running Windows The graphics card in the system was based on a nvidia Geforce 3 ti200 GPU with 64 megabytes of graphics memory. Some background tasks were present but none which should have had any noticeable effect on the results. The performance test is a virtual fly-through of a forest scene with a camera displayed in a windows of 800 times 600 pixels. The camera flies through the scene on a path like the path in figure 6.2. The camera is elevated 3-8 meters (though virtual meters, these are in relation to the size of the trees) above the ground (which is not flat) and constantly moves along the path so no two frames are created from the same position. The path takes exactly 60 seconds to complete and the movement along it is connected to the system clock and not to the rendering. The camera rotates independently of its movement, and the rotations are of different speed, the slowest rotations being of about 10 degrees/second and the fastest being of about 270 degrees/second. Figure 6.3 displays a screenshot of the performance test. As the image shows, no ground is rendered. The ground exists however, and the elevation of the trees is determined by it. The cache was at all times kept as large as allowed by the size of the graphics memory. This generally meant that about 250 impostors would fit in the cache. 6.3 Test results The first seven tests show how the frame-rate develops when the number of instances are increased. The target frame-rate (measured in frames/second - FPS) specifies the frame-rate which the impostoring engine tries to keep. The seven first tests only use a single tree model, of either or 4254 triangles. Using a single tree model minimizes the need to render new impostors but 49

50 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 6.3: A screenshot of the performance testing program also maximizes the overhead. The last test shows how the frame-rate develops with an increased number of unique models (with the total number of instances remaining constant). 50

51 CHAPTER 6. PERFORMANCE ANALYSIS OF GREMM 10 FPS target with 1 tree model of triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 20 32,8 30,1 0, ,2 14,5 0,3 80 6,0 9,4 0, ,5 9,9 0, ,3 10,2 1,9 51

52 GREMM: A REAL-TIME GRAPHICS ENGINE 20 FPS target with 1 tree model of triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 20 32,7 30,0 0, ,2 20,1 0,2 80 5,9 21,7 0, ,5 22,1 0,1 52

53 CHAPTER 6. PERFORMANCE ANALYSIS OF GREMM 30 FPS target with 1 tree model of triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 20 32,9 30,1 0, ,2 30,9 5,4 80 5,9 33,2 0, ,4 30,5 1,9 53

54 GREMM: A REAL-TIME GRAPHICS ENGINE 40 FPS target with 1 tree model of triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 20 32,8 33,6 2, ,2 38,7 4,3 80 6,0 41,5 1, ,4 32,5 5,0 54

55 CHAPTER 6. PERFORMANCE ANALYSIS OF GREMM 10 FPS target with 1 tree model of 4254 triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 80 26,3 12,4 0, ,0 4,9 0, ,9 5,3 0, ,1 5,7 0, ,6 7,0 0, ,8 4,2 0,1 55

56 GREMM: A REAL-TIME GRAPHICS ENGINE 20 FPS target with 1 tree model of 4254 triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 80 26,3 12,5 0, ,0 10,7 0, ,9 13,3 0, ,1 15,9 0, ,6 7,7 0,9 56

57 CHAPTER 6. PERFORMANCE ANALYSIS OF GREMM 40 FPS target with 1 tree model of 4254 triangles instances true rendering (FPS) impostored (FPS) failure rate (%) 80 26,3 18,5 0, ,0 24,5 0, ,9 35,2 0, ,1 21,7 0,8 57

58 GREMM: A REAL-TIME GRAPHICS ENGINE 15 FPS target with 640 instances of 8186 triangles in average unique models true rendering (FPS) impostored (FPS) failure rate (%) 1 2,3 11,1 0,0 2 2,3 12,9 0,0 4 2,3 14,2 0,0 8 2,3 15,0 0,0 16 2,3 19,7 30,0 58

59 CHAPTER 6. PERFORMANCE ANALYSIS OF GREMM 6.4 Analysis In many cases, the use of the impostoring engine is simply a loss of performance as well as visual quality. However, in some cases, the results are very promising. Such is the case in the 20 FPS test using the high resolution model. Using 20 or 40 instances, there is no or very little gain but at 80 instances the performance gain is about 250%. At 160 instances the gain is 500% with an failure rate of 0,1 %. The results at 20 FPS using the low-resolution models are equally interesting. At 640 instances the impostored system is more than four times faster, keeping 16 FPS with a low failure rate. The tests show that a higher target FPS increases the failure rate. To some extent this is expected, since a shorter frame time means that more objects with errors are ignored each frame. That alone does not completely explain the high failure rates. The main reason for the rapid rise in failure rate as the target frame-rate increases is probably a kind of render-time underrun. Critically important trees are always rendered using true rendering and when the total render-time available for each frame is less than the time needed to render the critical objects, then no new impostors will be generated. This may be so severe that it causes gaps of several seconds in the generation of impostors. A fairly easy way to avoid this problem would have been to modify the algorithm so that the engine will always generate at least one new impostor each frame regardless of the available time, since a slight drop in frame-rate is generally preferable to unsatisfactory visual quality. The test results clearly show that the engine does not hit the target framerate very well. The benchmark is created not to restrict the frame-rate (which the engine otherwise does by sleeping), so there is nothing strange with results that are above the target frame-rate. However, most results are actually below the target frame-rate and that is probably mainly caused by the overhead involved in the cache-lookup operations. Currently, the engine does not in any way compensate for the time lost in these operations. Supporting this is the test varying the number of models used. The frame-rate increases as the number of models increases (because the caches become smaller). However, at 16 unique models the failure rate suddenly increases to 30% from 0% at 8 models. This is because one or more of the caches is too small to fit the needed impostors. A higher number of unique models reduces the overhead, but also reduces the reuse of impostors. The tests using the low-resolution model had some unexpected results. In both the 10 and 20 FPS target frame-rate cases, the performance of the engine was improved as more instances were added. In the 10 FPS test, the frame-rate increased by 43% as the number of instances increased from 160 to 1280 and in the 20 FPS test, the increase was 49% as the number of instances increased from 160 to 640. Currently, the only explanation is that this is caused by the increased number if critical renderings. This reduces the available time which 59

60 GREMM: A REAL-TIME GRAPHICS ENGINE makes the greedy planning terminate faster. Also, critical renderings require no planning. Figure 6.4: A forest scene with a total of 1280 trees Some of the results are promising. Gains in frame-rate of up to 500% could translate into outdoor scenes five times more dense than previously possible. Figure 6.4 displays a screenshot of a fly-through in an extremely dense scene. The forest scene had a total of 1280 trees of 4254 triangles each and it was displayed at 7 FPS. 7 FPS is not good enough for entertainment applications, but the testing system was also far from cutting edge. Also, that particular scene is not wise to use as the trees are not rendered efficiently. Using individual triangles for each leaf in the tree is not an effective way to display trees. A more effective representation using textures could reduce the required complexity of the trees substantially, without loosing visual quality. 60

61 Chapter 7 Drawbacks with GREMM Nothing is free and higher frame-rates are no exception. techniques used in GREMM are the following: The costs for the Reduced visual quality. As previously mentioned, the visual quality is affected negatively in several ways. Less frequent updates may cause popping. The simplified shading does not look quite as good as true shading, and the transition between the impostor and the true geometry causes a slightly noticeable change in colour. The resolution of the impostor is rarely perfect for the screen, which causes some aliasing. The flat nature of the impostors causes incorrect depth values which may result in incorrect visibility (a solution to this was proposed by Schaufler [Sch97] but it would require an hardware implementation to be really efficient). All these causes for errors combined are not an insignificant impact on the visual quality. Less available graphics memory. The impostors are stored as uncompressed textures in the graphics memory. Even if the textures are reduced to a 16 bit colour depth, each 512x512 texture still uses 512 kilobytes of graphics memory. Fortunately, most of the impostors are distant and their textures smaller. 250 impostors may typically require about megabytes of memory. On a system with 64 MB:s of graphics memory, that is generally unacceptable. Overhead causing an increased CPU load. The task of finding the best available impostor for the visible objects of a certain type is an O(nm) task where n is the number of visible objects of that type and m is the size of the rendering cache. This should be performed once each frame so it can be a fairly heavy task. In some applications, this might be a problem if the CPU already is under heavy load of other important tasks (like the AI in entertainment applications). 61

62 GREMM: A REAL-TIME GRAPHICS ENGINE None of these three problems is likely to be solved with improvements to the engine, but some of the improvements discussed in section 8 are likely to reduce the problems. 62

63 Chapter 8 Future development of the engine The development of GREMM does not end with this thesis. It is intended for use in computer games and the development of a game using GREMM has already been started by Criosphinx Studios. The use of GREMM in game development requires optimization of the source code and simplification of its programming interface. There are also several new features planned, some which merely improve performance and others which add new effects, such as shadows. These features are alpha blended impostors, shadows, improved texture management, animation support, dynamic clustering, cluster-based impostors and more efficient cache lookup. 8.1 Alpha blended impostors Alpha blending is the most common technique to avoid popping in the transition between two images. Using alpha blending in the transition between the textured representations would reduce the popping, which is currently the greatest visual problem in GREMM. The drawback would be that each impostor would make use of several textures causing more texture memory to be used. Two different options are considered. The first possibility, and easiest, is that the impostor is blended during the transition based on weights that change with time. Figure 8.1 displays how the alpha weights could change with time. A transition starts as soon as a new view has been rendered and the old view may be discarded once its alpha weight is 0. Another possibility would be to always create the impostor by the blending of four sampled views as displayed in figure 8.2. The user is positioned at V and is looking at the object at O. The impostor of the object is created by blending the views rendered from the positions A, B, C and D which create a tetrahedron around the position V. When the viewer exists the tetrahedron, only one new view needs to be rendered (unless it already exists). Also, existing 63

64 GREMM: A REAL-TIME GRAPHICS ENGINE Figure 8.1: Time-based alpha blending for transition impostors may be refined by simply rendering a new view from the center of the existing tetrahedron, resulting in three new tetrahedrons. Figure 8.2: Alpha blending of sampled views The second alternative would require major modifications to the existing engine, but it is likely it would be far more effective than the current engine. 8.2 Shadows Even simple shadows greatly enhance the perceived realism. One interesting application for the rendering cache would be the reuse of the data in the cache for the creation of shadows. This could be by accomplished by the use of impostors in the shadow-map rendering, but other faster approaches based on 64

65 CHAPTER 8. FUTURE DEVELOPMENT OF THE ENGINE transforms of the data are perceivable. 8.3 Cluster-based impostors As the number of objects in the scene increases so does the heavy work needed to find the best impostors for these objects. This is the main reason for the use of cluster-based impostors. If distant objects are treated at a cluster level instead of at the individual level, then the number of objects requiring the expensive cache-lookup is reduced. Cluster-based impostors should be rendered using the impostors of the objects in the cluster, since rendering each object in the cluster using true rendering would likely be too time-consuming. Because of this, all objects in the cluster must have fairly good impostors which should affect the planning stage as objects should receive higher priority if they are part of a cluster needing an impostor. 8.4 Efficient cache lookup Currently, the rendering cache is implemented as a list. To find the best impostor, every item in the list is examined and that is simply the worst way to search the cache. Instead, the cache should be implemented as a tree structure branching (based on carefully selected intervals of the variables) first by the most important quality aspect and last by the least important as in figure 8.3. Figure 8.3: A rendering cache tree structure 65

66 GREMM: A REAL-TIME GRAPHICS ENGINE 8.5 Improved texture management Using OpenGL, the size of a texture must always be (a, b) where both a and b must be powers of two (64, 128, ). The problem is currently that a lot of texture area is wasted. There are two good solutions. One is that the image is stretched at rendering to fill the texture completely gaining higher resolution in one direction. Another, probably better, solution would be to add a texture management module that could make use of the unused parts of the texture as shown in figure 8.4. Such a module would allow allocation of arbitrary texture sizes at the cost of increased overhead in texture creation. Figure 8.4: More efficient texture allocation 8.6 Animation support GREMM must be extended to allow for impostors of animated objects. While some animated objects are clearly unsuitable for the techniques in the engine, a very suitable situation is the rendering of marching soldiers as seen from far away. Each model and pose is very likely to appear several times viewed from approximately the same direction. In some ways, the task is easier than rendering static objects as the popping is less noticeable. What is needed in GREMM to allow animated objects is a quality aspect function (section 5.3) for poses and probably a more efficient rendering cache. 66