Stroke-based Real-time. Non-Photorealistic Rendering

Transcription

1 Stroke-based Real-time Non-Photorealistic Rendering July 20th, 2011 Ben Ells

2 Introduction Research into non-photorealistic rendering (NPR) has often been very narrowly targeted at simulating particular artistic styles. More recently, generalized systems have emerged with much greater flexibility [1], even at interactive frame rates [2]. Commercial games are making increasing use of NPR, but only in very limited ways. Minor tweaks to shading models and texture style, with occasional delineation of silhouette edges is about as far as anyone goes. More extreme examples exist, but are few and far between. Much like early academic research in NPR, these use very specific, inflexible techniques. Implementing these techniques on a standard engine would require radical graphical modifications, if it were practical at all. Left: Borderlands [3] - a recent game employing typical cel-shading. Right: Okami [4] - a game well known for its stylized, sumi-e visuals. A stroke-based rendering system efficient enough to be used as part of a game engine but flexible enough to be easily adapted to different art-styles could massively boost graphical innovations in games. I present a system attempting to meet this pressing need by enabling fast, flexible, coherent stroke-based rendering of standard 3D models. Each frame, my system identifies interesting edgessilhouette edges, ridge edges and valley edgeswhich delineate the form of the model well. Note that by the term edge, I mean simply the geometric connection between two vertices. Contiguous interesting edges (edges sharing a vertex) are daisychained together into strokes in three dimensional space. Strokes are correlated across frames by their component vertices, and simplified to remove switchbacks (also called swallowtails ) and superfluous vertices, then split at hard angles into the straightest possible brush-paths. Strips of textured panels are laid down along these paths for rendering.

3 Related Work The problems of identifying silhouette edges and connecting them into clean strokes has been addressed before. Northrup and Markosian s approach [5] solves many of the same problems as my system. Edges are identified, simplified, and linked into paths, and geometry is generated and textured- broadly speaking, the same things are done. However, our specific approaches differ greatly. Unlike my system, they reject a brute-force edge-finding solution in favour of a randomized method exploiting temporal correspondences of silhouette edges between frames and spatial correspondences of silhouette edges within a mesh, presented by Markosian et al. [6]. This is very efficient, but cannot identify ridges and valleys. Linking and simplification is where our methods really diverge. Northrup and Markosian render an ID map, with each edge and face in a unique color. They analyze this image and construct strokes in two dimensions. This approach neatly bypasses the problems of simplification and visibility by only discovering visible edges, and linking based on pixel proximity in the final view. By contrast, my system links edges in world-space via shared vertices. Addressing problems of visibility and simplification at the vertices trades cheap but numerous pixel checks for a much smaller number of more expensive trigonometric tests. The WYSIWYG NPR system, produced by Kalnins et al. [2], expands on the earlier silhouette-stroke system by enabling non-silhouette strokes to be created. These can be created along existing edges or arbitrarily across the surface, without existing edges at all. These strokes must be defined by an artist, and cannot be generated on the fly. The system does not appear to support dynamic meshes. WYSIWYG uses vertices as control-points of a Catmull-Rom spline when generating stroke geometry, rather than simply extruding directly into strips of panels. The system can synthesize new random stroke textures from a set of examples. Furthermore, strokes can be parameterized to control things like thickness, color, or alpha. Jot [7], the expansion of WYSIWIG, addresses the issue of temporal coherence by placing sampling points every four pixels, and propagating them to the nearest corresponding stroke in the next frame. These techniques produce exceptional visual results. Ideally, many would eventually be incorporated into my system (performance allowing, of course).

4

5 Algorithms Pre-processing My system works with conventional 3D models, but they require some pre-processing. The result is stored in a.geo file, a simple extension of the.obj format. Geo files hold four main types of data- vertices, triangles, edges, and vertex-edges. As in.obj files, a vertex is a triplet of floats and a triangle is a triplet of vertex indices. An edge is two vertex indices and two triangle indices. A vertex-edge is a list of indices relating edges to that vertex. The added edge and vertex-edge data is needed to rapidly string connected edges together. Elevator Pitch At the highest level, my system iterates over the edges of the model and builds a list of the useful ones. Then it crawls the model from edge to contiguous edge, connecting them up into strokes. The strokes are extruded into strips of panels, textured, and rendered. Preliminaries Before anything can happen, the scene is rendered into a depth-map. This is a nontrivial fixed cost, and serves to cull hidden edges. Each vertex is transformed into screen-space and tested against the depth-map. Both results are stored for later. With the screen-space vertices, each triangle s normal in screen-space is calculated to determine whether the triangle is front-facing or back-facing. If the triangle is front-facing, a normal is calculated in the model s local coordinates as well, and stored for later. // Geometry::buildVertexScreenData for each vertex in model: calculate and store screen coordinates store visibility // Geometry::buildFaceNormalData for each triangle in model: calculate and store face direction if faces forwards: calculate and store normal Edge Assembly Now that we ve gathered what we need, we can build a list of useful edges. Edges are immediately discarded if they re between occluded vertices, or between two back-facing triangles. Edges between two front-facing triangles that are at a shallow angle to each other are discarded as well. Remaining edges, between a front-face and a back-face or between sharplyangled front-faces are considered useful. // Geometry::buildEdgeList

6 for each edge in model: if edge is hidden: continue if edge is between two front-faces: if angle between faces is shallow: continue add edge to usefuledges Each useful edge is examined one vertex at a time. The vertex in question is added to the current stroke. Then the vertex-edge data is consulted for a list of edges connecting to the vertex- if a useful edge is connected, the stroke-growing loop crawls into it and pushes its next vertex. This continues until there are no more interesting edges in that direction, then repeat with the other vertex of the initial edge. Useful edges are only examined once- a crawled edge is marked as used so it can be skipped when it comes up in the rotation. // Geometry::Strokify for each edge in usefuledges: if edge has been visited: continue mark edge as visited // crawl the model in one direction leadingedge = edge while leadingedge.vertexa has a usefuledge: pass vertexa in the front of the StrokeBuilder mark usefuledge as visited leadingedge = usefuledge // then crawl the other direction leadingedge = edge while leadingedge.vertexb has a usefuledge: pass vertexb in the back of the StrokeBuilder mark usefuledge as visited leadingedge = usefuledge for each stroke in StrokeBuilder: add stroke to strokelist reset StrokeBuilder The edge-assembly process produces a double-ended stream of vertices. These are

7 pushed into the StrokeBuilder, which oversees simplification and matching. Vertices can be pushed to either the front or the back- the process is the same either way. When a vertex is pushed, the model-space and screen-space versions are both sent. The model-space is there to root the actual geometry, while the screen-space is used to calculate the lengths and angles of the segments at various points. Note that the two ends of the StrokeBuilder are distinct from each other. Because strokes can be split, each end may well serve a different stroke. In the following discussion, references to previously pushed vertices mean specifically vertices that where previously pushed into the same end. Stroke Simplification Simplification is fairly straight-forward. Whenever a new vertex is pushed, the last four vertices of the stroke are examined. Several tests are applied to the three segments between them. If the newest segment is uselessly short or uselessly shallow, the new vertex simply overwrites the previous one. If both of the inside angles are sharp, it s a switchback, and the second-last and third-last vertices are averaged together to collapse the switch. If only the angle at the third-last vertex is sharp, the stroke is ended and a new one begun. // StrokeBuilder::simplifyFront / simplifyback // note: all operations deal with screen-space vertices length = distance between last and second-last vertices angle1 = angle at second-last vertex angle2 = angle at third-last vertex if length is too short OR angle1 is too shallow: remove second-last vertex from stroke else if angle1 and angle2 are both too sharp: average second-last and third-last vertices together else if angle2 is too sharp: split the stroke at the third-last vertex and carry on Stroke Matching Matching occurs after simplification. Every frame, StrokeBuilder remembers what Stroke each vertex was part of. Then, if that vertex appears again next frame, it can match the Stroke. The trick here is, if a new vertex appears that wasn t part of any Stroke last frame, my system needs to be able to indicate which Stroke it was part of in this frame, without revisiting it once a match is made. To accomplish this, Stroke IDs are reassigned every frame. The StrokeBuilder has two vectors for strokes- one for last frame, and one for this frame. When a match is made, it copies the stroke s data from last frame s vector, where it was stored under its old ID, across to this

8 frame s vector, where it s stored under its new ID. This way, every vertex knows which stroke it s part of, even if the stroke hasn t been matched yet. This saves the StrokeBuilder from having to backtrack to relabel previously unknown vertices when a match is finally made. If the stroke hasn t matched by the time it s extracted, a new stroke is created and stored in its place, ready to be matched next frame. When a new vertex is pushed, any of a number of things could happen, depending on whether the current stroke has been identified yet or not, and whether the last frame s stroke ID is a special value or an actual stroke identifier. There are three special values the stroke ID might take: unvisited, overloaded, and split-point. Unvisited is the null case. If there s no record of a vertex last frame, it s added to the end of the current stroke and marked as part of the current stroke for next frame. Overloaded is pretty simple as well. It signifies a vertex that was part of more than one stroke last frame. If a vertex was overloaded last frame, mark it overloaded this frame too. If a vertex is already marked as part of a stroke for the current frame, it s been visited before, and is overloaded. The existing stroke ID at the vertex is overwritten with overloaded to reflect this. Split-point is the fiddliest of the special values. If a vertex was a split-point last frame, it ll likely be a split-point again this frame. However, there cannot be two consecutive split-points in a stroke, because the segment between them would be effectively orphaned- unidentifiable because both vertices records have been overwritten with special values. Therefore, split-point vertices are only honoured on the next vertex push. If the next vertex is also a split-point, the previous split-point is overwritten with the current stroke ID for next frame. Otherwise, the stroke is split and things proceed as usual. In general, the last frame s stroke ID will be an actual stroke ID. If the current stroke hasn t been associated yet, the stroke data is copied across from last frame s list into this frame s list (at the newly established index). If the current stroke has already been matched and the new vertex corroborates the match, everything proceeds as usual. If the new vertex conflicts with the existing match, the stroke is split. // StrokeBuilder::matchStroke if previous vertex is a split-point: if new vertex is NOT a split-point: split the stroke at the previous vertex mark new vertex as part of the newly split-off stroke else if new vertex IS a split-point: overwrite prev vert s split-point entry with current stroke if new vertex is overloaded: overwrite existing record with overloaded return

9 if new vertex is unvisited: mark new vertex s entry with current stroke return if new vertex has a valid stroke ID from last frame: if current stroke hasn t been matched yet: copy strokedata from previous frame into new frame else if current stroke is already matched: if current stroke does NOT match new vertex s stroke ID: split stroke return mark new vertex s entry with current stroke Geometry Generation Stroke geometry generation is exactly as you d expect. A strip of panels is extruded perpendicular to both the stroke segment and the camera. The only trick involved is to scale the texture coordinates along the length to compensate for panels tilted away from the camera- the screen-space vertices are used to calculate the length of the panel on the screen and allocate a suitable length of texture.

10 Results Performance All testing was done on a 2 GHz Intel Core 2 Duo, with a GeForce 9500 GT and 2 GB RAM. All speeds are in milliseconds, and generally waver +/- a tenth of a millisecond in either direction. Render Process retrieve depth-map, 900 x 900 window Approx. Time 7.5 ms, fixed cost prep camera transformations 1 complete render of model vertex depth tests build triangle normals select useful edges assemble strokes 0-9 ms, fixed cost 3.6 ms each 0.4 ms 0.5 ms 0.5 ms 1.3 ms To render at 60 fps, each frame has to finish in 17 ms. Additional models have very predictable performance costs on any given run, though variance between runs is fairly high. The demo model is about 4000 triangles, 6000 edges, 2000 vertices. Depending on viewing angle, it ll produce between 130 and 220 strokes. models strokes ms to render frames / second The core functionality is very solid. Edge selection is accurate, reliable, and and efficient. Assembling contiguous edges into strokes is likewise very dependable. 1 This is exactly as nonsensical as it seems. The cost appears and disappears erratically. Could very plausibly be a bug in the profiling code, but the frame-rates are consistent with the reports.

11 Visual Results Using textures to lay down simulated brush strokes works quite well. Here we have three samples- a rendering in sharpie, in pencil, and in colorful stars.

12 This is a piece of lineart [8] that I tried to replicate. I thought the simple lines would work well, but lineweight was very important to the effectiveness of the drawing. Without parameterized strokes to give a measure of width control, the recreation was largely unsuccessful.

13 Oustanding Issues The stroke matching system is very flaky. Larger strokes are stable, but short, 2-4 vertex strokes can often be seen flickering. Using vertices to carry match data was convenient, but since each can only carry one match, data is lost from vertices that were used by more than one stroke. Longer strokes survive this, but shorter ones are very vulnerable. The system mostly stabilizes, but it fails often enough that at any given time there s usually at least one unstable stroke. Complex models strokes are also very prone to fragmentation as the camera pans around. Once a split-point is established, it ll survive until the vertex passes out of view- this accumulation causes strokes to get shorter with age. Hardy split-points are a boon to temporal stability- without them, strokes will fight for intersection vertices, growing and shrinking between frames. The shorter strokes that result can be visually unappealing, though. Transitions between strokes are sharp, and as they become more numerous the result is visually displeasing. Stroke stability is exacerbated by rasterization problems in the depth map. If a triangle is at a sharp enough angle to the screen, it can occlude its own vertices and cause false negatives. False negatives on vertex depth-tests cause dropped edges. I mitigate the problem by checking the four adjacent pixels as well. This is completely effective at the silhouette and on ridges, but fails completely in valleys.

14 Conclusions It s clear that the system is not fast enough to render large-scale scenes while leaving enough resources for a game engine. Roughly half the cost of the model render is in the stroke assembly; stroke matching and geometry generation in particular are very immature. I expect that their performance could easily be improved a great deal. The entire stroke assembly procedure calculates a great many lengths and angles that could likely be cached. It also seems likely that stroke geometry could be generated with a vertex shader. Even if the cost of stroke assembly were driven to zero, however, I have my doubts that the system as it stands would run well alongside other components of a game engine. In hindsight, it seems that while I cart both three-dimensional and two-dimensional vertices through the entire stroke assembly process, the three-dimensional vertices play very little role in the construction. It s possible that I could do away with the three-dimensional vertices entirely and construct everything in screen-space. Future Work The system as a whole is far from complete. Possibly the most important missing feature is stroke parameterization. Weighted vertices bound to stroke width or alpha will go a long way towards acheiving impressive visual results. Randomized texture synthesis from style samples as demonstrated by WYSIWYG [2], would be very nice, but is hardly required. Each of the textures here uses only a single sample, and there s no obvious repetition.

15 References 1. GRABLI S., TURQUIN E., DURAND F., SILLION F.: Programmable style for NPR line drawing. In Proc. EGSR (2004), pp KALNINS R. D., MARKOSIAN L., MEIER B. J., KOWALSKI M. A., LEE J. C., DAVIDSON P. L., WEBB M., HUGHES J. F., FINKELSTEIN A.: WYSIWYG NPR: drawing strokes directly on 3D models. In Proc. SIGGRAPH, ACM TOG (2002), pp Borderlands. Computer software. Vers K Games, 20 Oct Ōkami. Computer software. Capcom, 20 Apr NORTHRUP J. D., MARKOSIAN L.: Artistic silhouettes: a hybrid approach. In Proc. NPAR (2000), pp MARKOSIAN L., KOWALSKI M. A., TRYCHIN S.J., BOURDEV L.D., GOLDSTEIN D., HUGHES J.F.: Real-time nonphotorealistic rendering. In Proc. of SIGGRAPH 97, pages , August KALNINS R. D., "WYSIWYG NPR: Interactive Stylization for Stroke-Based Rendering of 3D Animation." Thesis. Princeton University, Delaney, Luka. Dyke-o-licious. Digital image. DeviantArt. 31 May Web. 16 July <