Image-Space Painterly Rendering

Transcription

1 Image-Space Painterly Rendering Janne Hellstén Helsinki University of Technology Abstract In this paper two painterly rendering techniques are described. Both techniques automatically transform images into painted-like images using impressionistic image filters. Litwinowicz concentrates in his work on video sequences and achieves frame-to-frame coherence by the use of optical flow in a novel way. Hertzmann presents an excellent image filter that creates images with a handpainted appearance using a multi-resolution brush stroke rendering method. 1 INTRODUCTION For years, computer graphics research has focused on creating images and animations that are photorealistic. However, photorealistic image synthesis is not in all applications required or even wanted. In some cases, an image that accentuates a certain look or feel can be more desirable than a photorealistic image. An image that depicts a subject too accurately can also be misleading and constraining, a good example of fighting this problem is Teddy Painter (Igarashi et al., 1999) Teddy Painter hides the technical side of modeling by using illustrated rendering (Markosian et al., 1997; Salisbury et al., 1997), achieving great results. Apart from pen-and-ink illustrated rendering there exist painterly rendering techniques, which aim for creating images having qualities unique to the art of painting. These techniques differ from pen-and-ink rendering in that shapes are distinguished by variations of color rather than by outline or contour. Most of the painterly rendering styles are inspired by the late 1800 s impressionistic artists, for example Claude Monet and Vincent van Gogh. These painters strove to capture their subjective impressions of the world on canvas instead of photographing precise images. 1.1 Related work In Haeberli s early work, he presents a painting program for creating impressionistic images (Haeberli, 1990). Although the presented technique produces striking results, the process of image creation requires vast amounts of user interaction. 1

2 In 1996, Meier proposed a brush rendering technique (Meier, 1996). The paper addresses the difficult problem of frame-to-frame coherence when displaying painterly rendered animation. However, the used technique is based on 3D-information and cannot as such be applied to 2D-images or 2D-image sequences. Brush stroke techniques are also used in Treavett s work. Although their brush model is rather primitive, they deserve credit for their statistical treatment of stroke placement (Treavett & M. Chen, 1997). Painterly rendering and non-photorealistic rendering techniques have been in use in commercial imaging software. Most of the techniques used, however, are usually very simple but such simple algorithms can be very effective when used with care. Adobe Systems has used many standard image processing filters to with great success in their Photoshop (Adobe Systems, 1996). 1.2 Overview To simulate a human painter, one must somehow simulate the process of painting a picture by hand. The one most important thing in the process of painting a painting is how an artist transforms his subjective picture or an impression of the world into the canvas. This transform usually involves the use of a brush and colors as the main tool. This process is strongly analogous to the way painterly rendering algorithms work the algorithm sees a parametrization of the world (usually a digital image) and must somehow transform the parametrization into a new parametrization having desired attributes. So the most obvious place to start is to simulate the use of colors and a brush digitally. The process of drawing an image using brush strokes is usually divided into separate tasks: the selection of brush strokes color, texture and shape; the rendering of the prepared strokes; and finally, in the case of animation, animating the brush strokes and creating new ones as needed. Some authors have even gone so far as to simulate the actual physics of a brush stroke (Strassman, 1986; Curtis et al., 1997), but the direction most researches have chosen is to approximate a stroke by some ad hoc method that produces satisfactory results. This is the way both Litwinowicz and Hertzmann have chosen. These two authors both present techniques for automatical brush stroke placement and stroke rendering. Litwinowicz also presents a novel way to handle brush stroke animation when rendering painterly animation out of ordinary videos. Neither of the techniques require any user interaction during the rendering process. In the next two sections we proceed to more detailed descriptions of the two techniques reviewed. And finally in section 4 we wrap up the subject and discuss possible future research directions. 2

3 Figure 1: Edge clipping. Note how the brush stroke gets clipped at the edges of the shape. 2 LITWINOWICZ In this section we describe the algorithm presented by Litwinowicz in more detail (Litwinowicz, 1997). The primary focus of the aforementioned paper is how to deal with temporal continuity when rendering painterly animation. 2.1 Rendering of brush strokes We start by defining a brush stroke. A brush stroke is a line having a radius of R, location in the image of (C x,c y ) and a direction of A. Suppose we now had a thousand of these brush stroke definitions, each stroke having additional one more attribute: stroke color (color). We could proceed straight to rendering the final image with a simple algorithm outlined below. for each stroke drawedgeclippedline((c x,c y ),A of width R with color color) Now, if all the parameters were set properly, we would have a nice painterly rendered image. What follows, is a description of an algorithm that given an input image creates all the necessary parameters for rendering the output image with the aforementioned algorithm. We assign a brush stroke for each two pixels in both X and Y directions in the source image, although this density can be a user-supplied parameter. The distribution of strokes must be dense enough to assure that strokes, when rendered, fully cover the output image. In order to render the actual strokes into the final image, a custom line drawing routine is needed in order to preserve edges in the source image. The line drawing routine has to first determine the endpoints of the stroke and then render the stroke. The endpoints are determined by an algorithm that starts from the center of the brush 3

4 stroke((c x,c y )) and traverses the source image in the direction of the stroke until an edge is found. 1 or a maximum line length is exceeded. See figure 1 for a depiction of the line-edge clipping process. To have the brush strokes follow the contours and shapes of the input image, a method for aligning each stroke to closely match the input image s local detail is needed. This alignment is carried out by finding a direction (angle A) for each stroke based on the local image gradient. Having the strokes oriented in the direction normal to the image gradient makes the strokes align to directions where there are relatively small changes in image intensity. The image gradient values are calculated from the Gaussian filtered source image. The blurring of the source image is performed to accommodate for noise in the source images, especially in the case of video sequences. This works well for some cases, but when the source image has large areas of constant or near constant color, the magnitude of image gradient approaches zero and causes distracting visual artifacts. This can be accommodated for by using a more sophisticated image gradient calculation method that uses neighbouring good image gradient values in areas of constant color. 2.2 Using optical flow to achieve frame-to-frame coherence When rendering painterly rendered animations, care must be taken to maintain frameto-frame coherence. In most non-photorealistic rendering systems, a subtle change in the input image can cause dramatic changes in the output image. This fact complicates the process of creating painterly animations out of video sequences; two sequential frames can differ too much when run through a painterly filter. This causes distracting artifacts when rendering painterly animations the animation does not seem smooth and continuous anymore when frames are changing incoherently. In Litwinowicz s work things are not any bit different. If the brush stroke rendering part of the paper would be applied for animation, the resulting animation would no doubt be scintillating. However, Litwinowicz suggests the use of optical flow to aid the animation process. Two dimensional optical flow vectors are typically used in machine vision systems (Sonka et al., 1993). These vectors describe the motion of each of portion of the image in the image plane. Optical flow can be directly applied to brush strokes. Instead of rendering each frame from scratch, calculated brush strokes are stored in memory and moved according to the optical flow. For each frame, some new brush strokes are introduced and some old ones are removed. The painterly animation rendering process begins with the calculation of the optical flow from the video sequence. The first frame of the output sequence is painted normally, as described in the previous section. As we proceed to painting the next frame we use the brush strokes calculated in the previous frame and move them in the direction of the optical flow. This will cause the strokes to sort of stick to the objects 1 A Sobel filtered Gaussian blurred version of the source image is used to detect the edges. 4

5 and shapes in the animation. However, as the the painting process continues, eventually some of the strokes will drift out of the image and some areas the image will be over-populated with strokes. To have the density of strokes remain constant over the image plane, new brush strokes need to be introduced and some old strokes need to be removed. An algorithm is needed for determining which areas are over-populated and which areas are underpopulated. In order to measure the brush stroke population over the image plane, a Delaunay triangulation is built based on the brush stroke center locations. Delaunay triangulation will cover the whole input image with triangles. The triangulation is then subdivided until a maximum area constraint is satisfied, that is, the mesh does not contain any triangles having a larger area than a predetermined maximum area. The new vertices resulting from this subdivision process are then used as new brush stroke center points. The removal of brush strokes from overly dense areas is also done through the use of the Delaunay triangulation. The edge list of the previously calculated triangulation is traversed and all brush strokes not satisfying a supplied minimum edge length constraint are discarded. The test is done by finding the length of each neighboring edge and testing that length against the predetermined minimum edge length. When the current brush stroke set is updated, the rendering of brush strokes will follow as described in the previous section. New colors and new orientations will be calculated for the strokes and the strokes will again be rendered in random order, with the exception of new strokes which are rendered after the old strokes from the previous frames to prevent popping artifacts. 3 HERTZMANN In this section a painterly rendering technique by Hertzmann is described (Hertzmann, 1998). Instead of using brush strokes of same size and shape like Litwinowicz did, Hertzmann uses brush strokes that are modeled using spline curves of varying thickness. The rendering is divided into multiple passes instead of rendering the whole picture in one go. This technique nicely captures small and large detail in the source image and is similar to some painting techniques where a painter first paints the image composition in coarse detail and then adding more brush strokes to areas needing more detail. In a sense, this progressive rendering technique is similar to Laplacian pyramid image-encoding (Burt & Adelson, 1983) Progressive rendering The image is rendered in multiple passes each pass having a different brush size. The first pass is rendered with the largest brush size and for each pass, the brush size is 2 To reconstruct an image out of a Laplacian pyramid, one first starts with the lowest, coarsest, level of the pyramid and then incrementally adds more high-frequency data. 5

6 progressively made smaller to capture more high-frequency detail. First the output image, canvas, is initialized to a constant color. For each pass, a reference image is constructed out of the source image by filtering with a Gaussian kernel of width w i. 3 Filtering is done to throw out detail that couldn t be captured with the current brush size. The current reference image is given as input to a sub-routine (named paintlayer, see pseudocode below) that will paint brush strokes of width R i on the canvas. This routine first determines the areas in the picture where more brushes need to be drawn. This is done by comparing the difference between the canvas and the input image. If a certain area s difference exceeds a predefined threshold T, a brush stroke will be added into that area. The brush stroke is placed in the location of largest error between the canvas and the reference image. This is to avoid missing small details in the input image and also to prevent visual regularities from appearing in the painted image. paintlayer(canvas, ref, src, R i ) { for x = 0 to imagewidth, x+=grid for y = 0 to imageheight, y+=grid { E,(x b,y b ) = meanerror(canvas,ref,x,y,grid) if E > T then { addbrushstroke(r i,(x b,y b ),ref,src) } } render brush strokes } In the above pseudocode, the whole image is traversed and brush strokes are added as needed. The function meanerror() returns the mean-square error between two source image in a given region. It also returns a location in the input region having the largest deviation from the canvas to the source image. The function addbrush- Stroke() is called for regions in the canvas that differ too much from the reference image. This function will do the actual rendering of the stroke, as described in the next section. 3.2 Brush stroke spline generation We will now continue to the details of rendering the actual brush strokes. Each brush stroke is a spline consisting of N control points. In order to render a stroke, an algorithm for determining the locations of the control points is needed. 3 The filter width w i is set equal or relative to the brush width R i. 6

7 Figure 2: Brush stroke generation. Control points for the brush stroke s spline are generated by walking along a path in the Sobel filtered reference image in the direction normal to the image gradient. On the left is the reference image and on the right is a filtered reference image showing the magnitude of the image gradient for each pixel. The algorithm first starts at a point (x 0,y 0 ) and picks a color for the stroke at that location from the source image. This point is the first control point in the spline. At (x 0,y 0 ), the Sobel filtered luminance of the reference image 4 is sampled and used to calculate the next control point (x 1,y 1 ) by moving R i pixels in the sampled direction. At each new location (x j,y j ) the same procedure is executed. This process is continued until either a maximum predetermined length of a stroke is reached or the difference between stroke color deviates too much from the current color at (x j,y j ). See figure 2 for a depiction of a brush stroke generation note how the stroke follows an edge in the reference image. When the control points are calculated, the brush stroke is rendered by rendering filled circles of radius R i along the spline. All the strokes are drawn in random order to minimize regularities in the output image. The random order can be achieved without storing the brush stroke splines by making the paintlayer-routine of the previous section to traverse the image in random order or by using a random Z-buffer as suggested in Hertzmann s paper. 3.3 Style parameters For an artist to use a painterly algorithm, the algorithm needs to be able to create output images of different styles there is no one style that would be the right choice for everything. Therefore the painterly algorithm must be ready to produce different styles based on user defined parameters. Consider a real painter. What are the charasteristics of his brush usage when he 4 See section 2.1 for discussion on using image gradients to guide brush strokes 7

8 Figure 3: Different styles. The image on the left uses smaller brush strokes while the image on the right uses bigger strokes making it more coarse. wants to paint in a certain style? The selection of a brush is one the painter might choose a larger brush for coarser regions in the painting. The painter must also choose the right medium whether he chooses oil or watercolor depends on his subjective preferences and the work at hand. Much of the painter s style is not, however, related to the choice of physical equipment but comes from the painter s mind. The artist might choose to paint in very short strokes or in some cases might want to use very long, exaggarated strokes. It is also up to the artist to decide how precise the painting should be the painter might want to accentuate some objects in the scene and perhaps make some objects less apparent. Hertzmann s algorithm is able to vary the output style by introducing style parameters using these parameters the artist can alter the visual style of the algorithm s output. Some of these parameters include approximation threshold (T) which controls how closely the output image resembles the original source image; brush size controls the choice of brush sizes in the rendering using very large brush sizes causes the output image to be approximated more coarsely; curvature filter can be used to constrain or exaggarate the brush strokes shape; minimum and maximum stroke lengths set a maximum and minimum length for a stroke e.g. having a maximum strength of 0 pixels for the strokes creates a style that generates pointilistic images. Figure 3 illustrates the effect of changing the brush size. The picture on the left uses a smaller brush stroke size which makes it more accurate. The picture on the right uses bigger brush strokes and is clearly more inaccurate. 8

9 4 DISCUSSION AND FUTURE WORK Although both of the techniques reviewed in this paper are rather good as such, there is still room for improvement. The brush stroke rendering of Litwinowicz s paper is rather primitive. As all the brush strokes are rendered using the same thickness for all the brushes, the output lacks a certain freedom of expression. There are no coarse, exaggerated strokes drawn, rather the output image s brush distribution is too uniform across the image. The fact that all the brush strokes are drawn using straight lines further adds to this problem. Litwinowicz s algorithm is also rather limited when it comes to varying the rendering style in practice, only the brush thickness can be controlled. Hertzmann has a lot more generic brush stroke model when compared to Litwinowicz s work styles can be varied, the brush strokes are long and curvy and rendered in different sizes. However, Hertzmann s technique does not properly handle animation. If applied to an animation sequence, the technique would suffer from bad scintillation artifacts due to the lack of frame-to-frame coherence treatment. The algorithm produces mediocre quality animation if the frame rate is lowered to approximately Hz but at higher frame rates the output starts to flicker as the viewer expects a certain continuity in the animation. At lower frame rates the discontinuities become less apparant to the viewer. Hertzmann s later work addresses this problem (Hertzmann & Perlin, 2000). In this paper he extends his previous painting technique by using optical flow to warp the brush strokes across multiple frames in an image sequence. The optical flow is used in a manner similar to Litwinowicz s work. Both of the techniques presented in this paper take only global user parameters, that is, parameters that affect the full output image. An artists might need more detailed, local control over the painterly style for example, parameters that can be used to give more detail to specific areas of the image to emphasize a certain character or an object. This kind of extensions could be easily integrated into Hertzmann s technique. E.g. having a mask image that would for each pixel tell the amount of detail desired. The mask image s value could then be used to control the error parameter T across the picture. Hertzmann has already taken a step towards this in his newer work where he proposes a relaxation procedure that given an energy function, tries to find a set of brush strokes that minimize the energy in the painting (Hertzmann, 2000). It would also be interesting to see painterly rendering in real-time applications. Algorithms have been presented on the pen-and-ink side that are fast enough to be used in real-time applications (Markosian et al., 1997). Meier s painterly rendering algorithm can also run at animation frame rates. However, applying either Litwinowicz s or Hertzmann s algorithms to real-time applications seems impractical for animation, both techniques require the use of optical flow which is computationally very expensive to calculate. The performance of the actual brush stroke computation can be further optimized, e.g. through the use of summed area tables in the filtering process. 9

10 REFERENCES Adobe Systems Adobe Photoshop 4.0. Burt Peter J., & Adelson Edward H The Laplacian Pyramid as a Compact Image Code. IEEE Transactions on Communications, Curtis Cassidy, Anderson Sean, Seims Joshua, Fleischer Kurt, & Salesin David H Computer-Generated Watercolor. SIGGRAPH 97 Conference Proceedings, Haeberli Paul Paint By Numbers: Abstract Image Representations. SIGGRAPH Annual Conference Proceedings, Hertzmann Aaron Painterly Rendering with Curved Brush Strokes of Multiple Sizses. SIGGRAPH 98 Conference Proceedings, Hertzmann Aaron Paint by Relaxation. Hertzmann Aaron, & Perlin Ken Painterly Rendering for Video and Interaction. First Internationl Symposium on Non-Photorealistic Animation and Rendering. Igarashi Takeo, Matsuoka Satoshi, & Tanaka Hidehiko Teddy: A Sketching Interface for 3D Freeform Design. SIGGRAPH 99 Conference Proceedings, Kaplan Craig S., & Salesin David H Escherization. SIGGRAPH 2000 Conference Proceedings, Litwinowicz Peter Processing Images and Video for An Impressionist Effect. SIGGRAPH 97 Conference Proceedings, Markosian Lee, Kowalski Michael A., Trychin Samuel J., Bourdev Lubomir D., Goldstein Daniel, & Hughes John F Real-time nonphotorealistic rendering. SIGGRAPH 97 Conference Proceedings, Meier Barbara Painterly Rendering for Animation. SIGGRAPH 96 Conference Proceedings, Salisbury Michael, Wong Michael, Hughes John, & Salesin David Orientable Textures for Image-Based Pen-and-Ink Illustration. SIGGRAPH 97 Conference Proceedings, Sonka Milan, Hlavac Vaclav, & Boyle Roger Image Processing, Analysis and Machine Vision Strassman S Hairy brushes. SIGGRAPH 86,

11 Treavett F., & M. Chen Statistical Techniques for the Automated Synthesis of Non-Photorealistic Images. Proc. 15th Eurographics UK Conference. 11