Autonomous Flock Brush for Non-Photorealistic Rendering

Transcription

1 Autonomous Flock Brush for Non-Photorealistic Rendering Hsueh En Huang, Yew-Soon Ong, Xianshun Chen Centre for Computational Intelligence School of Computer Engineering, Nanyang Technological University Singapore { hehuang1; asysong; chen0469}@ntu.edu.sg Abstract Non-photorealistic rendering systems strive to create compelling stylized effects from realistic images. We present an interactive process using flocks of autonomous agents to model a painter s brush. As flocks of agents glide across the canvas like bristles on a paint brush, a stylized picture can be produced by carefully directing the path of movement. The agents leave behind a trail of color resulting in painterly or pencil sketch looking images. I. INTRODUCTION Non-photorealistic rendering (NPR) is a well studied problem with different solutions [1]. It can be characterized as a transformation process. The chief aim is, given a digital photograph as input; output an artistic rendering of the photograph. With a creative picture as the goal, NPR has been the subject of researchers in the fields of computer graphics and evolutionary computing. Solutions have been proposed in both of these fields. These solutions include digital filters, strokebased approaches and the use of optimization techniques. However, to the best of our knowledge these systems have not explored using flocks of agents as the driving force for the NPR transformation process. In this paper, we propose a system that uses Swarm intelligence in the form of flocks of agents. The agents move in conjunction with steering forces to transform digital photographs into a creative image. At the heart of the system is a 2d brush modeled using a group of autonomous agents. A brush can be thought of as a collection of bristles. When these bristles move in a cohesive manner over a canvas, a stroke is formed. The spatial distribution of bristles form a defined brush pattern and the path marked by the individual bristles combine into a directed trail of color. The use of autonomous agents to represent the bristles of the paint brush presents an opportunity for creative and non-uniform interactions typical in artistic processes. The system serves to encourage its users to interact with art creation using a non-traditional mechanism; to think of the brush as a dynamic collection of agents. This unusual perspective may rouse new creative thoughts in the user. Some of the issues we hope to address include, exploring the application of distributed agents for non-photorealistic rendering. Modeling digital paint brushes as an interactive flock of agents. Lastly, examine the effects of different flock parameter settings on the resultant image. The remainder of the paper is structured as follows: In section 2 we begin with a brief introduction to related work on non-photorealistic rendering, steering behavior and flocking. An overview of how a union between these two ideas can be achieved is briefly discussed. Next, we explain the design of our flock brush system in Section 3. Then in Section 4 we discuss how the flock brush system can produce images of different styles. Finally, in Section 5 we draw some conclusions and explore possibilities for future research. II. RELATED WORK In this section, we describe some relevant works in the field. Traditionally, the field of Non-Photorealistic Rendering (NPR) is concerned with the harmonious union of digital images and art styles. This desire to create an algorithmic model for producing art styles computationally can now be observed in some works. Stroke driven algorithms have been shown to re-produce familiar art styles such as painterly and pencil sketch. Curved strokes, represented by spline curves, were later introduced to enhance the stylized output so it would resemble more closely to that of hand-painted pictures. Much of NPR research has explored techniques for artistic rendering of photo-realistic images. Analogous to paper based painting and drawing techniques, NPR techniques are varied and yield a variety of results. There is no "right way" of rendering a non-photorealistic image [2, 3]. Pioneering NPR systems used a stroke based approach to create a stylized version of an image. The output can be characterized as an ordered list of strokes. Each stroke is characterized by key parameters such as length, orientation, color and order of placement. Later, techniques were introduced to enhance the artistic qualities of the output image. Litwinowicz and Salisbury [4, 5] introduced stroke clipping for strokes that intersected with image edges. Rendered strokes extended directionally from a seed point and were clipped when edge pixels are encountered. Long curved strokes in the form of B-spline curved produced images which more likely resembled canvas paintings [3]. Image filter based approaches [6] have also been adopted to tackle the issue of image stylization. The attempts have yield significant results and filter based approaches are able to recreate similar styles as their stroked-based counterparts.

2 NPR is also a budding research area in the field of evolutionary computing. Recently, there has been an interest in applying evolutionary techniques to generate novel styles. This can be seen in the works of Collomosse [7], Izadi [8] and Semet [9]. The central idea of evolving images is common to the work presented in these three papers. Naturally, each work had a different focus. Collomosse focused on using genetic optimization to find the best painting given an input image. Semet used distributed agents or ants to create artistic styles based on a set of tunable parameters such as stroke length and angle. Izadi on the other hand, focused on evolving images by approximating input images using different sized triangles. This approximation process was then captured as a time-lapsed animation. It is in this similar context, evolving images over time using autonomous agents, that we present our work on flock-based art. Flock simulation was first pioneered by Reynolds [10, 11]. His model and its deviants have been used in various applications, from artificial life forms [12] to multi-agent systems [13, 14]. Flocking is commonly used as a tool for simulating the forces of interaction between natural flocks such as schools of fish or a flock of birds. Interactions between agents in the same flock are governed by a set of forces. The forces of cohesion, separation and alignment dictate the spatial interactions between agents in a flock. Flock - based art explores the idea of modeling the digital brush as a flock. Each agent in the flock represents a bristle on the brush. Common brush stroke parameters including shape, length and stroke angle are specified by the flock control parameters. These parameters can be defined in the system to produce different styles. III. INTERACTIVE FLOCK BRUSH SYSTEM The flock brush system performs a series of defined steps to generate stylized outputs from a digital image source. The user interacts with the system via a graphical user interface, see Fig. 1. In the preprocessing step, two guiding maps are generated. These include a luminosity map and a gradient map, see Fig. 2. Once pre-processing is complete, the user proceeds to specify the flock configuration parameters. This is followed by the flock painting or sketching process. Finally, the system outputs the result of the digital art process. A. Preprocessing Guiding the flock. 1. Generate luminance map Our system takes in a colored image as the input. After extracting the RGB values, the luminance values of each pixel are given by this weighted average formula suggested in [15]: Luminance = 0.299r g b. (1) The green component has the largest weight due to the nature of how humans perceive color. Cone cells in the human retina, responsible for color vision, are most sensitive to the color green [16]. FIGURE 1. FLOCK PAINT SYSTEM

3 2. Generate gradient map Using the luminance values from the previous step as a basis, we create the gradient map. This map is created by applying a straight-forward image processing filter, the Sobel filter [17], on every pixel of the input image. As we loop through each pixel, the neighborhood size remains constant. A 3 x 3 neighborhood is employed for its simplicity and low computational cost. FIGURE 2. A) LUMINANCE MAP B) GRADIENT MAP B. Controlling the Flock Art is a creative process. Hence we do not wish to impress a particular style on the user but allow different treatments on each input image. This gives rise to the possibility of collaboration between users through the system; an opportunity for creative expression on the part of the user. This collaboration is done through the use of flocking parameters. Table 1 shows a listing of the user configurable parameters provided in the flock brush system. These parameters are largely flocking specific. Some typical values used are provided in Table 1 to facilitate a visualization of their specific impacts on the resultant image. A flock consists of a constant number of agents. The flock size parameter controls the number of agents in the flock. Each agent also has an energy level to define how far the agent can move across the canvas. This energy level is a global parameter applied to all agents in the flock. Flock direction controls the general heading of the flock. Alignment, cohesion, seeking and separation are classic steering forces that dictate inter-agent behavior as multiple agents move along a directed path. All the parameters can be modified through the graphical user interface of the system. The four forces used in our application are represented as vectors in 2D space; see Fig. 3 for a pictorial representation. A summary of how each force functions is discussed in what follows. The alignment force allows agents to align its heading with other agents in the flock. Alignment is achieved by averaging the heading of nearby agents. This makes agents in the flock move in a similar direction. TABLE 1. USER SPECIFIED FLOCK PAINT CONTROL PARAMETERS. Flock Paint Parameters Type Range Flock Size Numeric [1,2 15] Agent Energy Pixels [1, ] Flock Direction Degrees [1,360] Alignment Force Numeric [0 1] Cohesion Force Numeric [0 1] Separation Force Numeric [0 1] Seek Force Numeric [0 1] Cohesion Range Pixels [0,1,2 100] Separation Distance Pixels [0,1,2 15] Gradient G 0 Numeric 80 Step Size Numeric [5,6 100] Color (from input image) - {RGB, grey scale} Movement Style - Fill/Edge drawing. The cohesion force ensures that agents within the flock remain relatively near each other. A cohesion range is defined for every agent. When cohesion force is applied to an agent, all agents are pulled towards a central position. This central position is calculated by averaging the position of all agents within the cohesion range. This force keeps the flock together. Cohesion draws agents near each other. The separation force however, prevents crowding and collision by enforcing a minimum distance between nearby agents by pushing away agents that come too close to one another. These two forces, cohesion and separation counterbalance each other as the flock moves so agents are neither crowding nor drifting away too much from the flock. In essence, a stroke is a directed line. It has a beginning and an end point. Therefore a force is required to steer the flock towards a particular point or position on the canvas. For this reason, a seek force is considered. The seek force represents a vector from an agent s current position that points towards a specific target or position on the canvas. The four forces, alignment (F Ali ), cohesion (F Coh ), separation (F Sep ) and seek (F Seek ) are summed and the total force is applied to each agent. It produces an acceleration which is then used to derive the agent position [11]. The weights used in (2) sum to one. Force Total = w 1 F Ali. + w 2 F Coh. + w 3 F Sep. + w 4 F Seek. (2) FIGURE 3. A ) SEPARATION B) COHESION C) ALIGNMENT D) SEEK

4 C. Using the flock brush to create stylistic effects How does the flock brush create stylistic effects? The process is akin to painting or sketching by hand, one stroke at a time, with each stroke rendered by a single flock of agents. As one flock terminates, another flock is initialized and put to task; this continues until the predefined termination conditions are met. As the flock moves around the canvas, its energy level decreases. The energy level is expressed as distance moved by the flock and is measured in pixels. The flock terminates when it is depleted of its energy. The input image is a digital photograph; the area in which the flock moves is the canvas which represents the generated artwork. The size of the canvas is generally the same size as the input image, if not larger. Input: C, canvas Output: Begin End Algorithm 1 - Flock Draw img, a reference image S, stylized image /* Pre-Processing */ Map Luminance Luminance( img ) Map Sobel Sobel ( img ) /* Flock Initilization */ cfg:= configuration object F CreateFlock( cfg ) /*F starts on a pixel with Gradient < G 0 */ While!Termination Conditions do Draw( C,F cfg, Map Sobel, img ) Navigate( C,F,cfg ) End while FIGURE 4. FLOCK PAINTING ALGORITHM First, the user configures the flock parameters given in Table 1. The first flock is then set in motion. The system locates a start pixel with gradient value below threshold, G 0. The flock brush is characterized by these parameters, flock size, steering force weights, separation distance and cohesion range. To begin, the flock brush is initiated and centered at the start pixel, i.e., refer to Fig. 4. The next step is to determine the direction of travel for the flock brush, refer to Fig. 5.The direction can be defined by the user manually or determined by the image gradient at the starting pixel. The flock brush then proceeds to move in the chosen direction of interest. Brush strokes are thus imprinted incrementally as the flock moves about the canvas, see Fig. 6. The resultant form and shape of the stroke is determined by a combination of user specified parameters and flock forces. Input: Begin End Input: Begin End C, canvas F, a flock Algorithm 2 - Navigate P Position( F ) /* Flock's current position */ angle Orientation( P, C, Map Sobel ) step StepSize( cfg ) move( angle, step ) update( F, P, step ) /* Update flock's state */ FIGURE 5. FLOCK NAVIGATE SUB - ROUTINE C, canvas F, a flock img, a reference image Algorithm 3 - Draw Map Sobel, a gradient map P Position( F ) /* Flock's current position */ angle Orientation( P, C, Map Sobel ) color Color( P, C, img ) length StrokeLength( cfg ) RenderStroke( P, angle, color, length ) FIGURE 6. FLOCK PAINT SUB ROUTINE The color of the stroke is sampled directly from the source image. So the resultant stroke share the same color as sampled at the flock s position. The termination condition for the flock brush is met when the energy of the flock has been depleted or the edge trail vanishes. After one flock terminates another will be created for the next stroke. The above process iterates and complete until the pre-defined number of strokes to create is fulfilled.

5 The shape of the stroke is defined by the steering forces. As the agents move in a flock, the steering forces may cause agents to overlay with another agent s path resulting in strokes that are non straight lines. Since artists in general do not draw or paint with rulers but instead uses a free-hand approach, this feature of the flock brush aids in creating a free-hand look. Real-life painters are also unable to paint continuously without refilling paint, so to simulate this nuance the color intensity of a stroke is set inversely proportional to the length of the stroke. Our method draws inspiration from previous work on stroke painting and image gradient following [2, 3, 9]. Our primary goal is about exploring the effects of synthesizing these stroke-based, gradient guided strategies with the use of a flock of autonomous agents as the style rendering mechanism. IV. RESULTS AND DISCUSSION We have conducted our experiments on an Intel Xeon 2.27GHz machine with 3GBytes memory running on a Windows XP operating system with standard Java application programming interface. The system is able to produce styles similar to those commonly seen in NPR systems. Namely, painterly and sketch style. See Figs. 11, 12 and Figs. 13, 14. In our experiments, the weights for (2) are set as follows: w 1 = 0.10, w 2 = 0.20, w 3 = 0.30 and w 4 = The above weights give emphasis to separation and cohesion as these two forces have a larger effect on the stroke path than the alignment force. The seek force is necessary to prevent the agents from wandering in random directions; therefore its weight is also configured to be larger. A. Stroke Styles Strokes produced by a flock are unlike those produced by a single agent. Single agents tend to produce uniform strokes with little variety, see Fig. 7. However a flock, when given the right number of parameters, could potentially yield countless stroke styles. By varying parameter values such as flock size, steering forces, agent energy level, stroke length, color and angle, different stroke styles can be achieved. Figs. 8, 9 and 10 showcase some of the more distinct results obtained. The corresponding parameter settings are listed in table 2 for reference. We have categorized the input parameter values as high (H), low (L), average (A); this is indicated by the subscript next to the value shown. This not only simplifies the range of observable values, it also makes it easier to identify consistent trends when applying the forces. The range for each parameter value can be found in Table 1. A Single Agent Flock of Agent FIGURE 7. LEFT ) UNIFROM STROKES CREATE BY A SINGLE AGENT RIGHT) NON-UNIFROM STROKES CREATED BY A FLOCK From the results obtained we can observe the following trends in the formed stroke: 1. A high cohesion range and a low separation distance create thin lines. This is because agents are tightly grouped together. See Fig. 8 strokes A high large flock size or high agent count significantly increases the thickness of the stroke. The reason for this is straight forward, more agents naturally leads to a flock that occupies a larger surface area on the canvas. See Fig. 8 stroke 6 and Fig 9 stroke A high separation distance ensures agents are spread out and remain separated. As agent move around the canvas the trails left behind tend to fan out and create a distinct shape that resembles a broom or mop. See Fig. 9 strokes Fig. 10 strokes are strokes made by the flock when agents are at a stationary position. Various flock sizes are represented in the figure. These are used simulate strokes resulting from a dabbing motion. FIGURE 8. STROKES USING A SMALL FLOCK SIZE

6 TABLE 2. STROKE STYLE PARAMETER SETTINGS FOR LOW AGENT COUNT H HIGH L LOW A AVERAGE. Agent Count = 2 Stroke No. Cohesion Range Separation Distance 1 25 L 12 H 2 55 A 12 H 3 80 H 12 H 4 25 L 2 L 5 55 A 2 L 6 80 H 2 L FIGURE 10. THICK LONG STROKES & POINT STROKES TABLE 4. STROKE STYLE PARAMETER SETTINGS FOR LONG STROKES AND POINT STROKES H HIGH L LOW A AVERAGE. Stroke No. Cohesion Flock Size Separation Range Distance A 2 L 8 H A 6 A 8 H FIGURE 9. STROKES USING A LARGE FLOCK SIZE TABLE 3. STROKE STYLE PARAMETER SETTINGS FOR HIGH AGENT COUNT H HIGH L LOW A AVERAGE. Agent Count = 10 Stroke No. Cohesion Range Separation Distance 7 25 L 12 H 8 55 A 12 H 9 80 H 12 H L 2 L A 2 L H 2 L A 12 H 8 H H 10 H 4 L H 10 H 4 L H 10 H 4 L To accurately observe how flock sizes, cohesion range and separation distance change the outcome of the stroke, parameters such as color and stroke angle are kept constant. Fig. 10 strokes are long strokes made with the flock brush system. These colored strokes feature high agent count, low separation and high cohesion. It also demonstrates the effects of stroke angles. Angle effects manifest in the form of an angled stroke, right angled stroke and a curve like stroke.

7 B. Painting FIGURE 11. LEFT ) ORIGINAL IMAGE RIGHT) PAINTERLY EFFECT FIGURE 14. LEFT ) ORIGINAL IMAGE RIGHT) PENCIL SKETCH EFFECT Typically, high energy agents in a flock brush have the effect of creating long thin strokes. The long strokes resemble the outline of a sketch. During subsequent strokes, the effect is enhanced via the use of shorter stroke that give the lines more weight. In Figs. 13 and 14, the flock brush are set to move in the same direction as the gradient of the underlying pixel and the agents are made to follow the edge of the image as it moves about the canvas. FIGURE 12. LEFT ) ORIGINAL IMAGE A painterly effect similar to that proposed by Haeberli [2] and Hertzmann [3] can be generated by using flocks of different sizes. Larger flocks produce thicker strokes and a smaller flock size results in thin strokes. The length of each stroke is determined by the agent s energy level. As the flock moves across the image, the flock places strokes of varying thickness and length on to the resultant image. Initially, we use a flock size of 10 to 15 agents to create thick and long strokes that approximate the input image. After which a flock of size 2 to 5 is used. This flock produces finer strokes and is used to enhance details and refine the output image. During this process, subsequent flock may produce strokes that over-write strokes created by previous flock. This causes the resultant image to generate an interesting colorblended appearance. C. Sketching FIGURE 13. LEFT ) ORIGINAL IMAGE RIGHT) PAINTERLY EFFECT RIGHT) PENCIL SKETCH EFFECT A pencil sketch looking image can be created by using a combination of grayscale sampling and varying the number of agents and their energy levels. D. A discussion on flock brush Flock parameters present a chance for the user to exercise a constrained creativity. In our experiments we have found some general trends with regards to the parameter values. A work on NPR and distributed agents was presented by Semet [9]. In their work, distributed agents or ants follow the image gradient and deposit ink onto the canvas to create stylistic effects. A single ant is used every iteration. The ant places strokes of uniform thickness on to the canvas. Our approach differs from previous work by considering a group of agents that models the digital brush more realistically than that of a single agent. Further, the flock is guided by multiple nature- inspired forces instead of merely following the image gradient. The position of an agent within the flock is determined relative to other agents within the flock. Overtime, this causes perturbations in the formation of the flock. These changes in formation make the resultant stroke formed by the flock non-uniform in thickness. See Fig Non-uniform strokes made by flocks bear a closer resemblance to real-life brush strokes. Realistic brush strokes begin with a certain thickness and then branch out and thin when the brush is lifted from the canvas. This effect can be observed when an artist uses quick short strokes. We now summarize how the different parameter values impact the resultant image. Flock size affects the size of the resultant stroke. A single agent will produce fine strokes; multiple agents will result in thicker strokes. Agents with high energy level will create longer strokes. The flock movement direction parameter can be guided by the image gradient or it can be user-defined. In general, changing the direction will change the orientation of the stroke but does not affect the placement of subsequent strokes. Color values can be based on the luminance or RGB values depending on the intended style. Decreasing separation and increasing cohesion values can generate thin and uniform strokes.

8 V. CONCLUSION AND FUTURE WORK This paper introduces a method of image stylization using flock brushes formed by autonomous agents. The method uses flock brushes to draw strokes guided by image gradient and steering forces to create non-photorealistic image styles. We did not focus on generating new styles with our method but instead on experimenting with autonomous agents as the style rendering mechanism for non-photorealistic rendering. The brush model presented in this work uses a flock of agents to model a 2D paint brush and local image gradients to guide the path of the flock. The path is translated to a painted stroke with source image color values. The shape of the stroke is constrained by formation of the flock and steering forces acting on the flock. Our preliminary results have shown that this brush model is capable of creating non-photorealistic digital images and the brush parameters can be tuned to yield different stylistic results. Painting and artistic rendering is a creative endeavor which presents many opportunities for research and expression. Avenues for future work present themselves in the following forms. Explore the use of alternative image features to generate the stroke path, for example, interpolation of image local gradient. Enhance artistic elements by introducing textures to simulate different types of drawing tools. This may allow the flock brush model to produce styles such as pastel drawing or charcoal sketch and sand painting. Drawing and painting on different surfaces or materials can lead to varied outcomes with the same tools. One can imagine that painting on an uneven surface would differ from that painting on a flat canvas. This type of phenomenon can be captured by applying additional forces to steer the flock. Lastly, as there are multiple parameters used for flock paint, we also wish to explore the possibility of using an interactive evolutionary process to search for innovative styles and allow for user evaluation of the artistic results. ACKNOWLEDGMENT The authors thank the staff of the Centre for Computational Intelligence School of Nanyang Technological University Singapore and Gambit Singapore for their support. Images used in this project are from Kodak Lossless True Color Image Suite [18] and online free wallpaper providers [19]. REFERENCES [1] Gooch, Gooch: Non-Photorealistic Rendering. AK-Peters (2001) [2] Haeberli, P.: Paint By Numbers: Abstract Image Representation. In: SIGGRAPH 90 Conference Proceedings. (1990) Orientable Textures for Image-based Pen-and-Ink Illustration Salisbury et al [3] Hertzmann, A.: Painterly rendering with curved brush strokes of multiple sizes.proceedings of SIGGRAPH 98 (July 1998) [4] Litwinowicz, P.: Processing images and video for an impressionist effect. Proceedings of SIGGRAPH 97 (August 1997) [5] Michael P. Salisbury, Sean E. Anderson, Ronen Barzel, and David H. Salesin. Interactive Pen-and-Ink Illustration. In ACM SIGGRAPH 94 Conference Proceedings, pages , July [6] Huiqin Wang, A Non-Stroke Based Method to Generate Sketching Style from Original Image Image and Signal Processing, CISP '08. Congress. [7] Collomosse, J. P. (2008). Evolutionary Search for the Artistic Rendering of Photographs. In Juan Romero and Penousal Machado, editor, The Art of Artificial Evolution, Natural Computing Series, pages Springer Berlin Heidelberg. [8] Ashkan Izadi, Vic Ciesielski, and Marsha Berry. Evolutionary non photo-realistic animations with triangular brushstrokes. In Jiuyong Li, editor, AI2010: Advances in Artificial Intelligence, Proceedings of the 23rd Australian Joint Conference, volume LNAI 6464, pages Springer, Berlin, [9] Semet, Y.O., Reilly, U.M., Durand, F.: An Interactive Artificial Ant Approach to Non-photorealistic Rendering. In: Deb, K., et al. (eds.) GECCO LNCS,vol. 3102, pp Springer, Heidelberg (2004) [10] Reynolds, C. W. (1987) Flocks, Herds, and Schools: A Distributed Behavioral Model, in Computer Graphics, 21(4) (SIGGRAPH '87 Conference Proceedings) pages [11] Reynolds, C.: Steering behaviors for autonomous characters. In: Game Developers Conference, pp (1999) [12] Xiaoyuan Tu and Demetri Terzopoulos, "Artificial Fishes: Physics, Locomotion, Perception, Behavior", Proc. of ACM SIGGRAPH'94, Orlando, FL, July, 1994, in ACM Computer Graphics Proceedings, 1994, p [13] Olfati-Saber, R.: Flocking for multi-agent dynamic systems: Algorithms and theory. IEEE Transactions on Automatic Control 51(3), (2006) [14] Ho, C.S., Nguyen, Q.H., Ong, Y.-S., Chen, X., Autonomous multiagents in flexible flock formation, Lecture Notes in Computer Science, 6459 LNCS, pp , [15] Foley, J., Van Dam, A., Feiner, S., Hughes, J.: Computer Graphics: Principles and Practice. 2nd edn. Addison Wesley (1997) [16] E. F. Schubert, Eye sensitivity function" in Light Emitting Diodes, 2nd edition, Cambridge, UK: Cambridge University Press, 2006, pp. 281 [17] Jain, R., Kasturi, R., Schunck., B.: Machine Vision. New York, NY, McGraw-Hill. (1995) [18] white lighthouse and dwelling, with blue sky and scattered clouds,kodak Lossless True Color Image Suite, [online] 1999, (Accessed: 20 March 2012). [19] Fresh Desktop Wallpapers,Wallpaper desktop encyclopedia [online] 2012, (Accessed: 20 March 2012).