Ink-and-wash Painting Based on the Image of Pine Tree using Mean Curvature Flow

Transcription

1 Ink-and-wash Painting Based on the Image of Pine Tree using Mean Curvature Flow IkSuk Yang Dept. of Computer Science & Computer Eng. Pusan National University Busan , Rep. of Korea DoHoon Lee Dept. of Computer Science & Computer Eng. Pusan National University Busan , Rep. of Korea YoungJung Yu Dept. of Computer Eng. Pusan University of Foreign Studies Busan , Rep. of Korea Abstract Ink-and-wash landscape painting in oriental has distinctive features to depict landscape comparing to the western painting. The pine tree is a main theme in oriental painting and its oriental style rendering remains unsolved problem to simplify the complex structure of its leaf based on image. In this paper, we propose a novel rendering method for ink-and-wash painting based on the image of pine tree using mean curvature flow (MCF). MCF is used to simplify the image for the segmentation of rendering components, and obtain primitive parameters to create strokes in pine needles. We provide primitive parameters to create a basic stroke and advanced parameters to describe more natural strokes. Using the combination of these parameters, the components of pine tree, which consist of the pine needles, trunk and branches and the shading of pine needles, are drawn. The system generates natural strokes for pine tree s leaves based on image and we show several results of simulated inkand-wash paintings. CR Categories: I.3.3 [Computer Graphics]: Picture/Image Generation Display Algorithms I.3.4 [Computer Graphics]: Graphics Utilities Paint systems; Keywords: ink-and-wash painting, image-based rendering, pine tree, mean curvature flow 1 Introduction Ink-and-wash landscape painting is one of the typical forms of expression in oriental painting and represents a theme of natural scenery using ink. It depicts the principle of harmony being formed with natural scenery such as mountain, pine tree, stone, water, cloud, and sky reinterpreting it by feelings of mind and body. It has aesthetic beauty of space by simplifying real whole view abstractly and omitting unnecessary objects, which is difficult to find in western painting. Many studies of oriental painting including ink-and-wash landscape painting have been done in the field of non-photorealistic rendering neoyang@pnu.edu yjyu@taejo.pufs.ac.kr dohoon@pnu.edu Copyright 2012 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) or permissions@acm.org. VRCAI 2012, Singapore, December 2 4, ACM /12/0012 $ (NPR) of computer graphics [Kim et al. 2005]. There are mainly two approaches for rendering in oriental painting according to input types: interactive simulation and image-based rendering [Wang and Wang 2007; Xie et al. 2010]. The First is interactive simulation [Yu et al. 2003b; Xie et al. 2010] uses a virtual brush with ink that can diffuse into the absorbent paper by input of user. The Second is Image-based rendering [Wang and Wang 2007; Yu et al. 2003a] that translates oriental painting using ink diffusion from input of a real image. In addition, creating 3D oriental painting [Way et al. 2002] with outline rendering and texture generation from 3D polygonal models. Pine Trees, the essential painting objects in ink-and-wash landscape painting can easily be presented both by simulation and 3D modeling with ink-and wash painting. The interactive simulation can draw the complicated pine tree structure easier by direct input of user. 3D modeling can create pine tree because of known information of vertices of pine tree. However, image-based rendering is difficult to extract information from pine tree in comparison to the two above-mentioned approaches. The reason is that it is difficult to separate pine needle, branch and trunk from pine tree on an image. The main problem is that the shape of pine tree is different from each image. There are wide variations in color due to light and shade, and extracting the whole pine tree from background looks too complex. For these reasons, most of existing studies on image-based ink-and-wash painting restrictively dealt with images of distant view. Studies of ink-and-wash painting based on the image of pine tree have never been done so far. In this paper, we present a novel ink-and-wash painting method based on the image of complicated pine tree structure which existing studies have never tried interpreting as the style of ink-andwash painting. The proposed method simplifies an image using mean curvature flow [Kang and Lee 2008]. We extract the region of pine needle from the simplified image and create strokes of pine needle considering the style of the ink-and-wash painting drawn by artists. Strokes have different shapes by parameters and they are composed with combinations of four types. Four types are to create strokes: adjusting the number of pine needle based on central direction, diffusing the stroke with ink and paper, controlling the curve of the stroke according to the parameter of tangent vector, and changing the number of strokes depending on density of pine needle on the image. The combination of strokes by parameter makes it possible to render a realistic ink-and-wash painting as if a real artist would have drawn it. In this paper, rendering system of ink-and-wash painting builds on our earlier work [Yu et al. 2003b]. This system uses the structure of fiber, hairy brush model and local equilibrium model to represent diffusing effects. The elements of rendering a pine tree are the pine needles, the shading for the bunches of the leaves, and the tree s trunk and branches. The rendering system creates ink-and-wash paintings reinterpreted as the style of the realistic ink-and-wash painting from input of such elements.

2 2 Related Work Existing simulation methods for oriental painting can be classified into interactive simulation and image-based rendering [Xie et al. 2010]. Interactive simulation for oriental painting models the physical properties of the virtual brush, the absorbent paper and diffusion effect with user interaction. Image-based rendering is called stroke-based rendering that converts a real image into an oriental painting. In this paper, we focus on the works related to rendering trees in these methods. Wang et al. [Wang and Wang 2007] proposed an image-based painterly rendering algorithm by automatically synthesizing an image with color ink diffusion. They created strokes using the luminance division and color segmentation. They simplified the information of the reference image, mix colors by the Kubelka-Munk theory, and simulated the result of the color ink diffusion in absorbent paper using the texture synthesis technique. The black ink result that has glass slightly similar to the shape of pine needles simulated by their method is close to a water painting [Curtis et al. 1997] rather than a ink-and-wash painting. 3 Rendering The proposed system of ink-and-wash painting mainly takes three stages. The first stage is simplifying the image using MCF, and creates the gradient vector field by extracting direction vector from the resulting image. The second stage is extracting the regions of pine tree s needles, trunk, and branches based on the resulting image, and creating strokes for rendering the components of pine tree using the combination of parameters. The last stage is converting these strokes into ink-and-wash painting using the rendering system from former study. Figure 1 shows the process by stage in the rendering system. Yu et al. [Yu et al. 2003a] described a new framework for synthesizing Chinese landscape painting using an image based approach. They created a few Brush Stroke Texture Primitives (BSTP) collected from samples of hand-made Chinese paintings, constructed the control picture, to provide color ID of mountains, and synthesized the fog image and draw mountains by mapping multiple layers of BSTP guided by the control picture. This method is suitable for the distant view such as mountains, and has limits with expressing a close view. Xie et al. [Xie et al. 2010] proposed an interactive sketch-based system for rendering oriental brush strokes of complex shapes. The user draws contours to represent complex shapes, and the system estimates the optimal trajectory of the brush automatically, and then renders them into oriental ink paintings. This method still requires talented people to present complex shapes such as pine trees in the same way as the interactive simulation. Way et al. [Way et al. 2002] presented a rendering method of automatically drawing trees with the Chinese ink painting style in 3D polygonal models. They used the information of the silhouette to render outline, generated textures using shade and orientation of 3D models surface to draw a particular tree, and established four reference maps to analyze the information for the bark texture. Such information of tree was already collected from 3D polygonal models. However, it is difficult to extract information of the tree in a 2D image. Conventional approaches for simplifying complex images deal with image segmentation, color quantization and feature-preserving smoothing. These techniques are focused to cluster or smooth pixels within homogeneous regions to simplify colors. However, they fail to simplify boundaries between heterogeneous regions and lose important information, such as directionality of features of an image. In this paper, we simplify the boundaries and preserve directionality in the image using the method of mean curvature flow (MCF) [Kang and Lee 2008]. Our earlier work [Yu et al. 2003b] as an interactive simulation for oriental painting, proposed local equilibrium model (LEM) that effectively calculates the movement of water and ink to represent diffusing effects. In this paper, rendering system for ink-and-wash painting was built on our earlier work. It is different from our earlier work in that its approach is stroke-based rendering of the image. Figure 1: Process in rendering system Image-based rendering for a ink-and-wash painting requires strokes in the same way as the interactive simulation. For extracting features from the complex input image necessary for generating strokes, we extract the region using the image simplified by MCF and the primitive information for strokes. We create realistic strokes with the combination of parameters that have several features. In this Section, we discuss the method to generate the strokes for inkand-wash painting based on the image of pine tree. 3.1 Image Simplification Preliminary process is to simplify the input image to find out the segmentation of the complex structure of pine tree. It is easy for a person to perceive the region of pine needle in the image. On the contrary, it is extremely difficult to distinguish the region of pine needle from the image full of the pixels of incoherent colors because of different color spaces. For this reason, we simplify the image to remove the noise and segment the region. MCF, which decreases the complexity of a geometric structure, is used as the method of simplifying the image [Kang and Lee 2008]. As shown in Figure 2, MCF smooths colors in the image like blur filter, and converts sharp shape into circular one. If an input image I(p) is regarded as the height field, where p = (x, y) denotes the location of pixel and height represents luminance, we can draw contours that have the same height in the image. The speed of applying MCF is proportional to the local isophote curvature. In other words, the part of high curvature on the curve is smoothened faster. Overall, complex shape is simplified, smoothed and eventually converted into a circular form by MCF (see Figure 190

3 4c shows the result image of extracting the region of the tree s trunk and branches, where hue range is from Orange & Brown (21 ) to Orange-Yellow (50 ) incorporating Brown color (30 ) and intensity is great than 70. Saturation in Figure 4 is ignored because it has no effect on these results. Figure 2: Mean curvature flow: (a) Input (b) MCF: 10 iterations (c) MCF: 60 iterations 2c). The evolution equation under MCF is defined as follows [Kang and Lee 2008]: I t = κ I, (1) where I t, the derivative of image I with respect to time t, is proportional to the absolute value of I = (I x, I y) which is the gradient of an image. κ denotes the local isophote curvature and is computed from the image [Kang and Lee 2008]: κ = I2 xi yy 2I xi yi xy + I 2 yi xx (I 2 x + I 2 y) 3/2. (2) Figure 3 shows gradient vector field in comparison to the input image and the resulting image by using MCF. The direction of gradient vector field in the input image is incoherent (see Figure 3a). However, the luminance of the resulting image by using MCF not only represents contour shapes in Figure 3b but also the gradient of its image shows the directionality of fan shape in Figure 3c. Figure 3: Gradient vector field in comparison to the input image and the resulting image by using MCF. (a) Direction of gradient vector field in the input image is incoherent. (b) Luminance of the resulting image by using MCF represents contour shapes. (c) Gradient vector field in comparison to the input image and the resulting image by using MCF. 3.2 Region Segmentation To create the strokes of pine needles, the region of pine needles needs to be distinguished from the result image, which was simplified by MCF. As shown in Figure 2c, the result image simplified by MCF is indistinguishable between the pine needles and the tree s trunk and branches. Searching the regions of them, we use HSI color space, which is better practical for humans to interpret than RGB color space. In HSI color space [WorkWithColor.com 2008] (Figure 4 a), we can adjust the range of hue and intensity in the MCF image to segment the region of pine needles because each image of pine needles has wide variations in colors due to illumination and shadow. Figure 4b shows the result image of extracting the region of pine needles in HSI color space, where hue range is from Yellow-Green (60 ) to Cyan (200 ) incorporating Pine Green color (175 ) and intensity is great than 80. In the same way, Figure Figure 4: Region Segmentation in HSI Space: (a) Hue ranges in HSI Color Space (b) Extracting the region of pine needles (c) Extracting the region of the tree s trunk and branches 3.3 Primitive Parameters Primitive parameters for creating a stroke of a pine needle are direction, start point, end point and length of the stroke. First, the direction of a stroke is calculated from the segmented region of the image simplified by MCF. We can observe the patterns of the direction of pine needles. Pine needles grow in the direction of pointing the sky, regularly spread out between 90 and +90 of the direction of branches. The direction of a pine needle in inkand-wash painting artworks also has the similar patterns mentioned above. Under these patterns, the pine needle starts from the end of branches and spread out in the central direction of them. The end of branches in the image of pine tree is densely encompassed by pine needle and well-hidden, which is why it is difficult to detect the direction of the pine needle. Nevertheless, inspired by that the luminance of the image applied by MCF represents contour shapes and the gradient of it shows the directionality of fan shape, we can calculate the direction of the pine needles. Hence, instead of finding the direction of the end of the branch, we let the gradient of the image in the region of pine needles be a direction vector of a pine needle (see Figure 5). The equation of calculating a direction vector from a MCF image is defined as follows: ( M(x, y) (L x, L y) =, x ) M(x, y), (3) y where L is the direction vector of a pine needle, and M denotes a MCF image. Supposing that current pixel is a start point, the length of a stroke can be expressed from the direction vector L on the pixel by the user, and an end point is derived by adding the direction vector L to the start point. Figure 5: Primitive parameters for a stroke calculated from MCF image MCF: direction, start point, end point and length. 191

4 3.4 Advanced Parameters A set of strokes is composed with the combinations of several parameters including primitive parameters to create natural strokes as if a real artist drew the pine needles. If strokes are created only by primitive parameters, the strokes are unnatural as if machines drew the strokes, not human beings. Advanced parameters for creating more natural strokes are the number of pine needles, diffusion effect, bending degree and the density of pine needles. We introduce four methods of creating strokes changed by advanced parameters. The first method is adjusting the number of pine needles in the central direction of representative pine needle. Figure 6 represents a set of strokes in the central direction calculated from the MCF image can express as if a real artist added several strokes for pine needles in the direction. A set consists of 3, 5 or 7 strokes. The second method is the diffusion effect of strokes, which is differentiated feature in ink-and-wash painting. Figure 6b represents the diffusion effect of seven strokes according to the concentration of ink and water on a piece of paper. We use the diffusion effect that was implemented in the rendering system of our earlier work [Yu et al. 2003b]. The third method is controlling the curve of a stroke by the parameter that represents bending degree (see Figure 6c). Pine needles in real world stretch in a straight line. Nevertheless, pine needles described in ink-and-wash painting is in the form of slight curve, similar to the shape of a line. We use Hermit curves to create the stroke of a pine needle. Hermite curves consist of two points and two tangent vectors. We already covered the two points of a stroke in primitive parameters and only need to deal with two tangent vectors. The bending degree of curve is controlled by the two tangent vectors of start point and of end point, and proportional to the range of random position. In addition, tangent vector connects the start point used in primitive parameters to the end point created by random position. The last method is changing the number of sets of strokes in accordance with the density of pine needles. We suppose that the density is in proportion to the number of sets of strokes, and is in inverse proportion to intensity. In other words, as intensity level increase, sets of strokes decrease in number. We adjust the number of strokes according to intensity level from the gray scale, which MCF result was converted into. Gray scale is divided into several levels to get intensity. We use histogram equalization for intensity that covers whole gray scale, and use quantization to divide into several level of intensity. The level of density is defined by the result and decides the number of sets of strokes (see Figure 6d). 3.5 Rendering Components of Pine Tree The components of pine tree, as shown in Figure 7, are the pine needles, the trunk, the branches, and the shading for the bunches of the leaves. Since the other components were covered in previous section, we will discuss the shading for the bunches of the leaves. We shade the bunches of the leaves before the rendering system draws the strokes of pine needles. Strokes for the shading are applied to every regions of pine needles and are created by increasing intensity. We add the needles, the trunk, and the branches over the shading for the bunches of the leaves. 3.6 Rendering System We took the earlier rendering system [Yu et al. 2003b], which had the form of interactive simulation, and extended and modified it for image based rendering in the system. Modified system simulates strokes extracted from image instead of generated stroke by brush. Figure 6: Methods of creating strokes by parameters: (a) the number of pine needles (b) diffusion effect (c) bending degree (d) the density of pine Figure 7: Components of pine tree: (a) Pine needles (b) Tree s trunk and branches (c) shading for the bunches of the leaves Since strokes were mentioned in previous section, we will introduce paper model and diffusion effect. In the paper model with fiber structure, each cell on the paper is divided into three layers: surface layer, absorption layer, and deposition layer. To represent the diffusion effect, Local equilibrium model (LEM) is used to calculate the movement of water and ink effectively. This rendering system has limitations of input images. The color of the pine needles, the trunk and the branches in the image must be distinguishable to the naked eye. In case it is difficult to separate the pine tree from the image, the background of the image is removed by Photoshop in advance to avoid interference colors. 4 Experimental Results Our system was implemented in C++ on Intel core i3 2.13GHz CPU with 2GB memory. As mentioned earlier, this system is imagebased rendering which coverts the complex shape of pine tree in an image into ink-and-wash painting. We used color images, relatively high-resolution, to draw the small, thin pine needle graphically. Figure 2 presents the result image simplified by MCF, which took about 3 seconds to complete 10 iterations, and took about 20 seconds to complete 60 iterations. Figure 8 shows the comparison with the direction of storkes by MCF iterations. The methods of constructing strokes by parameters are summarized in Figure 6. Figure 9 presents the various shapes of strokes applied 192

5 (a) input (b) MCF no iteration (c) MCF 30 iterations (d) MCF 60 iterations Figure 8: Comparison with the direction of storkes by MCF iterations (a) Bending = 10 (b) Bending = 20 (c) Bending = 30 (d) Bending = 40 (a) Input (b) No Background (c) Diffusion = 0 (d) Diffusion = 3 (e) Diffusion = 5 (b) Diffusion = 7 Figure 9: Bending degree in advanced Parameters. to bending degree in advanced parameters and Figure 10 shows several diffusion effects in the result images of ink-and-wash painting. In most cases, these parameters were set to the set of 7 strokes, diffusion = 3 and bending = 10 for more natural strokes. Figure 10: Diffusion effect in advanced Parameters; pine needles: 60 hue 180, intensity 230, saturation 0.2, MCF 60 iterations, set of 7 strokes, and bending = 10 and for the tree s trunk and branches: 220 hue 360 or 0 hue 50. The results of rendering components of pine tree are given in Figure 7. It takes about 20 seconds to create strokes of the pine needles, about 5 seconds to create the strokes of the trunk and the branches, and about 3 minutes to create background. Although it requires relatively longer time to draw many pine needles according to the directions obtained from MCF image for the shading, it is an essential component to make the strokes of pine needles stand out. Creating strokes for the pine needles and the shading is a novel and distinguished approach, in comparison to other rendering approaches. The result in Figure 7b is simple and crude in contrast to other results. The reason is that generating realistic strokes with the reconstruction of the trunk and the branches is unpoised in this study. iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 We used the original image without removing the background in Figure 11h and Figure 10 because the pine tree could be easily separated from the image. The result in Figure 10 was conducted under the conditions for pine needles: 60 hue 180, intensity 230, saturation 0.2, MCF 60 iterations, set of 7 strokes, and bending = 10 and for the tree s trunk and branches: 220 hue 360 or 0 hue 50, MCF no iteration. The result in Figure 11h was conducted under the conditions for pine needles: 60 hue 200, intensity 120, saturation 0.2, MCF 30 iterations, set of 7 strokes, diffusion = 3, and bending = 10 and for the tree s trunk and branches: 0 hue 72, intensity 120, MCF no iteration. Figure 11b, 11d and 11f represents diffusion effect using our rendering system after the background of input image is removed. The diffusion effect was used less than necessary because pine needles in real ink-and-wash paintings have little diffusion effect. Our results used conducted under the conditions in Figure 11: (b) 70 hue 270, intensity 20, MCF 60 iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 (d) 55 hue 180, 100 intensity 150, MCF 30 iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 (f) 60 hue 200, intensity 20, saturation 0.2, MCF

6 5 Conclusion The purpose of our study is to generate ink-and-wash paintings, by reinterpreting a complex pine tree in the style of the realistic ink-and-wash painting, which has never been studied so far. We proposed the method of simplifying complex shapes in the image, and obtaining the parameters of strokes by MCF. As for differentiated rendering method, we presented a method of creating natural strokes by combining parameters to describe pine needles, and suggested a mean to create the shading for the bunches of leaves which makes the pine needles stand out. Our system has a few limitations: using of the images of only pine tree, dealing with the images that a pine tree can be easily separated from, drawing crude result of trunk and branches in comparison to the result of pine needles, and being non real-time rendering. (a) Input (b) Output (c) Input (d) Output (e) Input (f) Output (g) Input (h) Output Based upon this study, primitive stroke could be used in expressing other types of trees besides pine trees, and extracting coherent direction from an image of complex shape. Future studies should focus on representing natural complex shapes of natural objects in realistic ink-and-wash painting. Acknowledgements This work was supported by the Brain Busan 21 Project in References C URTIS, C. J., A NDERSON, S. E., S EIMS, J. E., F LEISCHER, K. W., AND S ALESIN, D Computer-generated watercolor. In SIGGRAPH, vol. 31, K ANG, H., AND L EE, S Shape-simplifying image abstraction. Comput. Graph. Forum, K IM, S., L EE, J., K IM, B., K IM, H., AND KOO, B Recent trends in non-photorealistic rendering. Electronics and Telecommunications Trends 20, WANG, C.-M., AND WANG, R.-J Image-based color ink diffusion rendering. IEEE Transactions on Visualization and Computer Graphics 13, WAY, D.-L., L IN, Y.-R., AND S HIH, Z.-C The synthesis of trees in chinese landscape painting using silhouette and texture strokes. In WSCG 02, W ORK W ITH C OLOR. COM, Hue ranges. Website. http: // X IE, N., L AGA, H., S AITO, S., AND NAKAJIMA, M Ir2s: interactive real photo to sumi-e. In Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, ACM, New York, NY, USA, NPAR 10, Y U, J., L UO, G., AND P ENG, Q Image-based synthesis of chinese landscape painting. J. Comput. Sci. Technol. 18, 1, Figure 11: Ink-and-wash painting results; (b) 70 hue 270, intensity 20, MCF 60 iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 (d) 55 hue 180, 100 intensity 150, MCF 30 iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 (f) 60 hue 200, intensity 20, saturation 0.2, MCF 60 iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 (h) 60 hue 200, intensity 120, saturation 0.2, MCF 30 iterations, set of 7 strokes, length of stroke = 40, diffusion = 3, and bending = 10 and for the tree s trunk and branches: 0 hue 72, intensity 120, MCF no iteration. Y U, Y.-J., L EE, D.-H., L EE, Y.-B., AND C HO, H.-G Interactive Rendering Technique for Realistic Oriental Painting. Journal of WSCG 11, 1,