Image stylization with enhanced structure on GPU

Transcription

1 . RESEARCH PAPER. Special Focus SCIENCE CHINA Information Sciences May 2012 Vol. 55 No. 5: doi: /s y Image stylization with enhanced structure on GPU LI Ping 1*, SUN HanQiu 1*, SHENG Bin 1,2* & SHEN JianBing 3 1 Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China; 2 Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai , China; 3 School of Computer Science and Technology, Beijing Institute of Technology, Beijing , China Received October 24, 2011; accepted January 11, 2012; published online March 14, 2012 Abstract This paper presents a graphics processing unit (GPU) based stylization approach that preserves the fine structure between the original and the stylized images using gradient optimization. Existing abstraction and painterly stylization methods focused on contrast manipulation only, and thus the detailed salient structures of the input images are always destroyed when performing the current stylization techniques because of limitations like unavoidable salience information loss caused by contrast abstraction. We propose an image structure map to naturally model the fine structure existing in the original images. Gradient-based structure tangent generation and tangent-guided image morphology are used to construct the structure map. The image structure map, unlike an edge map, not only systematically models the boundary information within the imagery but also accentuates the underlying inner structure detail for further stylization. We facilitate the final stylization via parallel bilateral grid and structure-aware stylizing optimization on a GPU-CUDA platform in real time. In multiple experiments, the proposed method consistently demonstrates efficient and high quality image stylization performance. Keywords image stylization, non-photorealistic rendering, real-time processing, GPU, image abstraction Citation Li P, Sun H Q, Sheng B, et al. Image stylization with enhanced structure on GPU. Sci China Inf Sci, 2012, 55: , doi: /s y 1 Introduction The way we choose to portray images can have a significant effect on how accurately and efficiently we can communicate with viewers. Before the invention of photographic apparatus, painting was one of the most important ways to record both beautiful scenery and life itself. After hundreds of years of study, artists found that by making images look more non-photorealistic, with different stylization methods as a vehicle for abstraction, creativity and expression, they were able to enhance the audiences immersive feelings for the original story in real life on the canvas [1]. Recent advances in painterly rendering have demonstrated that certain artistic styles can be generated automatically [2 5]. The ability to automatically generate arts and illustrations for more effective communication with scenes, ideas, and actions would allow media creators of films and video games to attain a new level of creativity. In many cases, e.g. illustrations, stylized images have been proved to be more effective in communicating subtle information about phenomena than photorealistic images. *Corresponding author ( {pli, hanqiu, bsheng}@cse.cuhk.edu.hk) c Science China Press and Springer-Verlag Berlin Heidelberg 2012 info.scichina.com

2 1094 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No. 5 Figure 1 (a) The original image; (b) the stylized image by painterly stylization [1]; (c) our result: note that it faithfully preserves the structural details in the image, i.e. the eyes and clothes. All images have the same resolution of pixels. Abstraction and stylization algorithms are designed to deal with digital image watercolorization, oil painting, and cartoon style generation. The growing interest in non-photorealistic rendering (NPR) is because of the expressive power to illustrate shapes and spatial relationships, as well as for generation of artistic drawings and paintings. Recently, increasing numbers of researchers are focusing on the hot topics of NPR by introducing new computer graphic techniques to simulate the effects of traditional paintings using methods of abstraction [6 8] and painterly stylization [1,9,10]. However, the detailed salient structures of the original images are somehow always destroyed by the current image abstraction and painterly stylization methods because of limitations like unavoidable salience information loss caused by contrast abstraction manipulation, which may greatly influence the users understanding of visual art and illustration. There is an artistic theory called realism [11], according to which realist painters try to portray what they see without idealizing the subjects of their paintings. They seek to choose subjects from common objects in real life, and use brush and pencil to render the scenes realistically, with a care for the detailed structure of the targeted objects to be painted [12]. Therefore, the method inherently implies a belief that such quality and characteristics are ontologically independent of human conceptual feeling, ideas and recreation, and thus can be known by artists, who can in turn represent such reality faithfully using the brush and pencil on the canvas. Usually, realist painters position themselves against romanticism in the creation of artwork, seeking artwork that is undistorted by human bias [11]. Realism believes in the ideology of objective reality, which is the opposite of the exaggerated emotionalism in romanticism in painting, and thus the truth and accuracy of the object structure to be painted in real life become the goals of realism [12]. Such realistic artwork earns its popularity because of its differences compared with the true perspective of photography, because realism not only realistically represents the world with its fine object structures in the painting but also gives a certain range of aesthetic abstraction and artistic stylization. According to the popular artistic theory of realism [11,12], therefore, introduction of new approaches to deal with image stylization while maintaining the fine structure of the original images is very important. Careful inspection of the texture regions in Figure 1(b) (e.g. the eyes and the flowers on the clothes) reveals that the stylization technique destroys the characteristic patterns. Figure 1(c) shows the use of the image stylization technique that we introduce in this paper. As can be clearly seen, this image reproduces the correct stylization, and at the same time is faithful to the original texture appearance. Our NPR look is better, while on the face, the eyes look much clearer, and keeping the eyes and the mouth clear are essential in real painting. Also, on the child s clothes, we maintain a good-looking stylized texture with clear flower structures, and the overall effect of our rendering is visually pleasing. In this paper, we present a method to abstract images into a stylized manner while preserving the fine structures between the original and the stylized images, to make an effort through exploration of an image structure map to naturally model the underlying fine structure of the original input images. Gradientbased edge tangent generation and tangent-guided image morphology are then applied to build the structure map. Stylization is achieved through edge-aware bilateral grid processing with structure-aware stylization optimization using real-time GPU-CUDA programming. To the best of our knowledge, there is no previous method specifically designed to stylize images with a structure-awareprocess. Structure-aware stylization, as a more delicate form of image stylization, allows the viewer to understand the image content

3 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No more quickly and intuitively when they are organized as utilities like illustrations. This also eases many image analysis tasks. In the following, Section 2 briefly reviews the related techniques in image stylization and GPU acceleration. Section 3 describes our GPU-based image stylization, including gradient-based structure tangent generation, tangent-guided image morphology, structure-aware stylization, and GPUaccelerated processing. Section 4 presents the experimental results and a discussion of our approach in comparison with the state-of-the-art work. Finally, we summarize our work in Section 5. 2 Related work Image Stylization. This paper is made possible by the inspirations of previous work, and a comprehensive summary review of the state-of-art stylization theories and techniques is given in [13]. In general, two basic forms of stylization exist: the abstraction approach and painterly stylization. The former achieves the abstraction by reducing the contrast in low-contrast regions and increasing the contrast in high-contrast regions, while the latter applies tools like brush strokes to simulate the artistic styles of painters. Hertzmann [4,14] proposed a stroke-based method using energy function minimization to search for a painting with minimal energy, which allows the user to specify the painting style by varying the relative energy terms. The work is then extended to process images by learning from the real oil painting through the use of techniques called image analogies [15]. Bousseau et al. [2,16] proposed methods to deal with interactive watercolor rendering that recreate the specific visual effects of watercolors using images and 3D models. Wen et al. [10] introduced an NPR system to generate cartoon-like painterly sketches in a free-hand drawing style by minimizing the energy function based on an artist-drawn color database, artistic drawing rules and input image colors. The above methods provide efficient solutions for image/video stylization and abstraction. However, when performing stylization, image structure is always destroyed because of limitations like unavoidable salience information loss caused by abstraction, and this is actually essential information for better understanding of the material of stylized images. For instance, in illustrations of mechanical parts in educational books, where the detailed structure of the images is an important issue, even these images are stylized. DeCarlo and Santella [3] presented a computational approach to stylize images that responds in explicit terms to the design goal of clarifying the meaningful visual content using eye tracking. However, tracking the users eye movements and calibrating the eye tracker are usually very difficult tasks. Winnemöller et al. [7] presented a real-time video abstraction framework that abstracts imagery by modifying the contrast of the visually important features of luminance and color opponency. However, the detailed image structure of the original images is neglected when applying bilateral filtering to obtain the abstraction. Kang et al. [6] used flow-based bilateral filtering for edge-preservation and smoothing and a flow-based anisotropic difference-of-gaussian filter (FDoG) for line extraction; their work focused mainly on design of different filters for abstraction [17,18], and the edge information is used in their approach. Zhao et al. [8,19] extended their work and developed techniques of feature-aware and saliency-aware processing to improve the features of abstraction level specification. Huang et al. [20] proposed a method to enhance temporal coherence in video painting using a motion layer, where the layers for a single frame consist of a background layer and many object layers. They segmented the video into many motion layers, and placed strokes on these layers to produce the painting effects. They further extended their approach to painterly rendering [21], which simulates the artist s process of content-dependent painting, and the input image is segmented into non-uniform grids according to the related importance map. In their method, each brush has an individual color, and strokes rendered using the brushes will have multiple colors and variable textures. However, preserving the detailed underlying image structure is not a major concern in their work. Zhang et al. [22] presented a real-time online approach to generate temporally coherent abstracted videos with large regions of constant color and enhanced bold edges. Lu et al. [1] proposed an interactive system that converts images, videos and 3D animations into artistic renderings by painterly stylization, and the strokes are rendered as point sprites with textures in their system. However, this work still does not solve the problem of structure loss caused by abstraction and painterly stylization.

4 1096 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No. 5 Figure 2 Approach overview. Each step lists the function performed, and the sub-image on the right shows our structureaware image stylization of the koala image. GPU Acceleration. The graphic processor unit (GPU) is a highly parallel, multi-threaded, manycore processor with enormous computing power and high memory bandwidth. GPU stream processing allows applications and data manipulation to be easily exploited and investigated in the form of parallel acceleration. Considerable research has been carried out to use GPUs to investigate the image processing domain. Hachisuka and Jensen [23] used a GPU to carry out parallel progressive photon mapping with more accurate global illumination rendering. Bertal et al. [24] presented an anisotropic diffusion partial differential equation to be applied to images that allows real-time rendering. Kazhdan and Hoppe [25] proposed a novel streaming multi-grid GPU solver to solve the large linear systems that arise from image processing in a gradient domain. Orzan et al. [26] presented a GPU-based real-time system for rendering of images defined by a set of diffusion curves, which is a new vector-based primitive to create smoothshaded images. Mavridis and Papaioannou [27] introduced high quality texture filtering on a GPU based on the elliptical weighted average (EWA) filter. Their method uses the underlying anisotropic filtering hardware to construct a filter that closely matches the shape and properties of the EWA filter. Krähenbühl et al. [28] presented a novel integrated system for content-aware retargeting on the GPU CUDA (compute unified device architecture) platform, using an iterative multi-grid solver for feature estimation and energy minimization, and OpenGL for the EWA image synthesis. McCann and Pollard [29] proposed a GPU-based image editing program that allows artists to paint in the gradient domain with real-time feedback on megapixel images. For our work in structure-aware image stylization, local and global optimization is needed to preserve the underlying fine structure between the original input images and the stylized images, while performing bilateral filtering. Optimization processing is well known to be highly time-consuming, especially when processed together with bilateral filtering manipulation. These latest works on the use of GPUs have inspired us to carry out our structure-aware image stylization with optimization using GPU-CUDA acceleration. 3 Structure-aware stylization 3.1 Approach overview We propose a novel structure map to systematically model the detailed underlying fine structure that exists in the original images. Our goal is to stylize images by abstracting the visual content while preserving the elaborate image structure for important human perception, to ease structure-aware image stylization for artistic image viewing. Figure 2 shows the pipeline for our image stylization. Gradient domain information is first extracted to generate an erosion direction map for the mathematical morphology. We perform ellipse-shaped erosion in the directions computed using the gradient tangent, and the erosion magnitude, which here means the semi-major and semi-minor axes of the erosion ellipse, and is related to the corresponding image gradient magnitude. The morphology erosion is performed on an inverted gray scale image using local gradient-based optimization guided by the direction map to generate a detailed structure map. We use a real-time edge-aware bilateral grid to recursively process the original images to generate the stylized images in the L AB color space. Soft luminance quantization is carried out

5 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No to produce more non-photorealistic cartoon-like effects, and we add the edge computed using the FDoG proposed in [6] to enhance the edging effects in the initial stylized images. We then convert the stylized images back to the RGB color space for further processing. Note that image stylization at this stage mostly will not preserve the original fine structure because of limitations like luminance range reduction in the low-contrast regions caused by contrast manipulation. We propose image structure map guided global optimization processing to preserve the fine structure between the stylized and original images, and we perform this task as the final step in the pipeline to obtain the structure-aware image stylization effect. As a result of global optimization, a detailed image structure is maintained by our method, while the correct image stylization is abstracted appropriately, as shown by the koala example in the sub-image at the right side of Figure Gradient-based tangent generation Our goal is to construct a fine image structure map to perform structure-aware stylization, and thus we need to perform mathematical morphology operations on the anti-color gray level images to erode the non-structural information. Illustrations of the structure tangent generation and the ellipse-shaped erosion in the tangent direction are shown in Figure 3. As indicated in the illustrations, we need to compute for the erosion direction for morphology manipulation first. Here, we apply a gradient-based method to generate the structure tangent because the gradient direction is the normal for luminance intensity change, as shown in Figure 3, and we are then able to perform ellipse-shaped erosion according to the direction of the structure tangent. To compute the structure tangent, we first need to obtain the luminance map of the input image, and thus we compute the initial luminance map as I lu (x, y) =α R R(x, y)+α G G(x, y)+α B B(x, y), (1) where α R =0.213,α G =0.715,α B = We then use the real-time edge-aware bilateral grid (discussed in Subsection 3.4) to manipulate the initial luminance map as Ilu s = BilateralGrid(I lu) to obtain the smoothed luminance map. After the bilateral grid smoothing, the luminance map Ilu s is applied to estimate and find the locally consistent image structure gradient, and it is defined in terms of neighboring gradient values as I g (x, y) =( I gx (x, y), I gy (x, y)); { Igx (x, y) =Ilu s (x +1,y) Is lu (x, y), I gy (x, y) =Ilu s (x, y +1) Is lu (x, y). (2) We can thus get the directions of the fine image structure according to the image gradient, and an arc tangent function is applied to describe the direction of the image structure as an angle of θ (also shown in Figure 3): ( ) Igy (x, y) θ(x, y) = atan. (3) I gx (x, y) Using θ, we can perform gradient-based structure tangent generation efficiently using GPU-based bilateral grid processing. Figure 4 shows the visualization of a tree image gradient and tangent directions in terms of angle theta computed using the arc tangent function. We can see from Figure 4 that the generated image structure directions faithfully display the tendencies of the detailed structure in the tree image. 3.3 Tangent-guided image morphology Mathematical morphology has proved to be one of the best approaches to extract essential image structural components to represent and describe the original images. It is a set-theoretic method of image analysis, providing a quantitative description of detailed geometrical structures. Morphology can provide the boundaries of objects, their skeletons and convex hulls, and erosion is one of the elementary operators of mathematical morphology. The key mechanism under erosion is the local comparison of a shape, i.e. a structural element, with the object that is to be transformed. When positioning the structural element

6 1098 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No. 5 Figure 3 Illustrations of structure tangent generation (left), and ellipse-shaped erosion in the tangent direction (right). Figure 4 Visualization of image gradient, and tangent direction in terms of angle. (a) Tree image; (b) image gradient; (c) corresponding image tangent direction in terms of angle. at a given point, the structural element is included in the object, and then this point will appear in the results of the transformation. Here, we use the ellipse-shaped erosion of morphology to manipulate the anti-color gray scale image to extract the fine underlying image structure from the original images. We use the ellipse, in which the length of the semi-major axis a is twice that of the semi-minor axis b, asa structural element for erosion. Hence, the erosion structural element, namely the ellipse E (h,k) centered at position (h, k), has the equation in parametric form in canonical position as E (h,k) =(x e,y e ); { xe = h +2bcos t, y e = k + b sin t, π t π. (4) We perform ellipse-shaped erosion according to the directions of the gradient tangent, where the gradient tangent directions are used as the rotation angle for the semi-major axis a of the ellipse E (h,k) at position (h, k). To derive full structural information from images, we compute the semi-minor axis b using the image gradient magnitude as b = s I mag g = s Ig 2 x (x, y)+ Ig 2 y (x, y), (5) where s is the related scaling coefficient for normalization, and the length of the semi-major axis a =2b. Finally, erosion using the ellipse E (h,k) at position (h, k) as a structural element is performed on the color inverted gray scale image I gray according to the magnitudes of the gradient and the directions of the structure tangent. We extract the representative salient image structure map I struc, and formulate the problem as an optimization process for finding the I struc which differs least from (I gray ΘE) by finding

7 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No Figure 5 Local optimization based image structure building by adjusting the parameter s. (a) Tree image; (b) initial status of erosion optimization; (c) the intermediate result of erosion optimization; (d) erosion convergence using the optimal ellipses as computed, where a detailed image structure including tree branches is obtained. the optimal ellipses and adjusting the parameter s as arg min s {(I struc (I gray ΘE)) 2 }, I gray ΘE = {(h, k) :E (h,k) I gray }, (6) where Θ stands for the erosion process, and E stands for all the ellipses used. This is actually a local optimization process, and it is an iterative-based process. One example of the tree image structure building process is shown in Figure 5, where we see that the converged image structure (Figure 5(d)) faithfully represents the tree s branches in detail. 3.4 Structure-aware optimization We apply a real-time edge-aware bilateral grid to recursively process the original images to generate the stylized images in the L AB color space, and convert these stylized images back to the RGB color space for further processing. A bilateral grid is a primary data structure that enables a variety of edge-aware operations. With one more axis to represent the pixel intensity, the bilateral grid is able to achieve much faster image manipulation. The main feature of the bilateral grid is use of a 3D structure to represent 2D data, e.g. 3D representation of 2D images. Thus, axes x and y correspond to the pixel position in 2D images, and axis z corresponds to the pixel intensity. The Euclidean distance in the bilateral grid accounts for edges, with space distance (x, y) and intensity distance (z). The data structure was first introduced by Paris and Durand [30] as an auxiliary data structure for fast approximation of bilateral filtering, but the bilateral grid was simply applied as an algebraic re-expression of the original bilateral filter equation in their work. Chen et al. [31] treated the data structure differently, and expanded the bilateral grid as a primary data structure to enable a variety of edge-aware operations. They thus took greater advantage of the extra data dimension created from the original 2D images. We use a novel data structure, the GPU-based edge-aware bilateral grid filter, to recursively manipulate the L AB color space of original images to achieve initial image stylization abstraction. Soft luminance quantization is performed to generate better cartoon-like effects, and edge enhancement is carried out using the FDoG to strengthen the edging effects in the initial stylization. We then convert the stylized images back to the RGB color space for further global structure-aware stylization optimization using the image structure map. By lifting image stylization into a higher dimensional space, we are able to develop a stylization algorithm that naturally preserves the strong edges in images. Our image stylization method using the bilateral grid maps well onto GPUs and enables real-time stylization of high-resolution images. We developed our structure-aware image stylization method using the image structure map and the initial image stylization by bilateral grid filtering to perform stylization while preserving the image structure using global structure-aware energy minimization as E aware = (I strucaware (x, y) I style (x, y)/i struc (x, y)) 2, (7) x,y

8 1100 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No. 5 Figure 6 Image stylization of a female guitarist image and enlarged details of her clothes. (a) The input image; (b) salient region detected using region-based contrast in [32]; (c) DeCarlo and Santella [3]; (d) Winnemöller et al. [7]; (e) Zhao et al. [8]; (f) our structure-aware image stylization with less abstraction performed in the salient regions. The results generated from our method produce proper image stylization and preserve the fine image structure in the original image, but the texture of the clothes and the details of the face are preserved in different ways. The detailed underlying structure of the clothes is retained mainly because of our structure-aware optimization, and the details of the face are preserved mainly because less abstraction is performed in the highly salient region. where I strucaware (x, y) is the final structure-aware stylized image, I style (x, y) is the initial stylized image using recursive bilateral grid processing, and I struc (x, y) is the anti-color gray scale image structure map. Figure 6 shows our structure-aware image stylization of a female guitarist image in comparison with state-of-the-art work. As seen from the enlarged detail and the overall stylization, our method generates the correct image stylization while preserving the image structure between the original images and the stylized images. Features of the human visual system are added in addition to the gradient-based structure map by using a salient region detection method proposed by Cheng et al. [32], the region-based contrast approach. Therefore, we can preserve not only the detailed image structure of the clothes, but also the details of the human face. However, the reasons why these images are preserved are different: the underlying image structure of the clothes is retained mainly because of our structure-aware stylization processing, while the fine detail of the woman s face is preserved mainly because less image abstraction is performed in the highly salient region detected. 3.5 GPU-accelerated stylization We have developed GPU acceleration for image stylization with the bilateral grid using the latest NVIDIA CUDA version 4.0. For GPU CUDA coding, we set the total number of blocks as 128 and the total number of threads within a block to be 256, thus a total of threads are available in our program to compute the bilateral grid according to the object coordinates, which is sufficient for the bilateral grid manipulation and stylization processing. The pseudo code for computing the bilateral grid manipulation kernel on the GPU-CUDA for initial stylization is outlined as global static void GPU grid (float **objectsindex, float *inputintensity, float *outputintensity) { Get the block index BID and thread index TID; Initialize the bilateral grid; Cycle the image index (x, y) through all the color image space using BID and TID { Accumulate the input color intensity into each grid cell; Accumulate the number of pixels into each grid cell; } //Obtain new bilateral grid by convolution Convolute the grid with spatial Gaussian kernel along the spatial dimension; Convolute the grid with intensity Gaussian kernel along the intensity dimension; Cycle the image index (x, y) through all the color image space using BID and TID Extract the initial stylized image intensity by accessing the grid at (x/s, y/s, I(x, y)/i). }

9 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No Figure 7 Structure-aware stylization results for a flower image with salient region detected. (a) The input image; (b) the detected salient region of the flower image using a region-based contrast method described in [32]; (c) our structure-aware flower stylization with less abstraction carried out in the detected salient regions; (d) result from Huang et al. [21]; (e) result rendered using Hertzmann s method [14]; (f) result rendered using Zhao and Zhu s method [5]. Note that both (c) and (d) can produce realistic abstractions with the fine image structure concerned, but in the stamen area of the flower, our stylization can preserve the underlying structure better between the original and the stylized image because less abstraction is performed in the highly salient regions and because of the structure-aware stylization optimization in those regions. Unlike Huang s interactive foreground and background segmentation method, our approach can perform structure-aware optimization automatically in real time, with the automatic accurate salient region detection method proposed in [32]. The global qualifier declares the function GPU grid() as being a kernel, and the function is executed on the GPU, and is callable from the CPU only. The calling of the GPU kernel function GPU grid() from the CPU is in pseudo code as Get the current image index & color intensity according to bilateral grid construction; Copy the image index & color intensity data needed from main memory to GPU memory; GPU grid<<<grid, thread>>>(gpuobjectsindex, gpuinputintensity, gpuoutputintensity); Read the GPU computed results gpuoutputintensity from GPU memory back to main memory; Update the result matrix with the initial stylized image intensity values. There are grid.x grid.y blocks running synchronously in the GPU, and thread.x thread.y threads also running synchronously within each block. In total, there are (grid.x grid.y) (thread.x thread.y) threads running synchronously to operate the edge-aware bilateral grid manipulation kernel GPU grid() to compute for the bilateral filtering for all of the pixel stylization in parallel by the GPU-CUDA, achieving real-time processing for image stylization. Figure 7 shows our structure-aware image stylization of a flower image with the salient region detected in comparison with others. We use the region-based contrast method described in [32] to compute the salient regions automatically, and our method produces aesthetic stylization with the fine underlying image structure preserved between the original and the stylized images, especially in the stamen area of the flower. The image structure of the result rendered using the method of Hertzmann [14] seems fine, but the abstraction technique is not that advanced, and hence the stylization is not perfect. The result generated by Huang et al. [21] also can somehow maintain the image structure, but because we apply less abstraction in the highly salient area and because of the structure-aware optimization processing using our approach, our stylization process shows better structure awareness in the stamen area with a fully automatic process performed in real time with hardware GPU acceleration, even if enlarged.

10 1102 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No. 5 Figure 8 Image stylization results, where the top four rows show our structure-aware image stylization in comparison with others, and the last row shows some more results using our approach. It can be seen that our approach preserves fine image structures between the original and stylized images, while also providing high quality aesthetic image stylization. 4 Results and discussion In our experiments, we used C++ with GPU CUDA version 4.0 and MATLAB to develop our structureaware stylization. The example images have been tested on an Intel(R) Core(TM)2 Duo CPU 2.3 GHz PC with an NVIDIA GeForce 8800 GPU and 2 GB RAM. We have tested many example images for structureaware image stylization using gradient optimization. In Figure 8, we have compared our method with the latest state-of-the-art work. The first four rows show our structure-aware image stylization in comparison with others. The last row of sub-images in Figure 8 shows different types of scenes, including parrots, a lakeside area, and indoor fruit, in which our structure-aware stylization process produced aesthetic stylization with an elaborate image structure. In Figure 8, for the campus image, our approach presented proper image stylization while maintaining

11 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No Table 1 The performance comparison with previous methods Data Methods Recognition speed (s) Short term retention (d) Attractiveness Original Medium Child Lu et al. [1] Medium Our structure-aware High Original Medium Campus Lu et al. [1] Low Our structure-aware High Original Low Cat Kyprianidis et al. [17] Low Our structure-aware High Original Medium Tree Kyprianidis et al. [18] High Our structure-aware High Original Medium Guitarist DeCarlo and Santella [3] Low Our structure-aware Medium Original Medium Huang et al. [21] High Flower Hertzmann [14] Medium Zhao and Zhu [5] Low Our structure-aware High the fine image structure, e.g. the chairs and the windows. Compared with the existing methods, our stylization for the chairs, tree branches, and windows are better in terms of structural awareness, while at the same time our stylization effects for these objects are aesthetically pleasing for the audience. For the cat image, our method yielded good-looking NPR of the cat while providing very good hair structure preservation, and the use of the edge-aware bilateral grid means that our cat s outside edge is better; for the mouth of the cat, our approach generated a good structure for the teeth and tongue while the stylization effect remains obvious. For the tree image example, both methods look good, although the styles are a little different, but our method provided exquisite structure-aware stylization, where the tree is stylized but the structure of the tree is well preserved in the image, which is good for printing in, for example, educational books and newspapers. For the guitarist image, the stylization is very similar, but our method renders the lips well, which is essential in portraiture, and although the wall behind the guitarist is heavily stylized, our method still shows that it is a brick wall. These examples show the pleasing effects and the structural awareness of our approach, and some further examples are provided in the last row of Figure 8 to show the robustness of our method. To verify that our image stylization preserves or distills perceptually important information, we performed a user study of recognition speed and short-term memory retention. We invited 30 students (10 from an art school, 20 non-art-related) who knew nothing about our research to evaluate the image stylization. We gave them the images in Figures 1, 7, 8, along with other images of similar children, campuses, cats, trees, guitarists and flowers for evaluation recognition speed. All of the images were displayed on a computer with a 29 inch monitor. Along with recognition speed, we also tested short-term memory retention after initial recognition of the objects. All of the statistics are based on the average performance of the 30 students. We also printed and distributed 50 copies of the images in Figures 1, 7, 8 randomly on the street, asking people to mark the attractiveness of the images, where, 0 means not attractive at all and 10 means very attractive, and 46 copies of the printed materials were returned for evaluation. All of the statistics are based on the average scores of the 46 people, where the attractiveness is set to low if the average score of the image is less than or equal to 4, set to medium if the average score of the image is between 4 and 8, and set to high if the average score of the image is greater than or equal to 8. Finally, statistical information for the stylization is shown in Table 1. From Table 1, we

12 1104 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No. 5 see that our structure-aware stylization generally needed less time for picture recognition, and can help people remember the scenes for a few days longer in their short-term memory. The randomly sampled user study performed on the street also showed that our stylization was found to be more attractive by the people viewing the test images. In terms of stylization time performance, our approach can run in real time for a 1000 by 1000 pixel resolution image, at about 48 frames/second when tested on the GPU CUDA platform with parallel bilateral grid processing, and at about 46 frames/second if automatic salient region detection is applied. 5 Summary Previous abstraction and stylization methods have mostly focused on contrast manipulation, and the detailed salient structures of the original images are always destroyed while performing stylization. In this paper, we presented an image structure map to naturally represent the detailed underlying fine structure existing in the original images. We use an ellipse-shaped erosion method to extract the structure map using local energy minimization based on the gradient magnitude and the image structure directions. The intermediate stylization is produced using a recursive edge-aware bilateral grid in real time on a GPU CUDA platform. Finally, global optimization is carried out to ease the final structure-aware image stylization using the image structure map and the initial stylization generated by the bilateral grid. In multiple experiments and in user studies, our structure-aware stylization methods demonstrated efficiency in terms of ease of understanding and ease of remembering the stylized images. Acknowledgements The work was supported by an RGC research grant (ref ), UGC direct grants for research (Grant No , ), National Natural Science Foundation of China (Grant No ) and Key Program of NSFC-Guangdong Union Foundation (Grant No. U ). References 1 Lu J W, Sander P V, Finkelstein A. Interactive painterly stylization of images, videos and 3D animations. In: Proc ACM SIGGRAPH Sym Interactive 3D Graphics and Games (I3D 10), Bousseau A, Neyret F, Thollot J, et al. Video watercolorization using bidirectional texture advection. In: Proc SIG- GRAPH, DeCarlo D, Santella A. Stylization and abstraction of photographs. In: Proc SIGGRAPH, Hertzmann A. Paint by relaxation. In: Proc Computer Graphics International (CGI 01), Zhao M T, Zhu S C. Sisley the abstract painter. In: Proc ACM Sym Non-Photorealistic Animation and Rendering (NPAR 10), Kang H, Lee S Y, Chui C K. Flow-based image abstraction. IEEE Trans Vis Comput Graph, 2009, 15: Winnemöller H, Olsen S C, Gooch B. Real-time video abstraction. In: Proc SIGGRAPH, Zhao H L, Mao X Y, Jin X G, et al. Real-time saliency-aware video abstraction. The Visual Computer, 2009, 25: Kang H, Lee S Y, Chui C K. Coherent line drawing. In: Proc ACM Sym Non-Photorealistic Animation and Rendering (NPAR 10), Wen F, Luan Q, Liang L, et al. Color sketch generation. In: Proc ACM Sym Non-Photorealistic Animation and Rendering (NPAR 06), Prendeville B. Realism in 20th Century Painting. New York: Thames & Hudson, Stremmel K, Grosenick U. Realism. Koln: Taschen, Hertzmann A. Non-photorealistic rendering and the science of art. In: Proc ACM Sym Non-Photorealistic Animation and Rendering (NPAR 10), Hertzmann A. Painterly rendering with curved brush strokes of multiple sizes. In: Proc SIGGRAPH, Hertzmann A, Jacobs C E, Oliver N, et al. Image Analogies. In: Proc SIGGRAPH, Bousseau A, Kaplan M, Thollot J, et al. Interactive watercolor rendering with temporal coherence and abstraction. In: Proc ACM Sym Non-Photorealistic Animation and Rendering (NPAR 06),

13 Li P, et al. Sci China Inf Sci May 2012 Vol. 55 No Kyprianidis J E, Kang H. Image and video abstraction by coherence-enhancing filtering. In: Computer Graphics Forum (Eurographics 11), Kyprianidis J E, Kang H, Döllner J. Image and video abstraction by anisotropic Kuwahara filtering. In: Computer Graphics Forum (Pacific Graphics 09), 2009, Zhao H L, Jin X G, Shen J B, et al. Real-time feature-aware video abstraction. The Visual Computer, 2008, 24: Huang H, Zhang L, Fu T N. Video painting via motion layer manipulation. Computer Graphics Forum, 2010, 29: Huang H, Fu T N, Li C F. Painterly rendering with content-dependent natural paint strokes. The Visual Computer, 2011, 27: Zhang S H, Li X Y, Hu S M, et al. Online video stream abstraction and stylization. IEEE Trans Multimedia, 2011, 13: Hachisuka T, Jensen H W. Parallel progressive photon mapping on GPUs. In: Proc SIGGRAPH Asia Sketches, Bertal M, Fort P, Sanchez-Crespo D. Real-time, accurate depth of field using anisotropic diffusion and programmable graphics cards. In: Proc 3DPVT, Kazhdan M, Hoppe H. Streaming multigrid for gradient-domain operations on large images. In: Proc SIGGRAPH, Orzan A, Bousseau A, Winnemöller H, et al. Diffusion curves: a vector representation for smooth-shaded images. In: Proc SIGGRAPH, Mavridis P, Papaioannou G. High quality elliptical texture filtering on GPU. In: Proc ACM SIGGRAPH Sym Interactive 3D Graphics and Games (I3D 11), Krähenbühl P, Lang M, Hornung A, et al. A system for retargeting of streaming video. In: Proc SIGGRAPH Asia, McCann J, Pollard N S. Real-time gradient-domain painting. In: Proc SIGGRAPH, Paris S, Durand F. A fast approximation of the bilateral filter using a signal processing approach. In: Proc European Conference on Computer Vision (ECCV 06), Chen J, Paris S, Durand F. Real-time edge-aware image processing with the bilateral grid. In: Proc SIGGRAPH, Cheng M M, Zhang G X, Mitra N J, et al. Global contrast based salient region detection. In: Proc IEEE Conference on Computer Vision and Pattern Recognition (CVPR 11),