An Efficient Importance Map for Content Aware Image Resizing Abstract Ahmad Absetan 1* and Mahdi Nooshyar 2 1 Faculty of Engineering, University of MohagheghArdabili, Ardabil, Iran 2 Faculty of Engineering, University of MohagheghArdabili, Ardabil, Iran *Corresponding Author's E-mail: aabsetan@yahoo.com Recently, content aware image resizing methods were proposed to supplement content oblivious methods such as scaling or cropping. A content aware resizing operator relies on an importance map to preserve the important parts of the image at the expense of the lessimportant ones. The importance map is an image-based measure and is a core part of the resizing algorithm. Importance measures include image gradients, saliency and entropy, as well as high level cues such as face detectors, motion detectors and more. In this work we proposed a new method to calculate the importance map, the importance map is generated automatically using a novel combination of image edge density and Harel saliency measurement. Experiments of different type images demonstrate that our method effectively detects prominent areas can be used in image resizing applications to aware importance important areas while preserving image quality. Keywords: Image resizing, resizing algorithm, image edge density and Harel saliency measurement. 1. Introduction In the last decade, many content aware methods for image resizing have gained popularity. Due to the increase in the variety of commonly used display devices, and the prevalent use of mobile devices as available means for media intake, media needs to be adapted to different resolutions and aspect ratios. Uniform scaling methods (e.g., bicubic and bilinear) do not consider the content of images, therefore, content-aware image resizing becomes more and 786
more important. The purpose of content aware image resizing methods is that as far as possible to preserve the image content. The majority of these methods use an important map to calculate the amount of deformation of image areas, the significant map can be calculated by the combination of saliency, gradient, color clutter and etc., the importance of each pixel of the image is determined by the value of corresponding pixel in the importance map. The techniques typically start by computing an importance map which represents the relevance of every pixel, and then apply an operator that resizes the image while taking into account the importance map and additional constraints. Figure 1: The image retargeting steps. Most techniques follow this general flow [2]. It is clear that the definition of important can be subjective and can depend on the application in mind. Indeed, different works define different importance functions on images that contain both low level visual cues such as image edges and high level cues such as people s faces [1]. The common approach taken by all image resizing algorithm is composed of two steps: 787
1. The first step is the definition of an importance map and other constraints on the original media being retargeted. 2. The second step applies some operator to the media to change the size while taking into consideration the importance map and its constraints. Figure 1 shows the steps of retargeting problem. In this work we proposed a new measure of image importance that can better detect the prominent object of the input image. The rest of this paper is organized as follows: Section 2 reviews related works. We will discuss the image resizing problem in section 3 and then propos a new method for calculating importance map in section 4. Results and comparison to other methods are provided in Section 5. Finally, Section 6 concludes this paper. 2. Related Works Previous image retargeting methods have used various measures to determine the importance value of a pixel automatically. In the Feature-aware texturing proposed by Gal et al.[3], the user provides a feature mask that marks the parts of the image whose shape should be preserved. This method divides the input image to quads and tries to preserve the shape of all the quads contained in the features. Both Avidan and Shamir [4] and Wolf et al. [5] consider pixels with large gradient magnitudes as important. Rubinstein et al. [6] determine the pixel significance by accumulating the discontinuity of its neighbors if the pixel is removed. Visual saliency measure can identify important image regions that should remain intact while the image s aspect ratio is altered. There are two categories of approaches to automatically estimate saliency: bottom-up methods, and top-down methods [2]. Bottom-up methods are based on low-level features such as edge orientation, color, and intensities. A popular approach for computing bottom-up saliency was proposed by Itti et al. [7]. It is inspired by the human visual system, and is based on low-level features: color, intensity, and orientation. A multi-resolution pyramid of the image is built, and significant changes in the 788
features are searched for and combined into a single high-resolution map. Itti s method is so popular that from now on if we simply say some authors used saliency map without giving other references, they used that method. Stentiford [8] proposed a method for computing saliency based on dissimilarities between neighborhoods in the image. Ma and Zhang [9] introduced a heuristic-based method that analyzes contrast and is more efficient than Itti s method, while leading to similar results for image retargeting. Achanta and S usstrunk[10] proposed a saliency measure based on comparing pixels in a blurred version of the image to the average color of the original image in the Lab color space, which is useful when salient objects differ in color from the rest of the image. Harel et al. [11] proposed a graph-based visual saliency model. It consists of two steps: First, forming activation maps on certain feature channels, and then normalizing them in a way which highlights conspicuity and admits combination with other maps. Top-down methods make use of semantic information, such as the locations of important objects (e.g., faces, bodies, and text), structures, and symmetries. Fan et al. [12] also used a text detector as a component of their top-down saliency. Top-down approaches are often combined with bottom-up saliency to generate the importance map. Some retargeting methods used the combination of gradient and visual saliency to generate the importance map. Wang et al. [13] calculated it by multiplying two measures to determine the pixel significance: image gradient and saliency map [7]. The gradient indicates the presence of structures, while the saliency map determines the attractiveness of a region. In particular, the gradient magnitude can be misled by trivial and repeated structures, while the saliency measure also considers regions that are attractive but homogeneous as salient. By combining these two measurements, a region is considered significant only if it is both structural and attractive. Liang et al. [14] computed importance map as the sumof image edge and saliency. Image edge is computed with the Sobel Operator and saliency is calculated by the saliency of Harel [11]. 789
The ViSizer method [15] calculated importance map by multiplying the degree of interest (DOI) map and clatter map. DOI map was first introduced by Furnas [16] to indicate that visual items in visualization have different levels of importance. This framework employs an efficient method called Feature Congestion [17] to estimate the clutter magnitude in every local region. This method can produce an image called clutter map with the same resolution of the visualization for revealing the clutter magnitude at every pixel. It uses the level of feature congestion to indicate the degree of clutter in an image. 3. Image Resizing Problem As pointed by Shamir and Sorkin [1], the resizing problem can be stated as follows. Given an image of size, and a new size we would like to produce a new image of size which will be a good representative of the original image. However, to date, there is no clear definition or measure as to the quality of being a good representative of. In loose terms there are three main objectives for retargeting: The important content of I should be preserved in. The important structure of I should be preserved in. should be free of visual artifacts. Influenced by the three objectives stated above, the majority of image retargeting techniques follows a similar flow (Fig. 1).The definition of important can depend on the specific application being considered. As mentioned above, the first step of the content aware image resizing methods is the computation of an importance map, which quantifies the importance of every pixel in the image. Different papers define different importance measures that specify the level of importance of pixels in the image. Many of those described above. In the next section we proposed a new importance map for using in the first step of the image resizing algorithm. 790
4. Proposed Importance Map The importance map is an image-based measure and is a core part of the resizing algorithm [13]. Previous image retargeting methods have used various measures to determine the significance value of a pixel automatically. Avidan and Shamir [4] and Wolf et al. [5] used gradient value for calculating the importance of each pixel, Liang et al. [13] proposed a new measure of image importance by combining image edge and saliency [11]. The gradient can be calculated as: ( ) ( ) Where is the input image. Some methods have used instead of : ( ) ( ) ( ) Wang et al. [13] calculated important map by multiplying the gradient and the saliency as below: ( ) Where is the saliency that introduced by Itti et al. [7]. The gradient indicates the presence of structures, while the saliency map determines the attractiveness of a region. Liang et al. [14] computed importance map as: ( ) ( ) 791
Where is the saliency that introduced by Harel [11] and is computed with the SobelOperator., aconstantcoefficient,is set to. Figure 2: Overview of proposed algorithm. (a) original image, (b) edge density map, (c) saliency map and (d) is proposed importance map that is calculated by production of saliency and edge density. Here, we present a new measure of image importance that can better detect prominent image and has the best performance on resizing algorithms. We define the importance map as theproduct of the saliency of Harel[11] which is more powerful to predict human fixations on natural images and the edge density map. To indicate the presence of structures, the proposed algorithm use edge density map and to indicate the attractiveness of a region, it uses saliency of Harel [11]. Rosenholtz et al. [18] presented three measures of visual clutter, including feature congestion, subband entropy and the edge density measure. The edge density measure attempts to capture the notion of clutter as number of object by calculating the density of edge. We used the edge density to indicate the presence of structures of the input image for calculating the important map because it is less sharp than gradient and less remove the prominent objects while multiplying by saliency map. Let denotes the input image. We define the pixel importance map as: ( ) where is the Harel saliency map and is the edge density measure. We normalize the to obtain weighting value of each pixel of the input image within 0 and 1: 792
( ) ( ) Smaller values mean less importance. Fig. 2 shows an Overview of proposed algorithm. Figure 3: Comparison of our importance map (d) with Liang s importance map[14] (b) and Wang s importance map (c). (a) is original image. 5. Results and Discussions We compare our proposed importance map by other popular importance map that are the combonation of multi measures and described in Section 3. In Figure 3, we compare proposed method with Liang s et al. [14] and Wang s et al..as shown in Figure 3(b), the importance map that proposed by Lign s [14] is blured and the structure of the importance object of the input image is not evident. Because of the use of gradient, the importance map 793
that proposed by Wang et al. [13] the salient areas was removed by multiplying the gradient and saliency (see Figure 3(c)). As shown in Figure 3(d), the Important objects and structures of the input image are well recognized by using of the proposed algorithm. The reason of the good performance of proposed method is that edge density is less sharp than gradient and the prominent areas dos not removed while multiplying by saliency map. Image-resizing methods can be generally classified as discrete or continuous methods[15]. Seam carving [4], is the most popular in discrete group, we used the proposed importance map instead of the energy function in seam carving and we see that the important objects are better preserved is resized results (see Figure 4(e)). Like seam carving, optimal scale and stretch proposed by Wang et al. [13] is the most popular in continuous content aware image resizing method. We used the proposed importance map instead of the Wang s importance map and as show in the lower row of figure 5, it has better results in the resized images. Figure 4: Using proposed method instead of gradient in seam carving method [4]. (a) original image, (b) proposed importance map, (d) resized result by seam carving method by using of gradient as importance map, and (e) ) resized result by seam carving method by using of proposed importance map instead of gradient. 794
Fig. 3: Using proposed method instead of Wang soptimal scale and stretch method [13]. We define the importance map as the product of the edge density magnitude and the saliency measure. We compute the significance map of the original image (a) by two ways (the upper row of (b) shows the Wang s method [13] and the lower is ours). (c) shows the resized results of optimal scale and stretch resizing algorithm (the upper guided by Wang s significance map and the lower guided by our importance map). Notice that the width-height proportion preserved better with the use of our important map. 6. Conclusions and Future Works This paper introduced a new importance map for using in content aware image resizing method. The importance map is generated automatically using a novel combination of image edge density. To indicate the presence of structures, the proposed algorithm use edge density map and to indicate the attractiveness of a region, it uses saliency map. Compared with previous works, proposed method has better performance in detecting the content of image. 795
By using of the proposed importance map instead of existing importance maps in some content aware image resizing algorithms show that the resized images have better quality and the algorithm with our method better preserve the content of image. We plan to propose an effective algorithm to resize the image by using of this importance. References [1]. Shamir, A. and Sorkine, O., Visual media retargeting", SIGGRAPH ASIA Courses ACM, (2009), 1-11, Yokohama, Japan. [2]. D. Vaquero, M. Turk, K. Pulli, M. Tico, N. Gelfand, A survey of image retargeting techniques, Proc. SPIE 7798, Applications of Digital Image Processing XXXIII, (2010). [3]. R. Gal, O. Sorkine, AND D. COHEN-OR, Feature aware texturing,iproceedings of Euro graphics Symposium on Rendering,(2006), pp.297-303. [4]. S. Avidan, A. Shamir, Seam carving for content-aware image resizing, ACM Transactions on Graphics,Vol 26,3( 2007), pp.267-276. [5]. L. Wolf, M. Guttmann, D. Cohen-Or, Non-homogeneous content- driven video retargeting, IEEE International Conference on Computer vision, (2007), pp.1-6. [6]. M. Rubinstein, A. Shamir, S. Avidan, Improved seam carving for video retargeting, ACM Transactions on Graphics,Vol 27,3( 2008), pp.1-9. [7]. L. Itti, C. Koth, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. on Pattern Analysis and Machine Intelligence,Vol 20,11(1998), pp.1254-1259. [8]. Stentiford, F., An attention based similarity measure with application to content based information retrieval, SPIE Storage and Retrieval for Media Databases, (2003). [9]. Ma, Y.-F. and Zhang, H, Contrast-based image attention analysis by using fuzzy growing, ACM Intl. Conf. on Multimedia,( 2003), pp.374-381. [10]. Achanta, R. and S.usstrunk, Saliency Detection for Content-aware Image Resizing, IEEE Intl. Conf. on Image Processing, (2009). [11]. J. Harel, C. Koch, P. Perona, Graph-based visual saliency, Proceedings of the NIPS. (2006). [12]. Fan, X., Xie, X., Ma, W.-Y., Zhang, H.-J., and Zhou, H.-Q., Visual attention based image browsing on mobile devices,ieee Intl. Conf. on Multimedia and Expo, (2003). [13]. Y.S. Wang, C.L. Tai, O. Sorkin, T.Y. Lee, Optimized scale-and-stretch for image resizing,acm Transactions on Graphics, (2008), pp.1-9. 796
[14]. Y. Liang, Z. Su, X. Luo, Patchwise scaling method for content-aware image resizing,signal Processing,Vol 92(2012), pp.1243-1257. [15]. Y. Wu, X. Liu, S. Liu, AND Kwan-Liu Ma, ViSizer: A Visualization Resizing Framework, IEEE transactions on visualization and computer graphics, (2013), pp-278-290. [16]. G. Furnas, Generalized Fisheye Views, Proc. SIGCHI Conf. Human Factors in Computing Systems,(1986),pp. 16-23. [17]. R. Rosenholtz, Y. Li, J. Mansfield, and Z. Jin, Feature Congestion: A Measure of Display Clutter, Proc. SIGCHI Conf. Human Factors in Computing Systems, (2005), pp.761-770. [18]. R. Rosenholtz, Y. Li, L. Nakano, Measuring visual clutter, Journal of Vision,Vol 7,3(2007), pp. 1- Authors Ahmad Absetan received the B.S.c degree in Software Engineering from Hakim Sabzavari university, Sabzavar, Iran, in 2012. He is currently M.S.c student in Computer Architecture at University of Mohaghegh, Ardabily, Ardabil, Iran. His current research interests include digital image processing, machine vision and visualization. Mahdi Nooshyar received the B.Sc. degree from University of Tabriz, Tabriz, Iran, the M.Sc. degree from Tarbiat Modares University, Tehran, Iran, and the Ph.D. degree from University of Tabriz, all in Electrical Engineering in 1996, 1999, and 2010, respectively. He is currently an Assistant Professor of Electrical Engineering at University of Mohaghegh Ardabili, Ardabil, Iran. His current research interests include digital communications and information theory, digital image processing and machine vision, soft computing and its applications in electrical engineering. 797