Generalized Multiscale Seam Carving

Transcription

1 Generalized Multiscale Seam Carving David D. Conger #, Mrityunjay Kumar *2, and Hayder Radha #3 # Dept. of Electrical & Computer Engineering, Michigan State University East Lansing, MI, USA congerd2@egr.msu.edu, 3 radha@egr.msu.edu * Eastman Kodak Company Rochester, NY, USA 2 mrityunjay.kumar@kodak.com Abstract With the abundance and variety of display devices, novel image resizing techniques have become more desirable. Content-aware image resizing (retargeting) techniques have been proposed that show improvement over traditional techniques such as cropping and resampling. In particular, seam carving has gained attention as an effective solution, using simple filters to detect and preserve the high-energy areas of an image. Yet, it stands to be more robust to a variety of image types. To facilitate such improvement, we recast seam carving in a more general framework and in the context of filter banks. This enables improved filter design, and leads to a multiscale model that addresses the problem of scale of image features. We have found our generalized multiscale model to improve on the existing seam carving method for a variety of images. I. INTRODUCTION As widespread use of mobile media devices continues to increase there remains a pressing desire for an effective means of resizing images to fit arbitrary screen sizes. Traditional methods such as cropping or resampling either remove important image features or introduce significant visual distortion. Avidan and Shamir [] generated new interest in this area with the introduction of a content-aware image resizing technique called seam carving which iteratively removes/adds connected paths of pixels to achieve a desired target size. This was followed by the development of several other techniques ([4][][3]) which employed global resizing rather than the iterative method used in seam carving. While these techniques produced good results, they are less scalable an image must be reprocessed for each resolution change. Seam carving on the other hand enables one to store the locations of all seam paths removed/added in achieving some minimum/maximum resolution, allowing any resolution in between to be achieved by simply recalling the path locations and removing/repeating those pixels. For this reason and others, seam carving remains an effective image resizing tool. And although Rubinstein et al. [9] did improve on the original technique (as well as extend it to video), there are still cases where it has problems (cf. Figure (b)). MMSP'0, October 4-6, 200, Saint-Malo, France /0/$ IEEE (a) (b) (c) Figure : (a) Original image and the first 30 vertical seams selected for removal (shown in red) using (b) forward-energy seam carving and (c) our multiscale seam carving method using B fwd with N = 4. In this paper we present a generalized seam carving algorithm from the perspective of filter banks and develop a multiscale analysis model. This enables the use of many different families of filters and leads to an improvement in seam selection for many images (cf. Figure (c)). Section II details the proposed utility of filter banks for seam carving; section III discusses filter selection. Section IV describes the proposed multiscale model for seam carving. Section V presents our results and section VI gives concluding remarks. II. GENERALIZED SEAM CARVING MODEL To facilitate a more general seam carving model, we recast previous seam carving implementations in terms of 2D filter banks. Section II-A gives an overview of the most essential elements of the seam carving algorithm based on []. Section II-B develops the filter bank framework. Section II-C redefines backward-energy seam carving in this framework; section II-D redefines forward-energy seam carving. A. Seam Carving Overview A seam is defined as an 8-connected path of pixels from either top to bottom (vertical seam) or left to right (horizontal seam). In order to maintain an image s rectangular shape, a seam contains exactly one pixel per row (vertical seam) or per column (horizontal seam). The optimum path is determined by generating a cumulative energy map M and traversing from the bottom edge (or right edge) back to the top edge (or left

2 edge), maintaining a connected path and selecting the minimum value at each row (or column). This path of pixels is the seam and is removed. The process then starts over, continuing until the desired resolution is met. While there are several seam carving operations vertical seam removal/insertion, horizontal seam removal/insertion, combined vertical and horizontal seam removal/insertion they are all based on the process of identifying a single seam. Therefore, for the sake of brevity, development of our model will be based on the selection of vertical seams only, but can be easily extended to the other listed operations. (a) (b) B. Filter Bank Framework Formally, let B = {W k } 0 k K be a filter bank where W k is an h w mask. We define the result of filtering an m n image f with the mask W k as w/2 h/2 d k (x, y) = f(x + i, y + j)w k (i, j) i= w/2 j= h/2 which we will refer to as the k-th subband (or coefficient map). Note that this is the cross-correlation of f and W k and not a convolution a crucial distinction for the forwardenergy seam carving implementation defined in section II-D. Each coefficient map d k provides information about a particular image feature based on the structure of the mask W k ; image neighborhoods resembling the mask will result in large coefficient values while other neighborhoods will not. This leads to the idea that different combinations of coefficient maps can be used depending on the direction of a seam. Recall that, due to the nature of 8-connected paths, a seam can travel in one of three ways at each stop on its journey from edge to edge. For a vertical seam, at each row it can move down to the right, straight down, or down to the left. Therefore, it is possible to consult different coefficient maps for different directions. For convenience, we describe this discretionary ability using vector products. We define D i,j q([d 0 (i, j), d (i, j),, d K (i, j)]) where q is an element-wise operator used for normalization. Unless otherwise stated, we define q based on the L -norm: q(a) i,j a i,j where a i,j is the i, j-th element of the matrix A. This normalization is used to ensure that coefficients do not cancel and essentially typifies the energy metric. How and when the different coefficient maps are used comes into play during the generation of the cumulative energy map M(i, j) as defined in [2][9] and generalized here for vertical seam removal: M(i, j ) + D i,j p R M(i, j) = min M(i, j) + D i,j p C M(i, j + ) + D i,j p L where p R, p C and p L are K vectors whose k-th elements define the weighting factor of the k-th coefficient map. Essentially, these guidance vectors as we will refer to them designate which coefficient maps are used in steering a (c) (d) Figure 2: (a) Original image and a 30% width reduction using (b) forwardenergy seam carving and (d) inclusion of the Gabor filter with negative weight. (c) Result of filtering the image using the Gabor filter, shown in its frequency domain representation. seam down to the right, straight down or down to the left, respectively. The cumulative energy map can be defined similarly for horizontal seam removal; however, as mentioned above we will limit our discussion to the vertical case. C. Backward-Energy Seam Carving Filters The original seam carving algorithm presented in [] (and which was later named backward-energy in [9]), uses a single energy map primarily the L -norm of the image gradient to generate the cumulative energy map M. As there is more than one way to implement the gradient for discrete-time signals, we define the backward-energy filter bank in terms of general derivative masks for the vertical (H V ) and horizontal (H H ) directions: B back = {H V, H H } with p R = p C = p L = [,] T p. One can easily verify that D i,j p = d 0 (i, j) + d (i, j) which is the L -norm of the image gradient at the point (i, j). Notice that since p R = p C = p L, there is no distinction between the coefficient maps for different seam directions. D. Forward-Energy Seam Carving Filters The forward-energy seam carving algorithm defined in [9], is described in terms of the absolute pixel differences between disjoint pixels brought together by seam removal. Specifically, this is implemented directly into their definition of the cumulative energy map for vertical seam removal : Note that we have presented this with slightly different notation in terms of what is left and what is right, and have separated the cost C C (i, j) from the left and right costs in order to facilitate developing the filters.

3 Another observation gleaned from our model is that the guidance vectors can take on different weights to produce interesting results; we discuss this in section III-C. Figure 3: (left) 200 seams and (right) result of forward-energy seam carving (top) without and (bottom) with a center bias weighing factor of 5. where M(i, j ) + C R (i, j) + C C (i, j) M(i, j) = min M(i, j) + C C (i, j) M(i, j + ) + C L (i, j) + C C (i, j) C R (i, j) = f(i, j) f(i, j ) C C (i, j) = f(i, j + ) f(i, j ) C L (i, j) = f(i, j) f(i, j + ) This leads to the forward-energy seam carving filter bank with B fwd = 0 0, 0, p R = [,,0] T p C = [0,,0] T p L = [0,,] T As in section II-C, one can easily verify that D i,j p R = d 0 (i, j) + d (i, j) = C R (i, j) + C C (i, j) D i,j p C = d (i, j) = C C (i, j) D i,j p L = d (i, j) + d 2 (i, j) = C C (i, j) + C L (i, j) III. FILTER DESIGN Having redefined forward-energy seam carving in section II-D, one can see that it essentially combines shifted Roberts masks with a simple D difference mask. The Roberts masks discourage seams from passing through diagonal edges, while the simple difference mask discourages seams from passing through vertical edges. This observation naturally prompts the use of other seam carving filters, and while there are many possibilities, due to space limitations we highlight just a few specific cases (sections III-A and III-B). A. Texture Filters A significant problem with seam carving is its sensitivity to texture. Often there may be an image that contains a relatively smooth object set against a busy texture such as in Figure 2 (a). Using seam carving, the texture is mostly avoided and many seams pass through the person s hair and face, causing significant distortion (Figure 2 (b)). One way to address this problem is in terms of scale (which we discuss in section IV). Another approach is through the use of filters which are sensitive to texture, such as Laws texture masks or Gabor filters [3][7]. For the example in Figure 2, we used a Gabor filter with 90º phase, one octave bandwidth, and a center frequency of (Figure 2 (c)) let G be the mask associated with this filter. By augmenting this onto the forward-energy filters (i.e., B = B fwd {G }) and assigning it negative weight for diagonal directions, i.e., p R = [,,0, ] T p C = [0,,0,0] T p L = [0,,, ] T we can guide the seams through the texture and mostly avoid the person (Figure 2 (d)). Of course, there are other Gabor filters that could have been used, but this example is mainly meant to illustrate how incorporating a texture filter can improve the results. B. Roberts Masks As hinted above, the Roberts masks can also be used as an alternative to the forward-energy filters. Here, the goal is more about using simple filters than necessarily improving the results. Formally, we define B Roberts = 0 0, 0 0 with guidance vectors p R = [3,] T p C = [,] T p L = [,3] T which are based on those used for the forward-energy case if we define C C = d d 2, C R = d 0 and C C = d, D i,j p R = 3 d 0 (i, j) + d (i, j) = 2C R (i, j) + 2C C (i, j) D i,j p C = d 0 (i, j) + d (i, j) = 2C C (i, j) D i,j p L = d 0 (i, j) + 3 d (i, j) = 2C C (i, j) + 2C L (i, j) Despite their simplicity, we have found these filters to work quite well, comparable to B fwd and sometimes even superior (cf. Figure 8). C. Guidance Vector Weighting The guidance vectors control the weighting for each subband for a given direction; for general seam carving, they will usually be designed like the ones defined above. However, there may be cases where one might want to bias a particular direction. For example, compression methods using seam carving have been proposed which involve storing the

4 f j... W W K seam path locations [2]. By increasing the center bias, the seams will be straighter and thus decrease their entropy. While this may lead to a less optimal solution in terms of the forward-energy criterion, it may still produces comparable results and possibly better if the image contains objects with straight edges (cf. Figure 3). IV. EXTENSION TO MULTISCALE Having developed a filter bank framework for seam carving in section II, we extend our model to multiple scales using an N-level cascaded filter bank. Section IV-A outlines the filter bank; section IV-B describes how the cross-scale information is utilized; section IV-C addresses some of the design issues. A. Cascaded Filter Bank We describe the cascaded filter bank for multiscale analysis as shown in Figure 4. At each level j, the image f j is passed through the filters {W k } 0 k K. The output of the crosscorrelation of f j with is downsampled by L in both dimensions to produce f j+ which can then be filtered at the next level. Formally, w/2 Figure 4: Two levels of the cascaded filter bank. The blue shaded blocks are optional depending on design implementation. Figure 5: (left to right) Region from horse image and 4 levels of high-pass coefficients from fine to coarse for (top) a grassy area and (bottom) part of the horse s head. The coarser scales have been resized for comparison. h/2 f j+ d j+ K d j+ f j+ (x, y) = f j (Lx + a, Ly + b) (a, b) a= w/2 b= h/2... Thus, would assumedly be chosen as a low-pass filter and L should be chosen accordingly based on the type of lowpass filter. For example, if is a simple 3 3 average, then an appropriate choice for the scale factor would be L = 3. We have primarily used L = 2 for our simulations, with the exception of B fwd and B back for which we have used L = 3. The outputs of the remaining filters are the coefficients maps as described in section II and can be optionally downsampled depending on the desired implementation. For W W K f j+2 d j+2 K d j+2 example, to faithfully implement any form of the original seam carving algorithm for a single scale, downsampling should not be done. Without downsampling, this multiscale model is essentially the decimated à trous algorithm [8][0]. Usually, the à trous algorithm is presented in the context of wavelets with dyadic scaling, but here we make no formal requirements on the filter bank or scale factor. If downsampling is implemented, this model is very similar to the discrete wavelet transform, but again generalized to any filter bank. It is also important to see that there are several other possible variations of this model. For instance, one could use the normal à trous algorithm, which upsamples the filters at each level rather than downsampling the lowpass output. This would lead to improved space localization since the filtered outputs at every level would be of the same dimension; yet, such precision does not guarantee better results. Rather, we have found that the spatial location uncertainty associated with decimation to be beneficial. Generally speaking, if an image location is significant, it makes sense for a seam to avoid a larger neighborhood around that location, not just that single pixel. This has the effect of ensuring that seams stay some distance away from important features as much as possible. B. Energy Accumulation Map Having filtered the image using the cascaded filter bank, this brings us to the challenge of utilizing the information across multiple scales in some meaningful way. Up to this point we have not explicitly required that be a low-pass filter or that the remaining masks be high-pass filters (or bandpass filters covering higher frequencies); however, to facilitate our discussion, we will assume that this is the case from here on out. At each scale, the high-pass coefficient maps characterize the energy of the different features corresponding to each filter. Finer scales represent the fine details of an image the highest frequency content while coarser scales represent broader changes in the image. Consequently, the finest scales will be more susceptible to fine textures and noise which may be visually unimportant. This is exactly the case in Figure (b) the horse is much smoother than the grass and therefore makes for a less resistant path. Even though the edges of the horse produce high energy, it is not enough to make the cumulative path energy higher than a path through the grassy region. As we discussed above, the seam carving methods described in section II have the tendency to weight the fine details too heavily. On the other hand, it neither makes sense to give a lot of weight to the coarse details. Still, the goal is to give weight to significant edges and, as it is well known, such edges will span several scales. In particular, the wellestablished results by Mallat and Zhong have shown that edges appear across scales as modulus maxima, with the finest scale giving the precise location of the edge; as the scale increases, the edge diffuses [5][6]. This motivates the idea that the importance of some image point can be characterized not only by the significance of its energy at a certain scale, but also by the number of scales in which it is significant.

5 Consider the two different image regions in Figure 5 taken from the horse image. As the scale increases, the coefficients for the grassy region quickly decay, becoming nearly zero. In contrast, the coefficients for the region containing the horse s head are significant at all four scales. This brings us back to the challenge of combining the information across scales. While there are likely many ways to do this, we have found that a simple linear combination of the coefficient maps works quite well. The one challenge, of course, is that a coefficient map at level j is L times the size of one at level j + and therefore must be upsampled. This is implemented as shown in Figure 7. At each level, the coefficient map d j is normalized using q, weighted by α j, upsampled in both dimensions by L, low-pass filtered, and added to the weighted, normalized coefficient map at level j. Formally, the combination of levels j and j can be expressed as w/2 h/2 d (x, ȷ y) + d ȷ x+a a= w/2 b= h/2 L, y+b L W(a, b) where d ȷ α j q d j. If downsampling was implemented in the cascaded filter bank (see Figure 4), then we also upsample and filter at the last level to ensure that the output matches the size of the image f. We refer to the final map D k as the energy accumulation map corresponding to the filter W k (cf. Figure 6). Since it represents N scales of the coefficient map d k, we now redefine D i,j [D 0 (i, j), D (i, j),, D K (i, j)] Note that the normalization function q is absent in this definition since we have included it in the generation of the energy accumulation map. From this point, the rest of the algorithm proceeds as described in section II. C. Design While this may appear to be a fairly rigid method of utilizing the energy across scales, there are a few things that give it some flexibility. First, while there are no hard and fast rules on the number of levels N, the practical limit is given by log min{m, n} log(min{h, w} ) N log L where is the floor function. Second, the selection of the weight factors α j can be used to give more importance to certain scales if one is interested in preserving a certain spatial (a) (c) Figure 6: (a) Original image and its three energy accumulation maps using B fwd with N = 3; result of a 50% width reduction using (b) forward-energy seam carving and (c) our multiscale model with the aforementioned settings. frequency. For our purposes, we have generally given each scale equal importance, which requires something of the form, α j = L j. Also, the operator q can be adjusted to include normalization of each coefficient map, which we have done in our simulations. Specifically, q(a) = max i,j a a i,j i,j where a i,j is the i, j-th element of the matrix A. Finally, another option is that instead of linearly combining each scale, one could employ a product or some other non-linear operator. However, as mentioned above, we have found a linear combination to work well and have thus used it throughout this paper. V. RESULTS In Figure 8 we compare uniform resampling and forwardenergy seam carving to our multiscale seam carving method for the various filter banks defined above. Overall our method was more capable of avoiding important image features, and was less sensitive to fine texture. The top image shows a dramatic improvement: uniform resampling creates obvious distortion of the person s face and forward-energy seam carving nearly removes the face entirely; with either Roberts filters at four scales or the forward-energy filters at three scales, our method was able to preserve the person s face. The middle image showed similar results: with the forwardenergy filters at either three or four scales, the white horse is preserved. The bottom image is a particularly difficult case because of the many different objects and textures in it; (b) α N k q d N k L L... k α N k q d N L D k α k q d k Figure 7: Multiscale energy accumulation of the k-th subband. The blue shaded blocks are optional depending on design implementation.

6 B Roberts, N = 4 B fwd, N = B fwd, N = 3 B fwd, N = B back, N = 4 B Roberts, N = 6 (a) (b) (c) (d) (e) Figure 8: (a) Original image and a 30% width reduction using (b) uniform resampling, (c) forward-energy seam carving, and (d,e) our multiscale method. The filters and scale depths used are indicated under their respective images. uniform resampling and forward-energy seam carving again suffer the same fate obvious distortion and undesirable seam selection, respectively. Using the backward-energy filters at four scales, the duck and the person s face are avoided the primary objects in the image but the person s side is removed. The result is similar for the Roberts filters at six scales, except that the removal of the person s side is less noticeable because it happens near the image edge. In either case, this is a notable improvement over the prior methods. Overall, we have found the Roberts filters (at around five levels) and the forward-energy filters (at around three to four levels) to produce the best results. VI. CONCLUSIONS We have presented a new framework for seam carving based on filter banks. This generalized model encompasses the original implementations and provides for more design flexibility. In particular, it lays the groundwork for incorporating many different filters and their various combinations via the guidance vectors. Finally, we extended this model to multiple scales to reduce sensitivity to noise and improve seam selection. The result is a more robust seam carving algorithm. REFERENCES [] Avidan, S., and Shamir, A Seam carving for content-aware image resizing. In ACM Transactions on Graphics, Vol. 26, No. 3. [2] Avidan, S., and Shamir, A Seam carving for media retargeting. In Communications of the ACM, Vol 52, No.. [3] Grigorescu, S.E., Petkov, N., and Kruizinga, P Comparison of texture features based on Gabor filters. In IEEE Trans.Patt. Anal.Mach.Intell., Vol., No 0. [4] Guo, Y., Liu, F., Shi, J., Zhou, Z.H., and Gleicher, M Image retargeting using mesh parametrization. In IEEE Transactions on Multimedia, Vol., No. 4. [5] Mallat, S A Wavelet Tour of Signal Processing: The Sparse Way. Burlington, MA: Academic Press. [6] Mallat, S., and Zhong, S Characterization of signals from multiscale edges. In IEEE Trans.Patt.Anal.Mach.Intell., Vol. 4, No 7. [7] Randen, T., and Husoy, J.H Filtering for texture classification: a comparative study. IEEE Trans.Patt.Anal. Mach.Intell., Vol. 2, No 4. [8] Rao, R.M. and Bopardikar, A.S Wavelet transforms: Introduction to theory and applications. Reading, MA: Addison Wesley Longman, Inc. [9] Rubinstein, M., Avidan, S., and Shamir, A Improved Seam Carving for Video Retargeting. In ACM Transactions on Graphics, Vol. 27, No. 3. [0] Shensa, M.J The discrete wavelet transform: Wedding the à trous and Mallat algorithms. In IEEE Trans. Signal Processing, Vol. 40, No. 0. [] Simakov, D., Caspi, Y., Shechtman, E., and Irani, M Summarizing visual data using bidirectional similarity. In Proceedings of CVPR. [2] Tanaka, Y., Hasegawa, M., and Kato S Image coding using concentration and dilution based on seam carving with hierarchical search. In Proceedings of ICASSP. [3] Wolf, L., Guttmann, M., and Cohen-Or, D Non-homogeneous content-driven video-retargeting. In IEEE International Conference on Computer Vision (ICCV).