Intent-aware image cloning

Transcription

1 Vis Comput DOI /s ORIGINAL ARTICLE Intent-aware image cloning Xiaohui Bie Wencheng Wang Hanqiu Sun Haoda Huang Minying Zhang Springer-Verlag Berlin Heidelberg 2013 Abstract Currently, gradient domain methods are popular for producing seamless cloning of a source image patch into a target image. However, structure conflicts between the source image patch and the target image may generate artifacts, preventing the general practices. In this paper, we tackle the challenge by incorporating the users intent in outlining the source patch, where the boundary drawn generally has different appearances from the objects of interest. We first reveal that artifacts exist in the over-included region, the region outside the objects of interest in the source patch. Then we use the diversity from the boundary to approximately distinguish the objects from the over-included region, and design a new algorithm to make the target image adaptively take effects in blending. So the structure conflicts can be efficiently suppressed to remove the artifacts around the objects of interest in the composite result. Moreover, we develop an interpolation measure to composite the final image rather than solving a Poisson equation, and speed up the interpolation by treating pixels in clusters and using hierarchical sampling techniques. Our method is simple to use for X. Bie ( ) W. Wang M. Zhang State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, , China xiaohui@ios.ac.cn W. Wang ( ) whn@ios.ac.cn H. Sun Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, China H. Huang Google Inc., Mountain View, CA, 94043, USA X. Bie M. Zhang University of Chinese Academy of Sciences, Beijing, China instant and high-quality image cloning, in which users only need to outline a region of interested objects to process. Our experimental results have demonstrated the effectiveness of our cloning method. Keywords Intent representation Image cloning Image composition 1 Introduction Image composition has been widely applied for image generation and editing, which creates new images by pasting an object/region from a source image into a target image. For such needs, gradient domain methods [7, 10, 15] arewell known for their user-friendly interaction, by roughly sketching a boundary to select a region of interest in the source image. These methods work by solving a Poisson equation with Dirichlet boundary conditions. It is in fact to construct a harmonic interpolant (a smooth membrane) for diffusing the intensity discrepancies between the source and target images along the boundary of the patch to the entire cloned region. Thus, the intensity discrepancies on the boundary can be reduced considerably, achieving a seamless blending of the source image patch and the target image. However, artifacts may be resulted when the source patch has different structures from the target image. This is due to the fact that a smooth membrane cannot change the structure of the source image patch, which causes the structure differences noticeable in such composite, shown as artifacts. Though techniques have been proposed to optimize the conditions on the boundary for improving Poisson image cloning, such as the drag-and-drop pasting system [10] and using meanvalue coordinates [7], they cannot efficiently remove the

2 X. Bie et al. structure conflicts inside the patch, due to the produced a smooth membrane for image composition. Our work here is aimed at solving the structure conflicts in image cloning that could not be efficiently solved using existing techniques. We observe that structure conflicts in the source patch occur in the region outside the objects of interest, called over-included region. By image composition, we expect that the objects of interest are pasted into the target image, without the other contents. The existing gradient domain techniques let the source patch preserve its structures in the composite, by forming a smooth membrane to have the pixels inside the patch alter their intensities in similar extents. When the structure of the over-included region differs from that of the target image, the difference can be apparently noticeable and cause visual artifacts in final images. Our work is motivated to eliminate such artifacts of the over-included region in cloning images, so that structure conflicts will not appear around the objects of interest in the final composite. Unlike matting methods to produce a matte for image composition, we propose the novel cloning method that can re-produce the natural intent-aware cloning effects in composited images. It has a seamless blending of the source patch and the target image, with the colors of the source patch corrected by the target image. At the same time, we save laborious labeling work that are required for getting a matte. The pixels of the target image covered by the source patch are all allowed to take effect in image composition, unlike existing gradient domain methods to consider only the pixels of the target image along the boundary of the source patch. Initiated by [7] to interpolate the value at each interior pixel for fast image composition, we develop the interpolant for fast image cloning by executing the interpolant in clusters of pixels, and using hierarchical sampling techniques for further acceleration. In summary, our specific contributions are: An efficient method for cloning images with different structures, which extends the current methods to the general cases in selecting the source/target images for seamless cloning, making image composition more flexible and widely applicable. An easy-to-use approach for image composition, where the users only need to simply outline the region of interest in the source image, not aware of any structure conflicts between the source and target images. The approach is simple to use, fast, and also simple to develop for parallel computing. For the rest of the paper, we briefly review the related work in matting, gradient domain and image composition techniques in Sect. 2. The essential ideas in our intent-aware image cloning are presented in Sect. 3, including distinguishing pixels and blending images, and efficient processes in pixels clustering and hierarchical sampling are given in Sect. 4. The image cloning examples and experimental performance are described in Sect. 5, as well as the discussion of limitations. Finally, the summary is given in Sect Related work Matting methods are commonly used for image composition. They work by linearly interpolating the source and target image using weights specified by the alpha matte, and their effectiveness depends on the accuracy of the matte. Thus, some methods require the user to provide additional constraints using a trimap [8, 17] orasetofbrushstrokes to carefully get the matte [12, 13]. Others [14, 16] employ a controlled environment or a special device to reduce the inherent ambiguity of the matting problem. To leverage the knowledge of the target background to create a more successful matte, Wang et al. [20] proposed to combine matting and compositing into a single optimization process. However, matting methods cannot have the colors of the objects of interests corrected by the target image, which may lead to unnatural looking, and it is always very laborious to get a quality matte in treating complex objects, which prevents their interactive use. Gradient domain compositing methods are well-known for their ease to use and effectiveness on seamless composition. By solving the Poisson equation with the user-specified boundary condition, Poisson image cloning [15] can achieve seamless composition without visual seam along the boundary. To make Poisson image cloning more practical, many improvements have been made. In one aspect, many acceleration techniques [1, 7, 9, 11, 19] are proposed, and some [7, 21] even extended to seamless video composition. In another aspect, many approaches were studied to improve the composition quality, such as optimizing the boundary conditions in the Drag-and-Drop pasting system [10], or incorporating alpha matting to get hybrid solutions for texture smudging and color dis-matching problems [3, 5, 6, 22]. Our method also solves texture smudging and color dismatching problems, but compared with the hybrid solutions [3, 5, 6, 22], our method needs no user interaction to approximate the objects of interest, and can easily treat multiple objects of interests and complex structures in a source image patch, so more efficient for practical use. Recently, Sunkacalli et al. [18] proposed to harmonize the visual appearance of the source and target images before composition, to treat the inconsistency inside the composition region for a quality composite, where harmonization is achieved by transferring the appearance of one image to another. Since harmonizing visual appearance cannot effectively deal with structure conflicts between the source and

3 Intent-aware image cloning the target images, such a treatment may still produce evident artifacts around the interested objects in the output image. With regard to this, Darabi et al. [4] suggested to meld the source and target image using a synthesized transition region between the two images, to have inconsistent color, texture, and structural properties all change gradually from one source to the other for a smooth blending. However, the generated contents in the transition region may be different from the contents in the source and target images, so causing artifacts to impair the composite, though in some cases such transitions are valuable as treating the image of a squirrel inatreeholein[4]. Besides, these methods are quite time consuming, preventing their practical use. 3 Intent-aware image cloning We assume that the source patch Ω with a boundary Ω,in which there are some objects of interest Ω obj, is selected to paste into the target image, as illustrated in Fig. 1. As stated early, the artifacts resulted from structure conflicts always take place in the over-included region. If the objects of interest can be segmented precisely to paste onto the target image, no structure conflicts will appear to impair the final composite. However, precise segmentation of objects are often expensive and sometime even impossible. Thus, a simple outline of the source image patch is certain to have the over-included region. To efficiently remove the artifacts in the over-included region, we propose a novel gradient domain method that can suppress structure conflicts in the over-included region. It is a probability-based method that approximates the over-included region of the source image patch to guide image composition, and takes an interpolation computation to blend the source patch and the target image. In our approach, the surroundings of the objects of interest have the appearances determined mainly by the target image, so the structure conflicts are suppressed to achieve the natural and high-quality final composites. 3.1 Distinguishing pixels As we know, when the users outline a source patch for image composition, it is true that the objects of interest are always inside the patch, and the boundary usually has quite different appearances from the objects of interest. Based on the observation, we can estimate the probability for a pixel to be in the over-included region or not in the source image patch. To estimate such a probability for a pixel i, we first measure the affinity between the pixel i and any pixel j on the boundary, by the intensity difference and the spatial distance between them, and then get the probability by summing up all the affinity values for this pixel. If the pixel is farther from the boundary and with a larger appearance difference from the Fig. 1 For a source image patch, Ω\Ω obj is the over-included region that may have structure conflicts to impair the final composite quality boundary, it is more likely to be of the objects of interest, otherwise, more likely in the over-included region. As suggested in [2], the affinity z(i, j) between two pixels i, j is computed in the following formula: z(i, j) = exp ( f i f j 2) (1) where f i and f j are the feature vectors at pixels i and j, respectively, comprising their positions p i and p j (e.g., in x and y coordinates) and their appearances c i and c j (e.g., color in the Lab space) weighted by parameters σ p, σ c,expressed as f i = (p i /σ p, c i /σ c ) and f j = (p j /σ p, c j /σ c ). The parameters σ p and σ c are used to adjust the effect of the position difference and the appearance difference in distinguishing pixels, taking values between 0.0 and 1.0. In our tests in this paper, we generally have σ p = 1.0 and σ c = 0.2, shown with good experimental results. Finally, the probability S(i) for pixel i is computed as S(i) = γ (i, j)z(i, j) (2) j Ω where γ(i,j) is a spatial weighting, as suggested in [7] to enhance the effect of distances between pixels. It is computed as the mean value coordinates of pixel i with respect to boundary pixel j, as follows: / γ(i,j)= w j w j (3) where j Ω w j = tan(α j 1/2) + tan(α j+1 /2) p j p i and α j is the angle p j p i p j+1, as illustrated in Fig. 2. If the probability value at a pixel is smaller, the pixel is more likely to be of an object of interest, otherwise in the over-included region. By testing the images in our experiments, we find that the measure is very effective to approximate the over-included region in the source patch. Figure 3 illustrates three over-included region examples, which layered the solid basis to further develop our intent-aware image cloning method. (4)

4 X. Bie et al. Fig. 2 Angle definitions for mean-value coordinates Fig. 3 Separating the objects of interest from the over-included region: top row shows the source image patches, and bottom row shows the separation results 3.2 Blending images To blend the source image patch and the target image, Poisson image editing [15] interpolates the intensity function g of the source image and the intensity function of the target image f by taking into account the boundary condition, under guidance of the gradient field of the source image patch g. It is required to compute a function f by solving the Poisson equation: f = div g with f Ω = f (5) It is equivalent to solving the Laplace equation: f = 0 with f Ω = f g (6) and the final outcome of cloning can then simply be obtained as f = g + f. (7) This means that a smooth membrane f is constructed to diffuse the intensity difference f g between the target and source images on the boundary of the source image patch across the entire region Ω. As introduced earlier, the formulation may introduce structure conflicts in the overincluded region. To suppress structure conflicts in the region, we propose the intent aware interpolation membrane r instead of f to take different measures for the pixels in the over-included region and of the objects of interest, respectively. Consider apixeli Ω with a boundary Ω. The value of our membrane at this pixel r(i) is computed in the equation r(i) = 1 z(i, j) ( f g ) (i) + λ(i, j) ( f g ) (j) (8) N j Ω where N is the number of boundary pixels, and λ(i, j) = exp( s z(i, j)) is used to control how the intensity difference on the boundary between the source image patch and the target image take effect in cloning, and its coefficient s is for adjusting the controlling strength. In most cases, s is set to be 2.0 to allow the objects of interest to attain similar intensity diffusion as Poisson image editing. Often, not in all the cases such intensity diffusion on objects is desirable and positive. With regard to this, we permit the users to adjust the value of s to achieve the satisfactory composites, more discussion is given in the experimental results. With Eq. (8), we do not need to explicitly extract the over-included region, but construct a continuous membrane to treat the pixels in the over-included region or of the interested objects adaptively. Here, the probability for distinguishing pixel i is implicitly considered for cloning by using the affinity between the pixel and any pixel j on the boundary, and such an uniform treatment is convenient for computing the membrane. When pixel i is inside the over-included region, its z(i, j) is very possible to be high, so that the appearance of pixel i in the target image will take effect, and represented as (f g)(i), and λ(i, j) will be very small to reduce or almost remove the effect of the intensity difference on the boundary, (f g)(i), in computing the value at pixel i in the composite. On the contrary, when pixel i is of an object of interest, its z(i, j) will be very low, and λ(i, j) will be very high, so that the appearance of pixel i in the target image will take very little effect in image composition, and the appearance difference on the boundary will play an important role to have the objects of interest remain their structures in the source image. Using our membrane r computed in Eq. (8), the final result is computed by f = g + r (9) With Eq. (9), it is clear that the target image can print its structures in the region covered by the over-included region of the source image patch, and the objects of interest in the source patch can be also pasted into the target image seamlessly. 4 Efficient processing Using Eqs. (8) and (9) for image cloning, the computation complexity is O(nm), where n is the number of the pixels in the cloned region, and m is the number of boundary pixels. The process is rather time-consuming. To speed up the

5 Intent-aware image cloning processing, we present the efficient approach by grouping pixels in clusters and computing the affinity values of the interior pixels with hierarchical sampling. 4.1 Grouping pixels in clusters In the source image patch, the pixels with similar appearances will have their appearances changed similarly in the composite. Thus, we can group the pixels into clusters to speed up the cloning composition. In our treatment, the k- means clustering is applied in the 5-dimensional affinity space, where each pixel i is classified according to it s 5- dimensional feature vector f i, and ANN searching is used for acceleration. We then use the clusters to fast compute the membrane for image cloning. In computing the membrane, it is first approximately computed the membrane values for the clusters by their center pixels, respectively, and then these values are propagated to the pixels individually according to the affinities between the pixel and the center pixels of the clusters and the probability for distinguishing the pixel, to obtain the final membrane. In approximating the membrane value for a cluster, we compute it by modifying Eq. (8) as R(k) = 1 N z(k, j) ( f g ) (k) + λ(k, j) ( f g ) (j) j Ω (10) where k is the center pixel of a cluster. In computing the membrane value at a pixel, we select several clusters whose center pixels are very like the pixel to do approximation. As clustering is a probability-based computation, several clusters with very similar appearances can facilitate to reduce the disturbance from the clusters with very different appearances in approximation. In general, three clusters with the most similar appearances are sufficient to get very good composites, according to our tests. With the selected clusters, we first approximate the probability for distinguishing the pixel i according to the affinity values between the center points of the clusters and the pixels on the boundary. It is computed in the following two equations: S(k) = γ(k,j)z(k,j) (11) j Ω and A(i) = / z(i,k)s(k) z(i, k) (12) k C k C where A(i) is the approximated probability for pixel i, and C is the set of the selected clusters. Afterwards, with the approximated probability, we first compute how the membrane values of the selected clusters Fig. 4 Hierarchical sampling: the nearer the portion of the boundary to pixel A, the more the selected pixels in the portion used for distinguishing pixel A are propagated to pixel i in Eq. (13), and then get the membrane value at pixel i in Eq. (14), as expressed in the below. r (i) = / z(i,k)r(k) z(i, k) (13) k C k C r(i) = A(i) ( f g ) (i) + exp ( s A(i) ) r (i) (14) Finally, the cloning composite is obtained by Eq. (15) f = g + r (15) 4.2 Hierarchical sampling Using Eq. (2) to compute the probability for distinguishing a pixel, all the pixels on the boundary are required to investigate, which is very expensive. Motivated by the work [7], we take hierarchical sampling to select some pixels on the boundary to approximate the probability. The number of selected pixels on a portion of the boundary is inversely proportional to the distance from the portion to the pixel under investigation. With hierarchical sampling, considerable acceleration can be obtained, as discussed in [7]. An additional benefit is that small un-continuous regions intersecting the boundary can be excluded from the objects of interest, helping more to simply outlining of the source image patch for image cloning. As illustrated in Fig. 4, the pixels in the green region can be easily determined in the over-included region. In Fig. 5, an example is given to show that small regions intersecting the boundary can be efficiently excluded from the composite using hierarchical sampling, which will result in artifacts using uniform sampling. 5 Comparison and discussion We have applied our method on a variety of source and target images, and also compare the results with Poisson image editing [15], alpha matting [12], drag-and-drop pasting [10], and image melding [4].

6 X. Bie et al. Fig. 5 Hierarchical sampling is more helpful to detect the overincluded region than uniform sampling. (a) the source and target images; (b) by Poisson cloning; (c) with uniform sampling, note that the artifacts above the feet in the cloning region; (d) with hierarchical sampling, such artifacts are well removed in the composite Fig. 6 When the source image patches are pasted onto the target image (a), Poisson image editing causes blurring artifacts around the objects of interest in the composites (b), matting compositing brings the colors of the source image evidently into the composites (c), but our results are in high quality without these artifacts (d) Compared with Poisson image cloning Our method can efficiently remove the artifacts resulted from structure conflicts. As shown in Fig. 6 and Fig. 7, Poisson image cloning preserves the structures of the source image patch, so that evident artifacts appeared around the objects of interest in the final composite, while our method can produce better results with natural looking. Besides, our method allows the user to freely adjust the tone of the objects of interest until a satisfactory result is attained. As shown in Fig. 9, the color bleeding effect makes results of Poisson image cloning look unnatural, while with our method, the user can progressively optimize the result by modifying the parameter s (Fig. 9(d f)). Compared with alpha matting Our method has two main advantages. First, alpha matting relies on user s careful initialization work to extract an accurate alpha matte, which requires experience and usually involves intensive interactions. And sometimes it s difficult even impossible to attain an accurate alpha matte, as shown in the warship and fireworks examples in Fig. 6, where obvious artifacts resulted from un-clean alpha mattes make the composites far from realistic (Fig. 6(c)). However, using our method, the interactions are much more intuitive and simple, and our results are much better. Second, alpha blending directly copies the color of the foreground objects from the source image, so that tone dis-matching between the objects of interest and the target scene may appear, impairing the reality of the composite, as shown in the footprints example in Fig. 7(d). Fortunately, in our result, the color of the source patch will be corrected by the target image, so that our results are much better with natural looking, as shown in Fig. 7(e). Compared with drag-and-drop pasting Our method outperforms it in two aspects. First, drag-and-drop pasting sys-

7 Intent-aware image cloning Fig. 7 When the source image patches are pasted onto the target image (a), Poisson image editing generates unnatural structure properties around the interested objects (b), drag-and-drop pasting cannot paste multiple footprints simultaneously into the desert and exclude the overincluded region by a boundary optimization method (c), matting compositing cannot change the colors of the interested objects for a natural looking (d), butourresultedimagesareverygood withouttheseshortcomings (e) tem can avoid salient structure conflicts by optimizing the boundary, however, it is not suitable for treating multiple objects, as shown in the footprints example in Fig. 7(c), and when applied to the source image patch which has complex structures, drag-and-drop pasting system cannot optimize the boundary efficiently, and will lead to artifacts, as shown in the insect example in Fig. 7(c). Second, drag-anddrop pasting system requires the user to provide additional inputs to extract the objects of interest by using a grabcut method, while our method does not need any additional inputs. Compared with image melding Our method can bring the user two benefits. First, our method can save the user much selection work. As shown in Fig. 8, in which the results by image melding are copied from ref. [4], with image melding, the user should carefully select the objects of interest and the blending region as Fig. 8(b). As for our method, we only need a simple outline of the source image patch and paste it into the target image, shown in Fig. 8(c), which is intuitive and more easy to use. Second, as discussed in [4], the texture interpolation between very different textures in a large background margin around the object may result in artifacts, as shown in Fig. 8(d), where some appearances of the source image in the blending region may be preserved in the final result, smudging the final composite. But with our method, the wave properties in the desert are kept fairly well around the child, shown in Fig. 8(e). Besides, our method are more fast than image melding, so that it is more suitable for interactive image composition. 5.1 Performance In our system, we have developed intent-aware image cloning in two stages. Once a source patch is selected, the pixels are grouped into clusters. For each pixel, we search its three nearest clusters and compute the affinity values between the pixel and the clusters. These are computed in the preprocessing stage and used repeatedly in our image cloning later. When the source patch is pasted into a target image, the blending stage starts to run. Here, we computes the discrepancies f g on the boundary, the membrane value for interpolation, and the final composite using Eqs. (10) (15). In Table 1, we list the statistics data on the performance of our intent-aware image cloning method (preprocessing, compositing), not including disk I/O time. The experimental tests are performed on a personal computer with an Intel Q9400 CPU and 4 GB RAM. From the statistics data, our cloning method can perform image composition instantly. Clearly, for a source image patch used repeatedly, we can execute image composition in real time. The timing performance shows the feasibility of using our method for online and interactive media applications. Limitations Our method is based on the assumption that the objects of interest have different appearances from the

8 X. Bie et al. Fig. 8 Source and target image (a). The initialization work of image melding (b), where the user should carefully identify the kid and mark the blending region in magenta. The inputs of our method (c), a simple outline of the source patch and the target image. The result by image melding has the desert around the kid in a mess (d). Our result has a very realistic looking (e) Fig. 9 When the source patches are pasted onto the target image (a), Poisson image editing generates artifacts around the objects of interest (b), and our results have no structural artifacts and can have the tone adjusted freely with different parameters s for good looking (d f) Table 1 Performance statistics for intent-aware image cloning Examples Cloned pixels Boundary pixels Preprocessing time (ms) warship 107,231 1, insect 139,309 1, fireworks 418,080 2, footprints 136,024 1, kid (Fig. 8) 94,441 1, kid2 (Fig. 9) 73, swimming 85,651 1, Run time (ms) overall appearance of the boundary, so that the over-included region can be approximated to reduce their effects in composite, by which structure conflicts can be compressed to improve image cloning. It works very well on a wide variety of source and target images. However, this may not always be true. When the objects of interest have some parts with similar appearances as the boundary, our method will fail in separating them from the over-included region. As illustrated in Fig. 10, the dog has quite similar colors with the turbid water, so that our method may mistake some parts of the dog as in the over-included region, and have them disappeared in the composite (b). With regard this problem, we need a further study. Some possible ways may be investigated, such as having the user draw some strokes to refine the obtained objects of interest, or incorporating some advanced saliency detection techniques to improve obtainment of the objects of interest. Another case that may cause problems is that some objects of non-interest have very different appearances from the boundary. These objects may be mistaken as interest and lead to structure conflicts. As these objects are in very different appearances from the boundary, they are easy to recognize and so removed using image inpainting tools, after they are identified non-interest. However, how to efficiently distinguish these objects from the objects of interest need a further study. 6 Summary Gradient domain techniques are well known for userfriendly and seamless blending of images in image composition. However, existing techniques cannot well handle the structure conflicts between the source and target images, leading to artifacts in cloning composites thus preventing

9 Intent-aware image cloning Fig. 10 In the source image patch, the swimming dog has some parts in very similar colors with the turbid water. In this case, our method may fail in distinguishing these parts from the over-included region, making them rendered as transparent (b) practical uses. In this paper, we present the novel intentaware cloning method to address the challenging of structure conflicts in image cloning. Based on the observation that structure conflicts which will result in artifacts are mostly in the over-included region, we design the cloning method to suppress such structure conflicts in the over-included region to remove artifacts in the composite, in which the user s intent in outlining the source image patch is used efficiently. We then develop the interpolant for blending images, which incorporates the target image to efficiently suppress the structure conflicts in the cloning composite. Furthermore, our processes are accelerated by grouping pixels in clusters and also using hierarchical sampling techniques. By experiments, our intent-aware method can produce image cloning in real time, and without structure conflicts in natural and high-quality looking. Further, we will extend our cloning method to the latest GPUs and fast video processing for online/mobile applications, which will be studied in the near future. Acknowledgements The work is supported by the Knowledge Innovation Program of the Chinese Academy of Sciences, the National Social Science Foundation of China (Project No. 12AZD118), RGC research grants (ref , ) and UGC direct grant for research (no ). References 1. Agarwala, A.: Efficient gradient-domain compositing using quadtrees. In: SIGGRAPH 07: ACM SIGGRAPH, p. 94. ACM, New York (2007) 2. An, X., Pellacini, F.: Appprop: all-pairs appearance space edit propagation. In: SIGGRAPH 08: ACM SIGGRAPH, pp ACM, New York, NY, USA (2008) 3. Chen, T., Cheng, M.M., Tan, P., Shamir, A., Hu, S.M.: Sketch2photo: Internet image montage. ACM Trans. Graph. 28(5), 124:1 124:10 (2009) 4. Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31(4), 82:1 82:10 (2012) (Proceedings of SIGGRAPH 2012) 5. Ding, M., Tong, R.F.: Content-aware copying and pasting in images. Vis. Comput. 26(6 8), (2010) 6. Du, H., Jin, X.: Object cloning using constrained mean value interpolation. Vis. Comput. 29(3), (2013) 7. Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Lischinski, D.: Coordinates for instant image cloning. ACM Trans. Graph. 28(3), 1 9 (2009) 8. Gastal, E.S.L., Oliveira, M.M.: Shared sampling for real-time alpha matting. Comput. Graph. Forum 29(2), (2010) 9. Jeschke, S., Cline, D., Wonka, P.: A GPU Laplacian solver for diffusion curves and Poisson image editing. In: SIGGRAPH Asia 09: ACM SIGGRAPH Asia, pp ACM, New York (2009) 10. Jia, J., Sun, J., Tang, C.K., Shum, H.Y.: Drag-and-drop pasting. In: SIGGRAPH 06: ACM SIGGRAPH, pp ACM, New York (2006) 11. Kazhdan, M., Hoppe, H.: Streaming multigrid for gradientdomain operations on large images. ACM Trans. Graph. 27(3), 1 10 (2008) 12. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), (2008) 13. Levin, A., Rav-Acha, A., Lischinski, D.: Spectral matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(10), (2008) 14. McGuire, M., Matusik, W., Pfister, H., Hughes, J.F., Durand, F.: Defocus video matting. In: SIGGRAPH 05: ACM SIGGRAPH, pp ACM, New York (2005) 15. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), (2003) 16. Smith, A.R., Blinn, J.F.: Blue screen matting. In: SIGGRAPH 96: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp ACM, New York (1996) 17. Sun, J., Jia, J., Tang, C.K., Shum, H.Y.: Poisson matting. In: SIG- GRAPH 04: ACM SIGGRAPH, pp ACM, New York (2004) 18. Sunkavalli, K., Johnson, M.K., Matusik, W., Pfister, H.: Multiscale image harmonization. ACM Trans. Graph. 29(4), 125:1 125:10 (2010) (Proc. ACM SIGGRAPH) 19. Szeliski, R.: Locally adapted hierarchical basis preconditioning. In: SIGGRAPH 06: ACM SIGGRAPH, pp ACM, New York (2006) 20. Wang, J., Cohen, M.F.: Simultaneous matting and compositing. In: ACM SIGGRAPH 2006 Sketches, SIGGRAPH 06. ACM, New York (2006) 21. Xie, Z.F., Shen, Y., Ma, L.Z., Chen, Z.H.: Seamless video composition using optimized mean-value cloning. Vis. Comput. 26(6 8), (2010) 22. Zhang, Y., Tong, R.: Environment-sensitive cloning in images. Vis. Comput. 27(6 8), (2011) Xiaohui Bie received his bachelor degree in engineering mechanics form the Huazhong University of Science and Technology. He is currently a Ph.D. student at the Institute of Software, Chinese Academy of Sciences. His main research interests are computer vision and interactive image/video editing.

10 X. Bie et al. Wencheng Wang received the Ph.D. degree from the Institute of Software, Chinese Academy of Sciences in 1998, and been working as a professor at this institute from His research interests include computer graphics, visual analytics, and expressive rendering and editing. Haoda Huang received his Master degree from the Institute of Software, Chinese Academy of Sciences in After that, he joined Microsoft Research Asia and work there for about 5 years. He is currently a software engineer in Google Mountain View. His research interests include facial performance capture, hand deformation, image analysis and editing. Hanqiu Sun received the M.S. degree in electrical engineering from the University of British Columbia and the Ph.D. degree in computer science from the University of Alberta, Canada. She is now an associate professor in the Department of Computer Science and Engineering, Chinese University of Hong Kong (CUHK). Her current research interests include virtual and augmented reality, interactive graphics/animation, hypermedia, telemedicine, mobile image/video processing and navigation and realistic haptics simulation. Minying Zhang received his B.E. degree from Jilin University, P.R. China, in He is currently working toward the Ph.D. degree in computer science at the State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences. His research interests include image and video processing and computer graphics.