Free Appearance-Editing with Improved Poisson Image Cloning

Transcription

1 Bie XH, Huang HD, Wang WC. Free appearance-editing with improved Poisson image cloning. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY 26(6): Nov DOI /s Free Appearance-Editing with Improved Poisson Image Cloning Xiao-Hui Bie 1,3 ( ), Hao-Da Huang 2 ( ), and Wen-Cheng Wang 1 ( ), Member, CCF 1 State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing , China 2 Microsoft Research Asia, Beijing , China 3 Graduate University of Chinese Academy of Sciences, Beijing , China xiaohui@ios.ac.cn; hahuang@microsoft.com; whn@ios.ac.cn Received December 29, 2010; revised September 5, Abstract In this paper, we present a new edit tool for the user to conveniently preserve or freely edit the object appearance during seamless image composition. We observe that though Poisson image editing is effective for seamless image composition. Its color bleeding (the color of the target image is propagated into the source image) is not always desired in applications, and it provides no way to allow the user to edit the appearance of the source image. To make it more flexible and practical, we introduce new energy terms to control the appearance change, and integrate them into the Poisson image editing framework. The new energy function could still be realized using efficient sparse linear solvers, and the user can interactively refine the constraints. With the new tool, the user can enjoy not only seamless image composition, but also the flexibility to preserve or manipulate the appearance of the source image at the same time. This provides more potential for creating new images. Experimental results demonstrate the effectiveness of our new edit tool, with similar time cost to the original Poisson image editing. Keywords poisson image editing, appearance editing, edit propagation 1 Introduction Image composition is popular in creating new images by pasting an object or a region from a source image onto a target image. For seamless image composition, Poisson image editing [1] has been demonstrated as a very effective approach. By solving Poisson equations using the user-specified boundary condition, it can have the colors along the boundary between the source and target images changed slowly, and so blend the colors from both images without visible discontinuities around the boundary. However, such a reconstruction function interpolates the boundary conditions inwards, so the appearance of the objects in the source image would change, called color bleeding. In applications, such a color change is not always desired, because the interesting contents of the source image may be smudged to violate the desire of image composition, as illustrated in Fig.1, where the dog of the source image has its color changed in a miscellany of white and carnation with Poisson image editing. This may seriously prevent the use of Poisson image editing. For the color bleeding problem in Poisson image editing, we have not found efficient solutions in literatures till now, though it was listed as one of the future research directions in 2006 [2]. In this paper, we propose a new edit tool to well handle the color bleeding in Poisson image cloning. It cannot only well preserve the object appearance in the source image, but also allow the user to edit the object appearance freely. At first, it lets the user use strokes to specify the regions that will have their appearance preserved or edited. Then, a new energy function is designed for Poisson image editing by introducing two new energy terms, one for responding to the user s desire and the other for propagating the user s desire to the regions similar to the specified regions. In this way, the regions of interest can resist the influence from the target image so that their appearance can be preserved or edited freely. The new energy function can be still solved using the efficient sparse linear solver as in [1]. Our new tool can perform image composition quickly to soon respond to the user s interactions, and so is very helpful for the user to produce his desired images by adding constraints progressively. As shown in Fig.1, with our new edit tool, the dog can have its white color preserved or freely edited to be other pure colors. Therefore, our new edit tool provides more potential for the user to create new images with image composition. Short Paper This work was partly supported by the National Natural Science Foundation of China under Grant Nos , , Springer Science + Business Media, LLC & Science Press, China

2 1012 J. Comput. Sci. & Technol., Nov. 2011, Vol.26, No.6 Fig.1. Free appearance-editing in Poisson image cloning. Given a source image (a), it is pasted onto a target image (b), where the user draws strokes in orange to specify the dog to have its color preserved or edited. With Poisson image editing, the dog has its color changed to be in a miscellany of white and carnation (c), due to color bleeding. With our new method, the dog can retain its color (d), or be edited in yellow (e) or green (f) freely. 2 Related Work 2.1 Image Composition There are many approaches for image composition. By image matting, existing methods always require the user to provide additional constraints using a trimap [3] or a set of brush strokes [4], and some techniques [5] employ a controlled environment or a special device to reduce the inherent ambiguity of the matting problem. For seamless blending of the target and source images, the graph-cut algorithm has been adopted to find the best seams between the combined regions [6-7], and Poisson image editing is generally regarded as a more efficient technique [1]. When the source and target images are taken from different sources or shot under different conditions, simply using the above techniques may result in heavy inconsistencies. By using a multi-scale image harmonization method before composition [8], such inconsistencies can be successfully removed and more realistic composites will be produced. To make Poisson image editing more practical, many improvements have been made. In one aspect, many acceleration techniques have been proposed [9-12]. They all work very well as long as the boundary is suitable and the interior is smooth. In another aspect, boundary conditions are studied a lot. For example, Poisson matting [3] tries to efficiently treat blurry objects, and Drag-and-Drop pasting [2] enables the users to easily select the regions of interest by automatically reducing the conflict between the salient structures in the source and target images along the boundary. But none of them addresses the color bleeding problem in Poisson image editing. In comparison with these techniques, our new technique can efficiently support appearance preserving in Poisson image editing, and even appearance editing in seamless image composition. Experiments show that it is valid for a wide variety of images and able to run much quickly. This provides an efficient solution to the color bleeding problem in Poisson image editing. 2.2 Appearance Editing It is important to adjust the image appearance in image editing. To perform such adjustments in a particular region, e.g., increasing or decreasing the exposure locally, the user need first carefully select the region. To simplify the region selection process, several intuitive stroke-based image editing tools have been developed. In general, first the user marks the region of interest with a few strokes, and specifies constraints on the marked regions, then propagates the constraints to the entire image by solving an optimization problem. These methods have been proved effective for images [13-14] as well as materials [15-16]. However, they are generally limited to editing a single image or a kind of material, not addressing the editing problem in pasting a source image onto a target image. As for our new method, it also utilizes strokes for the user to specify the region of interest, and gives constraints to propagate for appearance preserving or editing. However, it tries to composite two images with appearance editing and image composition solved simultaneously in an unified framework for Poisson image cloning. Recently, Farbman et al. [17] proposed to use diffusion distances instead of generally used Euclidean distances for local edit propagation, because diffusion distances are more efficient to make edge information aware in propagating edits and helpful to promote edited results. It could be used for color composition, but, it needs two steps with the first to paste the source image onto the target image, and the second to propagate edits, where a suitable diffusion map has to be produced on the fly by setting good parameters according to the result of the first step. In comparison with this work,

3 Xiao-Hui Bie et al.: Free Appearance Editing in Image Cloning 1013 our method needs only one step without the expensive computation of the diffusion map. Thus, our method would be more convenient to composite images. 3 Free Appearance-Editing in Poisson Image Cloning We address the color bleeding problem in Poisson image editing in this section and try to propose an interactive tool for appearance editing in image composition. Using a few strokes, the user can easily specify the regions where he wants to keep or change the appearance. Then, the constraints are considered to form a new energy function to be minimized in the framework of Poisson image editing. As a result, the source image can be seamlessly blended into the target image while the specified regions can still retain their appearance or be changed as desired. 3.1 Poisson Image Editing In Poisson image editing [1], the membrane interpolant for blending the source image g and the target image f is defined as the solution of the minimization problem: min f v 2 dp with f Ω = f Ω, (1) p Ω where f is the resulting image, v = g is the guidance field for seamless cloning, Ω is the domain covered by the source image and Ω is the exterior boundary of Ω. These notations are illustrated in Fig.2. Fig.2. Guided interpolation notations [1]. Unknown function f interpolates the destination function f in domain Ω, under guidance of vector v, which might be or not be the gradient field of a source function g. For discrete images the above problem can be discretized using the underlying discrete pixel grids. Without ambiguity, suppose that S and Ω represent the pixels of the target and the source images respectively. For each pixel p in S, let N p be the set of its 4-connected neighbors which are in S, and let p, q denote a pixel pair such that q N p. The boundary of Ω is now Ω = {p S \ Ω : N p Ω }. Let f p be the value of f at p, the finite difference discretization of (1) yields the following discrete, quadratic optimization problem: for all p Ω, (f p f q v pq ) 2, with f p = fp, (2) min f Ω p,q Ω where v pq is the projection of v((p + q)/2) on the oriented edge [p, q], i.e., v pq = v((p + q)/2) pq. Its solution satisfies the following simultaneous linear equations: N p f p f q = fq + v pq, q N p Ω q N p q N p Ω for all p Ω. (3) As (3) forms a classical, sparse (banded), symmetric, and positive-definite system, it can be solved fast enough for interactive editing of medium-sized color image regions. 3.2 New Energy Function For appearance editing in Poisson image cloning, we put forward two energy terms to add to the (2), so as to form a new minimized energy function as follows: E = (f p f q v pq ) 2 + p,q Ω ( α ((f p g p ) e p ) 2 + p M β ) ((f p g p ) (f q g q )) 2 z pq, p Ω q N p (4) where α and β are two weights for the user to decide how to edit the appearance of the specified regions, M is the specified regions in the source image for appearance editing, g p represents the value at p in the original source image patch, e p represents the change that the user wants to assign at p, and z pq is for measuring the color similarity between the pixels p and q, computed in exp( ((R p R q ) 2 + (G p G q ) 2 + (B p B q ) 2 )/2σ 2 ) with σ being the variance of the color values at the pixels inside Ω. Obviously, when α = 0, it returns to the original Poisson image editing. For the term ((f p g p ) e p ) 2, it tries to retain the original value at p when e p = 0, or have the change at p aligned with e p. As for the term ((f p g p ) (f q g q )) 2 z pq, it is for propagating the editing constraints given by the user to the regions similar to the specified regions in Ω. By the new function, the solution would satisfy the following simultaneous linear equation: ( N p + α + αβ z pq )f p (1 + αβz pq )f q q N p q N p Ω

4 1014 J. Comput. Sci. & Technol., Nov. 2011, Vol.26, No.6 = q N p Ω (1 + αβz pq )f q + q N p (1 + αβz pq )v pq + αg p, for all p Ω. (5) In this way, the user s desire can be integrated into Poisson image editing for appearance preserving or manipulating in seamless image composition. 4 Results and Discussion With our new energy function, appearance editing can be easily implemented in Poisson image cloning. The related interpolations are also solved in Gauss- Seidel iteration with successive overrelaxation, as suggested in [1]. With the new tool, we conducted experiment on a wide variety of source and target images, and compared it with Poisson image editing. In the tests, α and β are set to be 1, 10 or 100 respectively. In fact, our new method is insensitive to their values by a lot of tests. By our tests on a personnel computer with an Intel Q9400 CPU and 4 G RAM, our new method ran at a similar speed to the original Poisson image editing. For example, in handling the examples Dog, Kid and Flower, our method took 0.29 s, 0.73 s and 0.25 s respectively for image composition, while our implementation Fig.3. Kid. (a) Source image. (b) Target image. (c) When the source image is pasted onto the target image, the kid is specified in strokes. (d) The membrane map to show the color change of the pixels covered by the source image patch after our new method is applied. (e) The result with Poisson image editing where the kid has her appearance changed a lot by the target image, due to color bleeding. (f) Our result where the appearance of the kid in the source image is well preserved. of original Poisson editing took 0.36 s, 0.67 s and 0.31 s respectively. Appearance Preserving. If the source and target images differ a lot in color and structure, the source image would be smudged with Poisson image editing, as shown in Fig.3(e). But with our new edit tool, the regions of interest can preserve the appearance shown in Fig.3(f) after they are specified in strokes in Fig.3(c). In Fig.3(d), the membrane map is given to show the color change in the source image with our new method, where the colors correspond to the differences in R, G and B at the pixels respectively. It is obvious that the specified regions have little or no change. This demonstrates that our new method can efficiently preserve appearance in Poisson image cloning. Appearance Editing. Because our new method can make the regions of interest resist the influence from the target image, forming independent areas irrespective of the blending of the source and target images, the user can freely edit the appearance of the regions of interest, as shown in Fig.1. As discussed in [2], the source image of fog plus a chimney is not easy to handle image matting. With such an image, our new method can easily preserve the appearance of the white fog, as shown in Fig.4(d), or edit the fog in other colors in Figs. 4(e) 4(g), respectively. Selective Preservation with Seamless Blending. Not all the color bleeding effects are disadvantageous to image composition. Sometimes, color bleeding effects by Poisson image editing are very helpful to create a photo realistic composite. With our method, we can selectively maintain the favorable color bleeding effects while remove the unfavorable ones by the user s intention. However, this is unable to achieve with image matting, since image matting cannot obtain color bleeding effects. As illustrated in Fig.5, our method is able to maintain the color bleeding effects under the water to achieve a seamless composition, while preserves the original appearance of the boy s head in the source image. But for matting, as illustrated in Figs. 5(e) 5(h), it is difficult to find a suitable matte to produce a photo realistic composite. Limitation. By our new method, the constraints that the user gives to the regions of interest will be propagated to the similar regions. When the regions of interest are similar to other regions in color, the edit constraints may be propagated to other regions to make the composite inconsistent along the boundary. 5 Conclusions Color bleeding in Poisson image cloning is not always desired in applications, which prevents the wide use of Poisson image editing. With regard to this, this

5 Xiao-Hui Bie et al.: Free Appearance Editing in Image Cloning 1015 Fig.4. Fog. (a) Source image. (b) The fog is specified in strokes when the source image patch is pasted onto the target image. (c) The result with Poisson image editing, where the fog has its color changed due to color bleeding. (d) Our result with appearance editing, where the fog remains its color in the source image. With our new method, the fog can be freely edited in various colors, as shown in (e) (g) respectively. Fig.5. Swimming boy. (a) Source image. (b) The boy s head is specified in strokes after the source image patch is pasted onto the target image. (c) The result with Poisson image editing, where the boy s head has its color changed due to color bleeding. (d) Our result with the appearance of the boy s head well preserved, while the boy s body has the appearance changed to achieve a seamless composition. (e) The alpha matte computed by the matting method [4] when we specify the boy s head, breast and left hand as foreground. (f) The result with the alpha matte obtained in (e), where much of the boy s body under the water cannot be seen. (g) Another alpha matte when we specify all the boy s body as foreground. (h) The result with the alpha matte obtained in (g), where the boundary of the matte is obvious and the source and target images are not blended seamlessly. paper proposed two new energy terms for Poisson image editing to well preserve or even freely edit the appearance of the regions of interest in the source image, while composing images seamlessly. Thus, it provides more potential for the user to create new images. This is a general framework. It can not only freely edit appearance in image composition, but also easily return to the original Poisson image editing when required. Experimental results have demonstrated its effectiveness. References [1] Pérez P, Gangnet M, Blake A. Poisson image editing. ACM Trans. Graph, 2003, 22(3): [2] Jia J, Sun J, Tang C K, Shum H Y. Drag-and-drop pasting. ACM Trans. Graph, 2006, 25(3):

6 1016 J. Comput. Sci. & Technol., Nov. 2011, Vol.26, No.6 [3] Sun J, Jia J, Tang C K, Shum H Y. Poisson matting. ACM Trans. Graph, 2004, 23(3): [4] Levin A, Rva-acha A, Lischinski D. Spectral matting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(10): [5] Mcguire M, Matusik W, Pfister H, Hughes J F, Durand F. Defocus video matting. ACM Trans. Graph, 2005, 24(3): [6] Boykov Y, Jolly M. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In Proc. ICCV 2001, Vancouver, Canada, July 7-14, 2001, pp [7] Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M. Interactive digital photomontage. ACM Trans. Graph, 2004, 23(3): [8] Sunkavalli K, Johson M K, Matusik W, Pfister H. Multi-scale image harmonization. ACM Trans. Graph, 2010, 29(4). [9] Kazhdan M, Hoppe H. Streaming multigrid for gradientdomain operations on large images. ACM Trans. Graph, 2008, 27(3). [10] Agarwala A. Efficient gradient-domain compositing using quadtrees. ACM Trans. Graph, 2007, 27(3). [11] Farbman Z, Hoffer G, Lipman Y, Cohen-Or D, Lichinski D. Coordinates for instant image cloning. ACM Trans. Graph, 2009, 28(3). [12] Jeschke S, Cline D, Wonka P. A GPU laplacian solver for diffusion curves and poisson image editing. ACM Trans. Graph, 2009, 28(5). [13] Levin A, Lischinski D, Weiss Y. Colorization using optimization. ACM Trans. Graph, 2004, 23(3): [14] Lischinshi D, Farbman Z, Uyttendaele M, Szeliski R. Interactive local adjustment of tonal values. ACM Trans. Graph, 2006, 25(3): [15] Pellacini F, Lawrence J. Appwand: Editing measured materials using appearance-driven optimization. ACM Trans. Graph. 2007, 26(3). [16] An X, Pellacini F. Appprop: All-pairs appearance-space edit propagation. ACM Trans. Graph, 2008, 27(3). [17] Farbman Z, Fattal R, Lischinski D. Diffusion maps for edgeaware image editing. ACM Trans. Graph, 2010, 29(6). Xiao-Bui Bie received his B.S. degree in engineering mechanics from Huazhong University of Science and Technology. He is currently a Ph.D. student at the Institute of Software, Chinese Academy of Sciences. His main research interests are computer vision and interactive image/video editing. Hao-Da Huang received his Master s degree from the Institute of Software, Chinese Academy of Sciences in He is currently an associate researcher of Microsoft Research Asia. His research interests include facial performance capture, hand deformation, image analysis and editing. Wen-Cheng Wang received his Ph.D. degree from the Institute of Software, Chinese Academy of Sciences in 1998, where he is currently a professor of the State Key Laboratory of Computer Science. His research interests include computer graphics, visualization, virtual reality and expressive rendering and editing.