Portraits Using Texture Transfer

Transcription

1 Portraits Using Texture Transfer Kenneth Jones Department of Computer Science University of Wisconsin Madison, USA ABSTRACT Texture transfer using a homogenous texture source image (e.g., white rice) can produce interesting results, particularly when applied to portraits. Skin and hair elements in portraits have distinct qualities, thus facial rendering might be more visually appealing if heterogeneous texture sources (e.g., partially mixed brown and white rice) were used to better represent such contrasting features. This paper proposes extensions to the basic texture transfer algorithm to better address this aspect. The goal is to synthesize more recognizable photo-realistic facial rendering using enhanced texture transfer techniques. Keywords: Texture Synthesis, Texture Transfer, Image Quilting, Facial Rendering 1. INTRODUCTION In Image Quilting for Texture Synthesis and Transfer, by Alexei Efros and William Freeman, a simple example-based algorithm is described that synthesizes a texture image by stitching together small tiles of an existing image. In an effort to hide the seams in this image quilt, they first overlap the image tiles by a parameterized percentage. To find a good match, they search through the source image for tiles that minimize a given distance metric in these overlapping regions to a certain threshold, randomly choosing one from the resulting set. To further reduce visible seems, they then find a minimum cost path along the overlap, which effectively becomes the boundary mask between the two tiles. The authors then go on to describe a method called texture transfer, which constrains the synthesis algorithm by requiring that each tile also satisfies an additional spatial map, referred to as a correspondence map, of some corresponding quantity over both the texture source and additional controlling target image. A particularly striking synthesized image from the paper, shown at the top of the next page, resulted from using the intensity values as the correspondence map to perform texture transfer of rice onto a portrait. The correspondence map may include information such as image intensity,

2 blurred image intensity, local image orientation, or other derived properties. This paper focuses on producing more visually appealing texture transfer portraits by exploring techniques to better render distinct qualities such as skin and hair elements using the contrasting features found in heterogeneous texture sources (e.g., partially mixed brown and white rice). Original Texture Transfer 2. MOTIVATION The general method of texture synthesis helps solve the need for photo-realistic textures by professionals in the game and film studios who need to produce high quality textures. Manual construction of textures by using scanned images is wrought with difficulties: they may not be the right size; they may contain non-uniform lighting, shadows, or geometry; and they may produce visible seams or repetitious elements if directly used for the construction of textures. Alternatively, texture synthesis allows the automatic construct of arbitrarily sized images in a general and user-friendly manner. Users typically specify a minimal number of parameters including an input exemplar image and the patch size to use during synthesis. In addition to an input texture exemplar, texture transfer also requires each patch in the output image matches an additional controlling target. Using texture transfer to perform automated artistic rendering of portraits could potentially have many commercial applications. For example, popular magazines and newspapers often feature portraits of the author near the beginning of each article, and technology oriented magazines, such as WIRED, make use of image filters to turn portraits into more visually aesthetic snapshots. This post-processing not only provides comfort to photo-shy authors but likely also adds to the bottom line by keeping an overall edgy aesthetic.

3 3. PROBLEM STATEMENT The techniques of texture quilting and texture transfer are capable of producing visually appealing photorealistic images. A variety of potential applications were presented in the original work by Efros, but in this paper we will focus on the specific task of applying texture transfer to the problem of rendering recognizable, photorealistic portraits. The task of rendering a recognizable portrait involves matching not only the large inter-element contrast between skin and hair but also the smaller intra-element contrast between differently lit areas of the face such the reflective forehead and the shaded cheek or shadowed neck. We will explore extensions to the basic technique that help to better distinguish skin from hair and better match the contrast in skin. We will also investigate post-processing the output of these techniques to render more artistic portraits. 4. RELATED WORK The general task of texture synthesis can be accomplished through several different approaches. One statistical approach views it as a problem of sampling from a probability distribution. Modeling texture as a Markov Random Field coupled with a Gibbs sampler for synthesis has been used by Zhu et. al. (1998). Gibbs sampling is commonly criticized for its lack of formal convergence measures and lack of speed. Random noise has been used to as an input to be molded into a texture by matching the filter response histograms at different spatial scales (Heeger and Bergen 1995), but they fail to capture structured texture patterns composed of larger elements. A multi-resolution filter-based method to match a texture patch at a finer scale by conditioning it on its coarser level parents has been investigated by De Bonet (1997). The technique is successful at maintaining at preserving inter-scale dependencies using a randomized sampling, but it again fails to capture higher levels of structure in the original texture and also requires texture inputs larger than the desired output, which seems somewhat self-defeating to the original desire to synthesis a larger output on a small input sample. A more mathematically sophisticated technique by Simoncelli and Portilla has produced good results, but they also do not adequately capture fine details on some highly structured patterns (1999). In contrast, the simple approach take by Efros capitalizes on forgoing the construction of a global model and instead growing an output to any size by continually sampling patches from the original texture conditioned by local constraints needed to match an overlap region (2001). This distribution constructed at each step works surprisingly well if a suitable patch size can be found, though the ideal patch size is a function of the specific texture

4 input and requires some empirical experimental to identify. The technique outlined by Efros also offers more flexibility as additional dynamic constraints can be placed on the patch search. These constraints can directly lead to the support of matching global features as seen in texture transfer. Work to render larger objects using a finite set of smaller, detailed elements has been done by Kim and Pellacini, where the task is cast into a problem of finding the best tiling (2002). This different perspective entails mathematically calculating optimal tiling arrangements with little regard for smooth transitions between contents of neighboring tiles regions. The results, though visually interesting, do appear as natural those obtained using texture transfer. Lorenzo de Carli also investigated accomplishing a similar task through different techniques as part of a University of Wisconsin Computational Photography project, but the results were mixed. 5. METHOD This paper proposes several extensions to the texture transfer technique to specifically address portraits. We will first make a distinction in the types of texture inputs used. Most work in the texture synthesis and texture transfer field has focused on what will be referred to here as homogenous textures. Such textures may contain high levels of structure, but the texture is composed primarily of a single texture element with limited changes in color such as white rice, gray rocks, and red bricks. Some examples of the homogenous patches used to render portraits in the experimental results section are shown below: When using such textures, we have a limitation in our ability to capture both high contrast features and lower contrast features in the target image. If we split our patches into just hair and skin elements, we will lose details in areas of the face, hence impeding the fine details need to recognize a particular individual. We will allow consider heterogeneous textures

5 consisting of multiple elements. Shown below are some of the heterogeneous source textures used for this paper: We want to promote the separation of distinct source texture element types to better capture the difference between skin and hair while at the same time maximizing the contrast of patches for a distinct texture element type to promote capturing the finer features. Texture synthesis uses the sum-of-squared-differences (SSD) between all pixels in the overlapping region of the partially synthesized output and corresponding pixels in all potentially matching patches found in the texture source. Texture transfer adds an additional constraint by comparing a spatial correspondence map at the location of the potential patch with a correspondence map of a controlling target image, so we if we can improve that comparison, we can improve the results. METHOD #1: GAUSSIAN FILTER The first extension that we will explore is using a Gaussian filter to compare blurred image intensities. Since the controlling target image serves to guide global patterns, and we are comparing only local patches at any one time, a blurred intensity value helps to better capture neighboring areas, potentially leading to a better final output. In the case of faces, neighboring interactions are very important to accurately synthesize facial features. Below we can see an example of corresponding regions of the controlling target, the source intensity values, the target with blur, and the source with blurred intensity values.

6 METHOD #2: HISTOGRAM EQUALIZATION The second extension that we will explore attempts to solve the problem of matching the typically higher levels of contrast present in the controlling target image than the texture source, which is particularly apparent when using homogeneous texture sources. Even in heterogeneous texture sources, equalizing contrasting regions in the texture should encourage element separation, a desirable property we have discussed earlier. Below we can see the histograms using ten bins for the controlling target image, the source, and the source after equalization. We perform histogram equalization by minimizing T in c 1 (T(k))-c 0 (k),where c 0 is the cumulative histogram of the texture and c 1 is the cumulative sum of target histogram for all intensities k. Below we can see the resulting intensity values in the controlling target, the source, and the source after histogram equalization.

7 METHOD #3: ARITIFICAL SHADING Skin typically has multiple shades due to lighting. If our texture source does not contain texture elements under varying lighting conditional, we might be limited in our ability to capture differences in intensity due to shadows. Our solution is to pre-process the texture image by artificially shading a segment to encourage better matching. We also want to keep enough untouched areas for normal texture synthesis to draw from, so we will only use one quarter of the textures source area. Too much shading is likely to stand out as artificial, so we will use a linear decrease in intensity until a parameterized maximum threshold is reached. This is just one of many options, and the success of any such method will vary depending on the properties of the texture source. Below we can see an original test and its corresponding shaded version. METHOD #4: ALPHA BLENDING Our next extension tries to better match the controlling target palette to the texture source. If we alpha blend using equal weights for each color channel in the target, we are likely to blend in colors not present in the texture, causing a markedly visible halo effect. We desire to alpha blend in channels proportional to their presence in the original texture source. To address this problem, we propose post-processing the synthesized image to blend a different amount for each RGB channel in target based on the mean of that channel in source texture. This modified alpha blending at a level of fifty percent seems to keep the target from bleeding in too much while still greatly enhancing distinctive facial features. METHOD #5: ARTISTIC BLENDING Our last extension goes in the opposite direction, making no attempt to keep the original texture source palette but instead using a black and white synthesized output image colorized using the controlling target. Using alpha blending at a level of fifteen percent, this method produces some visually pleasing results, though at the cost of loosening ties to the original texture source.

8 7. Experimental Results Given that we are concerned with facial detail but using patches composed of many texture pixels, we must consider what a pixel should be. Below are outputs using the original target image at 2x, 3x, and 4x (top to bottom) of the original size. We consider 3x (middle) to be the best trade-off between preserving visible texture detail and capturing controlling target details.

9 In an attempt to help the reader see the most detail of very high resolution images fitted for this paper, results are presented on the following three pages but discussed here. The only difference between all left and right images is the method in question, though both may have the benefit of additional extensions presented here. METHOD #1: GAUSSIAN FILTER We can see by contrasting this set of images that a greater level of detail is obtained as a result of this technique in the image on the right. For example, if we look at the mouth, we can see the contour of the lower lip is lost in the image on the left, but preserved in the one on the right. Additionally, we see a sharper edge at the chin. Lastly, notice how the hair on the left lower side of the face shows a smooth gradient transition while such detail is lost without blurring. METHOD #2: HISTOGRAM EQUALIZATION Homogeneous Example (green clovers) We can see in contrasting this set of images that the one on the left suffers greatly from a failure to match intensities between source and target. The output on the right clearly captures much finer facial details after equalization has been applied. Heterogeneous Example (mixed rice) Though they both benefit from Gaussian blur, we can see the addition of equalization does a better job of keeping heterogeneous elements separated. White and light brown rice render most of the face on the right, while black creeps in too much on the left, obscuring detail around the mouth. METHOD #3: ARITIFICAL SHADING Skin typically has multiple shades due to lighting. If our source image does not contain texture elements under varying lighting conditional, we might be limited in our ability to capture intensity levels. This shading example shows better details by making use of darker patches. METHOD #4: ALPHA BLENDING This heterogeneous pasta example already uses Gaussian blurring to keep the dark pasta mostly separate from the face. The addition of slight alpha blending weighted by channel greatly improves facial details without causing much of a halo effect. METHOD #5: ARTISTIC BLENDING This last example nicely illustrates how a little blending can preserve enough of the original texture while still giving a strong impression of the target. The synthesized image serves as a coarse outline using shading and plaster texture. This subtle effect allows a smaller amount of blending than might otherwise be needed and produces a very nice result.

10 Method #1: Without Blur Method #1: With Blur Method #2: Without Equalization Method #2: With Equalization

11 Method #2: Without Equalization Method #2: With Equalization Method #3: Without Shading Method #3: With Shading

12 Method #4 : Without Channel Blending Method #4 : With Channel Blending Method #5 : Without B&W Blending Method #5 : With B&W Blending

13 8. FUTURE WORK The extensions presented here are but a handful of possible options. To extend the shading method, one could pre-process the texture source by running standard texture synthesis to produce a larger input. With this larger input texture source, shading using several curves of varying slopes in different locations could better match the many skin shades in portraits. Many of these methods introduced new parameters, thus a systematic ways of determining the optimal Gaussian filter size or the amount of shading or the number of histogram bins would be desirable. A very large number of synthesized images resulted from the enumeration of combinations of such parameters, which then had to be visually inspected for quality. An automated way to rate the similarity of the synthesized image to the target image would also be desirable if a good way to determine optimal parameters cannot be found. 9. COCLUDING REMARKS When starting this project, initial research went into methods for detecting skin in the hope of automatically calculating bit mask regions for hair and skin. A two-pass method using one source texture in the first pass to render only skin and then a different source texture in the second pass to render only hair was thought to be ideal since the resulting two layers could then be alpha blended into a final result. After surveying the existing literature and finding few robust solutions, a shift in focus to maximize the effectiveness of a single heterogeneous texture was undertaken. The results produced here are arguable better than a two-pass method could hope to accomplish due to the natural transitions present when using a single texture, though this method requires identifying such suitable heterogeneous texture sources. More sophisticated techniques do not necessarily produce better results as the success of the relatively simple texture transfer algorithm has shown. Continuing in the same spirit, this paper found effective solutions through empirical investigation, producing visually compelling results in the process.

14 REFERENCES J. S. D. Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. In SIGGRAPH 97, pages , D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. In SIGGRAPH 95, pages , J. Malik and R. Rosenholtz. Computing local surface orientation and shape from texture for curved surfaces. International Journal of Computer Vision, 23(2): , K. Popat and R. W. Picard. Novel cluster-based probability model for texture synthesis, classification, and compression. In Proc. SPIE Visual Comm. and Image Processing, J. Portilla and E. P. Simoncelli. Texture representation and synthesis using correlation of complex wavelet coefficient magnitudes. TR 54, CSIC, Madrid, April S. C. Zhu, Y. Wu, and D. Mumford. Filters, random fields and maximum entropy (frame). International Journal of Computer Vision, 27(2):1 20, March/April A. A. Efros and W. T. Freeman: Image quilting for texture synthesis and transfer. In SIGGRAPH 01, pp , A. A. Efros, T. K. Leung: Texture synthesis by non-parametric sampling. In IEEE International Conferenceon Computer Vision, pp , C. Eisenhacher, S. Lefebvre, M. Stamminger: Texture synthesis from photographs. In Proceedings of the Eurographics conference, J. Kim and F. Pellacini: Jigsaw Image Mosaics. In SIGGRAPH 02, D. S. Ebert, F. K. Musgrave, D. Peachey, K. Perlin, S. Worly: Texturing and Modeling: A Procedural Approach. Morgan Kaufmann Publishers Inc., , pp. 1 8, 2008.