Simulating the Effect of Illumination Using Color Transformations

Transcription

1 Simulating the Effect of Illumination Using Color Transformations Maya R. Gupta a and Steve Upton b and Jayson Bowen a a Dept. of Electrical Engineering, University of Washington, Seattle, WA b Chromix, Seattle, WA ABSTRACT We investigate design and estimation issues for using the standard color management profile architecture for general custom image enhancement. Color management profiles are a flexible architecture for describing a mapping from an original colorspace to a new colorspace. We investigate use of this same architecture for describing color enhancements that could be defined by a non-technical user using samples of the mapping, just as color management is based on samples of a mapping between an original colorspace and a new colorspace. As an example enhancement, we work with photos of the 24 color patch Macbeth chart under different illuminations, with the goal of defining transformations that would take, for example, a studio D65 image and reproduce it as though it had been taken during a particular sunset. The color management profile architecture includes a lookup-table and interpolation. We concentrate on the estimation of the look-up-table points from minimal number of color enhancement samples (comparing interpolative and extrapolative statistical learning techniques), and evaluate the feasibility of using the color management architecture for custom enhancement definitions. Keywords: ICC profile, custom color enhancement, IME, color mapping, 3D UT 1. INTRODUCTION Digital design and graphics arts is an expanding field, due in part to cheap and easy access to high-quality color printing technologies and the growth and commercialization of the Internet. Design plays an indispensable role in perceptions of quality, design, and branding 1. 2 How do we empower digital designers to create custom color enhancements? Recently, there has been interest in computer-aided color image enhancement that allows users to create complicated or custom effects with limited work or technical knowledge. This paper explores the statistical learning issues that arise in learning a custom color enhancement based on a small number of sample input/output color pairs. In particular, we use the ICC profile architecture to define and implement a custom transformation. As an example application, this paper considers the problem of simulating illumination effects by a color transformation. ICC profiles 3 were designed for color management of devices, such as printers, which require empirical characterization and complex color transformations to model their sampled characteristics. The core of an ICC profile is a three-dimensional look-up-table (UT) that defines how input colors are transformed into output colors. There are two major advantages to making a custom color enhancement into an ICC profile. First, a three-dimensional look-up-table is a very flexible way to define a color transformation. Second, once defined, the color transformation can be implemented by any color management module. This allows users to implement transforms using non-proprietary software and view and edit transforms using color management applications like ColorThink. 4 ICC profiles are standardized by the International Color Consortium to provide an open, vendor-neutral device characterization for color management. Simulating illumination changes is an example of a complex color transformation that can be defined by sample pairs. For example, a designer might wish that a studio photo of a hamburger or model was taken under the light of a certain Namibian sunset. If the designer has captured the effect of that Namibian sunset and the effects of the studio light, then the effect of the sunset can be simulated on the studio photo. In this paper, Further author information: (Send correspondence to M.R.G.) M.R.G.: gupta@ee.washington.edu, S. U.: upton@chromix.com

2 we capture the effect of an illuminant on the 24 color samples of a Gretag Macbeth ColorChecker chart, and then learn a color transformation for the entire colorspace based on input/output pairs of, for example, D65 illumination to a particular sunset light. 2. REATED WORK Aiding designers to produce complex custom color enhancements is a popular area of research, and we reference only a few representative examples. One method to define a custom enhancement is to transform the color palette of an input image based on the color palette of a single reference image 5. 6 In such work, the output image spatially looks like the input image, but has inherited or moved towards the color palette from the reference image. Hertzmann et al. 7 use a pair of reference images to learn a transformation that can then be applied to a third image. They use multi-scale spatial features as well as pixel color values to create image transformations such as adding texture. UT s have previously been used to speed-up known functions that transform colors or images. 8 In Gatta et al. s opinion, 9 The UT transfer function is the simplest way to apply a global filtering to an image using a function previously devised. In contrast to prior work, this paper confines itself to colorspace transformations that can be implemented with a three-dimensional UT, and that are not already implementable by a known function or algorithm, so that the amount of data to learn from is relatively small sample of input/output color pairs (< 100), as might be defined by a graphic artist. This paper explores the use of statistical learning to create an accurate and visually smooth three-dimensional UT transformation. We use simulating illumination changes as an example application and data source. Another way to simulate illumination is to estimate the reference and destination illumination spectra, then divide out the reference spectra from the three color channels of the image, and multiply-in the destination illumination spectra. Direct illumination spectra estimation has been studied in work such as the scanner calibration literature. Since we are interested in an architecture for general color enhancement based on input and output samples, illumination spectra estimation is not general enough for our goals. Whether simulating the effect of an illuminant by estimating and applying the spectra, or as we do in this paper, by learning a color transformation based on samples, there remains the problem of metamers. Objects originally photographed that appear the same in the image may have physical properties that would make them not look the same under different illumination. For example, water and jeans might be metameric objects, both appearing the same color of blue in the image to be transformed. Without semantic knowledge of what the object is, all pixels of a certain color value will be transformed in the same way by the proposed color transformations. Work on semantic object recognition, 10 may one day be coupled with object specific transforms, but in the present work we ignore metameric complications. This work is also related to white balance for digital cameras. White balance algorithms are generally simple and serve a different purpose than the enhancements investigated here. 3. IUMINANT DATA Photographs of a standard 2004 Gretag Macbeth ColorChecker chart under different illumination conditions were used as the data for this project. A Sony DSC-F828 8 MegaPixel camera was used. For each picture, the colorchart and a standard photographer s gray card were set-up on a board. The camera was set to manual mode with the white balance set to daylight. The metering mode was set to spot and the angle of the board was adjusted until all the readings on the gray card were equal. Then the photo was taken. Example photos are shown in Fig. 1. A D65 photo was taken of the chart under a Gretag Macbeth SolSource D65 filtered lamp. For each of the twenty-four color squares in each image, the corresponding srgb pixel values from the camera were averaged, forming a data set of twenty-four srgb values for each illumination condition. The srgb values are then converted to CIEab values (D65 whitepoint) by standard formula. Differences in CIEab between pairs of illuminant samples are shown in Fig. 2.

3 Figure 1. eft: chart photographed under cloudy conditions. Right: chart photographed under a soft white incandescent light. b a b a Figure 2. eft: vectors show the differences between the 24 D65 color chart samples and 24 soft white incandescent color chart samples in three-dimensional CIEab space. Right: vectors show the differences between the 24 cloudy color chart samples and 24 soft white incandescent color chart samples in three-dimensional CIEab space.

4 4. ESTIMATING THE 3D UT Based on the known input/output CIEab color sample pairs, a 3D UT is estimated which maps a 21x21x21 grid in CIEab to corresponding output CIEab color values. Estimating the 3D 21x21x21 grid and its ouput values from only 24 sample pairs is a difficult learning problem, particularly because many of the grid points fall outside the convex hull of the set of 24 samples in the input space. The estimation issues are interpolation vs. extrapolation, a flexible enough fit to capture the underlying transform vs. overfitting the transform, and smoothness and monotonicity of the estimated transform. Because this is a color enhancement, smoothness of the transform around the neutrals is particularly important. In this paper we consider two radically different methods for estimating the 3D UT grid points, local linear regression, and linear interpolation with maximum entropy. Regression methods can be categorized as interpolative or extrapolative. For example, local linear regression 11 on a k-neighbor neighborhood fits a plane to the k samples nearest the grid point in the input space out of the total 24 original input samples. Since a plane is fit to the samples closest to each grid point, the method extrapolates to grid points that are outside the convex hull of the set of 24 samples in the input space. ocal linear regression has been shown to have nice properties near boundaries and for unevenly distributed sample points, 12 as is the case for this application. Consider, on the other hand, common tetrahedral interpolation sometimes used in color management, 13 which linearly interpolates the four samples closest to a particular grid point in the input space in order to estimate that grid point s output value. If a grid point is outside of the convex hull formed by its four nearest samples, then tetrahedral interpolation weights cannot be solved for, since the weights must sum to one. et x be a point on the grid that forms the 3D UT. et ŷ be the estimate of its transformed color. et x 1, x 2,..., x 24 be the 24 sample points in the input colorspace, where they have been indexed by their Euclidean distance to x in the input CIEab space. et y 1, y 2,..., y 24 be the corresponding output, or transformed, CIEab color values. Then tetrahedral interpolation calculates weights w j that solve, 4 j=1 w jx j = x 4 subject to w j = 1 j=1 w j 0 for all j. Once the tetrahedral weights w 1, w 2, w 3, w 4 have been found (if they exist), the tetrahedral interpolation estimate is 4 ŷ = w j y j j=1 As noted above, (1) cannot be solved for all grid points, tetrahedral interpolation can only be used if the grid point x being estimated is in the closure of the convex hull formed by its four nearest-neighbors. inear interpolation with maximum entropy (IME) 14 generalizes linear interpolation so that any number of sample points can be used and there is no problem if the test point x lies outside the convex hull of its neighbors. The IME weights attempt to solve the linear interpolation equations. If x is not in the convex hull of its neighbors, then the IME weights attempt to minimize the distance between the reconstructed point j w jx j and x, which in effect projects x onto the convex hull of the neighbors. If the linear interpolation equations are underdetermined (as may be when k > 4 for three dimensions), then there exist multiple solutions to the linear interpolation equations, and the IME weights are the solution that maximizes entropy. Formally, the IME weights w j solve, ( arg min w k j=1 w jx j x 2 + λ ) k j=1 w j log w j k subject to w j = 1 j=1 w j 0 for all j, (1)

5 where λ and k are algorithmic parameters. In this work, we set λ = 1e 9, so that the weights focus on minimizing the distortion between j w jx j and x. For k = 4, the IME weights will be numerically equivalent to the tetrahedral weights, if the tetrahedral weights exist. Given the IME weights w j, the color value estimated to be the output corresponding to input grid color x is, ŷ = k w j y j (2) j=1 5. RESUTS For different pairs of input/output illumination conditions, we compare local linear regression and IME estimates of the 3D UT by comparing how new points are transformed when the estimated 3D UT transformation is applied. To estimate a new color value z, the eight vertices of its grid cell are determined, and then z s corresponding output color is calculated using trilinear interpolation. 13 First, we consider a test case of neutral colors. After estimating the 3D UT for transforming D65 illumination to a sunset illumination, we transform the neutral axis in CIEab using the estimated 3D mappings. The results are shown in Fig. 3. It is clear from the plots that the estimation algorithm and the number of neighbors used significantly changes the estimation, even in the center of the colorspace. As expected, increasing the number of neighbors leads to a smoother fit, but may smooth out the desired color mapping. Using four near-neighbors with the local linear regression leads to a non-monotonic luminance transformation of the neutrals axis, and large hue variance in the mid-grays. Overall, the IME estimate is less variable with respect to the number of neighbors used. The IME change in hue and luminance is also smoother over the neutral axis for a given number of neighbors. However, some of that smoothness is due to the interpolative nature, which limits the darkest dark and brightest white to the convex hull of the original twenty-four samples. What happens on the edge of the D65 to sunset mapping? In Fig. 4, the output CIEab color values corresponding to the regular input CIEab grid are shown for the grid estimated by IME and the grid estimated by local linear regression, each using six nearest-neighbors. Here, the difference between the interpolative and extrapolative method is clear. Since it is doing only interpolation, the IME grid will map all input colors to a small gamut as defined by the convex hull of the original twenty-four samples. This can lead to images where very saturated object regions are clipped and appear flat. The local linear regression grid maps input colors to the entire space; in fact the colors shown have been clipped to be within [0, 100] and a, b [ 100, 100]. Many of the colors that the local linear regression maps to will be outside the gamut of standard monitors or printers, and thus many points will again be clipped upon display, leading to some flatness in extreme color regions of an image. Because the local linear regression grid extrapolates to the entire space, the average distance between output colors is much greater than the average distance between output colors from the IME grid. This can lead to false edges in local linear regression mapped images. In Fig. 5, a daylight photograph of a building (from the Foveon Image Gallery) is transformed using a D65 to sunset color transformation. The left transformations are with local linear regression, and the right transformations are with IME estimation. In the context of the image, the transformations look quite similar, with the most noticeable differences occurring in the brightest areas. In Fig. 6, a daylight photograph of a landscape is transformed from daylight (D65) to sunset using four different transformations. In both Fig. 5 and Fig. 6 the brightest regions of the local linear regression may strike some viewers as too bright compared to the rest of the image s palette. In Fig. 7, a clearer difference between the estimation methods is seen in this application of a soft white incandescent to cloudy color transformation. The specular highlights on the red pepper have turned blue in the IME images, and this will strike most viewers as unnatural. Though the off-color specular highlights are a very tiny portion of the image, they give the sense that the entire image is too blue, due to human s extreme sensitivity to highlights. ess offensive, but also unnatural, is the flat bright white region of the foreground garlic in the local linear regression images, which is inconsistent with the rest of the palette. Though not too

6 Figure 3. Plots show how the neutral CIEab axis is transformed under the estimated color mapping for D65 illumination to a sunset illumination. eft: local linear regression using 4, 6, and 9 neighbors (from top to bottom). Right: IME using 4, 6, and 9 neighbors (from top to bottom).

7 Figure 4. Both plots show the 3D grid mapping from D65 to a sunset illumination. eft: local linear regression output values corresponding to the 3D UT grid points, using a neighborhood of six of the twenty-four original input samples. Right: IME output values corresponding to the 3D UT grid points, using a neighborhood of six of the twenty-four original input samples. disagreeable, the extrapolated local linear regression highlights and bright areas seem again too bright for the color cast of the rest of the image. The differences between the six and nine neighbor images are not very noticeable in the images shown. 6. DISCUSSION In this paper we have investigated issues arising from estimating color transformations using a 3D UT based on relatively few samples. We have shown that either local linear regression or IME estimation can result in sensible transformations for realistic image conversions. An important estimation issue for this application is the interpolative vs. extrapolative nature of the estimation. Interpolation is severely limited by its very nature; for neutral images the interpolation fit may better reflect the desired transformation, but for images with extreme color values any interpolative method will lead to clipping and objects can acquire an appearance of flatness due to lost contrast. The clipping due to interpolation can also cause miscolored specular reflections to which the eye is extremely sensitive. Other regression or smoothing algorithms could be used to estimate the 3D UT. Due to the sparsity of data samples, polynomial or other spline approaches may have a negative extrapolative effect, resulting in high variance of the estimated color transformation for small changes in algorithm parameters. Further, extrapolative estimation risks assigning extreme output color values and failing to consistently capture the sense of the desired transformation. The key issue for this application is finding a middle way between interpolating and extrapolating that retains the sense of the transformation, and optimally fills the output colorspace, resulting in minimal clipping. Further, since the goal is an automated system that does not require the user to carefully analyze different estimated transformations, the estimation method should lead to a sensible transformation for a large class of possible original sample pairs. This goal is more likely to be achieved with estimation methods that are robust, in the sense that small changes in the estimation parameters (such as changing the neighborhood size) lead to small changes in the estimated transformation. Such robustness will help ensure consistently reasonable estimation behavior over a large class of input sample pairs.

8 Figure 5. Top left: original daylight (D65) image. eft: original image transformed using D65 to sunset mapping estimated with local linear regression (middle with six neighbors and bottom with nine neighbors). Right: original image transformed using D65 to sunset mapping estimated with IME regression (middle with six neighbors and bottom with nine neighbors).

9 Figure 6. Top left: original daylight (D65) image. eft: original image transformed using D65 to sunset mapping estimated with local linear regression (middle with six neighbors and bottom with nine neighbors). Right: original image transformed using D65 to sunset mapping estimated with IME regression (middle with six neighbors and bottom with nine neighbors).

10 Figure 7. Top left: original image taken under unknown lighting. eft: original image transformed using soft white incandescent to cloudy mapping estimated with local linear regression (middle with six neighbors and bottom with nine neighbors). Right: original image transformed using soft white incandescent to cloudy mapping estimated with IME regression (middle with six neighbors and bottom with nine neighbors).

11 Defining color image enhancements via 3D UT s based on a small set of sample pairs is an interesting and easy way to create custom image enhancements. The statistical estimation used must be robust for this to be useful in practice. REFERENCES 1. B. H. Schmitt and A. Simonson, Marketing Aesthetics: The Strategic Management of Brands, Identity and Image, Free Press, V. Postrel, The Substance of Style: How the Rise of Aesthetic Value Is Remaking Commerce, Culture, and Consciousness, Harpers-Collins, ICC webpage, ColorThink 2.1.2, Y. Chang and S. Saito, Example-based color stylization based on categorical perception, Proc. of the First Symposium on Applied Perception in Graphics and Visualization, pp , E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, Color transfer between images, IEEE Computer Graphics and Applications 21, pp , September A. Hertzmann, C. E. Jacobs, B. C. N. Oliver, and D. H. Salesin, Image analogies, SIGGRAPH, pp , D. M. A. Rizzi and. D. Carli, Multilevel brownian retinex colour correction, Machine Graphics and Vision, D. M. A. R. C. Gatta, S. Vacchi, Proposal for a new method to speed up local color correction algorithms, Proc. of the SPIE Y. i, J. A. Bilmes, and. G. Shapiro, Object class recognition using images of abstract regions, Proc. of the 17th Intl. Conf. on Pattern Recognition, T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical earning, Springer-Verlag, New York, T. Hastie and C. oader, ocal regression: automatic kernel carpentry, Statistical Science 8(2), pp , H. Kang, Color Technology for Electronic Imaging Devices, SPIE Press, United States of America, M. R. Gupta, Inverting color transformations, Proceedings of the SPIE Conf. on Computational Imaging, 2004.