Color Correction between Gray World and White Patch

Transcription

1 Color Correction between Gray World and White Patch Alessandro Rizzi, Carlo Gatta, Daniele Marini a Dept. of Information Technology - University of Milano Via Bramante, Crema (CR) - Italy - rizzi@dti.unimi.it a Dept. of Information Science - University of Milano Via Comelico, Milano - Italy ABSTRACT Color equalization algorithms exhibit a variety of behaviors described in two differing types of models: Gray World and White Patch. These two models are considered alternatives to each other in methods of color correction. They are the basis for two human visual adaptation mechanisms: Lightness Constancy and Color Constancy. The Gray World approach is typical of the Lightness Constancy adaptation because it centers the histogram dynamic, working the same way as the exposure control on a camera. Alternatively, the White Patch approach is typical of the Color Constancy adaptation, searching for the lightest patch to use as a white reference similar to how the human visual system does. The Retinex algorithm basically belongs to the White Patch family due to its reset mechanism. Searching for a way to merge these two approaches, we have developed a new chromatic correction algorithm, called Automatic Color Equalization (ACE), which is able to perform Color Constancy even if based on Gray World approach. It maintains the main Retinex idea that the color sensation derives from the comparison of the spectral lightness values across the image. We tested different performance measures on ACE, Retinex and other equalization algorithms. The results of this comparison are presented. Keywords: Retinex, Gray World, White Patch, Color correction 1. INTRODUCTION Our mode to see the world passes through our visual system (VS) and is influenced by its properties. Studying these properties, we note that our VS is a very poor measuring instrument, but a very efficient visual information extractor. This efficiency derives mainly from its adaptational mechanisms, among them lightness constancy (LC) and color constancy (CC). In fact, our VS, this wonderful sort of camera, is able to correctly expose its shots (LC) and record the real color of the objects regardless of the type of illuminant (CC). A digital camera can do almost the same task, but is not entirely autonomous. In particular, LC can fail in back-lighting conditions, and CC is usually realized by a pre-computed approximation of the illuminant type. These VS mechanisms are able to maximize the image s dynamics, as well as the information content of the perceived scene. Their effect is highly non-linear, since it can be global and local at the same time, solving problems of tone coding in situations with high luminance differences or gradients. 1 A computational model of these mechanisms can be very useful. Several imaging and computer vision techniques can take advantage of these capabilities. Otherwise, it can simply be used to produce well-balanced and pleasant images. In the attempt to develop a computational model of these mechanisms, we decided to set some boundaries. Computer displays are limited devices to investigate on color adaptation. Their limitations are not only due to the gamut fitting problem but, also, due to many adaptational cues, useful for our VS, which can be lost in the acquisition/display process, such as specular highlights, mutual reflections, etc. 2, 3 To overcome these limitations and to develop algorithms just for digital imaging, we decided to consider only a subset of features suitable to be used in digital image processing. It is in this way that we are interested in approaches that try to model the VS adaptation phenomena from a perceptual and not from a light-matter interaction point of view. Conversely, we are not interested in investigating the internal structure of our VS, preferring to take into account

2 its behavior under normal conditions, rather modeling its internal mechanisms using measurements taken under simplified conditions. For our purposes, visual tasks seem to be too complex to be faced bottom-up. Few VS models perform color recomputing solely according to the image content and spatial relation, but among them is Retinex. 4 In fact, the Retinex theory assumes that color perception is based on ratios of reflected light intensity in specific wavelength bands computed between adjacent areas and not on the image s spectral power distribution. 5 Starting with the Retinex model, we have developed a new model of unsupervised color correction, called ACE (for Automatic Color Equalization), that is presented and compared with Retinex in the following sections. In order to compare ACE with other models or just to test its efficacy, we had to choose a reliable measurement method. In our opinion, a color correction algorithm inspired by the human VS should perform LC and CC autonomously but not perfectly. In fact, like the real VS, it should perform incomplete adaptations and, in some cases, it should also cheat some of the image configurations, called visual illusions. 6 Thus, since no perfect CC is expected if the model has to perform image correction in a VS style, no precise colorimetric measurement can be useful. Measurements in comparison to a reference illuminant (usually, but not necessarily D65) do not solve the problem of measuring the appearance of the image objects, because they are biased by their mutual spatial relationship and then, in some cases, by a high-level interpretation and segmentation of the scene. An interesting trend of imaging research analyzes the production of filtered images in terms of their capability to please people. 7 Unfortunately, this analysis in terms of pleasantness of the output image deals with user subjectivity and faces the philosophical problem that we have in comparing the output of our VS model using our VS itself. Thus, waiting for a more suitable measurement method to evaluate the algorithm, we chose the classic E distance in CIELab between all the illuminants, avoiding a reference image. In the next section, we will introduce the characteristics taken into account to developing ACE, and in section 3, the basic algorithm, ending with our tests, test results, and conclusions. 2. GRAY WORLD, WHITE PATCH AND OTHER VISUAL MECHANISMS: ADAPTING TO THE IMAGE CONTENT Understanding the complex internal structure of the human VS is not necessarily useful for digital image filtering purposes. In this work we propose a computational model that only takes into account selected aspects of our VS, without mimicking its internal structure. In particular, we focused our attention on independent adaptation of the chromatic channels, lateral inhibition mechanism, White Patch adapting behavior, Gray World adapting behavior and Local/Global adaptation. Adaptation performs independently in each of the three visual chromatic channels, 8 corresponding approximately to RGB. Both Retinex and ACE computes the output pixel values separately on the three RGB channels. The LC adaptation makes us perceive as medium gray the objects which reflect the average luminance of a scene. In terms of histogram properties of a digital image, this corresponds to a level distribution which has its center mass around the middle value (e.g. 128 in eight bit channel depth). When this happens separately between the three chromatic channels, some global chromatic dominant can be eliminated. 9 We refer to this mechanism as Gray World (GW). This correcting mechanism fails if used alone (see Section 5), but is an important component in the whole visual process. Retinex does not consider any GW mechanisms, while ACE uses this approach to center the resulting dynamic. In some cases the VS normalizes its channel values, maximizing towards an hypothetical white reference area. We refer to this mechanism as White Patch (WP). 10 Retinex finds the white reference patch with the reset mechanisms, while ACE non linearly stretches towards the reference white with its relative lightness appearance function (see next Section). At first sight, GW and WP can be seen in opposition to each other when performing the adaptation, but they can actually be part of the same model. An important point of Retinex and ACE is that they take into account the spatial relationship within the scene areas, computing the final pixel values, according to the image areas distribution, size and mutual position. This results in a mixed range of local and global adaptations. The

3 WP mechanism is a global adaptational VS function, when considering a white global reference. However in many situations, if the reference white is local, the adaptation can be biased locally, allowing a more effective tone reproduction. In the same way our VS extracts useful information from the light areas in a backlight situation. This phenomenon is accounted for in the Retinex algorithm by the reset operation that finds the white reference local to a path. To realize this same effect in Ace, we introduced a weight function, which solves the problem of non-determinism of Retinex random path implementation, while introducing a new parameter. However, the choice of this function does not seem to be critical. 3. THE PROPOSED IMPLEMENTATION The algorithm structure is shown in Fig. 1, with a first stage that accounts for a chromatic spatial adaptation, responsible of the CC behavior, and a second phase of dynamic tone reproduction scaling that adjusts the output range in order to realize the LC behavior. The first phase realizes a sort of merging between the GW and WP approaches with positional weighting for the local/global filtering effect balancing, while the second phase maximizes the image dynamic, white referencing at a global level. This double phase structure is characteristic 11, 12 of a majority of the VS computational models in the field of digital imaging. No user supervision, no statistic and no data preparation are required to run the algorithm. Figure 1: Basic algorithm structure 3.1. Chromatic/Spatial adaptation The first phase, the Chromatic/Spatial adaptation, results in an image R in which every pixel is a floating point value recomputed according to the image content. As in Retinex, this phase is run separately on each chromatic channel, and the final pixel value results from the relative weighted differences of all the pixels in the image. In this way, the channel lightness of each pixel is computed, where the term lightness refers to Land s original meaning. 4 As opposed to the basic Retinex algorithm, ACE has no paths; the local effect is weighted by a distance function. Each pixel p of the output image R is computed separately for each channel c as follows: R c (p) = j Subset,j p r(i c (p) I c (j)) d(p, j) (1) where I c (p) I c (j) accounts for the lateral inhibition mechanism, d( ) is a distance function which weights the amount of local or global contribution, r( ) is the function, discussed below, that accounts for the relative lightness appearance of the pixel. The pixel computation can be extended to the whole image or restricted to a Subset.

4 Global/local weight: the d( ) function d( ) is the function in charge to balance between the global and the local adapting effects of the model. Global models, in fact, are not able to simulate local chromatic adaptation such as the simultaneous contrast or the Mach bands. Only a model with local spatial stimula interaction can account for these phenomena. 1 Different functions have been tested. Among the tested distances: r, e, Manhattan, Max, r 2, Manhattan 2, αr Max 2, r 3,wherer is the Euclidean distance. Only the last distances gave unsatisfactory results. Some functions seem to achieve better results than others, but no absolute definitive function has been found and consequently distance function choice requires a deeper investigation. For the tests in this paper we have chosen the Euclidean distance r The relative lightness appearance: the r( ) function For each pixel of the image r( ), with d( ), control the contrast interaction, accounting for the spatial channel lightness adaptation. It computes all the single contributions of the image content (weighted by d( )) to each final pixel value in the output image. To perform a GW behavior, r( ) has to be an odd function, while the WP behavior is obtained by non-linear enhancements of small differences in neighboring pixels, that result in a sort of local white referencing. In pursuing an effective WP behavior we have tested different r( ) functions: Linear, Truncation, Signum and Saturation, which are shown in Fig. 2. Figure 2: Tested r( ) functions We made a preliminary evaluation of the above r( ) functions performance by considering their chromatic correction and their dynamic enhancement capabilities. Results are shown in Table 1. As it can be noticed in Table 2, all the functions strongly decrease the chromatic dominant due to the illuminant, but Signum and Saturation give results with higher dynamic. Original Linear Truncation Signum Saturation Table 1: Sample image filtered with different s( ) function To test the r( ) functions CC capabilities, the mean E distance in CIELab between two synthetic images of a living room, computed with a photometric ray tracer 13 under the A and D65 CIE illuminant, before and

5 after the filtering have been computed. In the results shown in Table 2, Truncation has a threshold value 0.5 and Saturation has slope value 20. Original Linear Truncation Signum Saturation 48,74 7,26 11,88 4,54 6,94 Table 2. Mean E distance between two synthetic images under the A and D65 CIE illuminant, without any filtering (Original) and filtered with the various r( ) functions Signum performs the most effective CC, but, as it can be noticed in the enlarged detail of the synthetic image under illuminant A (Table 3), it acts as a sort of sharpening filter introducing high frequency noise and Mach bands effects. Linear and Signum functions can be seen as limit cases of Saturation function with unitary or infinite slope respectively. The higher the slope value, the more the contrast is enhanced. The tuning of this parameter has to be further investigated; for the test of this paper we chose Saturation with slope value 20. Figure 3: Enlarged details of synthetic image of Table 5 filtered using Signum (left) and Saturation (right) Reducing the border effect In the basic formula 1, no compensation is taken into account for the pixel distance from the border. The output value of each pixel is determined by N 1 contributors, where N are the pixels of the image; these contributors are weighted by the local/global balancing function d( ). Thus, the closer the pixel is to the border, the less the number of neighboring pixels with a high contribution weight are available. Consequently, the equation 1 has been modified with a normalization coefficient in the following way: R c (p) = r(i c(p) I c(j)) j Subset,j p d(p,j) r max j Subset,j p d(p,j) (2) where r max is the maximum value of r( ) Dynamic tone reproduction scaling This second phase maps the matrix R into the final output image O. In this stage not only a simple dynamic maximization can be made (linear scaling), also different reference values can be chosen in the output range to map into gray levels the relative lightness appearance values of each channel. According to the chosen reference point, an additional global balance between gray world and white patch is added. The following two scaling methods can be used to obtain a standard 24 bit output image from the floating point array R.

6 Linear scaling This first method linearly scales the range of values in R c independently on each channel to the range [0,255] by the formula: O c (x, y) =round[127.5+s c R c (x, y)], where s c is the slope of the segment [(m c, 0), (M c, 255)], with M c =max x,y R c (x, y) andm c =min x,y R c (x, y). In this case the linear mapping precisely fills the available dynamic range without further adaptation WP/GW scaling This alternative method linearly scales the values in R c with the same formula, but using M c =max x,y R c (x, y) as white reference (WP) and the zero value in R c as an estimate for the medium gray reference point (GW) to compute the slope s c. For this reason, the available dynamic could not be used entirely, otherwise, tones around the very dark values could be lost. Moreover, some values in O c could result negative values; in this case, these values are set to 0. This method adds a global GW adaptation in the final scaling, resulting in the dynamic of the final image always being centered around the medium gray Computational cost ACE has an O(N 2 ) computational cost where N is the pixels number of the image. On a AMD-K6-II@350Mhz, running Windows 98, with 128 Mb RAM, it takes 39 seconds to compute a 100x75 pixel image and about 12 minutes to compute a 192x192 pixels image. An optimizated version of the algorithm based upon a Local Look Up Table method is being written. A multilevel approach or an automatic selection of the image subset, for each pixel recomputing, could be two of the possible directions for its optimization. 4. TEST SETUP We measured the algorithm s capability to perform Color Constancy, computing the mean E distance across all the pixels between two images of the same size: E mean = sizex sizey x=0 y=0 E(I 1(x, y),i 2 (x, y)) sizex sizey where I 1 and I 2 are the images to be compared, sizex and sizey are the image dimensions in pixel. We have used two image sets: the University of East Anglia (UEA) uncalibrated colour image database and a set of six synthetic images generated by a photometric ray tracer program from the same 3D scene in six different lighting conditions. The UEA uncalibrated colour image database is a database of 392 design images made from 28 different designs images under three light sources using four digital cameras and two commercial scanners arbitrarily chosen. The images were acquired under a CIE A, D65 and TL84 lights. In this paper we used only the image set acquired with the Fuji Mx-700 digital camera. Effects due to device changing will be further investigated. The six synthetic images were generated with a photometric raytracing algorithm 13 from a 3D living room model. The light sources used were the standard Cie illuminant A, B, C, D65 and a Hg lamp; the latter image was obtained using a mix of these illuminants. 5. RESULTS On these two image sets we measured the mean E distance for any couple of the same image under the various illuminants, before and after the use of different filtering algorithms. We tested a classic histogram equalization, the global WP 10 and GW 9 algorithms, the Brownian random paths Retinex 6 and ACE. We chose this version of Retinex algorithm only because it was better known to the authors of this paper. However, different Retinex 14, 15 implementations have proven to perform comparable results. The test results are presented in the Table 3. The GW algorithm seems to give good results but performes a very poor equalization, as can be seen in Fig.4.

7 Figure 4: Sintetic image A GW filtered A B C D65 Hg Mix Mean No filtering Hist. Eq White Patch Gray World Retinex ACE Table 3: Mean E on synthetic images The same measure has been computed on the UEA database before and after the filtering with ACE and the Brownian Retinex, obtaining the values shown in Table 4. Original Retinex ACE ,12 Table 4: Mean E on UEA database A result comparison is shown in Fig. 5. A visual comparison of the test results is presented in Table 5 and CONCLUSIONS AND PERSPECTIVES A new algorithm for digital image automatic color equalization has been presented. ACE has been developed in the attempt of extending the Retinex model of color equalization, merging Retinex with the Gray World and the White Patch equalization methods. It is based on a schematization in two phases of a simple model of the human vision system. The first phase, the visual encoding, recovers the appearance of the scene areas and the second phase, the display mapping, normalizes the values of the filtered image, maximizing its dynamic. As well as Retinex, ACE performs simultaneously global and local filtering. Preliminary results are very promising: ACE has demonstrated to perform an effective color constancy correction. However it is an ongoing research, to tune some of its internal functions and parameters further investigations and tests are necessary. The main problem of ACE is its high computational cost. A multilevel approach or an automatic selection of the image subset, for each pixel recomputing, could be two of the possible directions for its optimization.

8 Illuminant A B C Original ACE Illuminant D65 Hg Lamp Mix Original ACE Table 5: Examples of ACE filtering on synthetic images Illuminant A D65 TL84 A D65 TL84 Original ACE Table 6: Examples of ACE filtering on UEA DB images

9 Figure 5: Ace result comparison with other equalization algorithms REFERENCES 1. J. Tumblin and H. Rushmeier. Tone reproduction for realistic image synthesis. IEEE Computer Graphics & Applications, November: , H.C. Lee. Method for computing the scene-illuminant chromaticity from specular highlights. Journal of Optical Society of America, 3: , M. D Zmura and P. Lennie. Mechanisms of color constancy. Journal of Optical Society of America, 3 (10): , E. Land. The retinex theory of color vision. Scientific American, 237-3:2 17, J.J. McCann. Color theory and color imaging systems: Past, present and future. Journal of Imaging Science and Technology, 42 (1):70 78, D. Marini and A. Rizzi. A computational approach to color adaptation effects. Image and Vision Computing, 18: , R.W.G. Hunt. How to make pictures and please people. In Proc. of Seventh Color Imaging Conference, J.J. McCann. Color constancy: Small overall and large local changes. In SPIE Proceedings Vol Human Vision, Visual Processing and Digital Display III, San Jose (California), G. Buchsbaum. A spatial processor model for object color perception. J.Franklin inst., 310 (1):1 26, B. Funt and V. Cardei. Committee-based color constancy. J.Opt.Soc.Am. A, 11 (11): , A. C. Hurlbert. Formal connections between lightness algorithms. J. Opt. Soc. A, 3: , S.N. Pattanaik, J.A. Ferwerda, M.D. Fairchild, and D.P. Greenberg. A multiscale model of adaptation and spatial vision for realistic image display. In Proc. of SIGGRAPH98, Orlando, Florida (USA), D. Marini, A. Rizzi, and C. Carati. Color constancy effects measurement of the retinex theory. In Proc. of Electronic Imaging 99, IS&T/SPIE s 11th International Symposium, S. Jose, California (USA), D. Marini, A. Rizzi, and L. De Carli. Multiresolution retinex: comparison of algorithms. In Proc. CGIP 2000, First International Conference on Color in Graphics and Image Processing, Saint-Etienne, FRANCE, October G. Ciocca, D. Marini, A. Rizzi, R. Schettini, and S. Zuffi. Retinex preprocessing of uncalibrated images for color based image retrieval. In CBMI2001 Second International IEEE Workshop on Content Based Multimedia and Indexing, Brescia (Italy), 2001.