Colour appearance and the interaction between texture and colour Maria Vanrell Martorell Computer Vision Center de Barcelona
2 Contents: Colour Texture Classical theories on Colour Appearance Colour and Texture for computer vision
3 Goal: How to deal with computer vision problems where we need to computationally represent the properties of colour-textured surfaces from a perceptual point of view?
4 Example 1: ATIC (Automatic Tile Classification) Goal: Automatic classification of porcelanic tiles where differences are due to small differences on texture and colour of uncontrolled conditions on the production process.
5 Example 2: CAOPPP: Colour Adjustment On Printed Paper Production First design Current design Last design Goal: Automatic adjustment of inks to get the same appearance on printed paper designs Improving time / similarity
6 Production Process: Color A + Texture 1 Texture 3 Texture 2 Texture 1 Color B + Texture 2 Color C + Texture 3 Printed Paper Platen 3 Platen 2 Platen 1 Neutral Paper Color C Color B Color A
7 In both cases, ATIC and CAOPPP, we need a computational representation of colour and texture that behaves as human perception does...
8 Questions to answer, how to work computationally with. COLOUR, TEXTURE, AND BOTH at the same time?
9 Questions to answer, how to work computationally with. COLOUR, TEXTURE, AND BOTH at the same time?
10 COLOUR, it has been deeply studied in Physics and there are a general agreement and a wide range of different spaces (RGB, HSI, CYM, ) to represent it and a lot of standard tools (Colorimeters, Spectroradiometers, Spectrophotometers, ) to measure colour in standard spaces (XYZ, CIELAB, CIELUV, CIECAM, )
11 Colour-matching experiment models the human colour perception and becomes the basis for the modern colour science (Wright 1929, Guild 1931, CIE 1931, CIE 1964, CIE 1971) 3 Primary lights Test light (monochromatic) Image taken from B. Wandell - SID Color Tutorial Notes http://white.standford.edu/~brian/refs.html#color
12 Conclusion: Human colour perception can be represented by a three-dimensional space given by three basic colours Y X Z that coincides with the number of different types of cones in the human retina
13.therefore, it is easy to build a computational representation of the colour property of a point using a three-dimensional numerical vector g ( X, Y, Z) ( L, u, v)k f ( R, G, B) ( H, L, S)K (Device-Independent) (Device-Dependent) f, g : Represent colour-conversion transformations that can be linear or non-linear.
14 There are different types of colour spaces: Uniforms (CIELAB, CIELUV) Device-dependent (RGB, CMY) With perceptual dimensions (HLS, HSV, ) Based on Phisiologycal evidences (Opponent) Distance Similarity ( R, G, B) = ( λ1r p, λ2gp, λ3b p ) ( R p, G p, B p ) primaries H V : : Hue Intensity O : 1 O O 2 3 L : S : Luminance Saturation Intensity : Red Green : Blue Yellow
15 Questions to answer, how to work computationally with. COLOUR, TEXTURE, AND BOTH at the same time?
16 Questions to answer, how to work computationally with. COLOUR, TEXTURE, AND BOTH at the same time?
17 TEXTURE, there is not an accepted definition for this visual cue, it is the property of some surfaces
18 We are referring to the appearance of texture images projected on the retina from a surface, not to physical/reflectance properties of a surface Yes Different texture images corresponding to the same physical surface α Not δ Images taken from http://www.cs.columbia.edu/cave/curet
19 Interesting properties of texture images Existence of a textural primitive Existence of a window that is invariant to translations + Regular + Random Existence of a texture scale or texture resolution
20 But, there is a lack of an accepted representation R Y COLOUR Colour Science B TEXTURE? G X??? Z PSYCHOLOGY?
21 Interesting Overview: Theories of Visual Texture Perception [J.R. Bergen. 1991]
22 Theories of texture perception in psychology: B. Julesz J. Beck B. Julesz Global representation Local entities representation J. Beck Texton theory Spatial Frequency Channels
23 Texton theory [Julesz 81] Texture discrimination is based on differences on texton densities. Textons are visible local features that allows to segregate textures and which are considered the basic elements for preattentive perception. The textons are: Blobs, elongated or rounded and its attributes (orientation, length, width and colour). Terminators (ends of elongated blobs). Crossings (elongated blobs intersection).
24 Differences between them can be easily represented by differences in their texton/attributes densities
25 however, texton theory does not explain how to represent differences between these two: Densities of textons are identical for these two textures!! The differences rely on Emergent patterns!!!
26 however, texton theory does not explain how to represent differences between these two: Densities of textons are identical for these two textures!! The differences rely on Emergent patterns PERCEPTUALLY GROUPED PATTERNS
27 Different stimulation of spatial frequency channels Texture representation can not only depend on first-order statistics of textons, but also on the orientation, size and brightness of emergent structures [Beck-86] The emergent structures can be the result of a convolution of the stimulus with a receptive field. I F θ F θ = ο 0 ο 45 ο 90
28 Conclusion: Textures could be represented by the densities of both elements: Blob attributes, and the attributes of their emergent patterns
29
30 Representation based on blob attributes Dark blobs (Non-elongated, High contras Light blobs (Non-elongated, High contrast) Elongated blobs (Orientation=45º, Low contrast) Elongated Blobs (Random orientations High contrast)
31
32 Representation based on attributes of emergent patterns Circles (Bright, Low Structure) Striped (Dark,High Structure) Chequered (Dark, Structured) Circles (Bright, Structured)
33 Multi-scale approaches are probably those that can better consider both aspects...
34 e.g. Computational model for preatentive texture perception [Malik&Perona-91] Family of filters I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) 1 1 2 1 3 1 4 1 5 1 6 1 Differences of offset gaussians scales I I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) 1 2 2 2 3 2 4 2 5 2 6 2 * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) I * f ( σ ) 1 3 2 3 3 3 4 3 Non linear mechanism 5 3 6 3
35 It combines both approaches: Statistical measurements of image blobs from: Small scales (do not break image structure) after applying blob and bar detectors. Statistical measurements of emergent blobs from: Large scales (capturing emergent properties) after applying the same blob and bar detectors. and it still probably needs more complex high-level operators as: corner, t-junctions, x-junctions, circle detectors,.
36 Now, we have an idea on how to represent image textures computationally, but still two questions remain to be answered What is the true dimension of the texture space? (There can be a lot of redundancy when we use a multichannel representation of images)?? Which are they???
37 There are some hipothesis: From psychophysical data [Rao et al. - 96] Dimension Reduction PCA / MDS... Granularity Contrast / Direccionalit Randomness From computational representations [Vanrell et al. - 96] Dimension Reduction MDS... Orientation Scale Contrast
38 A lot of work remain to be done on gray-level texture representation...
39 A lot of work remain to be done on gray-level texture representation...... and color is not considered yet.
40 Questions to answer, how to work computationally with. COLOUR, TEXTURE, AND BOTH at the same time?
41 Questions to answer, how to work computationally with. COLOUR, TEXTURE, AND BOTH at the same time?
42 Contents: Colour Texture Classical theories on Colour Appearance Colour and Texture for computer vision
43 Question: Are colour and texture independent visual cues?
44 IN COLORIMETRY, a lot of different models to built three dimensional perceptual colour spaces (XYX, CIELAB, CIELUV,..) have been developed, and a lot of DEVICES based on the integration of reflected light from a surface area.
45 IN COMPUTER VISION, a lot of measurements are based on histograms of the digital image represented by the RGB space.
46 But, some both approaches are not enough!!!
47 Red (255,0,0) Cyan (0,255,255) Green (0,255,0) Magenta (255,0,255) Blue (0,0,255) Yellow (255,255,0) Grey (128,128,128) Black (0,0,0) White (255,255,255) The same representation for colorimetrist devices and similar histogram statistics!!
48 Conclusion: We need to consider the effects of localisation and grouping of colours to deal with colour appearance
49 In colorimetry the colour appearance phenomena is a current goal: Mark D. Fairchild, Color Appearance Models Addison-Wesley 1998
50 While de CIE system of colorimetry has proven to be exptremely useful, i is important to remember that it has limitations. Most of its limitations are inherent in the design of a system of tristimulus values based on color matching. Such a system can accurately predict color matches for an average observer, but it incorporates none of the information necessary for specifying the color appearance of those matching stimuli. Such is the realm of color appearance models. Tristimulus values can be considered as a nominal scale for color. They can be used to state whether two stimuli match. The specification of color differences requires interval scales, and the description of color appearance requires interval scales (for hue) and raito scales (for brightness, lightness, colorfulness, and chroma). Additional information is needed, in conjunction with tristimulus values, to derive these more sophisticated scales. [Mark D. Fairchild, Color appearance Models, p.163, Addison Wesley, 98]
51 In Computer Vision spatial frequencies are usually processed separately in RGB channels of images. Attention to the following effects!!
52 Given three different channels...
53 C1 C2 C3 C1,C2,C3 C1,C1,C2 C3,C1,C1 C2,C3,C2 C1*G4,C2,C3 C1,C2*G4,C3 C1,C2,C3*G4 C1*G4,C2*G4,C3*G4
54 R G σ G G σ B G σ
55 R G σ G G σ B G σ
56
57 Direct extension of known measurements for gray level images to the colour image channels seems not to be a good solution Let s go deeply into the colour appearance phenomena...
58 COLOUR APPEARANCE PHENOMENA Simultaneous Contrast: Two identical gray patches presented on different backgrounds appear distinct. The black background causes the gray patch to appear lighter, while the white background causes the gray patch to appear darker.
59 Classical examples:
60 COLOUR APPEARANCE PHENOMENA Crispening: Is the increase in perceived magnitude of color differences when the background on which the two stimuli are compared is similar in color to the stimuli themselves.
61 COLOUR APPEARANCE PHENOMENA Spreading: Is the apparent mixture of a color stimulus with its surround. This effect is complete at the point of spatial fusion when the stimuli are no longer viewed as discrte but fuse into a single stimulus.
62 Why does it happen?
63 Color appearance depends on many different elements of the visual pathways, including the optical aberrations of the eye, light adaptation, and the neural computations that interpret objects, light sources, and istance relationships. [K-H Bäuml and Brian A. Wandell, Color appearance of Mixture Gratings, Vision Research, 96.
64 Causes: Light adaptation Chromatic aberration Neural computations to interpret objects... we will see the first two...
65 Light adaptation: Maximum range It is impossible to act on all this range simultaneously
66 Biological causes: Chromatic Aberration f Photoreceptor Mosaic does not present a uniform distribution
67 What does colorimetry propose?
68 Colour appearance models Define colour transformations that allow to go from colorimetric coordinates under one set of viewing conditions to colorimetric coordinates under a second set of viewing conditions VC 1 Colour appearancemodel ( X, Y, Z) ( X, Y, Z) VC 2
69 Colour viewing conditions: Spectral power distribution of the light source Luminance level Surround colour and relative luminance Background colour and relative luminance Image size and viewing distance Viewing geometry involves a large range of psychophysical experimentation.
70 Examples of colour appearance models: Bradford-Hunt 96S (simple model). Bradford-Hunt 96C (comprehensive model). CIECAM97s model. ZLAB colour appearance model.
71 but, in computer vision we usually do not know viewing conditions then, we propose a more computational approach...
72 To this end, let us go deeply on colour texture interactions
73 Image taken from http://ccp.uchicago.edu/~shevell/
74
75
76
77
78
79 What happens? Human perception presents the colour induction phenomena that changes the colour appearance of a stimulus due to the influence of the scene contents.
80 Different types of colour induction: Colour adaptation Colour contrast Colour assimilation Others...
81 Different types of colour induction: Colour adaptation Colour contrast Colour assimilation Others... Colour Constancy ability of the HVS due to global Illuminant effects
82 Image taken from http://www.system-concepts.com/ articles/colour.html
83 Image taken from http://www.system-concepts.com/ articles/colour.html
84
85 Different types of colour induction: Colour adaptation Colour contrast Colour assimilation More related to colour-texture interactions
86 Colour Contrast Appears when the chromaticity of the test stimulus changes away from the chromaticity of the inducing stimulus. Colour Assimilation Appears when the chromaticity of the test stimulus changes towards the chromaticity of the inducing stimulus.
87 Colour Contrast S1 S2 TS TS P1 P2
88 Colour Contrast S1 S2 S2 S2 P1 TS TS S1 TS TS S1 P2 r+g+b=1 P1 P2 r+g+b=1
89 Colour Assimilation
90 Colour Assimilation r+g+b=1 r+g+b=1
91
92 Conclusion: Colour contrast & Colour assimilation seem to be complementary effects Question: When they occur?
93 Some experiments show a relationship between spatial frequency of patterns and colour induction [Pokorny et al-2001]
94 Some experiments shows a relationship between spatial frequency of patterns and colour induction [Pokorny et al-2001] Assimilation Threshold Frecuency 4cpd Contrast
95...Now, the question is: How to consider induction effects on a computational colour-texture representation?
96...Now, the question is: How to consider induction effects on a computational colour-texture representation? Assimilation Blurring operator e.g. I A, σ = I G σ Contrast Sharpening operator 2 I C, σ = I e.g. 2 2 2 ( I G ) ( I G ) I = σ + σ σ 2 2 x I y
97 With these two operators we can simulate human inspection of colour texture surfaces Looking from a long distance a short distance Assimilation or Blurring Contrast or Sharpening Applications
98 Colour-Texture perception as a vision process: Short distance perception: I = I I A, σ n = I G A, σ1 σ1 = M I G σ n Global colour representation that consider texture influence Long distance perception: I = I I C C, σ n = I = I 2, σ σ 1 1 M 2 σn I I Local blob attributes as a texture representation that considers colour influence
99 Proposed Schema for a: Computational approach to colour texture perception Blurr p q......... Blurring output Global Measurements Sharp... I Blurr p q......... Colour-Texture representation Sharp Blurr Sharp p... q............ Sharpening output Local Measurements Blob Segmentation p assimilation scales q assimilation scales
100 Observer distancie + Chromatic Assimilation d < d δ Original Image Chromatic Contrast - d < d δ
101 Colour Perceptual Tower
102 but, there are still some important problems to be solved before to have a complete colour-texture perceptual approach
103 Problem 1: Which are the parameters of the operators that fit with human behaviour on contrast and assimilation? Spatial-CIELAB, has been proposed for colour assimilation by [Wandell et al.-96 ] I A, σ = I 3 i= 1 G σ i σ i : Measured and tabulated from psychophysical experimentation
104 Spatial CIELAB: models human colour assimilation process as a set of gaussian filters in an opponent space f ωi = ω k exp x 2 + y 2 i 2 k = 1K3 σ k i i i : viewing distance factor : normalisation factor ki Example:
105 Convolution with perceptual kernels f = k i w i E i O1 w i 0.921 σ i 0.0283 0.105 0.133 E i = k i 2 ( x + y e i 2 )/σ i O2 0.108 0.531 0.330 4.336 0.0392 0.494 Where the parameters are: O3 0.488 0.371 0.0536 0.386
106 Model behaviour (50, 150, 250) (50, 150, 250) (250, 250, 0) (250, 250, 0) (250, 250, 0) (50, 150, 250) (130, 190, 149) (240, 245, 12) Mean values of gratings after perceptual blurring
107 APPLICATION TO COMPUTER VISION [Boukouvalas-Petrou 96) 1. Remove PSF from the camera sensor 2. Convert from RGB to XYZ standard 3. Convert from XYZ to opponent colour 4. Compute a Perceptual Blurring 5. Compute all the measurements
108 Problem: colour contrast has not been psychophysically measured yet
109 Problem 2: What colour space is the best to fit the parameters? Colour opponent space has presented good properties to fit the parameters for colour assimilation [Wandell et al.-96]
110 Opponent space: represents the HVS pathways. - γ 3 γ 2 B G γ 1 β 3 + - I β 2 + β 1 R α 1 α 2 α 3 + Opp( p) = p Example: t 1 1 1 1 1 0 RGB 1 1 2 Opponent b y g r k w Opp(I)
111 OPPONENT-COLOUR MODEL O1= Y = 0.279X + 0.72Y + 0.107Z O2= RG = 0.279X + 0.72Y + 0.107Z O3= YB = 0.279X + 0.72Y + 0.107Z RG Y YB
112 Different blurring effects on the opponent-colour channels: (Y,RG,YB) (Y*G,RG,YB) (Y,RG*G,YB) (Y,RG,YB*G)
113 Problem 3: The sharpening operator needs to be improved for a true perceptual behaviour Some perceptual sharpening operators have been proposed by [Baldrich-01] I C, σ = I Interpolation( I) 2
114 A good sharpening operator has to consider colour contrast and mantain the structural properties of the image blobs. Example:
115 original Perceptual sharpening MS PhotoEditor moderate MS PhotoEditor strong Corel PhotoPaint
116 Example: blob segmentation with and without the perceptual sharpening original without Perceptually sharpened with
117 Proposed Schema for a: Computational approach to colour texture perception Blurr p q......... Blurring output Global Measurements Sharp... I Blurr p q......... Colour-Texture representation Sharp Blurr Sharp p... q............ Sharpening output Local Measurements Blob Segmentation p assimilation scales q assimilation scales