Color in Image & Video Processing Applications

Transcription

1 Color in Image & Video Processing Applications DAGM 2010 Darmstadt Joost van de Weijer Universitat Autonoma de Barcelona Computer Vision Center Why use Color? photometric invariance discriminative power saliency detection 1

2 Physics approach to color { RGB,, } e blue 14-green 15-red 16-yellow 17-magenta 18-cyan c b s s overview e c b THEORY PART I: features color spaces color feature etraction saliency detection. PART II: color constancy bottom-up color constancy top-down color constancy color constant features APPLICATIONS IMAGE CLASSIFICATION combining color and shape IMAGE SEGMENTATION deviations from the reflection model OBJECT RECOLORING comple reflection models 2

3 Applications classification segmentation Object recoloring human segmentation 2 Color Basics - human vision - physics based reflection model 3

4 Human vision Human vision Humans have two types of photoreceptors: rods: achromatic night time vision when light levels are low. cones: color vision during day time when light levels are high. 4

5 Human vision Humans have two types of photoreceptors: rods: achromatic night time vision when light levels are low. cones: color vision during day time when light levels are high. Normalized responsivity S M L opponent color system The information from these receptors is combined in the retina as colour opponent signals through the optic nerve. Retinal neurons appear to encode both achromatic contrast systems and chromatic contrast systems. g b R r 2 r g B b 2 r b G g 2 r g r g Y b 2 2 RG R G BY B Y ((r,g,b) normalized RGB) Opponent mechanism [Itti PAMI 95] 5

6 opponent color system The information from these receptors is combined in the retina as colour opponent signals through the optic nerve. Retinal neurons appear to encode both achromatic contrast systems and chromatic contrast systems. - black-white + + blue-yellow red-green O1 1 O2 1 O R 2G 1 B Opponent mechanism Linear combination of RGB CIE 1931 standard observer computing the matching function for humans primary stimuli: R=700nm G=546.1nm B=435.8nm Projected illuminants at sub-intervals = 5nm Image courtesy Bill Freeman 6

7 CIE 1931 standard observer Image courtesy Bill Freeman CIE 1931 standard observer Image courtesy Bill Freeman 7

8 CIE 1931 standard observer Image courtesy Bill Freeman CIE 1931 standard observer Image courtesy Bill Freeman 8

9 CIE 1931 standard observer Image courtesy Bill Freeman CIE 1931 standard observer Image courtesy Bill Freeman 9

10 CIE 1931 standard observer Image courtesy Bill Freeman CIE 1931 standard observer Eperiment result (Commission International de l Eclairage) Visuall field = 2 degrees Interval = [380,780] Sub-intervals = 5nm. primary stimuli: R=700nm G=546.1nm B=435.8nm Every position gives the miing weights for the primary stimuli. The resulting functions are the color matching functions r( ), g( ), b( ) 10

11 XYZ color space Imaginary space: XYZ (Standard colorimètric Observer (CIE 1931)) Goal: Avoid negative values in the color matching functions. This simplifies the constructions of color measurement devices. Correlate one of the functions to the intensity. Chromaticity.ee CIE diagram X X Y Z Y y X Y Z Z z X Y Z 1 1 1, y, z,, white:

12 MacAdam Ellipses If two stimuli are presented, one of them at the center of the ellipse, and the other one inside the ellipse, there is not a perceived difference. The space is not perceptually uniform (then the ellipses would be circles). Perceptually uniform space CIELUV and CIELAB are eamples of perceptually uniform color spaces. CIELUV 1 3 Y Y if k * L Yn Yn Y Y if k Yn Yn * X Y a 500 f f X n Yn * Y Z b 200 f f Yn Z n 3 if k f if k 116 k * E ab L * 2 a * 2 b * 2 12

13 srgb srgb: standard RGB color space (HP, Microsoft, Pantone,Corel etc). Linear relation with XYZ space: Rlinear X Glinear Y Blinear Z srgb images are assumed to have a gamma=2.2. In case of unknown origin people often assume srgb. For eample in the case of data sets collected from the internet. Device dependent color spaces RGB: output of majority of cameras and is dependent on the camera sensitivity response. CMY(K): Is the standard generally applied for printing devices. HSI: human perception (others HSV, HSL) G B 3 2 si R ma{ R, G, B} B R H si G ma{ R, G, B} RG si B ma{ R, G, B} 2 si L S 2 si L ma{ R, G, B} min{ R, G, B} I RG B 3 ma{ R, G, B} min{ R, G, B} 2 13

14 Device dependent color spaces RGB: output of majority of cameras and is dependent on the camera sensitivity response. CMY(K): Es the standard generally applied for printing devices. HSI: often used in computer vision Orthonormal opponent space: O1 2 2 R O2 G O3 B O1 H arctan O2 S I O3 2 O1 O2 2 O1 R B O3 O2 f S is not normalized with intensity! f G Projection of f on O3 is the intensity. Let f the projection on the plane O1-O2, its length is the satutation and its angle the hue. Guidelines application of color spaces be aware that many color spaces are devise dependent (industrial applications). apply CIELAB or CIELUV only when perceptually p uniformity of the color space is relevant. Learning color names tracking when you want to decorrelate only luminance use opponent color space. HSI spaces introduces non-linearities. Choose the color space which fits the photometric invariants your application requires. 14

15 2 Reflection Models Electromagnetic radiation spectrum log 10 ( ( m)) Radio TV Radar Microwaves Infrared Visible light Ultraviolate X rays Gamma rays (m) (nm)

16 visible light spectrum Purple green yellow orange nanometers 1nm10 9 m Spectra.ee slide credit: R. Baldrich Surface reflectance { RGB,, } e blue 14-green 15-red 16-yellow 17-magenta 18-cyan c b s? 16

17 Dichromatic reflection model Two types of reflection: Body reflection Surface reflection Normal Metals: mainly specular reflection. Dielectric: specular and diffuse reflection. It includes materials that are coloured using pigments: paints, plastic, ceramic, fabric, paper, etc. Body Reflectance Colorant particles Reflecting materials Body Reflectance Diffuse reflection, isotropic reflection. The spectral distribution depends on colorants. f (, ) m ( ) c ( ) b b b Surface Reflectance Specular reflection. The reflection angle is similar to the incident angle. Its spectral distribution depends on the illuminant. f (, ) m( ) c ( ) m( ) h s s s s slide credit: R. Baldrich 17

18 Dichromatic reflection model: b f(, ) f (, ) f (, ) b f (, ): Reflected light by the object body. It depends on the pg pigments used to colour the object and it s the one that makes the object look coloured. (Diffuse reflectance) fs(, ): Reflected light from the surface. It has a SPD nearly the : same as the incident light. (Specular or regular reflectance) Angles that depend on light source position, observer and surface The spectral and geometrical terms can be separated: f, m c m c b b f mbc b msc s s s s slide credit: R. Baldrich Dichromatic Reflection Model dichromatic model for matte surfaces: f mbc b RGB-histogram 18

19 Dichromatic Reflection Model dichromatic model for specular surfaces: f m c b m c s b s RGB-histogram color spaces: normalized RGB normalized RGB is given by: R G B rgb,, {,, } R GB RGB RGB invariant for shadow and shading variations (matte surfaces): R r R G B m c b b R m c m c b b R b b G m c b b B c b r c c b r b g c b b normalized RGB 19

20 color spaces: hue-saturation-intensity 3 R G defined as: hue arctan R G2B sat R G B RG RB GB 3 RGB i 3 hue is invariant for shading variations and specularities under white light: hue arctan m b b 3m s c c b R b s b s cr c cg c b s b c c 2c 2c G B s hue Take care of instabilities when working in different color spaces always take instabilities into account! Error propagation is a convenient tool for instability evaluation: q q() q ma q best q q q min best best best Suppose that t u,..., w are measured with corresponding uncertainties titi,..., to compute function q( u,..., w). The predicted uncertainty is defined by : 2 q q q u... w u w 2 u w 20

21 Take care of instabilities when working in different color spaces always take instabilities into account! Error propagation is a convenient tool for instability evaluation: E R G 2 3 R G hue arctan R G2B hue arctan R G 2B hue 2 hue 2 hue 2 hue R G B R G B R (assuming R G B) 2 sat Take care of instabilities when working in different color spaces always take instabilities into account! Error analysis is a convenient tool for instability evaluation: count The red bobsled is dominated by the blue sky and blue snow. hue sat saturation hue 21

22 References: photometric invariants S.A. Shafer. Using color to separate reflection components. Color research and applications, G.J. Klinker et al.. A physical approach to color image understanding. IJCV, M.J. Swain, D.H. Ballard. Color indeing. IJCV, 1991 T. Gevers, A.W.M. Smeulders. Color based object recognition. Pattern Recognition, J.M. Geusebroek et al. Color Invariance. PAMI, T. Gevers, H. Stokman. Robust histogram construction from color invariance for object recognition. PAMI, J. van de Weijer, C. Schmid. Coloring local l feature etraction. ti ECCV, 2006 B.A. Mawell et al. A Bi-illuminant Dichromatic Reflection Model for Understanding Images, CVPR, T. Zickler et al. Color Subspaces as Photometric Invariants. IJCV, Color Differential Structure 22

23 differential-based computer vision 1. How do we combine the differential structure of the various color channels? 2. How do we incorporate color invariance theory into the measurements of the differential structure while maintaining robustness? isoluminance luminance gradient: isoluminant edges are not detected. 23

24 Color Feature Detection 0-order structure 1-order structure Color Feature Detection from luminance to color vector: R + G = 0 + = red green 2 channel test image + = tensor: 2 R RR y R Ry 2 Ry + 2 G GG y G Gy 2 G y = 2 2 R G RRy GG y R Ry GGy 2 2 R y Gy DiZenzo. Note on the Gradient of a Multi-Image, Computer Vision, Graphics, and Image Processing,

25 feature detection in oriented patterns more tensor based features: Harris corner points symmetry points (star and circle structures) optical flow orientation estimation curvature estimation... oriented teture traditional orientation estimation: f y f y arctan arctan f f tensor based orientation estimation: 2f f y 2f f y arctan arctan 2 2 f 2 2 f y f f y differential-based computer vision 1. How do we combine the differential structure of the various color channels? 2. How do we incorporate color invariance theory into the measurements of the differential structure while maintaining robustness? 25

26 Photometric Invariant Edge Detection we differ between three types of edges 1. material edge 2. shadow/shading edge shading 3. specular edge shadow assumptions: 1. white illumination 2. neutral interface reflection 3. shadows are not colored. material specularity Computation of quasi-invariance IMAGE nonlinear transformation FULL INVARIANT FULL INVARIANT DERIVATIVE linear operation DERIVATIVES QUASI- INVARIANT DERIVATIVE hue hue 26

27 Dichromatic Model dichromatic model: b C b s C s intensity illuminant F e ( m C m C ) specular body e i i m C b b m C colorant first order photometric structure: b b F { R, G, B} m C em b b b em C em C material ( shadow shading ) specular i i Shadow-Shading-Specular Quasi-Invariant B B B R G R G = R G spherical coordinates shading variant shading invariant opponent colors specular variant specular invariant hue-saturation-intensity shading-specular variant shading-specular invariant 27

28 spherical coordinates For matte surfaces : f b b m c B all shadow-shading variation is in the radial direction uncertainty of c G R shadow/shading direction R r r 0 spherical f G r 0 r B sin 0 sin c 0 sin hue-saturation-intensity For specular surfaces : b b f m c s s m c there is no specular-shadow-shading variation in the hue-direction. uncertainty of h the hue direction f R sh 0 h hsi G s s s 0 B i i 0 h h

29 invariant edge detection applications Colo Feature Etraction Multi Image Applications image retrieval Single Image Applications Color Feature Detection snakes feature etraction Instabilities shadow-shading hd hdi invariance: i lim { R, G, B} 0 test-image invariant full quasi specular-shadow-shading invariance: lim { R, G, B } {1,1,1} 29

30 Edge Detection eperiments conducted on pantone colorset (1012) which is used to compose edges. edge detection is based on the maimum response path of the derivative energy. edges are tested on - edge displacement. - percentage of missed edges. Edge Detection eperiments conducted on pantone color set (1012) which is used to compose edges. edge detection is based on the maimum response path of the derivative energy. edges are tested on - edge displacement. - percentage of missed edges. shadow-shading: full quasi specular-shadow-shading: full quasi Conclusion: Quasi invariants more than half the edge displacement, and have higher discriminative power. 30

31 eperiments : canny edge detection luminance-gradient RGB-gradient eperiments : canny edge detection shadow-shading quasi-invariant shadow-shading-specular quasi-invariant 31

32 Edge Classification shadow edges F F F r, O3 Edge Classification specular edges F FO 3, 32

33 Edge Classification red - object edge green-shading/shadow edge Blue specular edge Photometric Invariant Corner Detection Harris corner detector combined with the quasi-invariants allows for photometric invariant corner detection RGB shadow-shading specularshadow-shading 33

34 eperiments : Hough transform RGB-gradient shadow-shading-specular quasi-invariant references: color differential structure S. DiZenzo. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing, G. Sapiro and D. Ringach. Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Image Processing, J.M. Geusebroek et al. Color Invariance. IEEE Trans. Pattern Analysis and Machine Intelligence, J. van de Weijer, Th. Gevers, J-M Geusebroek. Quasi-invariant edge and corner detection, IEEE Trans. Pattern Analysis and Machine Intelligence, J. van de Weijer, Th. Gevers, A.W.M. Smeulders, Robust Photometrical Invariant Features from the Color Tensor, IEEE T. Image Processing,

35 52 Color Salient Features Saliency Detection Goal: direct our gaze rapidly towards objects of interest in our environment. Visual attention is know to be driven by both bottom up (image based) and top-down (task based) cues. Bottom-up saliency uses simple visual attributes such as intensity, contrast, color opponency, orientation, direction and velocity of motion. What matters is feature contrast rather than absolute feature strength (as in center surround systems). 35

36 overview approach L.Itty, C. Koch Computational Modelling of Visual Attention, Nature Reviews Neurosciende, Computational Modeling of Visual Attention L.Itty, C. Koch Computational Modelling of Visual Attention, Nature Reviews Neurosciende,

37 black-white focus of detectors luminance-based points color-based points color distinctiveness the information content of an event, v, is equal to : I v log p v log( p f p f p f ) y v R G B equation differential-based salient point detectors : R G B R y H G y f, f B y y To change from strength to information content of edges: Color Boosting Saliency: p f pf ' gf gf ' 37

38 statistics of color images: The statistics of database. f is computed by looking of the images of the Corel RGB opponent colorspace Isosalient surfaces can be approimated by aligned ellipsoids in decorrelated color spaces. statistics of color images: Color Boosting Saliency: p f pf ' gf gf ' RGB opponent colorspace 38

39 color boosting: The statistics of Corel database. f is computed by looking of the images of the color boosting: J. van de Weijer, Th. Gevers, A. Bagdanov, Boosting color saliency in image feature detection, IEEE PAMI Color boosting: decorrelation t U whitening derivatives RGB color space color boosting: derivatives opponent color space derivatives color boosted space J. van de Weijer, Th. Gevers, A. Bagdanov, Boosting color saliency in image feature detection, IEEE PAMI

40 bottom-up color attention: eamples: input image color edges color boosted edges bottom up attention saliency points input car-image RGB-based (first 20 points) saliency boosting (first 4 points) 40

41 generality approach: global optimal regions RGB gradient color boosting eperiment: image retrieval Quantitative evaluation of color boosting on a retrieval eperiment. Nister database: around images detector: DoG (color boosted) descriptor: SIFT+hue 41

42 eperiment: image retrieval retrieval score theoretical maimum amount of boosting color boosting improves results between 5-10 percent the obtained maimum score is equal to the theoretical maimum. eperiment: edge detection Berkeley Segmentation Dataset and Benchmar 42

43 The do s and dont s of Color Features 1. Take care in combining different channels: Tensor-based features solve the opposing vector problem. 2. Look at what kind of photometric invariance your problem needs: Do not take derivatives of circular color spaces. Compute first derivatives, then color space transform. Quasi-invariants are more stable for feature detection. 3. When working with invariance take instabilities into account. Use error analysis to find certainty measures for your invariants. 4. When considering photometric invariance always also take discriminative power into account. 5. From information theory an optimal color space for salient feature detection can be derived. 6. Color information is highly corrupted in compressed data. In compression (jpeg, mpeg) chrominance is subsampled. Questions? 43