Introduction to Computer Vision Week 8, Fall 2010 Instructor: Prof. Ko Nishino
Midterm
Project 2 without radial distortion correction with radial distortion correction
Light Light Light! How do you recover the true color of a scene? " Recovering Lightness! How do you describe reflection? " Radiometry and Reflectance
Recovering Lightness
Land s Experiment (1959)! Cover all patches except a blue rectangle! Make it look gray by changing illumination! Uncover the other patches Color Constancy We filter out illumination variations
Color Cube Illusion D. Purves, R. Beau Lotto, S. Nundy Why We See What We do, American Scientist
Checker Shadow Illusion by Ted Adelson @ MIT
Lightness Recovery and Retinex Theory! Recover surface reflectance/color in varying illumination conditions " Use tools developed before " Sensing: intensity/color " Image processing: Fourier transform and convolution " Edge operators " Iterative techniques
Image Brightness (Intensity) scene irradiance scene reflectance scene radiance sensor response intensity Assume monochromatic light and b' ( x, y) = r' ( x, y)e' ( x,y) Reflectance as a multiplier to irradiance is often called lightness Note: our analysis can be applied to colored light (we ll simply have filters in front of the sensor) Also, assume everything is flat
Lightness Recovery b' ( x, y) = r' ( x, y)e' ( x,y) Can we recover and from? We need assumptions to constrain the solution!! Sharp changes in reflectance! Smooth changes in illumination Frequency spectrum (Fourier transform) We want to filter out e
Lightness Recovery Take logarithm Use Laplacian logb' x,y b' ( x, y) = r' ( x, y)e' ( x, y) ( ) = log r' ( x, y) + loge' ( x,y) ( ) = r( x, y) + e( x, y) b x, y d = " 2 b = " 2 r + " 2 e! Sharp changes in reflectance " 2 = #2 #x 2 + #2 #y 2 has 2 infinite spikes near edges and! Smooth changes in illumination elsewhere everywhere
Lightness Recovery (Retinex Scheme) Image: Laplacian : Thresholding : Reconstruction : (lightness) (reflectance)
Solving the Inverse Problem Image Laplacian Thresholding Lightness Find lightness l(x,y) from t(x,y) Poisson s Equation # % $ " 2 "x 2 + "2 "y 2 & ( l x, y ' ( ) = t( x, y) We have to find g(x,y) which satisfies ( ) = t u,v l x, y $ # "# $ # "# ( )g( x " u,y " v)dudv l( x, y) = t( x, y) " g( x, y)
Solving Poisson s Equation We have " 2 l( x, y) = t( x, y) and l( x, y) = t( x, y) " g( x, y) So "( u 2 + v 2 )L u,v Fourier transform? = T( u,v) and ( ) = T( u,v) and L( u,v) = T( u,v)g( u,v) L( u,v) = T( u,v)g( u,v) Thus G( u,v) = " 1 u 2 + v 2 g( x, y) = 1 2" log x 2 + y 2 + c See Robot Vision page 123,124
Lightness Recovery Log of image Which means: Normalize: Assume maximum value of Then: ( ) = k + r( x, y) l x, y (lightness) (reflectance) l' x, y l' x, y r' ( x, y) = (reflectance) l max ( ) = kr' ( x, y) ( ) corresponds to b' x, y e' ( x, y) = (illumination) r' x, y g( x, y) = 1 2" log x 2 + y 2 ( ) ( )
Computing Lightness (Discrete Case) Log of image 1 4 1 4-20 4 1 4 1 basically inverse of Laplacian mask But, inverse of a discrete approximation of the Laplacian would not be an exact inverse of the Laplacian Solve directly
Computing Lightness (Discrete Case) 1 4 1 4-20 4 1 4 1 1 4 1 4-20 4 1 4 1 "20l i, j + 4( l i+1, j + l i, j "1 + l i"1, j + l ) i, j "1 + ( l i+1, j +1 + l i"1, j +1 + l i"1, j "1 + l ) i+1, j "1 = 6t i, j Solve iteratively n l +1 i, j = 1 ( 5 l n + l n i+1, j i, j+1 + l n i"1, j + l ) n i, j"1 + 1 ( 20 l n + l n i+1, j+1 i"1, j+1 + l n i"1, j"1 + l ) n i+1, j"1 " 3 10 t i, j Use a discrete approximation to the inverse of Laplacian to obtain initial estimate of l
Lightness from Multiple Images taken under Varying Illumination Use spatial statistics of edges Illumination is not smooth Derivative operator responses are sparse ( ) = "r( x, y) + "e( x, y,t) "b x, y,t ( ) # median( "b( x,y,t) ) "r x, y t Y. Weiss ICCV 2001
Lightness from Multiple Images taken under Varying Illumination Y. Weiss ICCV 2001
Lightness from Multiple Images taken under Varying Illumination Y. Weiss ICCV 2001
Using Lightness w/o decomposition w/ decomposition Y. Weiss ICCV 2001
Using Lightness Y. Matsushita and K. Nishino CVPR 2003
Comments on Lightness Recovery! Not applicable to smooth reflectance variations! Not applicable to curved objects In general Intensity=f( Shape, Reflectance, Illumination )
Appearance! How do you describe reflection? " Radiometry and Reflectance! Can we recover geometry from shading? When can we do that? " Photometric Stereo
Radiometry and Reflectance
Radiometry and Image Formation To interpret image intensities, we need to understand Radiometric Concepts and Reflectance Properties " Image Intensities: Overview " Radiometric Concepts: " Radiant Intensity " Irradiance " Radiance " Bidirectional Reflectance Distribution Function (BRDF) " Image Formation Radiometry vs. Photometry?
Light!! Geometric Optics " Light rays " Emission, Reflection/Refraction, Absorption! Wave Optics " Electromagnetic wave (Maxwell s equations) " Diffraction, Interference, Polarization! Quantum Optics " Photons " Fluorescence, Phosphorescence
Geometric Optics heat Emission Reflection Refraction Absorption
Snell s law for air
Image Intensities source surface normal sensor image intensity surface element Image intensity understanding is an under-constrained problem!
Radiometric Concepts Solid Angle : Solid Angle subtended by : Foreshortened Area What is the solid angle subtended by a hemisphere?
Radiometric Concepts flux Radiant Intensity Light flux (power) emitted per unit solid angle : Flux Surface Irradiance Light flux (power) incident per unit surface area Does not depend on where the light is coming from
Radiometric Concepts Surface Radiance (Brightness) Light flux (power) emitted per unit foreshortened area per unit solid angle L = d" ( dacos# r )d$ foreshortened area ( W/m 2 str) : Flux L depends on direction Surface can radiate into whole hemisphere L is proportional to irradiance E Depends on reflectance properties of surface
Sun Example Total flux: 3.91x10 26 W" Surface area: 6.07x10 18 m 2 " Radiance by Steve Marschner
Sun Example Total flux: 3.91x10 26 W" Surface area: 6.07x10 18 m 2 " Radiance by Steve Marschner
Sun Example! Same flux on Earth and Mars?
Sun Example! Same flux on Earth and Mars? Radiance does not change along the ray! fall off of flux
Radiometric Image Formation (Scene Radiance to Image Irradiance) image plane lens scene Image Irradiance: Scene Radiance: Solid angles:
Radiometric Image Formation (Scene Radiance to Image Irradiance) image plane lens scene Image Irradiance: Scene Radiance: Solid Angle subtended by lens
Radiometric Image Formation (Scene Radiance to Image Irradiance) image plane lens scene Image Irradiance: Scene Radiance: Flux received by lens from = Flux projected onto (image) Image Irradiance
Radiometric Image Formation (Scene Radiance to Image Irradiance) We have Solid Angles: Solid Angle subtended by lens: Flux: Image Irradiance: We get image irradiance scene radiance Image irradiance is proportional to scene radiance Telephoto lenses have narrow field of view effects of are small