Lecture 24: More on Reflectance CAP 5415
Recovering Shape We ve talked about photometric stereo, where we assumed that a surface was diffuse Could calculate surface normals and albedo What if the surface isn t diffuse?
Basic Idea Assume illumination is distant Illumination only depends on orientation of the point on the surface A sphere can tell us the illumination at every surface normal (Image from Ron Dror)
Basic Idea How can I tell the orientation of every point on the bottle?
Results from the bottle
Key Advantage Can work with materials that are difficult to model mathematically
Estimating Reflectance This method works well When a probe is available If a probe is not available, then we have to use the reflectance of the object How can we recover the reflectance?
One option: Use the statistics of the real world Every image is a picture of the illumination of the real world Like images, the histograms of illuminations have a regular appearance (From Dror and Adelson)
Basic method looks at the response of steerable-pyramid responses In some sense, associates different materials with different textures%
Works surprisingly well
94% Accuracy on these materials Similar to human performance Classified based on a number of histogram measurements (percentile values, moments, etc.)
Limitations Works on objects of known geometry What if we have an arbitrary scene? How can we solve the problem then? Make a simplifying assumption: Surfaces are Lambertian Now we only need to recover the albedo
View the image as shading and albedo This is shading These stripes are reflectance changes
Take this image Goal: Find the albedo Observation Shading Information Albedo Information By splitting it into two images that represent the shading and albedo of the scene
Why? The effects of these characteristics are combined together in the image To understand this image, we need to be able to separate the effects of these characteristics
Other reasons Ability to reason about shading and reflectance independently is necessary for most image understanding tasks. Material recognition Image segmentation Want to understand how humans might do the task. An engineering application: for image editing, want access and modify the intrinsic images separately Intrinsic images are a convenient representation. More informative than just the image Less complex than fully reconstructing the scene
Your visual system is trying to solve a similar problem (From Edward Adelson) The squares marked A and B have the same intensity
What is shading? Shading is the interaction of the shape of the surface and the illumination Shading image should look like gray plaster
Albedo Albedo Fraction of light reflected at a point Albedo image represents the albedo at every point on the surface Points inside the squares should reflect less light
An image is formed as x Shading Reflec Image Albedo Image Observed Image
Our goal is to invert the process Observed Image Shading Image Albedo Image Using images to represent characteristics that are intrinsic to the scene s appearance Known as intrinsic image representation [Barrow and Tenenbaum 1978]
Estimating Shading and Albedo Images
First, Simplified Model of the World Observation Albedo Image Shading Image Albedo image Patches of constant albedo Called Mondrian Images (Land and McCann) Shading image Smooth
Given this observation, what s the albedo image? Observation
Estimation in 1-D It s easy to see how to estimate the albedo in 1-D Image Intensity Column in the image We ll work in the log domain, so the problem is additive
Now compute the derivative The locations of the changes in the albedo image are obvious Log Image Intensity Log Image Intensity Column in the image Column in the image
Eliminate the derivatives caused by shading changes Set everything but these derivatives to 0, then re-integrate Log Image Intensity Log Image Intensity Column in the image Column in the image Known as the Retinex Algorithm [Land and McCann] Estimate of the log albedo image
Basic Strategy for Estimating Images Pseudo-Inverse Classify Every Derivative as Shading or Albedo Change Estimate of the horizontal derivative Observation Estimate of the Albedo Image Estimate of the vertical derivative Similar to Horn s Implementation
Where Retinex Breaks Down Unfortunately, this assumption is often broken in the real-world
Retinex Most other work on this problem still uses the same basic assumption (Kimmel 03) Researchers have also tried augmenting the input Use multiple images, like time-lapse [Weiss 2001, Matsushita 2003] Color (Funt et al.) Extra information may not be available Estimating shading and albedo from a single, gray-scale image is a good test our ability to model images and surfaces
Beyond Mondrians Can go beyond the simple world of Mondrian images in two ways: Learn better models of local image statistics Look at an image patch rather than just a single derivative value Propagate information across the image
Why Propagate? One of these patches is an albedo change and the other is shading
Recovering Intrinsic Images Classify each x and y image derivative as being caused by either shading or a reflectance change Recover the intrinsic images by finding the leastsquares reconstruction from each set of labeled derivatives. (Fast Matlab code for that available from Yair Weiss s web page.) Original x derivative image Classify each derivative (White is reflectance)
Outline of our algorithm Gather local evidence for shading or reflectance Color (chromaticity changes) Form (local image patterns) Integrate the local evidence across space. Assume a probabilistic model and use belief propagation. Show results on example images
Probabilistic graphical model Unknown Derivative Labels (hidden random variables that we want to estimate)
Probabilistic graphical model Local evidence Local Color Evidence Derivative Labels Some statistical relationship that we ll specify
Probabilistic graphical model Local evidence Local Form Evidence Local Color Evidence Derivative Labels
Probabilistic graphical model Propagate the local evidence in Markov Random Field. This strategy can be used to solve other low-level vision problems. Local Evidence Hidden state to be estimated Influence of Neighbor
Local Color Evidence For a Lambertian surface, and simple illumination conditions, shading only affects the intensity of the color of a surface Notice that the chromaticity of each face is the same Any change in chromaticity must be a reflectance change
Classifying Color Changes Chromaticity Changes Angle between the two vectors, _, is greater than 0 Intensity Changes Angle between two vectors, _, equals 0 Red Red _ Blue Blue Green Green
Color Classification Algorithm 1. Normalize the two color vectors c 1 and c 2 c 1 c 2 2. If (c 1 c 2 ) < T Derivative is a reflectance change Otherwise, label derivative as shading
Result using only color information
Results Using Only Color Input Shading Reflectance Some changes are ambiguous Intensity changes could be caused by shading or reflectance So we label it as ambiguous Need more information
Utilizing local intensity patterns The painted eye and the ripples of the fabric have very different appearances Can learn classifiers which take advantage of these differences