Lighting and Photometric Stereo Photometric Stereo HW will be on web later today CSE5A Lecture 7 Radiometry of thin lenses δa Last lecture in a nutshell δa δa'cosα δacos β δω = = ( z' / cosα ) ( z / cosα ) π d cosα π d 3 Ω = = cosα 4 ( z / cosα ) 4 z δa cosα z = δa' cos β z' π d 3 δp = LΩδAcos β = LδAcos α cos β 4 z δp π d δa 3 E = = L cos α cos β CSE 5A, δa' Winter 4 006 z δa' d E = π cos 4 α L 4 z E: Image irradiance L: emitted radiance d : Lens diameter Z: depth α: Image Angle Irradiance of patch from optical axis BRDF With assumptions in previous slide Bi-directional Reflectance Distribution Function ρ(θ in, φ in ; θ out, φ out ) Ratio of incident irradiance to emitted radiance (θ in,φ in ) ^n Function of Incoming light direction: θ in, φ in Outgoing light direction: θ out, φ out ρ ( x θ, φ ; θ, φ ) CSE 5A, Winter 006 ; in in out out = L Lo ( x; θout, φout ) ( x; θ, φ ) cosθ dω i in in in (θ out,φ out ) CSE 5A, Winter 006 1
Light sources, shading, shadows Lighting Nearby Point Sources Line sources Area sources Function on 4-D space Distant Point sources Function on sphere Spherical Harmonics Shadows Standard nearby point source model ρ d ( x) N ( x) S( x) ( ) r x N S N is the surface normal ρ is diffuse (Lambertian) albedo S is source vector - a vector from x to the source, whose length is the intensity term works because a dot-product is basically a cosine CSE 5A, Winter 006 CSE 5A, Winter 006 Line sources Area sources Examples: diffuser boxes, white walls. The radiosity at a point due to an area source is obtained by adding up the contribution over the section of view hemisphere subtended by the source change variables and add up over the source See Forsyth & Ponce or a graphics text for details. radiosity due to line source varies with inverse distance, if the source is long enough CSE 5A, Winter 006 CSE 5A, Winter 006 Standard distant point source model Assume that all points in the scene are close to each other with respect to the distance to the source. Then the source direction doesn t vary much over the scene, and the distance doesn t vary much either. We can roll the constants together to get: N S ( x) ( N( x) S ) ρ d Illumination in terms of Spherical Harmonics Express lighting L as sum spherical harmonics Y l,m (θ,φ) Band l L( θ, φ) L Y ( θ, φ) = l= 0 m= l lm, lm, Ylm, ( θ, ϕ) l=0 1 l=1 y z x CSE 5A, Winter 006 CSE 5A, Winter 006 l=. xy yz 3z 1 zx x y m=- m=-1 m=0 m=1 m=
Shadows cast by a point source Area Source Shadows A point that can t see the source is in shadow For point sources, the geometry is simple Cast Shadow Attached Shadow 1. Fully illuminated. Penumbra 3. Umbra (shadow) CSE 5A, Winter 006 At the top, geometry of a gutter with triangular cross-section; below, predicted radiosity solutions, scaled to lie on top of each other, for different albedos of the geometry. When albedo is close to zero, shading follows a local model; when it is close to one, there are substantial reflexes. Irradiance observed in an image of this geometry for a real white gutter. Figure from Mutual Illumination, by D.A. Forsyth and A.P. Zisserman, Proc. CVPR, 1989, copyright 1989 IEEE Figure from Mutual Illumination, by D.A. Forsyth and A.P. Zisserman, Proc. CVPR, 1989, copyright 1989 IEEE Shading reveals 3-D surface geometry Photometric Stereo 3
An example of photometric stereo Shape-from-shading: Use just one image to recover shape. Requires knowledge of light source direction and BRDF everywhere. Too restrictive to be useful. Photometric stereo: Single viewpoint, multiple images under different lighting. 1. Arbitrary known BRDF, known lighting. Lambertian BRDF, known lighting 3. Lambertian BRDF, unknown lighting. Input Images Photometric Stereo: General BRDF and Reflectance Map Coordinate system Coordinate system n f(x,y) f(x,y) y y x Surface: s(x,y) =(x,y, f(x,y)) Tangent vectors: s( x, y) f = 1,0, s( x, y) f = 0,1, Normal vector s s n = = sx sy f f =,, 1 x Gradient Space: (p,q) f f p =, q = Normal vector s s f f n = =,, 1 1 n ˆ = ( p, q, 1) T p + q + 1 T 4
Image Formation E(x,y) n s a A Reflectance Map n s a E(x,y) For a given point A on the surface, the image irradiance E(x,y) is a function of 1. The BRDF at A. The surface normal at A 3. The direction of the light source. Let the BRDF be the same at all points on the surface, and let the light direction s be a constant. 1. Then image irradiance is a function of only the direction of the surface normal.. In gradient space, we have E(p,q). Example Reflectance Map: Lambertian surface E(p,q) For lighting from front Light Source Direction, expressed in gradient space. Reflectance Map of Lambertian Surface Two Light Sources Two reflectance maps Emeasured Emeasured 1 E.g., Normal lies on this curve What does the intensity (Irradiance) of one pixel in one image tell us? It constrains normal to a curve A third image would disambiguate match 5
Reflectance Map of Lambertian + Specular Surface Photometric stereo: Step1 1. Acquire three images with known light source direction.. Using known source direction & BRDF, construct reflectance map for each direction. 3. For each pixel location (x,y), find (p,q) as the intersection of the three curves. 4. This is the surface normal at pixel (x,y). Over image, this is normal field. Normal Field Plastic Baby Doll: Normal Field Next step: Go from normal field to surface Recovering the surface f(x,y) Many methods: Simplest approach 1. From estimate n =(n x,n y,n z ), p=n x /n z, q=n y /n z. Integrate p=df/dx along a row (x,0) to get f(x,0) 3. Then integrate q=df/dy along each column starting with value of the first row f(x,0) 6
What might go wrong? What might go wrong? Integrability. If f(x,y) is the height function, we expect that f f = Height z(x,y) is obtained by integration along a curve from (x 0, y 0 ). z( x, y) = z( x0, y0) + ( pdx + qdy) If one integrates the derivative field along any closed curve, on expects to get back to the starting value. Might not happen because of noisy estimates of (p,q) ( x, y) ( x0, y0 ) In terms of estimated gradient space (p,q), this means: p q = But since p and q were estimated indpendently at each point as intersection of curves on three reflectance maps, equality is not going to exactly hold Horn s Method [ Robot Vision, B.K.P. Horn, 1986 ] Formulate estimation of surface height z(x,y) from gradient field by minimizing cost functional: Image ( z p) + ( z q) dxdy x where (p,q) are estimated components of the gradient while z x and z y are partial derivatives of best fit surface Solved using calculus of variations iterative updating z(x,y) can be discrete or represented in terms of basis functions. Integrability is naturally satisfied. y We stopped here, but rest of slides are included for those looking to get a head start on the next assignment. Lambertian Surface ^s ^n II. Photometeric Stereo: Lambertian Surface, Known Lighting e(u,v) At image location (u,v), the intensity of a pixel x(u,v) is: e(u,v) = [a(u,v) ^n(u,v)] [s 0 ^s ] = b(u,v) s where a(u,v) is the albedo of the surface projecting to (u,v). n(u,v) is the direction of the surface normal. s 0 is the light source intensity. s is the direction to the light source. a 7
Lambertian Photometric stereo If the light sources s 1, s, and s 3 are known, then we can recover b from as few as three images. (Photometric Stereo: Silver 80, Woodham81). [e 1 e e 3 ] = b T [s 1 s s 3 ] i.e., we measure e 1, e, and e 3 and we know s 1, s, and s 3. We can then solve for b by solving a linear system. T b = [ e ][ ] 1 1 e e3 s1 s s3 Normal is: n = b/ b, albedo is: b What if we have more than 3 Images? Linear Least Squares [e 1 e e 3 ] = b T [s 1 s s 3 ] Rewrite as e = Sb where e is n by 1 b is 3 by 1 S is n by 3 Let the residual be r=e-sb Squaring this: r = r T r = (e-sb) T (e-sb) = e T e-b T S T e + b T S T Sb (r ) b =0 - zero derivative is a necessary condition for a minimum, or -S T e+s T Sb=0; Solving for b gives b= (S T S) -1 S T e Input Images Recovered albedo Recovered normal field Surface recovered by integration 8
Lambertian Photometric Stereo Reconstruction with albedo map Without the albedo map Another person No Albedo map III. Photometric Stereo with unknown lighting and Lambertian surfaces Covered in Illumination cone slides 9