Photometric Stereo. Photometric Stereo. Shading reveals 3-D surface geometry BRDF. HW3 is assigned. An example of photometric stereo

Transcription

1 Photometric Stereo Photometric Stereo HW3 is assigned Introduction to Computer Vision CSE5 Lecture 6 Shading reveals 3-D surface geometry Shape-from-shading: Use just one image to recover shape. Requires knowledge of light source direction and BRDF everywhere. oo restrictive to be useful. Photometric stereo: Single viewpoint, multiple images under different lighting.. Arbitrary known BRDF. Lambertian BRDF, known lighting 3. Lambertian BRDF, unknown lighting. An example of photometric stereo BRDF Bi-directional Reflectance Distribution Function ρ(θ in, φ in ; θ out, φ out ) Function of Incoming light direction: θ in, φ in Outgoing light direction: θ out, φ out (θ in,φ in ) ^n (θ out,φ out ) Input Images Ratio of incident irradiance to emitted radiance

2 Photometric Stereo: General BRDF and Reflectance Map Coordinate system Gradient Space (p,q) n f(x, f(x, y y x Surface: s(x, =(x,y, f(x,) angent vectors: s( x, f =,0, s( x, f = 0,, Normal vector s s n = = sx sy f f =,, x Gradient Space : (p,q) f f p =, q = Normal vector s s f f n = =,, n ˆ = ( p, q, ) p + q + Image Formation E(x, n s a P Reflectance Map n s a E(x, For a point p on the surface, the image irradiance E(x, under a distant source is a function of. he BRDF at p. he surface normal at p 3. he direction of the light source. Now, consider the BRDF to be the same at all points on the surface, and let the light direction s be in a constant direction over the surface.. hen image irradiance is a function of only the direction of the surface normal.. In gradient space, we have R(p,q).

3 Example Reflectance Map: Lambertian surface R(p,q) For lighting from front Light Source Direction, expressed in gradient space. Reflectance Map of Lambertian + Specular Surface Reflectance Map of Lambertian Surface E.g., Normal lies on this curve What does the intensity (Irradiance) of one pixel in one image tell us? It constrains normal to a curve wo Light Sources wo reflectance maps A third image would disambiguate match Emeasured Emeasured Photometric stereo: Step. Offline, use known source direction & BRDF to construct reflectance map for each direction.. Acquire three images with known light source direction. 3. For each pixel location (x,, find (p,q) as the intersection of the three curves. 4. his is the surface normal at pixel (x,. Over image, this is normal field. 3

4 Normal Field Plastic Baby Doll: Normal Field Next step: Go from normal field to surface Recovering the surface f(x, Many methods: Simplest approach. From normal field n =(n x,n y,n z ), p=n x /n z, q=n y /n z. Integrate p=df/dx along a row (x,0) to get f(x,0) 3. hen integrate q=df/dy along each column starting with value of the first row f(x,0) What might go wrong? Height z(x, is obtained by integration along a curve from (x 0, y 0 ). z( x, = z( x0, y0) + ( pdx + qd If one integrates the derivative field along any closed curve, on expects to get back to the starting value. Might not happen because of noisy estimates of (p,q) ( x, ( x0, y0 ) What might go wrong? Integrability. If z(x, is the height function, we expect that z z = In terms of estimated gradient space (p,q), this means: p q = But since p and q were estimated indpendently at each point as intersection of curves on three reflectance maps, equality is not going to exactly hold 4

5 Horn s Method [ Robot Vision, B.K.P. Horn, 986 ] Formulate estimation of surface height z(x, from gradient field by minimizing cost functional: Image ( z p) + ( z q) dxdy x where (p,q) are estimated components of the gradient while z x and z y are partial derivatives of best fit surface Solved using calculus of variations iterative updating z(x, can be discrete or represented in terms of basis functions. Integrability is naturally satisfied. y Iterative update Let S i,j be the height function at point i,j and p i,j and q i,j be the measurements. he error is: e [( ) ( ) ] i, j = Si+, j Si, j pi, j + Si, j+ Si, j qi, j 4 Let m i,j be a mask that is for pixels where reconstruction is to occur and 0 otherwise. otal error is: E = m i, jei, j i j Iteratively update the surface, starting at first iteration with S 0 i,j = 0. Lambertian Surface ^s ^n II. Photometeric Stereo: Lambertian Surface, Known Lighting e(u,v) At image location (u,v), the intensity of a pixel x(u,v) is: e(u,v) = [a(u,v) ^n(u,v)] [s 0 ^s ] = b(u,v) s where a(u,v) is the albedo of the surface projecting to (u,v). n(u,v) is the direction of the surface normal. s 0 is the light source intensity. s is the direction to the light source. a Lambertian Photometric stereo If the light sources s, s, and s 3 are known, then we can recover b from as few as three images. (Photometric Stereo: Silver 80, Woodham8). [e e e 3 ] = b [s s s 3 ] i.e., we measure e, e, and e 3 and we know s, s, and s 3. We can then solve for b by solving a linear system. b = [ e ][ ] e e3 s s s3 Normal is: n = b/ b, albedo is: b What if we have more than 3 Images? Linear Least Squares [e e e 3 e n ] = b [s s s 3...s n ] Rewrite as e = Sb where e is n by b is 3 by S is n by 3 Let the residual be r=e-sb Squaring this: r = r r = (e-sb) (e-sb) = e e-b S e + b S Sb b (r )=0 - zero derivative is a necessary condition for a minimum, or -S e+s Sb=0; Solving for b gives b= (S S) - S e 5

6 Input Images Recovered albedo Recovered normal field Surface recovered by integration Lambertian Photometric Stereo Reconstruction with albedo map 6

7 Without the albedo map Another person No Albedo map III. Photometric Stereo with unknown lighting and Lambertian surfaces How do you construct subspace? [ ] [ ] [ e e e 3 ] [e e e 3 ] = b [s s s 3 ] Given three or more images e e 3, estimate B and s i. How? Given images in form of E=[e e ], Compute SVD(E) and B* is n by 3 matrix formed by first 3 singular values. B Matrix Decompositions Definition: he factorization of a matrix M into two or more matrices M, M,, M n, such that M = M M M n. Many decompositions exist QR Decomposition LU Decomposition LDU Decomposition Etc. 7

8 Singular Value Decomposition Excellent ref: Matrix Computations, Golub, Van Loan Any m by n matrix A may be factored such that A = UΣV [m x n] = [m x m][m x n][n x n] U: m by m, orthogonal matrix Columns of U are the eigenvectors of AA V: n by n, orthogonal matrix, columns are the eigenvectors of A A Σ: m by n, diagonal with non-negative entries (σ, σ,, σ s ) with s=min(m,n) are called the called the singular values Singular values are the square roots of eigenvalues of both AA and A A Do Ambiguities Exist? Yes Is B unique? For any A GL(3), B* = BA also a solution For any image of B produced with light source S, the same image can be produced by lighting B* with S*=A - S because X = B*S* = B AA - S = BS When we estimate B using SVD, the rows are NO generally normal * albedo. Result of SVD algorithm: σ σ σ s Surface Integrability In general, B * does not have a corresponding surface. Linear transformations of the surface normals in general do not produce an integrable normal field. B A B* GBR ransformation Only Generalized Bas-Relief transformations satisfy the integrability constraint: λ 0 μ A = G = 0 λ ν 0 0 B G B f ( x, f ( x, = λ f ( x, + μx + νy Generalized Bas-Relief ransformations Objects differing by a GBR have the same illumination cone. Without knowledge of light source location, one can only recover surfaces up to GBR transformations. Uncalibrated photometric stereo. ake n images as input, perform SVD to compute B*.. Find some A such that B*A is close to integrable. 3. Integrate resulting gradient field to obtain height function f*(x,. Comments: f*(x, differs from f(x, by a GBR. Can use specularities to resolve GBR for non- Lambertian surface. 8