Photometric*Stereo* October*8,*2013* Dr.*Grant*Schindler* *

Photometric*Stereo* October*8,*2013* Dr.*Grant*Schindler* * schindler@gatech.edu*

Mul@ple*Images:*Different*Ligh@ng*

Religh@ng*the*Scene*

Credit:*Alex*Powell*

Froggies* Demo*

Geometry*of*Ligh@ng* Lamber@an*Surface*

Infinitely*Distant*Light*Source*

Acquiring)the)Reflectance)Field)of)a) Human)Face:)Paul)Debevec)et)al* hop://gl.ict.usc.edu/lightstages/*

More*Info* hop://www.pauldebevec.com/* hop://www.pauldebevec.com/research/ls/* hop://gl.ict.usc.edu/research/digitalemily/* hop://gl.ict.usc.edu/research/rhl/*

RealTTime*Photometric*Stereo* Surface*Normals* (Eigenvectors*via*Power*Itera@on)* 6*ms* Height*Field* (Integrate*Surface*Normals*via*GaussTSeidel)* 32*ms*

RealTTime*Photometric*Stereo*

Processing*Demo* Demo*

Reading** Szeliski*Book:*Sec@on*12.1.1*(page*580T583)*

3D*Scanner*in*Your*Pocket*

How*does*it*work?* Camera*(FrontTFacing)* Reflect*Light* Emit*Light*

How*does*it*work?* 2D* 3D*

Lambertian Surface The intensity of a pixel I(u,v) is: I(u,v) ^ s ^ n a I(u,v) = [a(u,v) ˆ n (u,v)] [s 0 ˆ s ] = b(u,v) s a(u,v) is the albedo of the surface projecting to (u,v). n(u,v) is the direction of the surface normal. s 0 is the light source intensity. s is the direction to the light source. Slide courtesy of David Kriegman

Orthographic Projection Simplification for light sources that are sufficiently far away from an object. All incoming light rays are parallel. Thus, while b vectors vary over the surface, s vector is constant. s b s s b b Pixels : b 1 T s, b 2 T s, b 3 T s,... Bs

Images Live in a 3-D Subspace of All Possible Images Images Surface Normals Light Directions = 3 #pixels #pixels 4 4 3

Background:*Linear*Algebra* hop://www.cc.gatech.edu/~phlosob/ transforms/* Keywords:*Singular*Value*Decomposi@on* Related:*Image*Compression*

Synthetic Sphere Images Five different lighting conditions Slide courtesy of David Forsyth

Recovered Albedo Slide courtesy of David Forsyth

Recovered Surface Normals Slide courtesy of David Forsyth

Recovered Surface Shape Recovery up to a constant depth error (not absolute depth) Slide courtesy of David Forsyth

Integra@on*Methods* TFrankotTChellappa*(FFT)* TGaussTSeidel*(Itera@ve*Condi@onal*Modes)* TPath*Integra@on*

High Quality Results George Vogiatzis and Carlos Hernandez

Photometric Stereo

Photometric Stereo s 1 s 2 s 3 b 2 b 1 b 3 Given multiple images of the same surface under different known lighting conditions, can we recover the surface shape? Yes! (Woodham, 1978)

Photometric Stereo Assume: A set of point sources that are infinitely distant A set of pictures of an object, obtained in exactly the same camera/object configuration but using different sources I 1 = Bs 1 ; I 2 = Bs 2 ; I 3 = Bs 3... A Lambertian object (or the specular component has been identified and removed)

Stereo for a Pixel I s 1 s 2 s 3 b For a pixel (x,y) we have n measurements : I 1 (x, y) =s 1 T b(x, y); I 2 (x, y) =s 2 T b(x, y)... I(x, y) = Sb(x, y) Solve an over-constrained linear system for b (with n>3)

What About Shadows? Shadowed pixels (e.g. attached shadows for a given light source position) are outliers. Max trick can be adapted for this case too: " I 1 (x,y) 0 0 %" I 1 (x, y) % " I 1 (x, y) 0 0 %" T s 1 % $ ' $ ' $ ' $ ' $ 0 0 ' $ ' = $ 0 0 ' $ ' b(x, y) T # $ 0 0 I n (x,y)&' # $ I n (x, y) &' # $ 0 0 I n (x, y) &' # $ s n & ' Pre-multiplying by a thresholded weight matrix zeros the contributions from shadowed pixels

Recovering the Albedo Recall that b(x, y) = a(x, y)ˆ n (x,y) a(x, y) = b(x,y) This gives a check on the normal recovery at a pixel If the magnitude of a(x, y) is greater than 1, there s a problem Then ˆ n (x, y) = b(x, y) /a(x, y)

Recovering the Surface Shape Depth map model (also called Monge patch): Graphic courtesy of David Forsyth z = f (x, y) Surface is set of points {x, y, f (x, y)}

Recovering a surface from normals - 1 If we write the known vector g as " g 1 (x, y) % (x,y, f (x, y)) g(x,y) = $ g 2 (x, y) ' $ ' # g 3 (x, y) & Then we obtain values for the partial derivatives of the surface: Recall the surface is written as This means the normal has the form: N(x,y) = " $ # 1 % " f x % f 2 x + f 2 ' $ f y ' y +1& $ ' # 1 & ( ) ( ) f x (x,y) = g 1 (x, y) g 3 (x, y) f y (x, y) = g 2 (x,y) g 3 (x,y) Slide courtesy of David Forsyth

Recovering a surface from normals - 2 Recall that mixed second partials are equal --- this gives us a check. We must have: ( g 1 (x, y) g 3 (x, y) ) = y ( ) g 2 (x, y) g 3 (x, y) (or they x should be similar, at least) We can now recover the surface height at any point by integration along some path, e.g. f (x, y) = y 0 x f x (s, y)ds + 0 f y (x,t)dt + c Slide courtesy of David Forsyth

Light Sources