Photometric Stereo.

Transcription

1 Photometric Stereo

2 Photometric Stereo v.s.. Structure from Shading [1] Photometric stereo is a technique in computer vision for estimating the surface normals of objects by observing that object under different lighting conditions. The technique was originally introduced by Woodham in Woodham, R.J Photometric method for determining surface orientation from multiple images. Optical Engineerings 19, I, The special case where the data is a single image is known as shape from shading, and was analyzed by B. K. P. Horn in B. K. P. Horn, Obtaining shape from shading information. In B. K. P. Horn and M. J. Brooks, eds., Shape from Shading, pages MIT Press.

3 Radiometry Units [7]

4 Radiometry Units

5 Radiometry Units

6 Radiometry Units [7] Per unit time Per unit solid angle Radiant energy Radiant power Radiant intensity Irradiance The amount of radiant power impinging upon a surface per unit area. Per unit area Radiant exitance Radiant emittance + reflected light Radiance Per unit projected-area Radiosity

7 Photometry Units [8] Photometry is the science of the measurement of light, in terms of its perceived brightness to the human eye. It is distinct from radiometry, which is the science of measurement of radiant energy (including light) in terms of absolute power; rather, in photometry, the radiant power at each wavelength is weighted by a luminosity function (a.k.a. visual sensitivity function) that models human brightness sensitivity.

8 Lambert s s cosine law [9] In optics, Lambert's cosine law says that the radiant intensity observed from a "Lambertian" surface is directly proportional to the cosine of the angle θ between the observer's line of sight and the surface normal. An important consequence of Lambert's cosine law is that when such a surface is viewed from any angle, it has the same apparent radiance. This means, for example, that to the human eye it has the same apparent brightness (or luminance). It has the same radiance because, although the emitted power from a given area (da) element is reduced by the cosine of the emission angle, the size of the observed area (da normal ) is decreased by a corresponding amount.

9 Lambertian scatters [9] When an area element is radiating as a result of being illuminated by an external source, the irradiance (energy or photons/time/area) landing on that area element will be proportional to the cosine of the angle between the illuminating source and the normal. A Lambertian scatterer will then scatter this light according to the same cosine law as a Lambertian emitter. This means that although the radiance of the surface depends on the angle from the normal to the illuminating source, it will not depend on the angle from the normal to the observer.

10 Lambert s s cosine law [9] Emission rate (photons/s) in a normal and off-normal direction. The number of photons/sec directed into any wedge is proportional to the area of the wedge.

11 Lambert s s cosine law [9] The observer directly above the area element will be seeing the scene through an aperture of area da 0 and the area element da will subtend a (solid) angle of dω 0. We can assume without loss of generality that the aperture happens to subtend solid angle dω when "viewed" from the emitting area element. This normal observer will then be recording I dω da photons per second and so will be measuring a radiance of photons/(s cm 2 sr). The observer at angle θ to the normal will be seeing the scene through the same aperture of area da 0 and the area element da will subtend a (solid) angle of dω 0 cos(θ). This observer will be recording I cos(θ) dω da photons per second, and so will be measuring a radiance of which is the same as the normal observer. photons/(s cm 2 sr),

12 BRDF [5]

13 Constraints on the BRDF [5]

14 Diffuse Reflection [5]

15 Diffuse Reflection [5]

16 Specular Reflection [5]

17 Diffuse Reflection and Lambertian BRDF [2]

18 Specular Reflection and Mirror BRDF [2]

19 Combining Specular and Diffuse [2]

20 Combining Specular and Diffuse [2]

21 BRDF Models [5] Phenomenological Phong Ward Lafortune et al. Ashikhmin et al. Physical Cook-Torrance Dichromatic He et al. Here we re listing only some well-known examples

22 Phong Illumination Model [5]

23 Measuring the BRDF [5] traditional design by Greg Ward Gonioreflectometer Device for capturing the BRDF by moving a camera + light source Need careful control of illumination, environment

24 Measuring the BRDF [5] MERL (Matusik et al.): 100 isotropic, 4 nonisotropic, dense CURET (Columbia-Utrect): 60 samples, more sparsely sampled, but also bidirectional texure functions (BTF)

25 Image Intensity and 3D Geometry [2]

26 Image Formation [6]

27 Reflectance Map [6]

28 Surface Normal [2]

29 Gradient Space [2]

30 Reflectance Map [2]

31 Reflectance Map [2]

32 Reflectance Map [2]

33 Reflectance Map [2]

34 Shape from a Single Image? [2]

35 Shape from a Single Image? [3] Take more images Photometric stereo Add more constraints Shape-from-shading

36 Photometric Stereo [2]

37 Photometric Stereo [2]

38 Solving the Equations [2]

39 More than Three Light Sources [2]

40 Color Images [2]

41 Light Source Direction [2] n s r-(rn)n r s=r-2(r-(rn)n) =r-2r+2(rn)n =2(rn)n-r

42 Normal Field to Surface [6]

43 Normal Field to Surface [6]

44 Results [2] 1. Estimate light source directions 2. Compute surface normals 3. Compute albedo values 4. Estimate depth from surface normals 5. Relight the object (with original texture and uniform albedo)

45 Shape from a Single Image? [3] Take more images Photometric stereo Add more constraints Shape-from from-shading

46 Human Perception [4] Our brain often perceives shape from shading. Mostly, it makes many assumptions to do so. For example: Light is coming from above (sun). Biased by occluding contours.

47 Stereographic Projection [4] (p,q)-space (gradient space) z (f,g)-space z p S s 1 n y N z 1 q f ŝ s 1 nˆ n y z 1 g x x Problem (p,q) can be infinite when 90 1 Redefine reflectance map as R f, g f 1 2 p 1 p 2 q 2 g 1 1 2q p 2 q 2

48 Occlusion Boundary [4] e n v n e, n v n e v e and v are known n e The n values on the occluding boundary can be used as the boundary condition for shape-from-shading

49 Image Irradiance Constraint [4] Image irradiance should match the reflectance map Minimize 2 e i I x, y R f, g dxdy image (minimize errors in image irradiance in the image)

50 Smoothness Constraint [4] Used to constrain shape-from-shading Relates orientations (f,g) of neighboring surface points Minimize e s f x 2 f y 2 image g 2 2 x g y dxdy f, g : surface orientation under stereographic projection f x f f g, f y, g x, x y x g y g y (penalize rapid changes in surface orientation f and g over the image)

51 Shape from Shading [4] Find surface orientations (f,g) at all image points that minimize weight e e e s i Minimize e image smoothness constraint image irradiance error f f g g I x, y R f, g x y x y 2 dxdy

52 Result [4]

53 With Colored Lights [10] In an environment where red, green, and blue light is emitted from different directions, a Lambertian surface will reflect each of those colors simultaneously without any mixing of the frequencies. The quantities of red, green and blue light reflected are a linear function of the surface normal direction.

54 With Colored Lights [10] For simplicity, we first focus on the case of a single distant light source with direction l illuminating a Lambertian surface point P with surface orientation direction n. The intensity measured at i-th sensor The spectral sensitivity of the i-th sensor The Spectral reflectance function at that point The energy distribution of the light source

55 With Colored Lights [10] in matrix form RGB normal mapping: Equation (2) establishes a 1-1 mapping between an RGB pixel measurement from a color camera and the surface orientation at the point projecting to that pixel. Our strategy is to use the inverse of this mapping to convert a video of a deformable surface into a sequence of normal maps. By estimating and then inverting the linear mapping M linking RGB values to surface normals, we can convert a video sequence captured under colored light into a video of normal-maps. We then integrate each normal map independently to obtain a depth map in every frame by imposing that the occluding contour is always at zero depth. At the end of the integration process, we obtain a video of depth-maps.

56 With Colored Lights [10] Tracking the surface Our approach is to use the first depth-map of the sequence as a 3D template which will be deformed to match all subsequent depth-maps. The deformations of the template will be guided by the following two competing constraints: the deformations must be compatible with the frame to frame 2D optical flow of the original video sequence, the deformations must be locally as rigid as possible.

57 References 1. Wikipedia, Photometric Stereo, available at 2. Yu-Wing, Photometric Stereo, KAIST Computer Vision (CS 770, Fall 2009) lecture material, available at 3. S. Narasimhan, Photometric Stereo, CMU Computer Vision(15-385, -685, Spring 2006) lecture material, available at s06/lectures/ppts/ 4. S. Narasimhan, Structure from Shading, CMU Computer Vision(15-385, -685, Spring 2006) lecture material, available at s06/lectures/ppts/ 5. Seitz, Light, Washington University Computer Vision (CSE 455, Winter 2004) lecture material, available at 6. David Kriegman, Photometric Stereo, UCSD Computer Vision (CSE152, Spring 2005) lecture material, available at 7. Wikipedia, Radiometry, available at 8. Wikipedia, Photometry, available at 9. Wikipedia, Lambert s cosine law, available at Carlos Hernandez, et at., Non-rigid Photometric Stereo with Colored Lights, 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, Oct 2007.