Lambertian model of reflectance II: harmonic analysis. Ronen Basri Weizmann Institute of Science

Transcription

1 Lambertian model of reflectance II: harmonic analysis Ronen Basri Weizmann Institute of Science

2 Illumination cone What is the set of images of an object under different lighting, with any number of sources? Images are additive and non-negative This set, therefore, forms a convex cone in R p, p number of pixels (Belhumeur & Kriegman) = 0.5* +0.2* +0.3*

3 Illumination cone Cone characterization is generic, holds also with specularities, shadows and interreflections Unfortunately, representing the cone is complicated (infinite degrees of freedom) Cone is thin for Lambertian objects; indeed the illumination cone of many objects can be represented with few PCA vectors (Yuille et al.)

4 Illumination cone is often thin Ball Face Phone Parrot # # # # # (Yuille et al.)

5 Lambertian reflectance is smooth (Basri & Jacobs; Ramamoorthi & Hanrahan) lighting reflectance

6 Reflectance obtained with convolution R v = S 2 k u, v l u du

7 Reflectance obtained with convolution R v = S 2 k u, v l u du

8 Spherical harmonics Y nm θ, = 2n + 1 4π p nm z = (1 z2 ) m/2 2 n n! n m! n + m! P nm(cos θ)e im d n+m dz n+m (z2 1) n Orthonormal basis for functions on the sphere n th order harmonics have 2n+1 components Rotation = phase shift (same n, different m) In space coordinates: polynomials of degree n Funk-Hecke convolution theorem

9 Spherical harmonics X + Y + Z 1 Positive values Negative values 1 Z X Y 2 2 Z XY XZ YZ X Y

10 Harmonic approximation Lighting, in terms of harmonics l θ, φ = Reflectance n=0 r θ, φ = k l n m= n 2 n=0 l nm Y nm (θ, φ) n m= n Approximation accuracy, 99.2% (Basri & Jacobs; Ramamoorthi & Hanrahan) k n l nm Y nm (θ, φ)

11 Harmonic transform of kernel 1.5 k(θ) = max (cos θ, 0) = n=0 k n Y n %

12 Subspace approximation Up to 2 nd order: 9 basis images Accuracy: 99.2% Up to 1 st order: 4 basis images: ambient + point source Accuracy: 87.5% In practice, due to self occlusions ~98% can be achieved with just 6 basis images (Ramamoorthi)

13 Scope Harmonic representations handle convex, lambertian objects with multiple light sources (including attached shadows) Harmonic representations do not model cast shadows and inter-reflections Accuracy is maintained for fairly close light sources Representing specular objects may require a very large basis

14 Applications We can use this theory to predict novel appearances under new lighting Harmonic lighting theory has led to applications in Face recognition Photometric stereo 3D reconstruction with prior Motion analysis

15 Harmonic faces Positive values Negative values ρ Albedo n Surface normal n ( n, n, n ) x y z n z nx ny 2 n z (3 1) ( n ) nn x y 2 2 n x y nn n n x z y z

16 Non-negative light We can enforce in addition that light is nonnegative, by projecting the illumination cone onto the harmonic space Closed-form constraints for 1 st order approximation Sampling method, or Toeplitz matrix (Shirdhonkar & Jacobs) for higher orders

17 Photometric stereo (Basri, Jacobs & Kemelmacher) S M L n z Image 1 Light 1 n z n y : : (3n 2 z -1) (n 2 x -n 2 y ) Image n Light n n x n y n x n z n y n z SVD recovers L and S up to an (r r) ambiguity

18 Photometric stereo

19 Reconstruction with a prior (Kemelmacher & Basri) Given just one image SFS is impractical Reconstruction is possible when a prior is available Energy min l,ρ,z Ω D + S Data term D = I ρl T Y(n) 2 Regularization 2 2 S = λ 1 Z Z prior + λ2 ρ ρ prior Solve as a linear PDE

20 Reconstruction with a prior

21 More

22 Mooney faces (Kemelmacher, Nadler & B, CVPR 2008)

23 Motion + lighting

24 Motion + lighting (Basri & Frolova) Given 2 images I p = ρl T n J p = ρl T Rn Take ratio to eliminate albedo J(p ) I(p) = lt Rn l T n If motion is small we can represent J p using a Taylor expansion around p

25 Small motion We obtain a PDE that is quasi linear in z az x + bz y = c Where a x, y, z = l 1 I θ zj x l 3 I b x, y, z = l 2 I θ zj x c x, y, z = l 3 I θ zj x l 1 I with J I I θ = θ Can be solved with continuation (characteristics)

26 Reconstruction

27 More reconstructions

28 Conclusion Understanding the effect of lighting on images is challenging, but can lead to better interpretation of images Harmonic analysis allows to model complex lighting in a linear model Various applications in recognition and reconstruction We only looked at Lambertian objects