Direct Surface Reconstruction using Perspective Shape from Shading via Photometric Stereo

Transcription

1 Direct Surface Reconstruction using Perspective Shape from Shading via Roberto Mecca joint work with Ariel Tankus and Alfred M. Bruckstein Technion - Israel Institute of Technology Department of Computer Science 11 December 2012

2 (Perspective) Shape from Shading Artistic Precursor of the Shape from Shading R. Mecca Direct Surface Reconstruction using Perspective SfS-PS

3 Problem introduction Differential model Not uniqueness

4 Main ingredients (Perspective) Shape from Shading Differential model Not uniqueness The SfS problem is described by the following irradiance equation: R(ρ, n) = I (1) where R is the reflectance function; n is the unit normal vector to the surface; ρ is the albedo; I is the image function; I : Ω [0, 1], with Ω is a compact closed domain (Ω R 2 open set).

5 Perspective transformation Differential model Not uniqueness Surface in the real world: h(x, y) = x y ẑ(x, y) z (x, y, ẑ) Surface in the perspective world: ξ x ẑ(x,y) f k(ξ, η) = η = y ẑ(x,y) f z(ξ, η) ẑ(x, y) y x where f is the focal length. (,, f) A. Tankus, N. A. Sochen, Y. Yeshurun: Shape-from-Shading Under Perspective Projection, International Journal of Computer Vision 63(1): (25)

6 Differential model Not uniqueness Hypotheses and Non-Linear Problem 1. only one light source placed at infinity in the direction of the unit vector ω = (ω 1, ω 2, ω 3 ) R 3 (with ω 3 < 0); 2. Lambertian surface; 3. albedo unknown; 4. no self-reflection. It is possible to get the following non-linear PDE with Dirichlet boundary condition: ρ(ξ, η) z ξ(fω 1 + ξω 3 ) z η (fω 2 + ηω 3 ) zω 3 f = I(ξ, η), on Ω p 2 (zξ 2 + z2 η) + (z + ξz ξ + ηz η ) 2 z(ξ, η) = g(ξ, η) for the PSfS problem. on Ω p (2)

7 Concave/convex ambiguity Differential model Not uniqueness

8 Differential model Not uniqueness One picture is not enough to obtain uniqueness

9 (Perspective) Shape from Shading Differential model Not uniqueness

10 (Perspective) Shape from Shading Differential model Not uniqueness...

11 The new differential approach Direct Surface Reconstruction using Multiple Image Differential model for the PSfS-PS using 2 images Let us overpass the problem of uniqueness of solution considering the approach using two light sources defined by the unit vectors ω = (ω 1, ω 2, ω 3) and ω = (ω 1, ω 2, ω 3 ). Using the information obtained by both images we can couple the two equations obtaining the following system of non-linear PDE: ρ(ξ, η) z ξ(fω 1 + ξω 3) z η (fω 2 + ηω 3) zω 3 f = I 1 (ξ, η), on Ω p 2 (zξ 2 + z2 η) + (z + ξz ξ + ηz η ) 2 ρ(ξ, η) z ξ(fω 1 + ξω 3 ) z η (fω 2 + ηω 3 ) zω 3 f = I 2 (ξ, η), on Ω p 2 (zξ 2 + z2 η) + (z + ξz ξ + ηz η ) 2 z(ξ, η) = g(ξ, η) on Ω p. (3)

12 Removing the non-linearity The new differential approach Direct Surface Reconstruction using Multiple Image Now, observing that the denominator of both equations is the same (i.e. it does not depend on the light source) and obviously always different from zero, we can explicit the non-linearity from the first equation for example f 2 (zξ 2 + z2 η) + (z + ξz ξ + ηz η ) 2 = z ξ(fω 1 + ξω 3) z η (fω 2 + ηω 3) zω 3 ρ(ξ, η) I 1 (ξ, η) and replacing it in the other we obtain the following linear equation [(fω 1 + ξω 3)I 2 (ξ, η) (fω 1 + ξω 3 )I 1 (ξ, η)] z (ξ, η) ξ + [(fω 2 + ηω 3)I 2 (ξ, η) (fω 2 + ηω 3 )I 1 (ξ, η)] z (ξ, η) η + (ω 3I 2 (ξ, η) ω 3 I 1 (ξ, η))z(ξ, η) = 0 (4)

13 Removing the non-linearity The new differential approach Direct Surface Reconstruction using Multiple Image Now, observing that the denominator of both equations is the same (i.e. it does not depend on the light source) and obviously always different from zero, we can explicit the non-linearity from the first equation for example f 2 (zξ 2 + z2 η) + (z + ξz ξ + ηz η ) 2 = z ξ(fω 1 + ξω 3) z η (fω 2 + ηω 3) zω 3 ρ(ξ, η) I 1 (ξ, η) and replacing it in the other we obtain the following linear equation [(fω 1 + ξω 3)I 2 (ξ, η) (fω 1 + ξω 3 )I 1 (ξ, η)] z (ξ, η) ξ + [(fω 2 + ηω 3)I 2 (ξ, η) (fω 2 + ηω 3 )I 1 (ξ, η)] z (ξ, η) η + (ω 3I 2 (ξ, η) ω 3 I 1 (ξ, η))z(ξ, η) = 0 (4)

14 Linear PDE (Perspective) Shape from Shading The new differential approach Direct Surface Reconstruction using Multiple Image That is, instead of two non-linear PDEs we can consider this linear PDE: { b(ξ, η) z(ξ, η) + s(ξ, η)z(ξ, η) = 0, on Ω p z(ξ, η) = g(ξ, η) on Ω p (5) where: b(ξ, η) = ((fω 1 + ξω 3)I 2 (ξ, η) (fω 1 + ξω 3 )I 1 (ξ, η), (fω 2 + ηω 3)I 2 (ξ, η) (fω 2 + ηω 3 )I 1 (ξ, η)) (6) and s(ξ, η) = ω 3I 2 (ξ, η) ω 3 I 1 (ξ, η). (7) Lemma If there aren t points (ξ, η) Ω p of black shadows for the image functions (i.e. I 1 (ξ, η) 0 and I 2 (ξ, η) 0), than b(ξ, η) 0.

15 Linear PDE (Perspective) Shape from Shading The new differential approach Direct Surface Reconstruction using Multiple Image That is, instead of two non-linear PDEs we can consider this linear PDE: { b(ξ, η) z(ξ, η) + s(ξ, η)z(ξ, η) = 0, on Ω p z(ξ, η) = g(ξ, η) on Ω p (5) where: b(ξ, η) = ((fω 1 + ξω 3)I 2 (ξ, η) (fω 1 + ξω 3 )I 1 (ξ, η), (fω 2 + ηω 3)I 2 (ξ, η) (fω 2 + ηω 3 )I 1 (ξ, η)) (6) and s(ξ, η) = ω 3I 2 (ξ, η) ω 3 I 1 (ξ, η). (7) Lemma If there aren t points (ξ, η) Ω p of black shadows for the image functions (i.e. I 1 (ξ, η) 0 and I 2 (ξ, η) 0), than b(ξ, η) 0.

16 Discontinuity of b and s The new differential approach Direct Surface Reconstruction using Multiple Image z! I( ) 1 ẑ 2 p 1 p 2 p 3 4 f x 12 5 p p 3 p p p 2 p p p

17 Sketch of the Proof (Perspective) Shape from Shading The new differential approach Direct Surface Reconstruction using Multiple Image Theorem Let γ(t) be a curve of discontinuity for the function b(ξ, η) (and s(ξ, η)) and let p = (ξ, η) be a point of this curve. Let n(ξ, η) be the outgoing normal with respect to the set Ω p +, than we have [ lim (ξ,η) (ξ,η) (ξ,η) Ω p + b(ξ, η) n(ξ, η) ][ lim (ξ,η) (ξ,η) (ξ,η) Ω p b(ξ, η) n(ξ, η) ] 0 (8) p + p + p + p + p p p p (, ) (, ) (, ) (, ) (t) (t) (t) (t)

18 Sketch of the Proof (Perspective) Shape from Shading The new differential approach Direct Surface Reconstruction using Multiple Image Theorem Let us consider the problem { b(ξ, η) z(ξ, η) + s(ξ, η)z(ξ, η) = 0, a.e. (ξ, η) Ω p ; z(ξ, η) = g(ξ, η) (ξ, η) Ω p. Let us suppose that (γ 1 (t),..., γ k (t)), the family of discontinuity curves for b(ξ, η) and s(ξ, η), are not characteristic curves (with respect to the previous problem). Then it exists a unique Lipschitz solution of the problem.

19 Multiple Images Formulation The new differential approach Direct Surface Reconstruction using Multiple Image If we consider the problem (5) for three images, we get the following three linear equations (one for each couple of images): b (1,2) (ξ, η) z(ξ, η) + s (1,2) (ξ, η)z(ξ, η) = 0, a.e. (ξ, η) Ω p b (1,3) (ξ, η) z(ξ, η) + s (1,3) (ξ, η)z(ξ, η) = 0, a.e. (ξ, η) Ω p b (2,3) (ξ, η) z(ξ, η) + s (2,3) (ξ, η)z(ξ, η) = 0, a.e. (ξ, η) Ω p (9) and simply by summing the previous three equations, we can resume the previous system in the following equation: { ( b (1,2) + b (1,3) + b (2,3)) z(ξ, η) + ( s (1,2) + s (1,3) + s (2,3)) z(ξ, η) = 0 z(ξ, η) = g(ξ, η) which continues to have a unique weak solution. (10)

20 The new differential approach Direct Surface Reconstruction using Multiple Image Weighted Perspective with 3 Images We define the ingredients for the Weighted PPS 3 problem (W-PPS 3 ) as follow: b w 3 (ξ, η) = q p (ξ, η)b p (ξ, η) and s w 3 (ξ, η) = q p (ξ, η)s p (ξ, η) p ( [3] 2 ) p ( [3] 2 ) (11) where ( ) [3] = {(1, 2), (1, 3), (2, 3)}. (12) 2 We have now completed the construction of the W-PPS 3 formulation with { b w 3 (ξ, η) z(ξ, η) + s w 3 (ξ, η)z(ξ, η) = 0, a.e. (ξ, η) Ω p z(ξ, η) = g(ξ, η) (ξ, η) Ω p (13).

21 W-PPS 3 with shadows The new differential approach Direct Surface Reconstruction using Multiple Image The presence of shadows in the images consists in a loss of information that could compromise the surface reconstruction. Since no ambient light is assumed, for each image the shadow sets are defined as follow: S i S = {(ξ, η) Ω p 1 0 S S 1 0 S : I i (ξ, S 0 η) = 0} = where Sj i S are open and disjoint sets. 4 n i s j= S4 S4 Sj, i S i = S 1, S 2, 3 (14) S4 S4 S4 S 0 2 S 0 2 S 0 2 S 0 1 S 0 1 S 0 1 S 0 3 S 0 3 S 0 3 S4 S4 I BW 1 I BW 2 I BW 3 Figure : In this black and white (i.e. 0 and 1) pictures we consider the images Ii BW = H(I i ) where H( ) is the Heaviside step function. S4

22 W-PPS 3 with shadows The new differential approach Direct Surface Reconstruction using Multiple Image Let us define the following sets: S (1,2) = S, S (1,3) = S and S (2,3) S = S S S (15) and introduce the functions 0 S 1 0 S 0 1 ( ) [3] q p (ξ, η) = 1 p [Ω \Sp ] (ξ, η), p 2 showed in Figure 2. S 0 3 S 0 3 S 0 3 S4 S4 S4 (16) S 0 1 S 0 2 S 0 2 S 0 2 S 0 S S 0 1 S 0 1 S S4 S4 S4 S 0 3 S 0 3 S 0 3 S4 S4 S4 q (1,2) q (1,3) q (2,3) Figure : Here a schematic representation of the non-smooth weight functions is given. Using the Heaviside function we can also write them as q (h,k) = H(I h )H(I k ).

23 Semi-Lagrangian Backward and Forward semi-lagrangian scheme Dividing both sides of the linear equation in (5) by b(ξ, η) and considering the definition of the directional derivative we arrive to the following fixed point semi-lagrangian schemes: z n+1 i,j = z n (ξ i hαi,j, 1 η j hαi,j) 2 b i,j (17) b i,j ± hs i,j where z n+1 (ξ i, η j ) = z n+1 i,j and α i,j = bi,j b i,j on a uniform discretization of Ω P (Ω p d ). The convergence of this schemes is proven exploiting the properties of the asymptotically non expansive map

24 Backward and Forward semi-lagrangian scheme Semi-Lagrangian : Convergence Definition Let K be a subset of a Banach space X. A transformation T : K K is said to be asymptotically nonexpansive if for each x, y K, T i (x) T i (y) k i x y (18) where {k i } is a sequence of real number such that lim i k i = 1. It is obvious that for asymptotically non expansive mapping it may be assumed the k i 1 and that k i+1 k i for i = 1, 2,.... Theorem Let K be a nonempty, closed, convex and bounded subset of a uniformly convex Banach space X, and let F : K K be asymptotically nonexpansive. Than F has a fixed point. K. Goebel and W. A. Kirk, A fixed point theorem for asymptotically nonexpansive mappings, Proceedings of the American Mathematical Society, 35 (1972), pp

25 Backward and Forward semi-lagrangian scheme Semi-Lagrangian : Convergence Definition Let K be a subset of a Banach space X. A transformation T : K K is said to be asymptotically nonexpansive if for each x, y K, T i (x) T i (y) k i x y (18) where {k i } is a sequence of real number such that lim i k i = 1. It is obvious that for asymptotically non expansive mapping it may be assumed the k i 1 and that k i+1 k i for i = 1, 2,.... Theorem Let K be a nonempty, closed, convex and bounded subset of a uniformly convex Banach space X, and let F : K K be asymptotically nonexpansive. Than F has a fixed point. K. Goebel and W. A. Kirk, A fixed point theorem for asymptotically nonexpansive mappings, Proceedings of the American Mathematical Society, 35 (1972), pp

26 Backward and Forward semi-lagrangian scheme Semi-Lagrangian : Convergence We are now able to prove the following: Theorem The numerical schemes defined by the operators Th(z)(ξ, b b(ξ, η) η) = z((ξ, η) + hα(ξ, η)) b(ξ, η) hs(ξ, η), T f b(ξ, η) h (z)(ξ, η) = z((ξ, η) hα(ξ, η)) b(ξ, η) + hs(ξ, η) are convergent. (19)

27 Numerical Tests on Synthetic Images Backward and Forward semi-lagrangian scheme ω : ϕ 1 = 0.1, θ 1 = 0.0 (I 1 ) ω : ϕ 2 = 0.1, θ 2 = π (I 2 ) ω : ϕ 2 = 0.1, θ 2 = 3 2 π (I 3 ) Table : Set of images used with the respective light sources described by their spherical coordinates. In this case the albedo mask is added for all the images together with back patches and 10% of Gaussian Noise. The size of the pictures goes from 5 5 pixels to 4 4 pixels.

28 Numerical Tests on Synthetic Images Backward and Forward semi-lagrangian scheme Figure : On the left the original surface and on the right the reconstruction with 10% noised images of 16 Megapixel. Forward s-l Backward s-l L error L 1 error time (sec) L error L 1 error time (sec) Table : In this table the errors in the L and L 1 norms are shown together with the computational time for the semi-lagrangian schemes. The rows are divided considering the size of the starting images.

29 Real Images (Perspective) Shape from Shading Backward and Forward semi-lagrangian scheme We use the presented algorithm on a well known set of images depicting the Beethoven s bust. The size of these pictures is very small, only and the time needed for the reconstruction is seconds. Figure : Starting data obtained respectively with the light sources ω : ϕ 1 = 0.263, θ 1 = 0.305, ω : ϕ 2 = 0.2, θ 2 = and ω : ϕ 3 = 0.281, θ 3 =

30 Real Images (Perspective) Shape from Shading Backward and Forward semi-lagrangian scheme Figure : Forward and Backward reconstruction.

31 Conclusion and Perspective Backward and Forward semi-lagrangian scheme a new differential formulation for the Perspective Shape from Shading problem via (with only two images) has been proven to be well-posed; the extension to the Multi-Image reconstruction has been formulated also considering shadows and occlusions; the algorithm implemented is able to solve in a few seconds the W-PPS 3 with starting images of several megapixels. Our algorithms are not based on parallel computing and a first improvement of this work could be a GPU implementation in order to have 3D reconstruction in real time. Another direction of research could be the improvement of the shading model employed. The initial setup can be improved by taking into account specular reflections and different kind of perspective deformations.