Analisi ed approssimazione di alcuni problemi appartenenti alla classe Shape-from-X

Transcription

1 Analisi ed approssimazione di alcuni problemi appartenenti alla classe Shape-from-X Silvia Tozza Assegnista INdAM Unità di ricerca INdAM presso Dip. di Matematica, Sapienza Università di Roma Convegno GNCS 2018, Hotel Belvedere, Montecatini Terme Febbraio 2018

2 Introduction - Shape-from-X Problems Goal: Reconstruction of the shape of an object starting from some kind of data. Shape-from-Shading Data: The grey-level measured in an image of the object Problem Surface Photo Findsurface(s)that givethesameimage(s) Shape-from-Polarization Data: A polarization image of the object.

3 Linear Differen*al Constraints for Photopolarimetric Height Es*ma*on Silvia Tozza William A.P. Smith Dizhong Zhu Ravi Ramamoorthi Edwin R. Hancock

4 Overview Polarimetric images of glossy object in uncontrolled, outdoor illumina#on Degree of polarisa#on Phase angle Unpolarised intensity Polarisa#on image Es#mated ligh#ng Es#mated depth map Texture mapped surface, novel pose A unified PDE system for height estimation Only need to solve large, sparse linear system A polarisation image from multichannel data Arbitrary uncalibrated illuminations (estimated) Remark Uncalibrated two source photometric stereo solvable with two polarisation images.

5 Overview Polarimetric images of glossy object in uncontrolled, outdoor illumina#on Degree of polarisa#on Phase angle Unpolarised intensity Polarisa#on image Es#mated ligh#ng Es#mated depth map Texture mapped surface, novel pose A unified PDE system for height estimation Only need to solve large, sparse linear system A polarisation image from multichannel data Arbitrary uncalibrated illuminations (estimated) Remark Uncalibrated two source photometric stereo solvable with two polarisation images.

6 Outline Introduction a. Polarimetric image capturing b. Polarisation image Multichannel polarisation image estimation Photo-polarimetric height constraints A Unified PDE formulation Two source lighting estimation Numerical Experiments

7 Introduction - Shape from polarisation (SfP) Problem Assumptions: Orthographic projection Refractive index of the surface known Diffuse polarisation model and diffuse reflectance model Dielectric (i.e. non-metallic) material Notations: x = (x, y) is an image point z(x) surface height Light source direction s and viewer direction v, with s v Unit surface normal n(x) formulated via surface gradient n(x) = [ p(x) q(x) 1]T 1 + z(x) 2, where p(x) = x z(x) and q(x) = y z(x).

8 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

9 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

10 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

11 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

12 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

13 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

14 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

15 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

16 Introduction - Polarimetric image capturing Figure: Rotate linear polarising filter in front of camera Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle Figure: Intensity varies sinusoidally

17 Introduction - Polarimetric image capturing The measured intensity at a pixel varies sinusoidally with the polariser angle ϑ j, j {1,..., P}, with P 3: i ϑj (x) = i un (x) ( 1 + ρ(x) cos(2ϑ j 2φ(x)) ) Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle The polarisation image is obtained by decomposing the sinusoid at every pixel into three quantities [Wolff, 1997]. ρ(x) = I max(x) I min (x) I max (x) + I min (x) and i un (x) = I max(x) + I min (x) 2

18 Introduction - Polarisation image 9:% ;9:% <=9:% 8% 0.35 /&)*$+,*-&.%47*#"% !"#$""%&'%(&)*$+,*-&.% /0*,"%1.#)"% 2.(&)*$+,"3%4.5".,+56% Remark Using beam splitters or custom CCDs it is possible to make the required measurements in a single shot.

19 Multichannel polarisation image estimation Colour images (3 channels), polarisation images with two different light source directions (2 channels) or both (6 channels). ρ and φ constant over the channels (they depend only on surface geometry). i un will vary between channels. Multichannel observations in channel c with polariser angle ϑ j i c ϑ j (x) = i c un(x)(1 + ρ(x) cos(2ϑ j 2φ(x))). The system of equations is linear in the unpolarised intensities and, by a change of variables, can be made linear in ρ and φ [Huynh et a., 2010].

20 Multichannel polarisation image estimation We wish to solve a bilinear system and do so in a least squares sense using interleaved alternating minimisation: 1 Fixing ρ and φ and then solve linearly for iun c in each channel 2 Fix the unpolarised intensities and solve linearly for ρ and φ using all channels simultaneously. Concretely, for a single pixel, we solve min C [ I i 1 un (x),..., iun(x) ] C T 2 di, where i 1 un (x),...,ic un (x) C I = (1 + ρ(x) cos(2ϑ 1 2φ(x)))I C., (1 + ρ(x) cos(2ϑ P 2φ(x)))I C d I = [ i 1 ϑ 1 (x),..., i C ϑ 1 (x), i 1 ϑ 2 (x),..., i C ϑ P (x) ] T. and I C denoting the C C identity matrix.

21 Multichannel polarisation image estimation Then, with the unpolarised intensities fixed, we solve for ρ and φ using the following linearisation: min a,b C ρφ [ ] a 2 d b ρφ, where [a b] T = [ρ(x) cos(2φ(x)), ρ(x) sin(2φ(x))] T, iun(x) 1 cos(2ϑ 1) iun(x) 1 sin(2ϑ 1) iϑ 1 1 (x) iun(x) 1... i 1 C ρφ = un(x) cos(2ϑ P ) iun(x) 1 sin(2ϑ P ) i iun(x) 2 cos(2ϑ 1) iun(x) 2 ϑ 1, d sin(2ϑ ρφ = P (x) i 1 un(x) 1) iϑ 2 1 (x) i 2. un(x)... iun(x) C cos(2ϑ P ) iun(x) C sin(2ϑ P ) iϑ C P (x) iun(x) C We estimate ρ and φ from the linear parameters using φ(x) = 1 2 atan2(b, a) and ρ(x) = a 2 + b 2.

22 Multichannel polarisation image estimation The multichannel result is visibly less noisy than the single channel performance. Input Single channel estimation Input Multichannel estimation Figure: Multichannel polarisation image estimation. Left to right: an image from the input sequence; phase angle (φ) and degree of polarisation (ρ) estimated from a single channel; phase angle (φ) and degree of polarisation (ρ) estimated from three colour channels and two light source directions.

23 Photo-polarimetric height constraints - I Diffuse polarisation model ρ d (x) = 4 cos(θ(x)) where η is the refractive index. ) 2 sin(θ(x)) ( 2 η 1 η η 2 sin(θ(x)) 2 sin(θ(x)) ( 2 η + 1 η ) η2 + 2, Typical values for dielectrics η [1.4, 1.6]. We assume η = 1.5 for the rest of the talk. Degree of polarisation constraint We rearrange the previous equation arriving to cos(θ(x)) = n(x) v = f (ρ d (x), η) = 2 ρ + 2 η 2 ρ 2 η 2 + η 4 + ρ η 2 ρ 2 η 4 ρ 2 4 η 3 ρ (ρ 1) (ρ + 1) + 1 η 4 ρ η 4 ρ + η η 2 ρ η 2 ρ 2 η 2 + ρ ρ + 1 where we drop the dependency of ρ d on x for brevity.

24 Photo-polarimetric height constraints - II Lambertian reflectance model where γ(x) is the albedo. Shading constraint i un (x) = γ(x) cos(θ i ) = γ(x)n(x) s, (1) Writing n(x) in terms of z, (1) can be rewritten as follows: i un (x) = γ(x) z(x) s + s z(x), with s = (s 1, s 2 ). (2) 2 Remark If s v (a configuration physically impossible to achieve precisely) then this equation provides no more information than the degree of polarisation. Hence, we assume s v.

25 Photo-polarimetric height constraints - III The second degree of freedom of the surface normal direction is the azimuth angle and this is related to the phase angle of the sinusoid Intensity /4 /2 3 /4 5pi/4 3 /2 7 /4 2 Polariser angle The azimuth angle is either equal to the phase angle or shifted by pi.

26 Photo-polarimetric height constraints - III The polarisation cue restricts n(x) to two possible directions!"#$#%&'&(&)*"+! *,-&./)0#& Collinearity condition: n(x) [cos(φ(x)) sin(φ(x)) 0] T = 0. (3) Since the nonlinear normalisation term is always 0, we can write (3) in terms of the surface gradient arriving to Phase angle constraint p(x) cos(φ(x)) + q(x) sin(φ(x)) = 0

27 Linearisation of height constraints Combining the degree of polarisation and the shading constraints, we can arrive at a linear equation DOP ratio constraint z(x) ṽ + v 3 f (ρ d (x), η) = γ(x) z(x) s + s 3. (4) i un (x) We can rewrite (4) arriving to the following PDE: b(x) z(x) = h(x), (5) where b(x) :=b (f,iun) = i un (x)ṽ γ(x)f (ρ d (x), η) s, (6) h(x) :=h (f,iun) = i un (x)v 3 γ(x)f (ρ d (x), η) s 3. (7) with ṽ = (v 1, v 2 ) and s = (s 1, s 2 ).

28 Linearisation of height constraints Let us consider two unpolarised intensities, i un,1, i un,2, taken from two different light source directions, s, t. By applying the shading constraint twice, once for each light source, we get Intensity ratio constraint i un,2 ( z(x) s + s 3 ) = i un,1 ( z(x) t + t 3 ). (8) we can rewrite (8) as a PDE in the form of (5) with b(x) := b (i un,1,i un,2 ) = i un,2 (x) s i un,1 (x) t, (9) where t = (t 1, t 2 ), and h(x) := h (i un,1,i un,2 ) = i un,2 (x)s 3 i un,1 (x) t 3. (10)

29 A Unified PDE formulation B(x) z(x) = h(x), where B : Ω R J 2, h : Ω R J 1, Ω is the reconstruction domain, J = 2, 3 or 4 depending on the cases. Single light and polarisation formulation [Smith et al., 2016] [ ] (f,i b un) (f,iun) B = 1 b 2, h = [h (f,iun), 0] T, cos φ sin φ with b (f,iun) and h (f,iun) defined by (6) and (7). Here, we need a single polarisation image uniform γ(x)

30 A Unified PDE formulation B(x) z(x) = h(x), where B : Ω R J 2, h : Ω R J 1, Ω is the reconstruction domain, J = 2, 3 or 4 depending on the cases. Proposed 1: Albedo invariant formulation B(x) = [ (i b un,1,i un,2 ) 1 b (i ] un,1,i un,2 ) 2, h(x) = cos φ sin φ [ ] h (i un,1,i un,2 ), 0 where b (i un,1,i un,2 ) and h (i un,1,i un,2 ) defined as in (9) and (10). Here, we need two unpolarised images taken from two different light source direction, s and t.

31 A Unified PDE formulation B(x) z(x) = h(x), where B : Ω R J 2, h : Ω R J 1, Ω is the reconstruction domain, J = 2, 3 or 4 depending on the cases. Proposed 2: Phase invariant formulation B = [b (f,i un,1), b (f,i un,2), b (i un,1,i un,2 ) ] T, and h = [h (f,i un,1), h (f,i un,2), h (i un,1,i un,2 ) ] T, Here, we need two unpolarised images taken from s and t. Knowledge of the albedo map s, t, v non-coplanar to have B not singular. Note that f = f (ρ d (x), η) is the same for the two required images.

32 A Unified PDE formulation B(x) z(x) = h(x), where B : Ω R J 2, h : Ω R J 1, Ω is the reconstruction domain, J = 2, 3 or 4 depending on the cases. Proposed 3: Most constrained formulation b (f,i un,1) 1 b (f,i un,1) 2 B = cos φ b (f,i un,2) 1 b (f,i un,2) 2 b (i un,1,i un,2 ) 1 b (i un,1,i un,2 ) 2 sin φ, h = Here, we combine all the previous constraints. h (f,i un,1) h (f,i un,2) h (i un,1,i un,2 ). We require known albedo. Nevertheless, it is possible to first apply proposed method 1, estimate the albedo and then re-estimate surface height. 0

33 Height estimation via linear least squares We discretize the gradient via finite differences, arriving to Discrete linear system in z Az = h, A = BG, with A R JM M, M the number of pixels, J = 2, 3, or 4 depending on the cases. G R 2M M the matrix of finite difference gradients B R JM 2M is the discrete per-pixel version of B(x), h R JM 1 is the discrete per-pixel version of h(x), z R M 1 the vector of the unknown height values. The system is large but sparse. A is a full-rank matrix for each choice of B that comes from the proposed formulations. A related to [Smith et al., 2016] is full-rank except in one case: s 1, s 2 0, s 1 = s 2 and φ = π/4 at least in one pixel.

34 Two source lighting estimation It is possible to estimate both light source directions simultaneously, in an albedo invariant manner. From the intensity ratio, we have 1 equation per pixel and 6 unknowns. Hp: The intensity of the light source remains constant in each colour channel across the two images the length of the light source vectors is arbitrary. So, we constrain them to unit length. In spherical coordinates: (θ s, α s ) and (θ t, α t ), such that [s 1, s 2, s 3 ] = [cos α s sin θ s, sin α s sin θ s, cos θ s ] and [t 1, t 2, t 3 ] = [cos α t sin θ t, sin α t sin θ t, cos θ t ]. This reduces the number of unknowns to four. We have two possible surface normal directions at each pixel and therefore two possible gradients: p(x) ± cos φ(x) tan θ(x), q(x) ± sin φ(x) tan θ(x).

35 Two source lighting estimation The residuals at pixel x j in channel c are given by either: r j,c(θ s, α s, θ t, α t) =i c un,1(x j)( p(x j)t 1 q(x j)t 2 + t 3) i c un,2(x j)( p(x j)s 1 q(x j)s 2 + s 3) or q j,c(θ s, α s, θ t, α t) =i c un,1(x j)(p(x j)t 1 + q(x j)t 2 + t 3) i c un,2(x j)(p(x j)s 1 + q(x j)s 2 + s 3). Minimisation problem for light source direction estimation min min[r 2 θ s,α s,θ t,α t j,c(θ s, α s, θ t, α t ), qj,c(θ 2 s, α s, θ t, α t )]. j,c This optimisation is non-convex, but even with a random initialisation, it almost always converges to the global minimum. Convex/concave ambiguity (s, t) is a solution (Ts, Tt) is also a solution (with T = diag([ 1, 1, 1])).

36 Numerical Tests - Synthetic case Input Input (uniform albedo) (varying albedo) Prop. 1 Prop. 2 Prop. 3 Prop. 1+3 Varying albedo Uniform albedo [Smith et al., 2016] Ground truth height Figure: Qualitative results on synthetic data. S. Tozza - INdAM Analisi ed approssimazione di problemi appartenenti alla classe Shape-from-X

37 Numerical Tests - Synthetic case Table: HeightS. and Tozzasurface - INdAMnormal Analisi errors ed approssimazione on synthetic di problemi data. appartenenti alla classe Shape-from-X Setting Uniform albedo, known lighting Uniform albedo, estimated lighting Unknown albedo, known lighting Unknown albedo, estimated lighting Method σ = 0% σ = 0.5% σ = 2% Height Normal Height Normal Height Normal (pix) (deg) (pix) (deg) (pix) (deg) [Smith et al., 2016] Prop Prop Prop Prop [Smith et al., 2016] Prop Prop Prop Prop [Smith et al., 2016] Prop Prop Prop Prop [Smith et al., 2016] Prop Prop Prop Prop

38 Numerical Tests - Real objects Input Estimated Normals Estimated Surface Estimated Albedo [Smith et al., 2016] Figure: Qualitative results on real objects with varying albedo obtained by using Prop. 1+3 and comparison to [Smith et al., 2016]. S. Tozza - INdAM Analisi ed approssimazione di problemi appartenenti alla classe Shape-from-X

39 Numerical Tests - Real objects Input Estimated Normals Estimated Albedo Figure: Qualitative results on real objects with varying albedo obtained by using Prop. 1+3 and comparison to [Smith et al., 2016]. S. Tozza - INdAM Analisi ed approssimazione di problemi appartenenti alla classe Shape-from-X

40 Numerical Tests - Real objects Estimated Surface [Smith et al., 2016] Figure: Qualitative results on real objects with varying albedo obtained by using Prop. 1+3 and comparison to [Smith et al., 2016].

41 Conclusions We proposed a unified PDE formulation for recovering height from photo-polarimetric data We proposed a variety of methods that use different combinations of linear constraints The used equations are linear, so depth estimation is simply a linear least squares problem. We proposed a more robust way to estimate a polarisation image from multichannel data We showed how to estimate lighting from two source photo-polarimetric images Together, our methods provide uncalibrated, albedo invariant shape estimation with only two light sources.

42 Work in progress/future Perspectives 1 Moving to a perspective projection 2 Considering more complex reflectance models 3 Exploiting better the information available in specular reflection and polarisation 4 Allowing mixtures of the two polarisation models (diffuse and specular) 5 Adding multiview polarisation images From a single polarisation image [Smith et al., 2016], estimating the albedo [Smith et al., submitted].

43 References Wolff, L.B., Polarization vision: a new sensory approach to image understanding, Image Vision Comput., 15(2): 81 93, Huynh, C.P., Robles-Kelly, A., Hancock, E., Shape and refractive index recovery from single-view polarisation images, In: Proc. CVPR, pp , W. A.P. Smith, R. Ramamoorthi, S. Tozza, Linear depth estimation from an uncalibrated, monocular polarisation image, Lecture Notes in Computer Science 9912, pp , Springer S. Tozza, W. A.P. Smith, D. Zhu, R. Ramamoorthi, E. R. Hancock, Linear Differential Constraints for Photo-polarimetric Height Estimation, 2017 IEEE International Conference on Computer Vision (ICCV), pp , W. A.P. Smith, R. Ramamoorthi, S. Tozza, Height-from-Polarisation with Unknown Lighting or Albedo, IEEE Transactions on Pattern Analysis and Machine Intelligence, submitted.

44 Shape-from-Shading: Works in progress Resolution of the SfS problem via Filtered schemes (in collaboration with M. Falcone, G. Paolucci (Sapienza)) Multi-view/SfS/PS via minimization techniques (in collaboration with Y. Quèau (TUM Munich) and Jean-Denis Durou (IRIT, Toulouse)) Resolution of the SfS problem via Learning? (in collaboration with E. Rodolà (Sapienza))