Re-rendering from a Dense/Sparse Set of Images

Transcription

1 Re-rendering from a Dense/Sparse Set of Images Ko Nishino Institute of Industrial Science The Univ. of Tokyo (Japan Science and Technology) kon@cvl.iis.u-tokyo.ac.jp

2 Virtual/Augmented/Mixed Reality Three consistencies Photometric Geometric Time

3 Photometric Object Modeling? Observation Display

4 Approaches: Appearance-based Image-based Rendering Light rays as a?(<7)d function Compression Interpolation Unknown geometry

5 Approaches: Appearance-based cont d 3D Photography Surface light fields Compression Interpolation Known geometry

6 Approaches: Model-based Inverse Rendering Extract photometric information from images Texture BRDF Lighting Render

7 Our Approach Take full advantage of knowing the geometry of the target 1. From a dense set of images 2. From a sparse set of images

8 Eigen-Texture Rendering Re-rendering from a Dense Set of Images color images geometric model Eigenspace virtual object compression synthesis

9 Cutting Out Texture color images project range images mesh model

10 Cell Image Sequence 357 cell #202 0

11 Compression k eigen-textures Cell image sequence coefficients x 1,1 x 1,2 x 1,3N x 2,1 x 2,2 x 2,3N x 1,1 x 1,2 x 1,3N x 2,1 x 2,2 x 2,3N x k,1 x k,2 x k,3n x M,1 x M,2 x M,3N

12 Synthesis k eigen-textures coefficients synthetic image a M,0 + a M,1 + a M,3 = Linear combination

13 Determining the Dimensions synthetic input If planar and Lambertian 3 Photometric Stereo [Woodham 78] In real life Non-planar Specularity Interreflections Selfshadows Adaptive dimensions based on eigen-ratios

14 Cell-adaptive Compression Input Synthetic Average dimensions Compression ratio RMS error

15 Cell-adaptive Compression Input Synthetic Average dimensions Compression ratio RMS error

16 Interpolation in Eigenspace

17 Sampling Lights +

18 Integration

19 Re-rendering from a Sparse Set of Images As less input images as possible? Assumptions on BRDF Inverse rendering (a hard one) Estimate all three elements

20 Inputs 6 HDR images Camera params Mesh model

21 Reflection on a Surface specular reflection diffuse reflection

22 Reflection Components Radiance of a object surface point I = I D + I S diffuse component specular component = + Image irradiance Scene radiance

23 Diffuse Component Lambertian model isotropic N θ i I D Ω ( θ, φ ) cosθ dω ] = max[ 0, K L D i i i i i

24 Specular Component Torrance-Sparrow model Micro-facets Gaussian dist. α N θ r θ i I S K FG α 2 = S L i Ω cosθ 2σ 2 r ( θ, φ ) exp d i i i ω

25 Reflection Component Separation Intensity specular component diffuse component occluded image

26 Diffuse Texture Map 3D grids in each triangular patch min

27 Diffuse-texture-mapped Images

28 Specular Images - = Ignore interreflections

29 Illumination Hemisphere

30 Initial Illumination Hemisphere

31 Simplify and Discretize T-S I S K FG 2 α 2σ () S, v v L ( ) i θi, φi exp d i = Ω 2 cosθ r Assume F and G are constant Uniform illuminant color L Illumination as a set of point light sources ω I () v I ()L v S = S 2π K () S v L ( θ ) l l, φl N = L exp I S 2 N L cosθ r l 2σ α 2

32 Illumination and Reflectance Parameter Estimation Straightforward minimization ( ) ( ) K T S l S N k N N t s S L K k t s I k t s I,, 2,,,,,, arg σ min s,t : image coords k: image number

33 Specular Reflection as Convolution I S 2π K N cosθ N L S ( s, t, k) = L ( θ, φ ) g BS( s, t, k, θ, ) C( s, t, k) L r l l l l ( ) 1 l φl 2σ 2 specular images z = h u +η filter noise true lighting 2D convolution on the surface of the illumination hemisphere

34 Illumination and Reflectance Parameter Estimation min L, σ f N K L N ( ) = ( ) ( ) + L 2 L, σ min I S s, t, k I s, t, k 2 ρ L( θl, φl ) L, σ k N l L l Alternative Minimization fix estimate σ L fix estimate Robustness L σ M-estimation Conjugate gradient

35 Estimated Results σ = fisheye view

36 View-dependent Rendering 1. Render diffuse reflection image 2. Construct shadow maps Shadow z-buffer 3. Render specular reflection image 4. Add 1 and 3

37 View-dependent Rendering Result

38 Relighting = N L l v l l v l D v v D L M,,,, cosθ K I Diffuse reflection under the original ill-hemipshere Diffuse reflection under a new ill-hemisphere = N L l v l l v l D v v D L M ~,,,, cos ~ ~ ~ θ K I = L L N l v l l v l N l v l l v l D v D v L M L M ~,,,,,, cos ~ ~ cos ~ θ θ I I

39 Relighting Result

40 Relighting Result

41 Relighting Result

42

43

44 Conclusion Re-rendering from a Dense Sampling Appearance-based On the object surface (triangular patch based) Re-rendering from a Sparse Sampling Estimate texture, parametric BRDF and lighting Deconvolution on a spherical surface

45 Pros and Cons Appearance-based Any kind of BRDFs Requires dense sampling Model-based Less input images & compact representation Assumptions on BRDFs Assumptions on lighting

46 Future Work From a dense sampling Higher correlation global eigenspace? From a sparse sampling Relighting under different color illuminants spatio-spectral distribution

47 Collaborators Katsushi Ikeuchi Yoichi Sato The Univ. of Tokyo Zhengyou Zhang Microsoft Research

48 More Info. Ko Nishino Moving soon (Columbia Univ.)

49 Alternating Minimization Joint regularization min u, h f 2 = 2 u, h L Ω ( u, h) min h u z +α TV ( u) + α TV ( h) ( ) Total Variation (TV) norm 1 2 TV ( u) Ω u dxdy AMUH (AMHU) 0 initialize h 0 = u f ( k 1 ) k 1 u, h u Convergence? 0 = h f ( k ) k u 1, h h