CSCI 1290: Comp Photo

Transcription

1 CSCI 1290: Comp Photo Fall Brown University James Tompkin Many slides thanks to James Hays old CS 129 course, along with all of its acknowledgements.

2 What do we see? 3D world 2D image Point of observation Figures Stephen E. Palmer, 2002

3 What do we see? 3D world 2D image Point of observation Painted backdrop

4 On Simulating the Visual Experience Just feed the eyes the right data No one will know the difference! Philosophy: Ancient question: Does the world really exist? Science fiction: Many, many, many books on the subject, e.g. slowglass from Light of Other Days Latest take: The Matrix Physics: Slowglass might be possible? Computer Science: Virtual Reality To simulate we need to know: What does a person see?

5 What is light? Electromagnetic radiation (EMR) moving along rays in space R(l) is EMR, measured in units of power (watts) l is wavelength Useful things: Light travels in straight lines In vacuum, radiance emitted = radiance arriving i.e. there is no transmission loss

6 The Plenoptic Function Figure by Leonard McMillan Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let s start with a stationary person and try to parameterize everything that he can see

7 Grayscale snapshot is intensity of light P(q,f) Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer)

8 Color snapshot is intensity of light P(q,f,l) Seen from a single view point At a single time As a function of wavelength

9 A movie is intensity of light P(q,f,l,t) Seen from a single view point Over time As a function of wavelength

10 Holographic movie is intensity of light Seen from ANY viewpoint Over time As a function of wavelength P(q,f,l,t,V X,V Y,V Z )

11 The Plenoptic Function P(q,f,l,t,V X,V Y,V Z ) Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function

12 Sampling Plenoptic Function (top view) Just lookup Google Street View

13 Model geometry or just capture images?

14 Ray Let s not worry about time and color: 5D 3D position 2D direction P(q,f,V X,V Y,V Z ) Slide by Rick Szeliski and Michael Cohen

15 How can we use this? Lighting No Change in Radiance Surface Camera

16 Ray Reuse Infinite line Assume light is constant (vacuum) 4D 2D direction 2D position non-dispersive medium Slide by Rick Szeliski and Michael Cohen

17 Only need plenoptic surface

18 Synthesizing novel views Slide by Rick Szeliski and Michael Cohen

19 Lumigraph / Lightfield Outside convex space 4D Empty Stuff Slide by Rick Szeliski and Michael Cohen

20 Lumigraph - Organization 2D position 2D direction q s Slide by Rick Szeliski and Michael Cohen

21 Lumigraph - Organization 2D position 2D position s u 2 plane parameterization Slide by Rick Szeliski and Michael Cohen

22 Lumigraph - Organization 2D position 2D position t s,t s,t u,v v 2 plane parameterization s u u,v Slide by Rick Szeliski and Michael Cohen

23 Lumigraph - Organization Hold s,t constant Let u,v vary An image s,t u,v Slide by Rick Szeliski and Michael Cohen

24 Lumigraph / Lightfield

25 Lumigraph - Capture Idea 1 Move camera carefully over s,t plane Gantry see Lightfield paper s,t u,v Slide by Rick Szeliski and Michael Cohen

26 Lumigraph - Capture Idea 2 Move camera anywhere Rebinning see Lumigraph paper s,t u,v Slide by Rick Szeliski and Michael Cohen

27 Lumigraph - Rendering For each output pixel determine s,t,u,v either use closest discrete RGB interpolate near values s u Slide by Rick Szeliski and Michael Cohen

28 Lumigraph - Rendering Nearest closest s closest u draw it Blend 16 nearest quadrilinear interpolation s u Slide by Rick Szeliski and Michael Cohen

29 Unstructured Light Field D4PA5plo

30 Stanford multi-camera array pixels 30 fps 128 cameras synchronized timing continuous streaming flexible arrangement

31 Light field photography using a handheld plenoptic camera Commercialized as Lytro Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan

32 Conventional versus light field camera Marc Levoy

33 Marc Levoy Conventional versus light field camera uv-plane st-plane

34 Prototype camera Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125μ square-sided microlenses pixels lenses = pixels per lens

35

36 Marc Levoy Digitally stopping-down the aperture Σ Σ stopping down = summing only the central portion of each microlens

37 Marc Levoy Digital refocusing Σ Σ refocusing = summing windows extracted from several microlenses

38 Example of digital refocusing Marc Levoy

39 Marc Levoy Digitally moving the observer Σ moving the observer = moving the window we extract from the microlenses Σ

40 Example of moving the observer Marc Levoy

41 P(x,t) by David Dewey

42 2D: Image What is an image? All rays through a point Panorama? Slide by Rick Szeliski and Michael Cohen

43 Image Image plane 2D position

44 Spherical Panorama See also: 2003 New Years Eve All light rays through a point form a ponorama Totally captured in a 2D array -- P(q,f) Where is the geometry???

45 Other ways to sample Plenoptic Function Moving in time: Spatio-temporal volume: P(q,f,t) Useful to study temporal changes Long an interest of artists: Claude Monet, Haystacks studies

46 Space-time images Other ways to slice the plenoptic function t y x

47 The Theatre Workshop Metaphor (Adelson & Pentland,1996) desired image Painter Lighting Designer Sheet-metal worker

48 Painter (images)

49 Lighting Designer (environment maps) Show Naimark SF MOMA video

50 Sheet-metal Worker (geometry) Let surface normals do all the work!

51 working together clever Italians Want to minimize cost Each one does what s easiest for him Geometry big things Images detail Lighting illumination effects

52 Link to other slide deck James Tompkin s section from here: VIDEO FOR VIRTUAL REALITY LIGHT FIELD BASICS JAMES TOMPKIN

53 Slides from Alexei A. Efros, James Hays, and others Modeling Light

54 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection? λ light source James Hays

55 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ light source James Hays

56 A photon s life choices Absorption Diffuse Reflection Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ light source James Hays

57 A photon s life choices Absorption Diffusion Specular Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ light source James Hays

58 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ light source James Hays

59 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ light source James Hays

60 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ 2 λ 1 light source James Hays

61 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection λ light source James Hays

62 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection t=n t=1 light source James Hays

63 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering Phosphorescence Interreflection (Specular Interreflection) λ light source James Hays

64 Lambertian Reflectance In computer vision, surfaces are often assumed to be ideal diffuse reflectors with no dependence on viewing direction. Like in stereo depth estimation! James Hays

65 Fast Separation of Direct and Global Images Using High Frequency Illumination Shree K. Nayar Gurunandan G. Krishnan Columbia University Michael D. Grossberg City College of New York Ramesh Raskar MERL SIGGRAPH Conference Boston, July 2006 Support: ONR, NSF, MERL

66 Direct and Global Illumination participating medium surface source D B camera A E translucent surface P C A : Direct B : Interrelection C : Subsurface D : Volumetric E : Diffusion

67 Direct and Global Components: Interreflections source surface j i camera L[ c, i] = L [ c, i] L [ c, i] d + radiance direct global g L = g[ c, i] A[ i, j] L[ i, j] P BRDF and geometry

68 High Frequency Illumination Pattern source surface i camera + L [ c, i] = L [ c, i] L [ c, i] d + g fraction of activated source elements

69 High Frequency Illumination Pattern source surface i camera + L [ c, i] = L [ c, i] L [ c, i] d + - L [ c, i] = ( 1 ) L [ c, i] g g fraction of activated source elements

70 Separation from Two Images + L [ c, i] = L [ c, i] L [ c, i] d + - L [ c, i] = ( 1 ) L [ c, i] g g fraction of activated source elements = 1 : 2 L d = L max L min, L g = 2Lmin direct global

71 Diffuse Interreflections Specular Interreflections Diffusion Volumetric Scattering Subsurface Scattering

72 Scene

73 Scene Direct Global

74 High Frequency Illumination Pattern source surface i camera + L [ c, i] = L [ c, i] L [ c, i] d + - L [ c, i] = ( 1 ) L [ c, i] g g fraction of activated source elements

75 Pipeline projector surface 1. Actually use more unique chequerboard patterns Global effects from local lighting 2. Pick L_min and L_max from across all unique patterns for each pixel 3. Factor in projector s light floor camera i

76 More Real World Examples:

77 Eggs: Diffuse Interreflections Direct Global

78 Wooden Blocks: Specular Interreflections Direct Global

79 Kitchen Sink: Volumetric Scattering Volumetric Scattering: Chandrasekar 50, Ishimaru 78 Direct Global

80 Direct Global Peppers: Subsurface Scattering

81 Hand Skin: Hanrahan and Krueger 93, Uchida 96, Haro 01, Jensen et al. 01, Cula and Dana 02, Igarashi et al. 05, Weyrich et al. 05 Direct Global

82 Face: Without and With Makeup Without Makeup Direct Global With Makeup Direct Global

83 Blonde Hair Hair Scattering: Stamm et al. 77, Bustard and Smith 91, Lu et al. 00 Marschner et al. 03 Direct Global

84