Focal stacks and lightfields
|
|
|
- Phebe Mathews
- 5 years ago
- Views:
Transcription
1 Focal stacks and lightfields , , Computational Photography Fall 2018, Lecture 11
2 Course announcements Homework 3 is out. - Due October 12 th. - Any questions? Two extra late days for everyone. Can be used for this or subsequent homeworks. Project logistics on the course website. Additional guest lecture next Monday: Anat Levin, Coded photography.
3 Overview of today s lecture Leftover from last lecture: efficiently solving the Poisson equation Quick reminder: pinhole vs lens cameras. Focal stack. Confocal stereo. Lightfield. Measuring lightfields. Plenoptic camera. Images from lightfields. Some notes on (auto-)focusing.
4 Slide credits Many of these slides were adapted from: Fredo Durand (MIT). Gordon Wetzstein (Stanford). Marc Levoy (Stanford).
5 Pinhole vs lens cameras
6 Everything is in focus. Very light inefficient. Pinhole camera
7 focus plane Lens camera Only one plane is in focus. Very light efficient. How can we get an all in-focus image?
8 Focal stack
9 Focal stack Capture images focused at multiple planes. Merge them into a single all in-focus image. Analogous to what we did in HDR Focal stack instead of exposure stack.
10 Homework 5: focal stack imaging
11 Focal stack imaging 1. Capture a focal stack 2. Merge into an all in-focus image
12 Focal stack imaging 1. Capture a focal stack 2. Merge into an all in-focus image
13 How do you capture a focal stack? lens focal length f lens aperture f/# lens-object distance D lens-sensor distance D Which of these parameters would you change (and how)?
14 How do you capture a focal stack? lens focal length f rotate lens focus ring (not zoom!) lens aperture f/# lens-object distance D lens-sensor distance D Which of these parameters would you change (and how would you achieve that)?
15 How do you capture a focal stack? lens focal length f lens aperture f/# lens-object distance D lens-sensor distance D Which of these parameters would you change (and how would you achieve that)?
16 In-focus plane in each stack image Capturing a focal stack
17 Focal stack imaging 1. Capture a focal stack 2. Merge into an all in-focus image
18 How do you merge a focal stack?
19 How do you merge a focal stack? 1. Align images 2. Assign per-pixel weights representing in-focus -ness 3. Compute image average
20 How do you merge a focal stack? 1. Align images 2. Assign per-pixel weights representing in-focus -ness 3. Compute image average
21 Why do we need to align the images? Image alignment
22 Image alignment Why do we need to align the images? When we change focus distance, we also change field of view (magnification). Also, scene may not be static (but we will be ignoring this for now). change in magnification change in focus
23 Image alignment
24 Image alignment
25 Image alignment Why do we need to align the images? When we change focus distance, we also change field of view (magnification). Also, scene may not be static (but we will be ignoring this for now). change in magnification change in focus Assume we know f and all D values. How do can we align the images?
26 Image alignment Why do we need to align the images? When we change focus distance, we also change field of view (magnification). Also, scene may not be static (but we will be ignoring this for now). change in magnification change in focus m = 1 f D f D + 1 D = 1 f resize using these equations Assume we know f and all D values. How do can we align the images?
27 How do you merge a focal stack? 1. Align images 2. Assign per-pixel weights representing in-focus -ness 3. Compute image average
28 Weight assignment How do we measure how much in-focus each pixel is?
29 Weight assignment How do we measure how much in-focus each pixel is? Measure local sharpness. This is also how auto-focus works. How do we measure local sharpness?
30 Weight assignment How do we measure how much in-focus each pixel is? Measure local sharpness. This is also how auto-focus works. run Laplacian operator do some Gaussian blurring (why?)
31 Weight assignment How do we measure how much in-focus each pixel is? Measure local sharpness. This is also how auto-focus works. run Laplacian operator Just one example, many alternatives possible. do some Gaussian blurring (so that nearby pixels have similar weights)
32 How do you merge a focal stack? 1. Align images 2. Assign per-pixel weights representing in-focus -ness 3. Compute image average
33 Focal stack merging
34 Some results example image from stack all in-focus image
35 Another example Focal stacking is very useful in macrophotography, where depths of field are very shallow
36 Another example middle image from stack all in-focus image
37 Another look at the mixing weights What do the mixing weights look like?
38 Another look at the mixing weights Depth from defocus = determining sharpest pixel in focal stack
39 Depth from defocus on a mobile phone Use focal stack from autofocus [Suwajanakorn et al., CVPR 2015]
40 Another look at the mixing weights What is a problem of these depth maps?
41 Another look at the mixing weights What is a problem of these depth maps? Blurry because we to process entire neighborhoods to compute sharpness. Can we use any extra information to get per-pixel depth?
42 Confocal stereo
43 Confocal stereo Capture a 2D stack by varying both focus and aperture Analyze each pixel s focus-aperture image to find true depth [Hassinof and Kutulakos, ECCV 2006]
44 aperture α ( aperture α i, focus f j ) pixel p Aperture-Focus Image of pixel p aperture α focus f focus f
45 AFI model fitting Key idea: When a point is in focus, its color and intensity remain the same for all apertures. This property is called confocal constancy. We can find correct depth by checking intensity variation across apertures for each focus setting. focus hypothesis (equi-blur regions) AFI best model fit to AFI absolute difference RMS error
46 Lightfield
47 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
48 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
49 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
50 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
51 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
52 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
53 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints How would you merge these images into a lens, defocused image?
54 Lightfield: all rays in a scene reference plane aperture plane What is the dimension of the lightfield? Parameterize every ray based on its intersections with two planes. [Levoy et al. SIGGRAPH 1996, Gortler et al. SIGGRAPH 1996]
55 Lightfield: all rays in a scene reference plane (s, t) aperture plane (u, v) 4-dimensional function L(u, v, s, t) Parameterize every ray based on its intersections with two planes.
56 Lightfield: all rays in a scene reference plane (s, t) aperture plane (u, v) sensor plane (s, t) 4-dimensional function L(u, v, s, t) (conjugate of scene-based function) Parameterize every ray based on its intersections with two planes.
57 Lightfield slices aperture plane (u, v) sensor plane (s, t) What does L(u = u o, v = v o, s, t) look like? 4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
58 Lightfield slices aperture plane (u, v) sensor plane (s, t) What does L(u = u o, v = v o, s, t) look like? a pinhole image from a certain viewpoint 4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
59 Lightfield slices reference plane (s, t) aperture plane (u, v) sensor plane (s, t) aperture coordinates u = u o, v = v o Lightfield slice L(u = u o, v = v o, s, t)
60 Lightfield slices aperture plane (u, v) sensor plane (s, t) What does L(u = u o, v = v o, s, t) look like? a pinhole image from a certain viewpoint What does L(u, v, s = s o, t = t o ) look like? 4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
61 Lightfield slices aperture plane (u, v) sensor plane (s, t) What does L(u = u o, v = v o, s, t) look like? a pinhole image from a certain viewpoint What does L(u, v, s = s o, t = t o ) look like? radiance emitted by a certain (in-focus) point at various directions 4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
62 Lightfield slices reference plane (s, t) aperture plane (u, v) sensor plane (s, t) reference/sensor coordinates s = s o, t = t o reference/sensor coordinates s = s o, t = t o Lightfield slice L(u, v, s = s o, t = t o )
63 Lightfield slices aperture plane (u, v) sensor plane (s, t) What does L(u = u o, v = v o, s, t) look like? a pinhole image from a certain viewpoint What does L(u, v, s = s o, t = t o ) look like? radiance emitted by a certain (in-focus) point in various directions 4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
64 Lightfield visualization L(u = u o, v = v o, s, t) is a pinhole image from a certain viewpoint L(u, v, s = s o, t = t o ) is the radiance emitted by a certain (in-focus) point at various directions Demo:
65 Lightfield: all rays in a scene reference plane (s, t) aperture plane (u, v) sensor plane (s, t) How can you capture the lightfield of a scene? 4-dimensional function L(u, v, s, t) (conjugate of scene-based function)
66 Measuring lightfields
67 How can you do this? How to capture a lightfield?
68 Option 1: use multiple cameras Stanford camera array ( synthetic aperture ) [Willburn et al., SIGGRAPH 2005]
69 Option 1: use multiple cameras Stanford camera array What kind of lens would you use for this? ( synthetic aperture ) [Willburn et al., SIGGRAPH 2005]
70
71 Option 2: take multiple images with one camera Single camera mounted on LEGO motor. Demo: portrait_lf/preview.zip&zoom=1 How would you move the camera?
72 Plenoptic camera
73 Option 3: use a plenoptic camera plenoptic = plenus (Latin for full ) + optic (Greek for seeing, in this case)
74 Making a plenoptic camera reference plane (s, t) aperture plane (u, v) sensor plane (s, t) reference/sensor coordinates s = s o, t = t o reference/sensor coordinates s = s o, t = t o Lightfield slice L(u, v, s = s o, t = t o )
75 Making a plenoptic camera reference plane (s, t) aperture plane (u, v) sensor plane (s, t) each pixel measures ray L(u, v, s = s o, t = t o ) for different (u, v) reference/sensor coordinates s = s o, t = t o reference/sensor coordinates s = s o, t = t o Lightfield slice L(u, v, s = s o, t = t o )
76 Making a plenoptic camera reference plane (s, t) aperture plane (u, v) sensor plane (s, t) each pixel measures a ray L(u, v, s, t) each pinhole corresponds to a slice L(u, v, s = s o, t = t o ) Lightfield L(u, v, s, t) How can we make this more light efficient?
77 Making a plenoptic camera reference plane (s, t) aperture plane (u, v) sensor plane (s, t) each pixel measures a ray L(u, v, s, t) each pinhole corresponds to a slice L(u, v, s = s o, t = t o ) Lightfield L(u, v, s, t) How can we make this more light efficient? replace pinholes with lenslets
78 History of the plenoptic camera First conceptualized by Gabriel Lippmann, who called it integral photography. Original article appeared in French journal three years earlier, what is shown here is an American re-print.
79 History of the plenoptic camera First conceptualized by Gabriel Lippmann, who called it integral photography. Original article appeared in French journal three years earlier, what is shown here is an American re-print. Notice the date: more than a century ago.
80 History of the plenoptic camera Reappeared under different forms and names throughout the century. The left paper is from 1930, the right one from 1970.
81 History of the plenoptic camera Re-introduced to computer vision and graphics in 1992, which was also when the term plenoptic camera was coined.
82 History of the plenoptic camera Figure from the 1992 paper, which shows the version of the plenoptic camera with pinholes instead of lenslets on the image plane.
83 History of the plenoptic camera Re-re-introduced to computer vision and graphics in 2005, when the term light field photography was also coined.
84 Prototype plenoptic camera [Ng et al., Stanford Technical Report 2005]
85 Commercial plenoptic camera
86 Commercial plenoptic camera lenslet array
87 Commercial plenoptic camera newer version with higher resolution
88 Industrial plenoptic cameras Plenoptic cameras have become quite popular in lab and industrial settings. Much higher resolution, both spatial and angular, than commercial cameras. Support interchangeable lenses. Can do video. Very expensive.
89 Shack-Hartmann wavefront sensors Completely different use: measuring how close a wavefront is to being planar Exactly the same optics as a plenoptic camera. Common instrument in optics labs and medical imaging.
90 Question Given that the plenoptic camera design was known since 1908, why did it take a century for such a camera to be made commercially available?
91 Question Given that the plenoptic camera design was known since 1908, why did it take a century for such a camera to be made commercially available? Difficult to manufacture good lenslet arrays at high resolution (unless attached to sensors). Digital sensors did not have sufficiently high resolution. No known good use for them besides depth sensing. - The introduction of the handheld plenoptic camera was strongly influenced by the 1996 lightfield papers, which demonstrated all of the useful photographic operations one could replicate by having access to the entire lightfield. And of course, nobody thought of commercializing them.
92 Making a plenoptic camera Is it possible to make a plenoptic camera using an array of pinholes?
93 Making a plenoptic camera Need to be able to separate different rays falling on same pixel. Is it possible to make a plenoptic camera using an array of pinholes? Yes, under certain conditions on the images being captured. [Georgiev, ECCV 2008]
94 Images from lightfields
95 A plenoptic image What are these circles?
96 A plenoptic camera reference plane (s, t) aperture plane (u, v) sensor plane (s, t) each pinhole corresponds to a slice L(u, v, s = s o, t = t o ) Lightfield L(u, v, s, t)
97 Simulate different viewpoints? How do I
98 Simulate different viewpoints? Pick same pixel within each aperture view How do I
99 Changing the viewpoint Viewpoint change is limited by the aperture of each of the lenslets.
100 How do I Simulate different viewpoints? Pick same pixel within each aperture view Simulate different aperture sizes?
101 How do I Simulate different viewpoints? Pick same pixel within each aperture view Simulate different aperture sizes? Sum more than one pixels within each aperture view
102 How do I Simulate different viewpoints? Pick same pixel within each aperture view Simulate different aperture sizes? Sum more than one pixels within each aperture view Simulate different focus depths?
103 How do I Simulate different viewpoints? Pick same pixel within each aperture view Simulate different aperture sizes? Sum more than one pixels within each aperture view Simulate different focus depths? Shift the areas you integrate relative to each other
104 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
105 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
106 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
107 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
108 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints
109 Measuring rays We can capture the same set of rays by using a pinhole camera from multiple viewpoints How would you merge these images into a lens, defocused image?
110
111
112
113 [Ng, SIGGRAPH 2006]
114 Refocusing example
115 Refocusing example
116 Refocusing example
117 Synthesizing images from a lightfield Many more examples with demo:
118 Three ways to measure a lightfield 1) Use a plenoptic camera 2) Use a camera array 3) Use one camera multiple times What are the pros and cons of each?
119 Some notes on (auto-)focusing
120 Different cameras have different focusing processes
121 Manual focus in rangefinder cameras Focusing based on triangulation: when the image is in focus, you will see the two copies aligned. Very accurate but very painstaking. Different perspective than that of the main lens. standard in Leica cameras
122 Manual focus in (D)SLR cameras Same view as main lens. Just rotate the focusing ring until you are satisfied by the sharpness. Viewfinder indicators can help this process. grid of points where sharpness is evaluated
123 Manual focus in (D)SLR cameras Same view as main lens. Just rotate the focusing ring until you are satisfied by the sharpness. Viewfinder indicators can help this process. instead of a grid, you can also focus based on a single point
124 Active auto-focus: time-of-flight sensors Basically how SONAR works (we ll also see time-of-flight sensors later in class. Method used in Polaroid cameras, which used ultrasound waves. Energy inefficient. Limited range. Multi-path interference (e.g., glass surfaces back-reflected the waves).
125 Passive auto-focus: phase detection As the lens moves, ray bundles from an object converge to a different point in the camera and change in angle. This change in angle causes them to refocus through two lenslets to different positions on a separate AF sensor. A certain spacing between these double images indicates that the object is in focus. Demo:
126 Passive auto-focus: contrast detection Sensors at different image distances will see the same object as high-contrast if it s in focus, or low-contrast if it s not. Move the lens until the high-contrast subimage falls on the middle sensor, which is conjugate to the camera s main sensor. Compute contrast using local differences of pixel values. Demo:
127 High-end DSLRs use phase detection Distance between sub-images allows lens to move directly into focus, without hunting Many AF points, complicated algorithms for choosing among them: generally use closest point, but also consider position in FOV.
128 Low-end cameras (and phones) use contrast detection Nowadays it is mostly done using main camera sensor instead of dedicated sensors. Requires repeated measurements as lens moves, which are captured using the main sensor (an autofocus stack ). Equivalent to depth-from-focus in computer vision. Slow, requires hunting, suffers from overshooting. But People have come up with creative uses for the autofocus stack (depth-from-defocus on a cell phone, HDR+ on Android). [Suwajanakorn et al., CVPR 2015; Hasinoff et al., SIGGRAPH Asia 2016]
129 Should you use autofocus?
130 Quick answer: Yes. Should you use autofocus?
131 Should you use autofocus? Quick answer: Yes. More detailed answer: Yes, except for certain special circumstances. You are using a lens that does not have an autofocus motor (e.g., vintage or otherwise old lenses, high-end lenses, industrial and machine vision lenses). You are trying to capture an image under conditions where autofocus is prone to fail (e.g., macrophotography, poorly-lit scenes, imaging through glass or occluders). You intentionally want some part of the scene to be out of focus (e.g., for artistic effect, or because you want a face or other featured to be obscured). You are in an once-in-a-lifetime opportunity to photograph something, and you cannot afford to risk autofocus failing. This additionally assumes that: - Your scene is static enough that you can take the time to focus manually. - You are experienced enough so that the probability of manual focus failing is smaller than the probability of autofocus failing.
132 References Basic reading: Szeliski textbook, Section , Additional reading: Nayar et al., Real-Time Focus Range Sensor, PAMI One of the (relatively) early papers on depth from defocus. Suwajanakorn et al., Depth from Focus with Your Mobile Phone, CVPR Implementing depth from defocus on a mobile phone using the autofocus focal stack. Hasinoff and Kutulakos, Confocal Stereo, IJCV The paper on high resolution depth from a focus and aperture stack. Kuthirummal et al., Flexible Depth of Field Photography, PAMI Continuously change focus within one exposure, without stopping to capture a stack. Levoy and Hanrahan, Light Field Rendering, SIGGRAPH Gortler et al., The Lumigraph, SIGGRAPH The two papers introducing the light field. Adelson and Wang, Single Lens Stereo with a Plenoptic Camera, PAMI The paper (re)-introducing the plenoptic camera to computer vision and graphics. Ng et al., Light field photography with a hand-held plenoptic camera, Stanford TR The paper (re)-(re)-introducing the plenoptic camera, and the precursor to Lytro. Ng, Fourier Slice Photography, SIGGRAPH The paper on frequency-space analysis of refocusing and lightfield measurements. Wilburn et al., High Performance Imaging Using Large Camera Arrays, SIGGRAPH The camera array paper. Georgiev et al., Unified Frequency Domain Analysis of Lightfield Cameras, ECCV Make a lightfield camera from a pinhole array (and many other interesting stuff about lightfield cameras).
Geometric camera models and calibration
Geometric camera models and calibration http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2018, Lecture 13 Course announcements Homework 3 is out. - Due October
Computational Photography
Computational Photography Matthias Zwicker University of Bern Fall 2010 Today Light fields Introduction Light fields Signal processing analysis Light field cameras Application Introduction Pinhole camera
Computational Photography: Real Time Plenoptic Rendering
Computational Photography: Real Time Plenoptic Rendering Andrew Lumsdaine, Georgi Chunev Indiana University Todor Georgiev Adobe Systems Who was at the Keynote Yesterday? 2 Overview Plenoptic cameras Rendering
Modeling Light. Michal Havlik : Computational Photography Alexei Efros, CMU, Fall 2007
Modeling Light Michal Havlik 15-463: Computational Photography Alexei Efros, CMU, Fall 2007 The Plenoptic Function Figure by Leonard McMillan Q: What is the set of all things that we can ever see? A: The
Radiance Photography. Todor Georgiev Adobe Systems. Andrew Lumsdaine Indiana University
Radiance Photography Todor Georgiev Adobe Systems Andrew Lumsdaine Indiana University Course Goals Overview of radiance (aka lightfield) photography Mathematical treatment of theory and computation Hands
Computational Cameras: Exploiting Spatial- Angular Temporal Tradeoffs in Photography
Mitsubishi Electric Research Labs (MERL) Computational Cameras Computational Cameras: Exploiting Spatial- Angular Temporal Tradeoffs in Photography Amit Agrawal Mitsubishi Electric Research Labs (MERL)
The Focused Plenoptic Camera. Lightfield photographers, focus your cameras! Karl Marx
The Focused Plenoptic Camera Lightfield photographers, focus your cameras! Karl Marx Plenoptic Camera, Adelson 1992 Main lens focused on microlenses Plenoptic Camera, Adelson 1992 Microlenses focused on
Modeling Light. Slides from Alexei A. Efros and others
Project 3 Results http://www.cs.brown.edu/courses/cs129/results/proj3/jcmace/ http://www.cs.brown.edu/courses/cs129/results/proj3/damoreno/ http://www.cs.brown.edu/courses/cs129/results/proj3/taox/ Stereo
Plenoptic camera and its Applications
Aum Sri Sairam Plenoptic camera and its Applications Agenda: 1. Introduction 2. Single lens stereo design 3. Plenoptic camera design 4. Depth estimation 5. Synthetic refocusing 6. Fourier slicing 7. References
Computational Photography
Computational Photography Photography and Imaging Michael S. Brown Brown - 1 Part 1 Overview Photography Preliminaries Traditional Film Imaging (Camera) Part 2 General Imaging 5D Plenoptic Function (McMillan)
Title: The Future of Photography is Computational Photography. Subtitle: 100 years after invention, Integral Photography becomes feasible
Title: The Future of Photography is Computational Photography Subtitle: 100 years after invention, Integral Photography becomes feasible Adobe customers are creative and demanding. They expect to use our
Chapter 3-Camera Work
Chapter 3-Camera Work The perfect camera? Make sure the camera you purchase works for you Is it the right size? Does it have the type of lens you need? What are the features that I want? What type of storage
Image-Based Rendering
Image-Based Rendering COS 526, Fall 2016 Thomas Funkhouser Acknowledgments: Dan Aliaga, Marc Levoy, Szymon Rusinkiewicz What is Image-Based Rendering? Definition 1: the use of photographic imagery to overcome
CIS 580, Machine Perception, Spring 2015 Homework 1 Due: :59AM
CIS 580, Machine Perception, Spring 2015 Homework 1 Due: 2015.02.09. 11:59AM Instructions. Submit your answers in PDF form to Canvas. This is an individual assignment. 1 Camera Model, Focal Length and
Coding and Modulation in Cameras
Mitsubishi Electric Research Laboratories Raskar 2007 Coding and Modulation in Cameras Ramesh Raskar with Ashok Veeraraghavan, Amit Agrawal, Jack Tumblin, Ankit Mohan Mitsubishi Electric Research Labs
Chapter 12-Close-Up and Macro Photography
Chapter 12-Close-Up and Macro Photography Close-up images all taken with Hipstamatic on the iphone Close-up and Macro Examples All taken with close-up filter or reverse mount lens Macro Close-up Photography
Plenoptic Cameras. Bastian Goldlücke, Oliver Klehm, Sven Wanner, and Elmar Eisemann. 5.1 Introduction
Plenoptic Cameras Bastian Goldlücke, Oliver Klehm, Sven Wanner, and Elmar Eisemann 5.1 Introduction The light field, as defined by Gershun in 1936 [Gershun 36] describes the radiance traveling in every
Modeling Light. Michal Havlik
Modeling Light Michal Havlik 15-463: Computational Photography Alexei Efros, CMU, Spring 2010 What is light? Electromagnetic radiation (EMR) moving along rays in space R( ) is EMR, measured in units of
Structured light , , Computational Photography Fall 2017, Lecture 27
Structured light http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 27 Course announcements Homework 5 has been graded. - Mean: 129. - Median:
DSLR Cameras and Lenses. Paul Fodor
DSLR Cameras and Lenses Paul Fodor Camera Principle of a pinhole camera: Light rays from an object pass through a small hole to form an image on the sensor: 2 Aperture and Focal Length Aperture is the
More and More on Light Fields. Last Lecture
More and More on Light Fields Topics in Image-Based Modeling and Rendering CSE291 J00 Lecture 4 Last Lecture Re-review with emphasis on radiometry Mosaics & Quicktime VR The Plenoptic function The main
Modeling Light. Michal Havlik
Modeling Light Michal Havlik 15-463: Computational Photography Alexei Efros, CMU, Fall 2007 What is light? Electromagnetic radiation (EMR) moving along rays in space R(λ) is EMR, measured in units of power
AN O(N 2 LOG(N)) PER PLANE FAST DISCRETE FOCAL STACK TRANSFORM
AN O(N 2 LOG(N)) PER PLANE FAST DISCRETE FOCAL STACK TRANSFORM Fernando Pérez Nava +, Jonás Philipp Lüke + Departamento de Estadística, Investigación Operativa y Computación Departamento de Física Fundamental
Modeling Light. Michal Havlik : Computational Photography Alexei Efros, CMU, Fall 2011
Modeling Light Michal Havlik 15-463: Computational Photography Alexei Efros, CMU, Fall 2011 What is light? Electromagnetic radiation (EMR) moving along rays in space R(λ) is EMR, measured in units of power
Lecture 19: Depth Cameras. Visual Computing Systems CMU , Fall 2013
Lecture 19: Depth Cameras Visual Computing Systems Continuing theme: computational photography Cameras capture light, then extensive processing produces the desired image Today: - Capturing scene depth
http://www.diva-portal.org This is the published version of a paper presented at 2018 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), Stockholm Helsinki Stockholm,
Image-based modeling (IBM) and image-based rendering (IBR)
Image-based modeling (IBM) and image-based rendering (IBR) CS 248 - Introduction to Computer Graphics Autumn quarter, 2005 Slides for December 8 lecture The graphics pipeline modeling animation rendering
Stereo and structured light
Stereo and structured light http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2018, Lecture 20 Course announcements Homework 5 is still ongoing. - Make sure
Computational Methods for Radiance. Render the full variety offered by the direct observation of objects. (Computationally).
Computational Methods for Radiance Render the full variety offered by the direct observation of objects. (Computationally). Methods for Plenoptic 1.0 Computing with Radiance Goal: Render the full variety
But, vision technology falls short. and so does graphics. Image Based Rendering. Ray. Constant radiance. time is fixed. 3D position 2D direction
Computer Graphics -based rendering Output Michael F. Cohen Microsoft Research Synthetic Camera Model Computer Vision Combined Output Output Model Real Scene Synthetic Camera Model Real Cameras Real Scene
DD2423 Image Analysis and Computer Vision IMAGE FORMATION. Computational Vision and Active Perception School of Computer Science and Communication
DD2423 Image Analysis and Computer Vision IMAGE FORMATION Mårten Björkman Computational Vision and Active Perception School of Computer Science and Communication November 8, 2013 1 Image formation Goal:
Light transport matrices
Light transport matrices http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2018, Lecture 18 Course announcements Homework 5 has been posted. - Due on Friday
VIDEO FOR VIRTUAL REALITY LIGHT FIELD BASICS JAMES TOMPKIN
VIDEO FOR VIRTUAL REALITY LIGHT FIELD BASICS JAMES TOMPKIN WHAT IS A LIGHT FIELD? Light field seems to have turned into a catch-all term for many advanced camera/display technologies. WHAT IS A LIGHT FIELD?
CMPSCI 670: Computer Vision! Image formation. University of Massachusetts, Amherst September 8, 2014 Instructor: Subhransu Maji
CMPSCI 670: Computer Vision! Image formation University of Massachusetts, Amherst September 8, 2014 Instructor: Subhransu Maji MATLAB setup and tutorial Does everyone have access to MATLAB yet? EdLab accounts
CS201 Computer Vision Lect 4 - Image Formation
CS201 Computer Vision Lect 4 - Image Formation John Magee 9 September, 2014 Slides courtesy of Diane H. Theriault Question of the Day: Why is Computer Vision hard? Something to think about from our view
SD Cards = Your Film. Always turn off your camera before removing! First thing, format your card (erases card)
Core Concepts SD Cards = Your Film Always turn off your camera before removing! First thing, format your card (erases card) Formatting your card Menu Button Top Wheel To Wrench 1 Back Wheel to Format Card
Non-line-of-sight imaging
Non-line-of-sight imaging http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 25 Course announcements Homework 6 will be posted tonight. - Will
Light Field Spring
Light Field 2015 Spring Recall: Light is Electromagnetic radiation (EMR) moving along rays in space R(l) is EMR, measured in units of power (watts) l is wavelength Useful things: Light travels in straight
Depth Estimation with a Plenoptic Camera
Depth Estimation with a Plenoptic Camera Steven P. Carpenter 1 Auburn University, Auburn, AL, 36849 The plenoptic camera is a tool capable of recording significantly more data concerning a particular image
DEPTH, STEREO AND FOCUS WITH LIGHT-FIELD CAMERAS
DEPTH, STEREO AND FOCUS WITH LIGHT-FIELD CAMERAS CINEC 2014 Frederik Zilly, Head of Group Computational Imaging & Algorithms Moving Picture Technologies Department, Fraunhofer IIS Fraunhofer, Frederik
Introduction to 3D Concepts
PART I Introduction to 3D Concepts Chapter 1 Scene... 3 Chapter 2 Rendering: OpenGL (OGL) and Adobe Ray Tracer (ART)...19 1 CHAPTER 1 Scene s0010 1.1. The 3D Scene p0010 A typical 3D scene has several
The CAFADIS camera: a new tomographic wavefront sensor for Adaptive Optics.
1st AO4ELT conference, 05011 (2010) DOI:10.1051/ao4elt/201005011 Owned by the authors, published by EDP Sciences, 2010 The CAFADIS camera: a new tomographic wavefront sensor for Adaptive Optics. J.M. Rodríguez-Ramos
Modeling Light. On Simulating the Visual Experience
Modeling Light 15-463: Rendering and Image Processing Alexei Efros On Simulating the Visual Experience Just feed the eyes the right data No one will know the difference! Philosophy: Ancient question: Does
Real-time Generation and Presentation of View-dependent Binocular Stereo Images Using a Sequence of Omnidirectional Images
Real-time Generation and Presentation of View-dependent Binocular Stereo Images Using a Sequence of Omnidirectional Images Abstract This paper presents a new method to generate and present arbitrarily
Light Field Occlusion Removal
Light Field Occlusion Removal Shannon Kao Stanford University kaos@stanford.edu Figure 1: Occlusion removal pipeline. The input image (left) is part of a focal stack representing a light field. Each image
Next Generation CAT System
Next Generation CAT System Jaewon Kim MIT Media Lab Advisor: Ramesh Raskar Associate Professor of Media Arts & Sciences MIT Media Lab Reader: V. Michael Bove Principal Research Scientist of Media Arts
Projective Geometry and Camera Models
/2/ Projective Geometry and Camera Models Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem Note about HW Out before next Tues Prob: covered today, Tues Prob2: covered next Thurs Prob3:
CSCI 1290: Comp Photo
CSCI 1290: Comp Photo Fall 2018 @ Brown University James Tompkin Many slides thanks to James Hays old CS 129 course, along with all of its acknowledgements. What do we see? 3D world 2D image Point of observation
5 main methods. 1 Dedicated Macro lens. 2 Extension tubes. 3 Close up Filters. 4 Teleconverters. 5 Stacking lenses.
Macro Photography The technical part Macrophotography. MacroPhotography is the capturing of an image of an object at lifesize or up to 10 times life size on the recording medium, that is the film or digital
3D Surface Reconstruction Based On Plenoptic Image. Haiqiao Zhang
3D Surface Reconstruction Based On Plenoptic Image by Haiqiao Zhang A thesis submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Master of
Interactive Light Field Editing and Compositing
Interactive Light Field Editing and Compositing Billy Chen Daniel Horn* Gernot Ziegler Hendrik P. A. Lensch* Stanford University MPI Informatik (c) (d) Figure 1: Our system enables a user to interactively
Overview of Active Vision Techniques
SIGGRAPH 99 Course on 3D Photography Overview of Active Vision Techniques Brian Curless University of Washington Overview Introduction Active vision techniques Imaging radar Triangulation Moire Active
Computational Imaging for Self-Driving Vehicles
CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz--------Ramesh Raskar--------Achuta Kadambi--------Guy Satat Computational Imaging for Self-Driving Vehicles Jan Kautz--------Ramesh Raskar--------Achuta
Visual Communication: Basics of Photography Appendix 2
11. Basics of Photography Synopsis This module provides an overview of the basics of photography.. Lectures 11.1 Cameras 11.2 Aperture 11.3 Shutter Speed 11.4 Light 11.5 Composition 11.6 Creative Exercises
ECE-161C Cameras. Nuno Vasconcelos ECE Department, UCSD
ECE-161C Cameras Nuno Vasconcelos ECE Department, UCSD Image formation all image understanding starts with understanding of image formation: projection of a scene from 3D world into image on 2D plane 2
CS6670: Computer Vision
CS6670: Computer Vision Noah Snavely Lecture 5: Projection Reading: Szeliski 2.1 Projection Reading: Szeliski 2.1 Projection Müller Lyer Illusion http://www.michaelbach.de/ot/sze_muelue/index.html Modeling
Image or Object? Is this real?
Image or Object? Michael F. Cohen Microsoft Is this real? Photo by Patrick Jennings (patrick@synaptic.bc.ca), Copyright 1995, 96, 97 Whistler B. C. Canada Modeling, Rendering, and Lighting 1 A mental model?
Why is computer vision difficult?
Why is computer vision difficult? Viewpoint variation Illumination Scale Why is computer vision difficult? Intra-class variation Motion (Source: S. Lazebnik) Background clutter Occlusion Challenges: local
L16. Scan Matching and Image Formation
EECS568 Mobile Robotics: Methods and Principles Prof. Edwin Olson L16. Scan Matching and Image Formation Scan Matching Before After 2 Scan Matching Before After 2 Map matching has to be fast 14 robots
Lecture 15: Image-Based Rendering and the Light Field. Kayvon Fatahalian CMU : Graphics and Imaging Architectures (Fall 2011)
Lecture 15: Image-Based Rendering and the Light Field Kayvon Fatahalian CMU 15-869: Graphics and Imaging Architectures (Fall 2011) Demo (movie) Royal Palace: Madrid, Spain Image-based rendering (IBR) So
Ping Tan. Simon Fraser University
Ping Tan Simon Fraser University Photos vs. Videos (live photos) A good photo tells a story Stories are better told in videos Videos in the Mobile Era (mobile & share) More videos are captured by mobile
3D Photography: Active Ranging, Structured Light, ICP
3D Photography: Active Ranging, Structured Light, ICP Kalin Kolev, Marc Pollefeys Spring 2013 http://cvg.ethz.ch/teaching/2013spring/3dphoto/ Schedule (tentative) Feb 18 Feb 25 Mar 4 Mar 11 Mar 18 Mar
Structure from Motion and Multi- view Geometry. Last lecture
Structure from Motion and Multi- view Geometry Topics in Image-Based Modeling and Rendering CSE291 J00 Lecture 5 Last lecture S. J. Gortler, R. Grzeszczuk, R. Szeliski,M. F. Cohen The Lumigraph, SIGGRAPH,
Analysis of Two Representative Algorithms of Depth Estimation from Light Field Images
Analysis of Two Representative Algorithms of Depth Estimation from Light Field Images by Yutao Chen A Report Submitted in Partial Fulfillment of the Requirements for the Degree of MASTER OF ENGINEERING
Optical image stabilization (IS)
Optical image stabilization (IS) CS 448A, Winter 2010 Marc Levoy Computer Science Department Stanford University Outline! what are the causes of camera shake? how can you avoid it (without having an IS
The Light Field and Image-Based Rendering
Lecture 11: The Light Field and Image-Based Rendering Visual Computing Systems Demo (movie) Royal Palace: Madrid, Spain Image-based rendering (IBR) So far in course: rendering = synthesizing an image from
Mosaics, Plenoptic Function, and Light Field Rendering. Last Lecture
Mosaics, Plenoptic Function, and Light Field Rendering Topics in Image-ased Modeling and Rendering CSE291 J00 Lecture 3 Last Lecture Camera Models Pinhole perspective Affine/Orthographic models Homogeneous
L1a: Introduction to Light Fields
L1a: Introduction to Light Fields 2018 IEEE SPS Summer School on Light Field Data Representation, Interpretation, and Compression Donald G. Dansereau, May 2018 Schedule 2 Outline Lecture 1a: Introduction
Advanced Vision Guided Robotics. David Bruce Engineering Manager FANUC America Corporation
Advanced Vision Guided Robotics David Bruce Engineering Manager FANUC America Corporation Traditional Vision vs. Vision based Robot Guidance Traditional Machine Vision Determine if a product passes or
Introduction to Photography
Topic 5 - Basic DSLR Settings Learning Outcomes In this topic, you will learn about the functions of the buttons on the top, the side and the back of your camera. This will give you the platform to begin
A Frequency Analysis of Light Transport
A Frequency Analysis of Light Transport Frédo Durand MIT CSAIL With Nicolas Holzschuch, Cyril Soler, Eric Chan & Francois Sillion Artis Gravir/Imag-Inria & MIT CSAIL Our research 3D rendering Light transport
Introduction to 3D Machine Vision
Introduction to 3D Machine Vision 1 Many methods for 3D machine vision Use Triangulation (Geometry) to Determine the Depth of an Object By Different Methods: Single Line Laser Scan Stereo Triangulation
Multiple View Geometry
Multiple View Geometry Martin Quinn with a lot of slides stolen from Steve Seitz and Jianbo Shi 15-463: Computational Photography Alexei Efros, CMU, Fall 2007 Our Goal The Plenoptic Function P(θ,φ,λ,t,V
3D Photography: Stereo
3D Photography: Stereo Marc Pollefeys, Torsten Sattler Spring 2016 http://www.cvg.ethz.ch/teaching/3dvision/ 3D Modeling with Depth Sensors Today s class Obtaining depth maps / range images unstructured
Projective Geometry and Camera Models
Projective Geometry and Camera Models Computer Vision CS 43 Brown James Hays Slides from Derek Hoiem, Alexei Efros, Steve Seitz, and David Forsyth Administrative Stuff My Office hours, CIT 375 Monday and
Announcements. Light. Properties of light. Light. Project status reports on Wednesday. Readings. Today. Readings Szeliski, 2.2, 2.3.
Announcements Project status reports on Wednesday prepare 5 minute ppt presentation should contain: problem statement (1 slide) description of approach (1 slide) some images (1 slide) current status +
CAMERAS. ADVANTAGES Access to a wide range of lenses/focal lengths. High-resolution sensors. Full range of manual controls and Raw file capture.
ESSENTIALS essentials cameras 10 CAMERAS When shooting on film was the only option for landscape photographers, bigger was always considered to be better. Large-format cameras loaded with sheet film provided
Computer Vision CS 776 Fall 2018
Computer Vision CS 776 Fall 2018 Cameras & Photogrammetry 1 Prof. Alex Berg (Slide credits to many folks on individual slides) Cameras & Photogrammetry 1 Albrecht Dürer early 1500s Brunelleschi, early
CS4670: Computer Vision
CS467: Computer Vision Noah Snavely Lecture 13: Projection, Part 2 Perspective study of a vase by Paolo Uccello Szeliski 2.1.3-2.1.6 Reading Announcements Project 2a due Friday, 8:59pm Project 2b out Friday
Image Based Rendering
Image Based Rendering an overview Photographs We have tools that acquire and tools that display photographs at a convincing quality level 2 1 3 4 2 5 6 3 7 8 4 9 10 5 Photographs We have tools that acquire
3D Particle Position Reconstruction Accuracy in Plenoptic PIV
AIAA SciTech 13-17 January 2014, National Harbor, Maryland 52nd Aerospace Sciences Meeting AIAA 2014-0398 3D Particle Position Reconstruction Accuracy in Plenoptic PIV Timothy W. Fahringer and Brian S.
HTC ONE (M8) DUO CAMERA WHITE PAPERS
HTC ONE (M8) DUO CAMERA WHITE PAPERS 1 HTC ONE (M8) DUO CAMERA WHITE PAPERS INTRODUCTION In a few short years, the smartphone has become the predominant image-capturing device for millions of people. At
Multi-View Stereo for Community Photo Collections Michael Goesele, et al, ICCV Venus de Milo
Vision Sensing Multi-View Stereo for Community Photo Collections Michael Goesele, et al, ICCV 2007 Venus de Milo The Digital Michelangelo Project, Stanford How to sense 3D very accurately? How to sense
LET S FOCUS ON FOCUSING
LET S FOCUS ON FOCUSING How A Lens Works The distance between the center of the lens and the focal point is called the FOCAL LENGTH. Images are only sharp where the focal plane meets the focal point. To
Optical image stabilization (IS)
Optical image stabilization (IS) CS 178, Spring 2014 Begun 4/29/14, finished 5/1. Marc Levoy Computer Science Department Stanford University Outline what are the causes of camera shake? how can you avoid
Image correspondences and structure from motion
Image correspondences and structure from motion http://graphics.cs.cmu.edu/courses/15-463 15-463, 15-663, 15-862 Computational Photography Fall 2017, Lecture 20 Course announcements Homework 5 posted.
Light Fields. Johns Hopkins Department of Computer Science Course : Rendering Techniques, Professor: Jonathan Cohen
Light Fields Light Fields By Levoy and Hanrahan, SIGGRAPH 96 Representation for sampled plenoptic function stores data about visible light at various positions and directions Created from set of images
What have we leaned so far?
What have we leaned so far? Camera structure Eye structure Project 1: High Dynamic Range Imaging What have we learned so far? Image Filtering Image Warping Camera Projection Model Project 2: Panoramic
Dense Lightfield Disparity Estimation using Total Variation Regularization
Dense Lightfield Disparity Estimation using Total Variation Regularization Nuno Barroso Monteiro 1,2, João Pedro Barreto 2, and José Gaspar 1 1 Institute for Systems and Robotics, Univ. of Lisbon, Portugal
Macro Lens, Extension Tubes, or Close-Up Filters?
Macro Lens, Extension Tubes, or Close-Up Filters? By Ron Ross IACC Macro Special Interest Group. November 13, 2017 Introduction You ve decided that you want to try macro photography. Which tool should
two using your LensbAby
two Using Your Lensbaby 28 Lensbaby Exposure and the Lensbaby When you attach your Lensbaby to your camera for the first time, there are a few settings to review so that you can start taking photos as
Another option is a clamping device that has a camera screw adapter attached to it.
Tripods and Camera Supports When beginners first start using their cameras seriously, they may tend to think of a tripod or other camera support only being necessary when the shutter speed gets to slow
Digital Imaging Study Questions Chapter 8 /100 Total Points Homework Grade
Name: Class: Date: Digital Imaging Study Questions Chapter 8 _/100 Total Points Homework Grade True/False Indicate whether the sentence or statement is true or false. 1. You can change the lens on most
Today s lecture. Image Alignment and Stitching. Readings. Motion models
Today s lecture Image Alignment and Stitching Computer Vision CSE576, Spring 2005 Richard Szeliski Image alignment and stitching motion models cylindrical and spherical warping point-based alignment global
lecture 10 - depth from blur, binocular stereo
This lecture carries forward some of the topics from early in the course, namely defocus blur and binocular disparity. The main emphasis here will be on the information these cues carry about depth, rather
Announcements. Mosaics. Image Mosaics. How to do it? Basic Procedure Take a sequence of images from the same position =
Announcements Project 2 out today panorama signup help session at end of class Today mosaic recap blending Mosaics Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html
DEPTH AND ANGULAR RESOLUTION IN PLENOPTIC CAMERAS. M. Damghanian, R. Olsson, M. Sjöström
This material is published in the open archive of Mid Sweden University DIVA http://miun.diva-portal.org to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein
SP AF 17-50mm F/2.8 XR Di-II VC LD Aspherical [IF] (Model B005)
![SP AF 17-50mm F/2.8 XR Di-II VC LD Aspherical [IF] (Model B005)](/thumbs/72/66333745.jpg "SP AF 17-50mm F/2.8 XR Di-II VC LD Aspherical [IF] (Model B005)")
SP AF 17-50mm F/2.8 XR Di-II VC LD Aspherical [IF] (Model B005) Thank you for purchasing the Tamron lens as the latest addition to your photographic equipment. Before using your new lens, please read the
Computer Vision I - Image Matching and Image Formation
Computer Vision I - Image Matching and Image Formation Carsten Rother 10/12/2014 Computer Vision I: Image Formation Process Computer Vision I: Image Formation Process 10/12/2014 2 Roadmap for next five
20mm F2 FE MF / 20mm F2 FE AF
20mm F2 FE MF / 20mm F2 FE AF FíRIN is Tokina s new Premium lens series, designed and manufactured for mirrorless cameras. FíRIN is a variation of Fírinne, the Old Irish word for truth, which was coined