Image-Based Modeling and Rendering Image-Based Modeling and Rendering MIT EECS 6.837 Frédo Durand and Seth Teller 1 Some slides courtesy of Leonard McMillan, Wojciech Matusik, Byong Mok Oh, Max Chen 2 Final projects Final report due tomorrow Friday at 5pm Next Tuesday: Best project presentation in class IBMR Image-Based Modeling and Rendering Reuse images to generate new views Use images to create 3D models New area of computer graphics (mid nineties) Lot of research at MIT 3 4 What we have learnt so far Graphics is about producing images from a scene description Geometry + reflectance property + light From continuous description to a discrete image What IBMR is about Focus on discrete descriptions Forget (mostly) about geometry IBMR is about samples cf. lecture on sampling and antialiasing 5 6
Motivations Geometry is costly Triangles become tiny Triangles are not good for complex objects, e.g. plants Photos are photorealistic Well, we could argue about that another time Modeling is tedious IBR Principles Consider images as a collection of rays, rather than a collection of pixels. 7 8 The Plenoptic Function Complete view for each point of space 3D (for position) * 2D (for view) *1D for wavelength * 1D for time Image-based rendering is about signal reconstruction rather than physical simulation. 9 5-D for static scenes in general 4-D in empty regions of static scenes 6-D for dynamic scenes 10 Geometry vs. sample Geometry exploits the structure of the scene Compression (information theory point of view) IBMR is brute force Closer to final images Simpler to render More costly to store Harder to manipulate, create, edit To decrease dimensionality Add geometry for structure Forget some visual phenomena We already know some IBR Texture mapping! 11 12
Overview Questions? Image-based rendering Various dimensionalities Various structure Image-based Modeling Image-Based Modeling and Editing 13 14 QuickTime VR & Panoramas 2D reconstruction only: 1 single complete view Video 3D slice See http://www.panoguide.com/gallery/ 15 16 Light field and Lumigraph 4D representation Plane of cameras 2-plane parameterization Lines in space have 4D 17 From [Gortler et al 96] 18
2-plane parameterization Sample this 4D space of light rays What do we miss in the plenoptic function? Light field reconstruction Compute an image similar to ray casting Simple lookup Can be optimized 19 20 Light field Light-field/Lumigraph slice Collection of rays Collection of images Dynamic Reparameterization Demo 21 22 Structure Structure Which ray is best? Which ray is best? s,t u,v s,t u,v New ray New ray 23 Geometry matters! 24
Depth correction: Lumigraph Without depth correction With depth correction From [Gortler et al. 96] 25 Light Field Acquisition Motion Platforms Precise positioning Calibrated digital camera Expensive (> $10K) Very Slow (~20 mins) Light field cameras Less precise Calibration per aperture Inexpensive (~ $100) Slow (> 3 mins) 26 LF Rendering Questions? Advantages Simple/Fast rendering algorithm View-dependent shading High-quality reconstructions Disadvantages Huge memory footprint (compression) Incompatible with traditional graphics 27 28 Images with Depth [Chen & Williams 1993, McMillan & Bishop 1995] 2D representation (image) + depth per pixel View warping to reproject pixels Tradeoff between dimensionality and geometric information View warping [Chen & Williams 1993, McMillan & Bishop 1995] Reprojects pixels with depth 2 Types of problems Disocclusion Undersampling Undersampling can be alleviated using splatting Instead of displaying a pixel, display a splat (ellipse) Color Depth 29 30
Layered Depth Images (LDI) Same as Images with depth, but multiple samples in depth for one (x,y) pixel Solves many disocclusion artifacts Acquiring Depth Images Economical laser scanners Demo 31 Images and video courtesy of the University of North Carolina at Chapel Hill, Office of the Future Project, Wei-Chao Chen, Henry Fuchs, Lars Nyland, Herman Towles and Greg Welch. 32 Visual Hulls What is a Visual Hull? Intersection of all silhouette cones 33 34 Why Use a Visual Hull? Finding silhouettes is simple and robust Blue-screen methods Image differenceing Contains actual object Can seed more sophisticated methods Acquisition Several cameras with overlapping views Geometric calibration Photometric calibration Synchronization background + foreground background - = foreground 35 36
Representing a Visual Hull Geometric Model + Know how to render Calculating intersections isn t robust High polygonal complexity Volumetric Model + Easy to compute Expensive storage Result seldom predicts inputs Image-based Visual Hulls Volume-like Self-consistent Discretediscretecontinuous 37 38 Shading Visual Hulls Results View-dependent illumination Visibility 39 A range of different virtual viewpoints of a visual hull computed from four cameras in real-time. Top images show depth maps and bottom images show shaded visual hulls. The background is a textured polygonal model. 40 Results Video Summary Plenoptic function is mostly 5D Images and panoramas are 2D Video is 3D Light fields are 4D Structure can help Images with depth Visual hull 41 42
Questions? Overview Image-based rendering Various dimensionalities Various structure Image-based Modeling Image-Based Modeling and Editing 43 44 Image-based Modeling From a set of 2D images Build a 3D polygonal model Using vision techniques & user intervention Polygonal Representation Modeling and Rendering Architecture from Photographs [Debevec et al. 1996] From [Debevec et al. 1996] 45 From [Debevec et al. 1996] 46 Image-based modeling Video Write down the equations Optimize Camera coordinate system Edge location From [Debevec et al. 1996] 47 48
Questions? Overview Image-based rendering Various dimensionalities Various structure Image-based Modeling Image-Based Modeling and Editing Graphics Durand 49 50 Image-based modeling & editing Image-Based Representation Images with depth per pixel [Chen 93] Layers Joint work with Oh, Chen & Dorsey Create an Image-Based representation Editing of sampled representation Our context: Single image as input Color Input image New viewpoint Depth Layers Relighting 51 Workflow 52 Depth Assignment Tool Tool that assigns or modifies the depth of pixels Similar to tools of 2D photo editing But on the depth channel Apply depth Input image Segment Edit, relight MIT EECS 6.837, Teller and Durand Clone brush holes 53 Color channel Depth channel Side view 54 9
Depth Assignment and Selection Going Beyond Painting Arbitrary selection/segmentation restricts the affected pixels Painting absolute depth is hard Hybrid geometric tools But still pixel based (flexible, use of selection) Geometry is temporary Selection area Color channel Depth channel Side view Initial image (side view) Correlate Update Read &6.837, align depth z-buffer 3Dand data geometry MIT EECS Teller Durand 55 56 Ground Plane Tool Vertical Tool The ground plane is easy to infer (horizon) Will be used as a reference Uses ground plane as reference Draw contact between ground and object in image vertical shape user input user input user input Reference view Depth channel ground plane Side view user input Reference view Depth channel 57 Other Geometric Primitives 58 Organic Shapes Sphere, cylinder, box, pyramid, etc. Possible snapping to constrain verticality Level set method [Wil98,IMT99] Distant depth at boundary, closer depth towards center Layer Side view 59 Depth channel 60 10
Generic Geometry Tool 3D template User defined point correspondences 3D pose optimization Refinement through 2D morphing Depth Painting & Chiseling Paint on depth channel [Kan98] Relative or absolute Local smoothing, sharpening User defined point correspondences Optimized pose Side view 61 Layer Depth channel 62 Refined Example Clone brushing Apply depth Input image Segment Clone brush holes Coarse depth Refined depth 63 MIT EECS 6.837, Teller and Edit, Durand relight 64 2D Clone Brush 2D Clone Brush Copy via brush interface Copy via brush interface source pixel destination pixel source pixel destination pixel 65 66
2D Clone Brush 2D Clone Brush Copy via brush interface Copy via brush interface source pixel destination pixel source pixel destination pixel 67 68 2D Clone Brush Copy via brush interface Limitations of 2D Clone Brushing Distortions due to foreshortening and surface orientation source pixel destination pixel 69 70 Coping with Distortions Examples We have depth information Minimize distortion v Non-distorted parameterization Iterative optimization (u,v) u v (u,v) u 71 72
Results Hotel Lobby Results Dali 73 74 Tuesday Best project presentations Images vs. geometry Photos Film Texture mapping Environment mapping 75 76