Image-Based Modeling and Rendering

Image-Based Modeling and Rendering Richard Szeliski Microsoft Research IPAM Graduate Summer School: Computer Vision July 26, 2013

How far have we come?

Light Fields / Lumigraph - 1996 Richard Szeliski Image-Based Rendering and Modeling 3

Environment matting - 2001 Richard Szeliski Image-Based Rendering and Modeling 4

Photo Tourism - 2006 Richard Szeliski Image-Based Rendering and Modeling 5

Ambient Point Clouds - 2010 Richard Szeliski Image-Based Rendering and Modeling 6

Photo Tours - 2012 Richard Szeliski Image-Based Rendering and Modeling 7 [Kushal et al., 3DIMPVT 2012]

The Visual Turing Test Video Richard Szeliski Image-Based Rendering and Modeling 8 [Shan et al., 3DV 2013]

Where are we going?

A personal retrospective Panoramas and 360 Video Walkthroughs Light Fields and Lumigraphs LDIs, Sprites, and Layered Video Image-Based Modeling Environment Mattes and Matting Photo Tourism and Photosynth Point-Based Rendering and NPR Reflections and Transparency [2012] Richard Szeliski Image-Based Rendering and Modeling 10

Pre-history (?): view interpolation Depth maps and images: view extrapolation [Matthies,Szeliski,Kanade 89] input depth image novel view View interpolation: warp and dissolve [Beier & Neely 92; Chen & Williams 93] Richard Szeliski Image-Based Rendering and Modeling 11

Panoramic image stitching Convert image sequence into a cylindrical image + + + = Richard Szeliski Image-Based Rendering and Modeling 12

Panoramic image stitching Key technical breakthroughs: interactive viewing [(Lippman 80), Chen 95] general camera motions [1997 ] parallax compensation [1998 ] fully automated feature detection and matching [Brown & Lowe 03] seam selection [1975 (1998), 2001 ] Laplacian and Poisson blending [1984, 2003] Early example of computational photography Richard Szeliski Image-Based Rendering and Modeling 13

VR today: Gigapan Richard Szeliski Image-Based Rendering and Modeling 14

VR today: 360 cities Richard Szeliski Image-Based Rendering and Modeling 15

VR today: Photosynth Richard Szeliski Image-Based Rendering and Modeling 16

Moving beyond single (or linked) panoramas

Panoramic Video-Based Tours [Uyttendaele et al. 2004] 2D 3D Light Field (?)

Video-Based Walkthroughs Move camera along a rail ( dolly track ) and play back a 360 video Applications: Homes and architecture Outdoor locations (tourist destinations) Richard Szeliski Image-Based Rendering and Modeling 19

Surround video acquisition system OmniCam (six-camera head) [ Point Grey Ladybug ] Richard Szeliski Image-Based Rendering and Modeling 20

Acquisition platforms (then) Robotic cart Wearable Richard Szeliski Image-Based Rendering and Modeling 22

Acquisition platforms (today) Richard Szeliski Image-Based Rendering and Modeling 23

Acquisition platforms (future?) Richard Szeliski Image-Based Rendering and Modeling 24

Acquisition platforms (now) 10 camera panoramic rig One bubble every 2 meters Reproject into cube maps Richard Szeliski Image-Based Rendering and Modeling 25

Lightfields and Lumigraphs True 4D (with lots of slides from Michael Cohen)

Modeling light How do we generate new scenes and animations from existing ones? Classic 3D Vision + Graphics : take (lots of) pictures recover camera pose build 3D model extract texture maps / BRDFs synthesize new views tons of great work and demos as 3DIMPVT Richard Szeliski Image-Based Rendering and Modeling 27

Computer Vision Output Model Real Scene Real Cameras Richard Szeliski Image-Based Rendering and Modeling 28

Computer Graphics Output Image Synthetic Camera Model Richard Szeliski Image-Based Rendering and Modeling 29

Combined Output Image Synthetic Camera Model Real Cameras Real Scene Richard Szeliski Image-Based Rendering and Modeling 30

But, vision technology falls short Output Image Synthetic Camera Model Real Cameras Real Scene Richard Szeliski Image-Based Rendering and Modeling 31

and so does graphics. Output Image Synthetic Camera Model Real Cameras Real Scene Richard Szeliski Image-Based Rendering and Modeling 32

Image Based Rendering Output Image Synthetic Real Scene Images+Model Camera Real Cameras -or- Expensive Image Synthesis Richard Szeliski Image-Based Rendering and Modeling 33

Lumigraph / Lightfield Outside convex space Empty Stuff 4D Richard Szeliski Image-Based Rendering and Modeling 34

Lumigraph - Organization 2D position 2D position s u 2 plane parameterization Richard Szeliski Image-Based Rendering and Modeling 35

Lumigraph - Organization 2D position 2D position t s,t s,t u,v v u,v 2 plane parameterization Richard Szeliski Image-Based Rendering and Modeling 36 s u

Lumigraph - Rendering For each output pixel determine s,t,u,v either find closest discrete RGB interpolate near values s,t u,v Richard Szeliski Image-Based Rendering and Modeling 37

Lumigraph - Rendering For each output pixel determine s,t,u,v either use closest discrete RGB interpolate near values Richard Szeliski Image-Based Rendering and Modeling 38 s u

Lumigraph - Rendering Nearest closest s closest u draw it Blend 16 nearest quadrilinear interpolation u Richard Szeliski Image-Based Rendering and Modeling 39 s

Lumigraph - Rendering Depth Correction quadralinear interpolation new closest like focus u Richard Szeliski Image-Based Rendering and Modeling 40 s

Lumigraph Image Effects Image effects: parallax occlusion transparency highlights Richard Szeliski Image-Based Rendering and Modeling 43

Unstructured Lumigraph What if the images aren t sampled on a regular 2D grid? can still re-sample rays ray weighting becomes more complex [Buehler et al., SIGGRAPH 2000] Richard Szeliski Image-Based Rendering and Modeling 45

Surface Light Fields [Wood et al, SIGGRAPH 2000] Turn 4D parameterization around: image @ every surface pt. Leverage coherence: compress radiance fn (BRDF * illumination) after rotation by n Richard Szeliski Image-Based Rendering and Modeling 46

Surface Light Fields [Wood et al, SIGGRAPH 2000] Richard Szeliski Image-Based Rendering and Modeling 47

2.5D Representations Images (and panoramas) are 2D Lumigraph is 4D 3D 3D Lumigraph subsets (360 video tours) Concentric mosaics 2.5D Layered Depth Images Sprites with Depth (impostors) Layered (Virtual Viewpoint) Video View Dependent Surfaces (see Façade) Richard Szeliski Image-Based Rendering and Modeling 49

Layered Depth Image 2.5 D? Layered Depth Image Richard Szeliski Image-Based Rendering and Modeling 50

Layered Depth Image Rendering from LDI [Shade et al., SIGGRAPH 98] Incremental in LDI X and Y Guaranteed to be in back-to-front order Richard Szeliski Image-Based Rendering and Modeling 51

Sprites with Depth Represent scene as collection of cutouts with depth (planes + parallax) Render back to front with fwd/inverse warping [Shade et al., SIGGRAPH 98] Basis of Virtual Viewpoint Video [Zitnick et al. 2004] Richard Szeliski Image-Based Rendering and Modeling 53

Virtual Viewpoint (3D) Video [Zitnick et al., SIGGRAPH 2004]

Scenarios for 3D video Sports events, e.g., CMU s 30-camera EyeVision system at SuperBowl XXXV) Concert performances, plays, circus acts Games Instructional video, e.g., golf, skating, martial arts Interactive (Internet) video Richard Szeliski Image-Based Rendering and Modeling 55

3D video: Challenges Capture: record multiple synchronized camera streams Processing and representation: ensure photorealism Compression: exploit redundancy Rendering: provide interactive viewpoint control Richard Szeliski Image-Based Rendering and Modeling 56

cameras concentrators hard disks controlling laptop Richard Szeliski Image-Based Rendering and Modeling 58

Matching pixels Easy Richard Szeliski Image-Based Rendering and Modeling 59

Harder Richard Szeliski Image-Based Rendering and Modeling 60

Really Really Hard Occlusion Richard Szeliski Image-Based Rendering and Modeling 61

Matting Some pixels get influence for multiple surfaces. Background Surface Foreground Surface Close up of real image: Image Camera Multiple colors and depths at boundary pixels Richard Szeliski Image-Based Rendering and Modeling 62

Find matting information: 1. Find boundary strips using depth. 2. Within boundary strips compute the colors and depths of the foreground and background object. Background Foreground Strip Width Richard Szeliski Image-Based Rendering and Modeling 63

Why matting is important No Matting Matting Richard Szeliski Image-Based Rendering and Modeling 64

Representation Main Boundary Main Layer: Color Depth Strip Width Boundary Layer: Color Alpha Depth Richard Szeliski Image-Based Rendering and Modeling 65

Richard Szeliski Image-Based Rendering and Modeling 66

Outline Panoramas and 360 Video Walkthroughs Light Fields and Lumigraphs LDIs, Sprites, and Layered Video Image-Based Modeling Environment Mattes and Matting Photo Tourism and Photosynth Point-Based Rendering and NPR Reflections and Transparency [2012] Richard Szeliski Image-Based Rendering and Modeling 67

Image-Based Modeling

Image-Based Modeling Richard Szeliski Image-Based Rendering and Modeling 69

Façade 1. Select building blocks 2. Align them in each image 3. Solve for camera pose and block parameters (using constraints) Richard Szeliski Image-Based Rendering and Modeling 71

View-dependent texture mapping 1. Determine visible cameras for each surface element 2. Blend textures (images) depending on distance between original camera and novel viewpoint Richard Szeliski Image-Based Rendering and Modeling 72

Graz reconstruction Fully automated multi-view stereo, ca. 2009 Richard Szeliski Image-Based Rendering and Modeling 74

Non-photorealistic experiences

Photo Tourism Images on the Internet Computed 3D structure [Snavely, Seitz, Szeliski, SIGGRAPH 2006] Richard Szeliski Image-Based Rendering and Modeling 76

Photo Tourism Richard Szeliski Image-Based Rendering and Modeling 77

Internet Computer Vision Use the Internet as an image and/or annotation source to solve challenging vision problems. Richard Szeliski Image-Based Rendering and Modeling 78

Internet Computer Vision Richard Szeliski Image-Based Rendering and Modeling 79

Internet Computer Vision Richard Szeliski Image-Based Rendering and Modeling 80

Recent research

Piecewise planar proxies [Sinha, Steedly, Szeliski ICCV 09] Richard Szeliski Image-Based Rendering and Modeling 82

Building Interiors [ Furukawa et al., ICCV 09] Richard Szeliski Image-Based Rendering and Modeling 84

Internet-Scale Stereo [ Furukawa et al., CVPR 10] Richard Szeliski Image-Based Rendering and Modeling 85

What s missing? Reflections Gloss? Richard Szeliski Image-Based Rendering and Modeling 86

Reflections Richard Szeliski Image-Based Rendering and Modeling 87

Image-Based Rendering for Scenes with Reflections Sudipta N. Sinha Johannes Kopf Michael Goesele Daniel Scharstein Richard Szeliski

Wrapping up

Graphics/Imaging Continuum Concentric mosaics Geometry centric Image centric Fixed geometry Viewdependent texture Viewdependent geometry Sprites with depth LDI Lumigraph Light field Polygon rendering + texture mapping Warping Interpolation Richard Szeliski Image-Based Rendering and Modeling 90

What works? What doesn t? Automatic 3D pose estimation Aerial (+ active ground level) modeling Accurate boundaries & matting Reflections and transparency User-generated content Casually acquired photos and videos New sensors Active sensing (Kinect ); Plenoptic cameras Integration with recognition Richard Szeliski Image-Based Rendering and Modeling 91

What about Computational Photography? Richard Szeliski Image-Based Rendering and Modeling 92

Image-Based Rendering Panoramas and 360 Video Walkthroughs Light Fields and Lumigraphs Layered Representations and Matting Image-Based Modeling Environment Mattes and Matting Photo Tourism and Photosynth Point-Based Rendering and NPR Reflections and Transparency Where next? How to use in your applications? Richard Szeliski Image-Based Rendering and Modeling 93

Graphics/Imaging Continuum Concentric mosaics Geometry centric Image centric Fixed geometry Viewdependent texture Viewdependent geometry Sprites with depth LDI Lumigraph Light field Polygon rendering + texture mapping Warping Interpolation Richard Szeliski Image-Based Rendering and Modeling 94

( Questions ) [ The End ]