Traditional Computer Graphics Image-Based Modeling and Rendering Thomas Funkhouser Princeton University COS 426 Guest Lecture Spring 2003 How would you model and render this scene? (Jensen) How about this one? (Jensen).. and this one? (Kim ) What about this one? (Jensen)
How about this one? (Louvre) Geometric Modeling When doesn t this pipeline work? (M cm illan) Reflectance Modeling It is hard to create meshes for complex objects... It is hard to acquire good BRDFs for complex surfaces... (Levoy) Light Transport Simulation (Square) What Else Can We Do? It is hard to compute all light paths for complex illumination... (RenderPark) (M cm illan) 2
Image-Based Rendering (IBR) Model scene as set of reference images Render novel views by resampling pixels IBR Rendering Pipeline IBR Model Geometric Model Novel view Reference images Plenoptic function (): o Describes the radiance traveling along a ray - to/from any point (x, y, z), - in any direction (φ, θ), - at any frequency (λ), - at any time (t) (φ,θ) (x,y,z) Consider only directions from viewpoint Image Panoramas Consider only directions from viewpoint Cylindrical Panorama (Quicktim evr)
Image Panoramas Image Panoramas Kang 99 Virtual tours of Princeton (Brow n) Texture Mapping Map photographs onto surfaces Consider only positions on surfaces Photographs Geometry + Texture Maps Texture Mapping Consider only directions or positions on surfaces (D ebevec)
Create novel images by resampling photographs o Reference images sample plenoptic function Method: o Warp nearby reference images to novel viewpoint o Blend warped images Derived Frame Reference Frame Reference Frame R1 N R2 This is just a morph where the warp is defined by pixel correspondences! How define warp for one view to another? o Use depth at pixel to project into scene, or... How define warp for one view to another? o Use depth at pixel to project into scene, or... R1 N R2 How define warp for one view to another? o Use depth at pixel to project into scene, or... o Use pixel correspondences How define warp for one view to another? o Use depth at pixel to project into scene, or... o Use pixel correspondences
Finding pixel correspondences o Coarse model o Sparse image features o Depth at every pixel Disparity Problems with view interpolation: o Changes in visibility o Disocclusions Left Right (Szeliski) Disocclusions Partial solutions: o Fill holes by interpolating nearby pixels Disocclusions Partial solutions: o Fill holes by interpolating nearby pixels o Use more photographs Lumigraph (Gortler) Light Field / Lumigraph If observer stays in free space, plenoptic function reduces to o Exterior of the convex hull of an object o Interior of an environment Consider only directions or positions on surfaces F(r, α, φ, θ)
Representing a Light Field Two-plane parameterization () Representing a Light Field Two Interpretations of a Light Field Creating a Light Field Capturing a Light Field Capturing a Light Field (Stanford University) (BennettW ilburn,m ichalsm ulski,m ark Horowitz)
Capturing a Light Field Capturing a Light Field (BennettW ilburn,m ichalsm ulski,m ark Horowitz) (BennettW ilburn,m ichalsm ulski,m ark Horowitz) Rendering a Light Field Resampling problem o Interpolation o Avoid aliasing Rendering a Light Field Video (Gortler96) (Levoy & H anrahan) Rendering a Light Field Other IBR Representations Demo Consider only directions or positions on surfaces (Levoy & H anrahan)
Sea of Images Dense sampling with hemispherical camera moving in environment on eye-height plane Robotic Capture Device Captured viewpoints Walkthrough viewpoints (with Daniel Aliaga) IBR Trade-offs Advantages o Photorealistic - by definition o Do not have to create detailed model o Do not have to do lighting simulation o Performance independent of scene Disadvantages o Real-world scenes only o Difficult for dynamic scenes o Difficult to change rendering parameters o Difficult for scenes with specularities, etc. o Limited range of viewpoints o Limited resolution IBR Applications Historical Site Preservation Remote Education Virtual Tourism Future Image-Based Rendering Analyze & Reproject Simulate Frank Lloyd Wright Fallingwater House, PA Thomas Jefferson Monticello, VA Inside Independence Hall, Philadelphia, PA Computer Vision Computer Graphics Layered Depth Images Multiple samples per pixel at different depths (Shade98)