CS 684 Fall 2005 Image-based Modeling and Rendering Ruigang Yang
Administrivia Classes: Monday and Wednesday, 4:00-5:15 PM Instructor: Ruigang Yang ryang@cs.uky.edu Office Hour: Robotics 514D, MW 1500-1600 Prerequisite: Graduate standing, CS 635 (or equivalent) Recommended Reference Book: Computer Vision: a modern approach by Forsyth & Ponce OpenGL Programming Guide the red book Webpage: http://vis.uky.edu/~ryang/teaching/cs684-fall05/index.htm (slides and more) Fall 2005 CS684-IBMR 2
Goals and Objectives To introduce a new interdisciplinary research topic that is both technically challenging and empirically important. To introduce the fundamental problems and main concepts and techniques to solve those. To enable participants to implement solutions for reasonably complex problems To enable participants to do research in IBMR Fall 2005 CS684-IBMR 3
Some Results from Last Year David Guinnip and Shuhua Lai (SIGGRAPH 2004 Poster, Pacific Graphics 2005) Shunnan Chen, Jesus Caban, Xinyu Huang, George V. Landon (Virtual Reality 2005, SIGGRAPH 2005 Poster) Mingxuan Sun et al. (ICCV 2005) Fall 2005 CS684-IBMR 4
Course Organization Lectures (~1.5 months) Research Reading and Presentation Research Investigation Fall 2005 CS684-IBMR 5
Grading Class participation 15% Presentation 25% Project Proposal 10% Final Project 50% No final exam Fall 2005 CS684-IBMR 6
The Evolution of Computer Graphics Sketchpad, 1963 Ivan Sutherland Fall 2005 CS684-IBMR 7
The Evolution of Computer Graphics Fall 2005 CS684-IBMR 8
Traditional Rendering 3D model Rendering New New image New image 2D image image Real-time Rendering on GeForce FX, NVIDIA 2003 Photo Realism Fall 2005 CS684-IBMR 9
Image-Based Modeling and Rendering 3D model Rendering New New image New image 2D image image Image Image Image Image Image Fall 2005 CS684-IBMR 10
Computer Vision A discipline related to Artificial Intelligence Let computer see Obtaining 3D models A Branch of CV that focuses on reconstruction of 3D model from 2D images Fall 2005 CS684-IBMR 11
Computer Graphics What is the goal of computer graphics again Given a 3D model, we ll make a 2D image Fall 2005 CS684-IBMR 12
Bridge the Gap between Computer Graphics & Computer Vision Fall 2005 CS684-IBMR 13
Image-Based Modeling and Rendering (IBMR): Advantages Photographs are easy to obtain Photographs are already photo-realistic Allow cheating to circumvent the need for a 3D model, which is hard to obtain. More predictable performance Fall 2005 CS684-IBMR 14
IBMR: A Historical Perspective Theoretical Foundations Jim Kajiya (Caltech) The Rendering Equation 1986 Edward Adelson, James Bergen The Plenoptic Function and the Elements of Early Vision, 1991 Fall 2005 CS684-IBMR 15
IBMR: A Historical Perspective Texture- and Environment Mapping (Blinn 78) With and without texture-mapping Environment Mapping Fall 2005 CS684-IBMR 16
IBMR: A Historical Perspective Image Warping and Morphing (Wolberg 90) Fall 2005 CS684-IBMR 17
IBMR: A Historical Perspective Image Mosaics (Chen 95) combination Higher resolution or lager image Fall 2005 CS684-IBMR 18
Fall 2005 CS684-IBMR 19 IBMR: A Historical Perspective Formalization by McMillan and Bishop (1995) ),,,, ( z y x V V V P p φ θ = ),,,, ( z y x V V V P p φ θ = Viewer s position Rotation and elevation of light ray
Image Mosaics Different Images combination Higher resolution or lager image Fall 2005 CS684-IBMR 20
Mosaic Image Representation Planar Image Cube Cylinder Sphere Fall 2005 CS684-IBMR 21
Quick-time VR approach Uses a cylinder mapping of the mosaic Virtual camera is at a fixed center-of-projection You can move around the mosaic by rotation and tilting up/down the virtual camera Fall 2005 CS684-IBMR 22
Image-Based Modeling Rendering Mosaics are large images with a single center of projection How about imagery from arbitrary views? Solution One: Create a 3D model! 3D model Rendering New New image New image 2D image image Image Image Image Image Image Fall 2005 CS684-IBMR 23
3D Shape Recovery Stereo Depth Map Correlation profile u 1 u 1 Left Camera Right Camera Pioneered by Marr and Poggio, 1976 Fall 2005 CS684-IBMR 24
Advantages of Stereo Well-studied, many algorithms Passive, requiring only two images Can be done in real-time Depth Map Yang & Pollefeys CVPR 2003 Fall 2005 CS684-IBMR 25
State of the art Show video (from Marc Pollefeys, 2000) Fall 2005 CS684-IBMR 26
Image-Based Modeling Rendering Stereo is not robust Any alternative for imagery from arbitrary views? Solution Two: Pre-record them all 3D model Rendering New New image New image 2D image image Image Image Image Image Image Fall 2005 CS684-IBMR 27
Image-based Modeling and Rendering (IBMR) McMillan & Bishop 1995 New Viewpoint Fall 2005 CS684-IBMR 28
Light Field Rendering Any surface, any scene Photo realism Hundreds or thousands of images Levoy & Hanrahan 1996 Fall 2005 CS684-IBMR 29
A Continuum of IBMR methods Geometry-based Image-based Image Image Image Image Image Shape Recovery 3D model Computer Graphics New 2D image New 2D image New 2D image Image Image Image Image Image Additional constraints New 2D image New 2D image New 2D image Stereo Space Carving Façade Lumigraph Light Field More geometric primitives More image samples Fall 2005 CS684-IBMR 30
FAÇADE: Fall 2005 CS684-IBMR 31
FAÇADE: Video Fall 2005 CS684-IBMR 32
View Morphing Mona Lisa Stereo Pair (cross your eyes and fuse them!) Steve Seitz, Chuck Dyer 1996 Show Video Fall 2005 CS684-IBMR 33
Other Issues in IBMR: Lighting Ravi Ramamoorthi, 2002 Yizhou Yu, 1998 Fall 2005 CS684-IBMR 34
Other Issues in IBMR : Properties of Surfaces ω i ωr i Diffuse Reflection ω Mirror Reflection ωr ω i i Specular Reflection D. McAllister, 2003 ωr ω ωr Retroreflection Fall 2005 CS684-IBMR 35
Other Issues in IBMR : Properties of Surfaces Measurement Fall 2005 CS684-IBMR 36
Course Lecture Outline Introduction to 3D Reconstruction Camera basics Passive algorithms Stereo, Space Carving, Shape from X Active techniques Image-based Rendering Image Warping Plenoptic function Lighting Surface Reflectance Properties Capturing and Synthesizing Reality (smoke, water,etc) Fall 2005 CS684-IBMR 37