Computer Graphics -based rendering Output Michael F. Cohen Microsoft Research Synthetic Camera Model Computer Vision Combined Output Output Model Real Scene Synthetic Camera Model Real Cameras Real Scene Real Cameras 1
But, vision technology falls short Output Output and so does graphics. Synthetic Camera Model Real Cameras Real Scene Synthetic Camera Model Real Cameras Real Scene Based Rendering Ray Output Constant radiance time is fixed Synthetic Camera Real Scene s+model Real Cameras -or- Expensive Synthesis 5D 3D position 2D direction 2
All Rays Line Plenoptic Function all possible images too much stuff! Infinite line 4D 2D direction 2D position Ray Discretize What is an image? Distance between 2 rays Which is closer together? All rays through a point Panorama? 3
plane 2D position of rays has been fixed direction remains 2D position Object plane Light leaving towards eye 2D position 2D just dual of image 4
Object Object All light leaving object 4D 2D position 2D direction Object Lumigraph All images How to organize capture render 5
Lumigraph - Organization Lumigraph - Organization 2D position 2D direction q s 2D position 2D position s u 2 plane parameterization Lumigraph - Organization Lumigraph - Organization 2D position 2D position t v Hold constant Let vary An image 2 plane parameterization s u 6
Lumigraph - Organization Discretization higher res near object if diffuse captures texture lower res away captures directions Lumigraph - Capture Idea 1 Move camera carefully over plane Gantry see Lightfield paper Lumigraph - Capture Lumigraph - Rendering Idea 2 Move camera anywhere Rebinning see Lumigraph paper For each output pixel determine, either find closest discrete RGB interpolate near values 7
Lumigraph - Rendering For each output pixel determine, Lumigraph - Rendering Nearest closest s closest u draw it either use closest discrete RGB interpolate near values s u Blend 16 nearest quadrilinear interpolation s u High-Quality Video View Interpolation Using a Layered Representation Larry Zitnick Sing Bing Kang Matt Uyttendaele Simon Winder Rick Szeliski Interactive Visual Media Group Microsoft Research Current practice free viewpoint video Many cameras vs. Motion Jitter 8
Current practice free viewpoint video Video view interpolation Many cameras vs. Motion Jitter Fewer cameras and Smooth Motion Automatic Real-time rendering Prior work: IBR (static) Prior work: IBR (dynamic) Plenoptic Modeling McMillan & Bishop, SIGGRAPH 95 Light Field Rendering Levoy & Hanrahan, SIGGRAPH 96 Stanford Multi-Camera Array Project Virtualized Reality TM Kanade et al., IEEE Multimedia 97 Dynamic Light Fields Goldlucke et al., VMV 02 The Lumigraph Gortler et al., SIGGRAPH 96 Concentric Mosaics Shum & He, SIGGRAPH 99 -Based Visual Hulls Matusik et al., SIGGRAPH 00 Free-viewpoint Video of Humans Carranza et al., SIGGRAPH 03 3D TV Matusik & Pfister, SIGGRAPH 04 9
System overview cameras OFFLINE Video Capture concentrators hard disks controlling laptop Stereo Representation Compression File ONLINE Selective Decompression Render Calibration Input videos Zhengyou Zhang, 2000 10
Key to view interpolation: Geometry correspondence 1 2 Stereo Geometry 1 2 Leg Correct Good Incorrect Camera 1 Camera 2 Virtual Camera Wall Bad Match Score Match Score Local matching 1 2 Global regularization A Create MRF (Markov Random Field): 1 2 Low texture B C A E D F P Q A S U R T color A color B z A z B Each segment is a node Number z A z P of, zstates Q, z S = number of depth levels 11
Iteratively solve MRF Depth through time Matting Background Interpolated view without Surface matting Rendering with matting Foreground Surface No Matting Matting Background Strip Width Background Alpha Foreground Foreground Bayesian Matting Chuang et al. 2001 Camera 12
Representation Main Background Boundary Boundary Layer: Strip Width Foreground Main Layer: Massive Arabesque videoclip Color Color Alpha Depth Depth 13