IMAGE-BASED MODELING & RENDERING (1)

Transcription

1 CS580: Computer Graphics KAIST School of CompuDng IMAGE-BASED MODELING & RENDERING (1) Some next lecture slides are adopted from slides of Prof. Ravi Ramamoorthi (USCD), Marc Levoy (Google), Michael Cohen (Facebook), Leonard McMillan (UNC) and Richard Szeliski (Facebook). 2 1

2 Elements of Computer Graphics Geometry Material LighDng Photography Rendering 3 Modeling light How do we generate new scenes and animadons from exisdng ones? Classic 3D Vision + Graphics : take (lots of) pictures recover camera pose build 3D model extract texture maps / BRDFs synthesize new views 4 2

3 Computer Graphics Output Image Synthetic Camera Model slides from Michael Cohen 5 Computer Vision Output Model Real Scene slides from Michael Cohen Real Cameras 6 3

4 Visual CompuDng (Combined) Output Image Synthetic Camera Model Real Cameras Real Scene slides from Michael Cohen 7 But, vision technology falls short Output Image Synthetic Camera Model Real Cameras Real Scene slides from Michael Cohen 8 4

5 and so does graphics. Output Image Synthetic Camera Model Real Cameras Real Scene slides from Michael Cohen 9 Image-Based Rendering Output Image Synthetic Camera Images+Model Real Scene Real Cameras -or- Expensive Image Synthesis slides from Michael Cohen 10 5

6 TradiDonal Modeling and Rendering 11 Next few slides courtesy Paul Debevec; SIGGRAPH 99 course notes TradiDonal Modeling and Rendering 12 6

7 Image-based modeling & rendering Patrick Paczkowski,, Yann Morvan, Julie Dorsey, Holly Rushmeier, and Carol O'Sullivan Insitu: sketching architectural designs in context. In Proceedings of the 2011 SIGGRAPH Asia Conference (SA '11). ACM, New York, NY, USA,, ArDcle 182, 10 pages. 13 Image-Based Modeling and Rendering 14 7

8 IBM / IBR Image-based modeling (IBM) & Image-Based Rendering (IBR) The study of image-based modeling and rendering is the study of sampled representacons of geometry. 15 Geometry-based vs. image-based rendering conceptual world model construcdon geometry geometry-based rendering rendering computer vision real world image acquisidon images image-based rendering flythrough of scene flythrough of scene 16 8

9 Shortcujng the vision/graphics pipeline real world vision pipeline geometry image-based rendering graphics pipeline views (from M. Cohen) 17 Game1: Increase the dimensionality 18 9

10 Game2: Replace the Quality Represented 19 Image-based representadons: the classics more geometry less geometry 3D model + texture/reflectance map [Blinn78] model + displacement map [Cook84] volume rendering [Levoy87, Drebin88] 2D + Z range images [Binford73] disparity maps [vision literature] 2.5D sprites [vis-sim, games] n 2D 2D epipolar plane images [Bolles87] movie maps [Lippman78] environment maps, a.k.a. panoramas [19th century] 20 10

11 Recent addidons more geometry less geometry full model view-dependent textures [Debevec96] surface light fields [Wood00] Lumigraphs [Gortler96] sets of range images view interpoladon [Chen93, McMillan95, Mark97] layered depth images [Shade98] relief textures [Oliveira00] feature correspondences plenopdc edidng [Seitz98, Dorsey01] camera pose image caching [Schaufler96, Shade96] sprites + warps [Lengyel97] light fields [Levoy96] no model outward-looking KAIST CS580 Computer QTVR Graphics [Chen95] 21 Spectrum of IBMR 22 11

12 IBR: Pros and Cons Advantages Easy to capture images: photorealisdc by definidon Simple, universal representadon Oren bypass geometry esdmadon? Independent of scene complexity? Disadvantages WYSIWYG but also WYSIAYG Explosion of data as flexibility increased Oren discards intrinsic structure of model? Today, IBR-type methods also oren used in synthedc rendering (e.g. real-dme rendering PRT) General concept of data-driven graphics, appearance Also, data-driven geometry, animadon, simuladon Spawned light field cameras for image capture 23 Image-Based Models: What do they allow? 24 12

13 IBR: A brief history Texture maps, bump maps, environment maps [70s] Poggio MIT 90s: Faces, image-based analysis/synthesis Mid-Late 90s Chen and Williams 93, View InterpolaDon [Images+depth] Chen 95 QuickDme VR [Images from many viewpoints] McMillan and Bishop 95 PlenopDc Modeling [Images w disparity] Gortler et al, Levoy and Hanrahan 96 Light Fields [4D] Shade et al. 98 Layered Depth Images [2.5D] Debevec et al. 00 Reflectance Field [4D] Inverse rendering (Marschner,Sato,Yu,Boivin, ) Today: IBR hasn t replaced convendonal rendering, but has brought sampled and data-driven representadons to graphics 25 FOUNDATIONS FOR IBMR Some next lecture slides are adopted from slides of Prof. Ravi Ramamoorthi (USCD), Marc Levoy (Google), Michael Cohen (Facebook), Leonard McMillan (UNC) and Richard Szeliski (Facebook)

14 Images as a CollecDon of Rays 27 Warping slides courtesy Leonard McMillan, SIGGRAPH 99 course notes The PlenopDc FuncDon 28 14

15 Image-based Rendering is about 29 Where to Begin? 30 15

16 Mapping from Rays to Points 31 Correspondence 32 16

17 Warping in AcDon 33 Visibility 34 17

18 ParDDon Reference Image 35 EnumeraDon 36 18

19 ReconstrucDon 37 IMAGE-BASED MODELING Some next lecture slides are adopted from slides of Prof. Ravi Ramamoorthi (USCD), Marc Levoy (Google), Michael Cohen (Facebook), Leonard McMillan (UNC), Richard Szeliski (Facebook), and Giljoo Nam (KAIST)

20 3D Imaging passive shape from stereo shape from focus shape from modon, etc. acdve texture-assisted shape-from-x trianguladon using structured-light Dme-of-flight 39 A General Pipeline for 3D Imaging Pre-processing Camera/Projector Model ParLal Geometry TriangulaDon Camera/Projector CalibraDon Structured Light Imaging Geometry Enhancement Photometric Stereo Complete 3D Geometry RBT + ICP OpDmizing Geometry Surface ReconstrucDon 40 20

21 Line Closest Point p projector Line n Line m camera projector coordinates camera coordinates This method can reconstruct the 3D point, but it is not robust to noise. More pracdcal approach usually exploits line-plane intersecdon point 41 Line-Plane IntersecDon p projector Line m camera projector coordinates camera coordinates Except for parallel situadon, they always intersect at a point Line : a ray from camera Plane : a plane from projector 42 21

22 Line-Plane IntersecDon light plane t P= { p: n ( p q P ) = 0} projector p camera ray L= { p= q +λv} C camera projector coordinates v camera coordinates q P n q C t t n ( p q ) = n ( λv+ q q ) = 0 t n ( qp qc) λ = t nv p = q + λv C P C P 43 Line-Plane IntersecDon p projector Line m camera projector coordinates camera coordinates Plane sweep We sweep the plane and capture each plane with the camera Laser 3D scanners 44 22

23 Camera-Projector CalibraDon Results Y c (mm) Z c1 O Z c1 p X O c1 p Y p Y c X c (mm) X p Z c2 O c Y c2 X c Z c (mm) Moreno, D., & Taubin, G. (2012, October). Simple, accurate, and robust projector-camera calibralon. In 3D Min Imaging, H. Kim Modeling, Processing, VisualizaDon and Transmission (3DIMPVT), 2012 Second InternaDonal Conference on (pp ). IEEE. 45 Structured LighDng: Encoding & Decoding Scheme p projector Line m camera projector coordinates camera coordinates Encoding on a projector Encoding the corresponding points of a projector Decoding on a camera Decoding the corresponding points of a camera 46 23

24 Structured LighDng: Encoding Paverns N 47 Structured LighDng: Decoding Paverns N color coding Pavern decoded 48 24

25 Structured LighDng per View 49 EsDmaDng Normals: Photometric Stereo [Farooq et el. 2008] First proposed by [Woodham 1980] Use at least 3 light sources and one camera Assume diffuse reflecdon Woodham, R. J. (1980). Photometric method for determining surface orientalon from mullple images. OpCcal engineering, 19(1), Farooq, A. R., Smith, M. L., Smith, L. N., & Midha, S. (2005). Dynamic photometric stereo for on line quality KAIST control CS580 of ceramic Computer Lles. Computers Graphics in industry, 56(8),

26 Enhance Geometry using Normals DirecDonal light Point light source Radial intensity distribudon Surface Surface Surface Where does the low frequency bias come from? DirecDonal light assumpdon 51 Enhance Geometry using Normals Where does the low frequency bias come from? DirecDonal light assumpdon PerspecDve camera projecdon Etc

27 Diffuse ReflecDon I = ρ(!"! N!" L ) A diffuse surface!"! N!" L The apparent brightness of a diffuse surface is the same regardless of the observer's I : observed view intensity ρ : diffuse reflectance direcdon.!"! N :surface normal (unit vector)!" L : light direction (unit vector) 53 Diffuse ReflecDon I = ρ(!"! N!" L ) Light #1!"!!"! L N 1 I1 L1, x L1, y L 1, z N x I = ρ L L L N 2 2, x 2, y 2, z y I 3 L3, x L3, y L 3, z N z I : observed intensity ρ : diffuse reflectance!"! N :surface normal (unit vector)!" L : light direction (unit vector) 54 27

28 Diffuse ReflecDon I = ρ(!"! N!" L ) Light #2!"! L 2!"!!"! L N 1 I1 L1, x L1, y L 1, z N x I = ρ L L L N 2 2, x 2, y 2, z y I 3 L3, x L3, y L 3, z N z I : observed intensity ρ : diffuse reflectance!"! N :surface normal (unit vector)!" L : light direction (unit vector) 55 Diffuse ReflecDon I = ρ(!"! N!" L ) Light #3!"! L 3!"! L 2!"!!"! L N 1 I1 L1, x L1, y L 1, z N x I = ρ L L L N 2 2, x 2, y 2, z y I 3 L3, x L3, y L 3, z N z I : observed intensity ρ : diffuse reflectance!"! N :surface normal (unit vector)!" L : light direction (unit vector) 56 28

29 Diffuse ReflecDon I = ρ(!"! N!" L )!"! L 3!"! L 2!"!!"! L N 1 ρn = ( L! L) 1 L! I N = normalize(n) ρ = ρn I : observed intensity ρ : diffuse reflectance!"! N :surface normal (unit vector)!" L : light direction (unit vector) 57 Photometric Stereo normal map reflectance map normal map color coding: { R,G, B} = N +1 x, N +1 y, N +1 z diffuse reflectance per channel: { }! 3 ρ = ρ R,ρ R,ρ R ρ R,ρ R,ρ R 0,

30 Photometric Stereo Summary acquisidon setup observadon output normal map reflectance map [Farooq et el. 2008] Farooq, A. R., Smith, M. L., Smith, L. N., & Midha, S. (2005). Dynamic photometric stereo for on line quality control of ceramic Lles. Computers in industry, 56(8), Light CalibraDon Mirror reflective chrome ball Reflective Light Calculating light direction! L θ i! N θ r! R(=! V ) Chrome Ball! L = 2(! N! R)! N! R! R =! V =[0,0,1] T 60 30

31 Photometric Stereo observadon normal map close up 61 Enhance Geometry using Normals Enhance Geometry [Nehab et el. 2005] presented in SIGGRAPH 2005 Geometry : Good Low Frequency + High Frequency Errors Normal : Low Frequency Bias + Good High Frequency Nehab, D., Rusinkiewicz, S., Davis, J., & Ramamoorthi, R. (2005, July). Efficiently combining posi1ons and normals for precise 3D geometry. In ACM Min TransacCons H. Kim on Graphics (TOG) (Vol. 24, No. 3, pp ). ACM

32 Enhance Geometry using Normals Original geometry Normals OpLmized geometry + = 63 A General Pipeline for 3D Imaging Complete 3D Geometry RBT + ICP Surface ReconstrucDon Register all pardal geometries into a common global coordinates (RBT + ICP) Get implicit funcdon for 3D surface (Poisson reconstrucdon) Mesh indexing (Marching cubes) Done 64 32

33 Merging MulD-view Geometries 65 Merging Two Geometries Suppose the object is rigid, then this problem is considered as rigidbody transformadon Select at least 3 pairs of same points of the object automadcally or interacdvely 66 33

34 Merging MulD-view Point Clouds 67 Surface from Implicit funcdon Empty verdces: the implicit funcdon value is posidve Yellow verdces: the implicit funcdon value is negadve Then, depending on combinadon of each cell, draw the surface as figures above 68 34

35 3D Imaging Result 3D object 3D geometry 3D printed 69 Image Based Models 70 35

36 Image-Based Modeling Create 3D model (and texture maps) from images automated (structure from modon, stereo) interacdve Façade system 71 Façade 1. Select building blocks 2. Align them in each image 3. Solve for camera pose and block parameters (using constraints) 72 36

37 View-dependent texture mapping 1. Determine visible cameras for each surface element 2. Blend textures (images) depending on distance between original camera and novel viewpoint 73 Model-based stereo Compute offset from block model Some more results: 74 37

38 Image-Based Faces EsDmate shape from images Match metrics to shape Project video onto shape Texture map Animate 75 Graphics/Imaging ConDnuum Concentric mosaics Geometry centric Image centric Fixed geometry Viewdependent texture Viewdependent geometry Sprites with depth LDI Lumigraph Light field Polygon rendering + texture mapping Warping InterpolaDon 76 38

39 KAIST-VCLAB IBMR Example See video 77 39