CS 260 Computer Graphics

Transcription

1 Craig M. Wittenbrink Lecture 6: Image Based Rendering Techniques cont.: Seitz and Dyer (Shade et al., Wittenbrink et al., finish from last time) CS 260 Computer Graphics &6Ã:LQWHUÃ:LWWHQEULQNÃOHFWÃ Overview Shade et al. Layered depth images, finish discussion Wittenbrink et al. Karhunen-Loeve Transform 2D texture mapping from satellite images Seitz and Dyer, Voxel Coloring HP s Voxel Coloring work Conclusions

2 Layered Depth Images (LDI s) Attempt to handle more disocclusions and large amounts of parallax contains potentially multiple depth pixels per pixel location Farther pixels help to fill holes (disocclusions) Use linked during construction, and packed for rendering Fig. 5 reproduced from Shade et al. Copyright SIGGRAPH 98 Creation of LDI s a)use synthetic ray tracer that provides depth per pixel b) Or scan conversion and read z-buffer Choose one camera position as LDI camera and warp images to that camera 2)Use Less regular sampling with ray tracer which rays to choose? 3)Or use Computer Vision from multiple images Modified Seitz and Dyer algorithm, view centered voxelization 2

3 Reconstruction Common events: ) disocclusions as viewpoint changes 2) surfaces that cover large areas of the screen Cube of possible new viewpoints Define an LDI from each cube face (they don t appear to actually do this) Creation with ray caster cont. Parametrize the rays Use cosine weighted direction over the hemisphere use Stratified Stochastic sampling divide uniformly into NxNxNxN strata for each stratum cast m rays N=32, m=6, gives 32^4*6=6million 3

4 Sampling with ray caster Main point, get lots of rays to cover many viewpoints Rays are in all directions LDI from real images (not that you could tell ;) ) Seitz and Dyer dinosaur toy 4

5 Rendering Layered Depth Images Splatting used Space efficient representation pack LDI, bottom-to-top, left-to-right in screen space, and back-to-front along ray Store offsets for fast access ) beginning of scanline 2) pixel in scanline Incremental Warping Computation Given 4x4 matrix for LDI view and 4x4 matrix for desired new view You can transform a point in the LDI view to the world coordinates, and then to the new view x2w y2w z2w w = C2C x y z a2 C 2 a Aworld C C2 C 5

6 6 Incremental Warping Reuse matrix results by factoring New start is simply incremented depth z start T z y x T z y x C C w w z w y w x,2, = + = = xincr start T y x T y x T newstart + = + = + = ,2,2,2 Rendering of LDI For each pixel For number of layers result=start +z*depth (location) clip either behind camera or out of frustum splat (pick appropriate splat size) increment for next pixel on scanline Splat size chosen by projected pixel area approximiation Put approximations in lookup table ],, [ 2 d n n lookup z size y x

7 LDI Rendering Results LDI s max 0 layers per pixel,.24 average depth complexity 300x300 resolution, at 8-0 frames/second on Pentium II, 300 MHz LDI Rendering Results Cross eyed stereo pairs. LDI s from Rayshade raytracer. LDI has. million depth pixels. 4-0 frames/second on Pentium 7

8 LDI Rendering Results Cross eyed stereo pairs. LDI s from Rayshade raytracer. LDI has. million depth pixels. 4-0 frames/second on Pentium Wittenbrink et al. SPIE mm slides Feature extraction of clouds from GOES satellite data for integrated model measurement visualization, Craig M. Wittenbrink, Glen Langdon, Jr., and Gabriel Fernandez, In Proceedings of IS&T/SPIE Symposium on Electronic Imaging: Image and Video Processing IV 996, Vol. 2666, pages , R. Stevenson and M.I. Sezan, San Jose, CA

9 Video, Craig s Image Based Rendering 3D Clouds as rendered with texture mapping on SGI Seitz and Dyer, Voxel Coloring Steven Seitz and Charles Dyer, Photorealistic Scene Reconstruction by Voxel Coloring, in Proc. Computer Vision and Pattern Recognition Conf. pp , 997. All IBR work is trying to address limitations of current methods Voxel Coloring: Works well with disparate camera views Has explicit occlusion modelling Integrates many images But Requires excellent camera calibration diffuse lighting restricted camera positioning 9

10 Voxel Coloring From Seitz and Dyer Copyright IEEE, 97, Figure. Example camera configurations Camera Volume Definition-Convex hull of camera centers Outward looking Looking down from a plane Voxel Coloring Cameras are all in plane Ordinal Visibility Constraint Invariant voxels/colors are uniquely defined Scene, S, is a collection of voxels that are consistent with the colors in all cameras Complete scene color(p,i) = color(s(p),s) for every image I and every pixel, p in I. How to compute this constraint? 0

11 Voxel Coloring / Ordinal Visiblity Constraint Use a norm. to define For all scene points P and Q for input image I, P occludes Q in I only if P < Q For geometry before, use distance to plane P occludes Q if on segment: CQ C P Q Voxel Coloring / Colors and Scene points No Scene point is allowed to be contained within the convex hull of the camera centers The uniqueness property is hard to solve So, define with color uniqueness Allows consistent solution Ambiguity problems 2 viewpoints or or consistent views

12 Ambiguity-Indistinguishable viewpoints No points in common shared point with different color assignment each point has same color in both scenes From Seitz and Dyer, Copyright IEEE, 997, Figure 2. Hard variants Color matches in all scenes in which it s visible Can t be so restrictive Voxel coloring defined as set of voxels that is consistent with all of the images S = Vp p Ii, Voxel coloring is not a minimal solution { i m} versus 2

13 Voxel consistent Inductive argument, take all voxels closer to cameras than V to be Sv If Sv is consistent, and V is consistent, add V to Sv Snew = V U Sv (union) So, algorithm, incrementally move farther from cameras and add voxels to consistent set Method guarantees a consistent reconstruction is found (doesn t mean it s correct ;) ) Color Consistency The consistency of a voxel is determined by a statistical test Use a threshold to account for variance in images To reconstruct, consider background removal so that background voxels are not created In other words, don t correlate on background color 3

14 Voxel Coloring algorithm S = φ for I=, r do (for every uniform distance) for every V in d do (for all voxels at that distance) V i C project to I, Im compute λ V if λ V < thresh then S = S {} V use bit mask in images to decide occlusion Seitz and Dyer results Run time O(voxels*images) = O(N^3*N^2*m) Space complexity O(images) = O(N^2*m) Model Images Run time resolution Toy 2 3minutes 7million- >72,000 Toy 2 <sec?->980 Room sec?- >50,000 4

15 Seitz and Dyer results: Toy dinosaur photograph voxel coloring Left: input image Middle: reconstruction from same viewpoint Right: new viewpoint Seitz and Dyer results: Rose photograph voxel coloring Left: input image Middle: reconstruction from same viewpoint Right: new viewpoint 5

16 Seitz and Dyer results: synthetic room voxel coloring Reconstruction Original model HP Results on Voxel Coloring Bruce Culbertson Tom Malzbender John Harvey Cindy Chen Zoe Wood Greg Slaubagh and others 6

17 HP s Voxel coloring outline Chroma glyphs Resolving issues of colors Cusps Adding geometry View dependent texture mapping Pose Estimation : ChromaGlyph Wallpaper Inexpensive, Simple Automatic detection + identification Highly overconstrained Supports arbitrary foreground geometry Occlusion handling Glpyhs: 3 of 6 prototype colors (20) 7

18 ChromaGlyph Wallpaper Applied Glyph Extraction Glyph Identity 3D Coord. Glyph Position 2D Coord Tsai/Wilson Camera Calibration Transform Matrix Detecting Glyphs Use pixel statistics to: Eliminate partially occluded glyphs. Prevent foreground objects from being identified as glyphs. Only N=3 protoype colors are seen in the region. Disregard any region with more than 5% of its pixels not in the top 3 ranked, classified colors. The number of pixels in the top 3 ranked colors corresponds to a distribution expected from a glyph geometry of concentric overlapping discs. Disc regions are concentric. The spatial centroid of the top 3 ranked pixel colors should be within a few pixels of each other. Pixels in the outer band should have a larger standard deviation of position than those in the inner bands. The rank of the the top N=3 color histogram must match the rank of the top 3 spatial standard deviations. A glyph must exceed some minimum size to be usable. The glyph ID implied by the top 3 ranked colors must have actually occurred on the particular chromaglyph sheet imaged. 8

19 Imaging Methodology Print Wallpaper on DesignJet 2500 Printer + Assemble Scene Photograph onto Film Scan onto PhotoCD Downsample to 024x536 (from up to 4096x644) Recover Pose * Note: Color Transformations Bi-Plane Glyph pattern used in practice PhotoCD: + Resolution + No demosaicing - Distortion - Center of projection Chromaglyph Color Space 9

20 Color Misclassification Due to Aliasing *PhotoCD Nearest Neighbor Classifiers 20

21 Pure Colors Threshold = 0.4 * distance to nearest prototype, (/T2 = 6.0), adequate Voxel-coloring the challenge Input: A handful of calibrated images Output: Images from new viewpoints Animations 2

22 Voxel-coloring image sets Ls 2 images synthetic dinosaurs 2 images from Steve Seitz towers 36 images trains 20 images shoes 30 images Voxel-coloring schematic input images volume to reconstruct scan direction 22

23 Voxel-coloring how is self-occlusion handled? visit occluding voxel before occluded voxel ignore occluded pixels Voxel Coloring - rough description Given N images and a voxel array: - Traverse voxel space away from each camera - For each voxel determine which images yield an unoccluded view of the voxel. - Project voxel into each image and measure color statistics to determine if this voxel is a surface voxel. (Projection consistency) 23

24 Voxel-coloring algorithm for all images /* clear occlusion bitmaps */ set all pixels to not-occluded for each layer of voxels along major axis { for each voxel V in layer { for each image i { find the set P(i) of pixels in the projection of V for all pixels of P(i) which are not-occluded add contribution to colormean and colorstandarddeviation } if (colorstandarddeviation < threshold) { /* if colors match */ mark V opaque and color V with the colormean for each image i set pixels in P(i) to occluded } else mark V transparent /* if colors do not match */ } } Voxel-coloring finds a surface consistent with the input images, not necessarily the actual surface reconstructed surface actual surface reconstructed surface actual surface 24

25 Voxel-coloring problem: edges and other abrupt color variations solutions Simple solution special treatment of edges If a voxel projects onto an edge in all images then color it. More general solution adaptive coloring threshold When considering whether to color a voxel V, compute the standard deviation S i in the color of the projection of V in each individual image i. Compute the mean M across all S i. Use a coloring threshold T proportional to M: T = k M + k 2 fixed component adaptive component Voxel-coloring is RGB the right color space? In RGB, dark colors are closer together than bright colors, regardless of chromaticity. Is there commensurately less information in the dark regions of our images? Surfaces do not, in general, radiate equally in all directions. With Lambertian surfaces, most of the variation is in luminosity. Hence, voxelcoloring might perform better in a color space that places less weight on luminosity than on chromaticity. Hence, we are experimenting with alternative color spaces. So far, we only have preliminary results. Reconstruction using RGB Reconstruction using CIELab 25

26 Voxel-coloring Enhancements: how do we know if they are better? A measurement technique. ( - ) & unused image reconstruction matte = difference, RMS error Voxel-coloring A measurement technique Experimental results Algorithm original special edge treatment adaptive threshold CIELab dinosaurs RMS percent error improvement shoes RMS error percent improvement

27 Voxel Coloring Results Cyclic Dependencies Can we expect a single pass to yield correct results? Current Estimate of Geometry Occlusion Relationships Color Statistics Among Voxel Footprints. Which images to project V i into? * Voxel coloring assumption: If (and only if) we are moving away from all cameras, we can expect additional geometry not to change occlusion relationships for already computed voxels. 27

28 Voxel Traversal Order Octant Case Specifies traversal in 3 axes. Half Space Case Quadrant Case Specifies traversal in 2 axes. Surrounded Case Specifies traversal in axis. * Embedded Case exists as well. Polygon Coloring Motivation Voxel Coloring Assumptions: Ordinal Visibility Constraint Voxel is a point sample regarding traversal order. Voxel is a small cube when projected into images. Polygon Coloring Well defined polygonal geometry built on top of volumetric representation (arbitrary topology). Ability to incorporate complex traversal strategies. 28

29 Marching Cubes Surface Tessellation Lookup table based Each 2x2x2 neighborhood classified into one of 256 cases. Ensures closed objects of arbitrary topology. Polygon Coloring Incremental Geometry Binary Occupancy Volume 29

30 Layered Depth Images (Not stored) (Depth, Polygon ID ) (Depth, Polygon ID ) Allows us to add and delete geometry incrementally for occlusion purposes. Polygon Coloring Results Original View Reconstruction Original View Reconstruction 30

31 Polygon Coloring Results View Dependent Texture Mapping Calculate the Barycentric coordinates of this new view relative to the three closest views. Due to changes in visibility weights vary per polygon. *.49 *.27 *.24 3

32 Conclusions Shade et al. Layered depth images (complete) Wittenbrink et al. Karhunen-Loeve Transform 2D texture mapping from satellite images Seitz and Dyer Photorealistic Scene Reconstruction by Voxel Coloring CVPR 97 HP Voxel Coloring Next time: Curves and Surfaces, (Watt & Watt Chapter 3) Conclusions 32