Point Based Graphics State of the Art and Recent Advances

Transcription

1 Point Based Graphics State of the Art and Recent Advances Markus Gross Computer Graphics Laboratory ETH Zürich A SIGGRAPH 2009 Course Acknowledgements Mario Botsch, Gael Guennebaud, Simon Heinzle, Richard Keiser, Oliver Knoll, Edouard Lamboray, Matthias Müller, Miguel Otaduy, Cengitz Oeztireli, Mark Pauly, Denis Steinemann, Matthias Teschner, Tim Weyrich, Martin Wicke, Stephan Wuermlin, Matthias Zwicker Leo Guibas, Stanford University Leif Kobbelt, RWTH Aachen Andy Nealen, Marc Alexa, TU Darmstadt Bart Adams, Phil Dutre, Uni Leuven Hanspeter Pfister, Jeroen van Baar, MERL 2

2 Purpose of this Course Points are useful primitives for graphics and modeling! 3 Overview 1. Motivation 2. Representation 3. Processing, Editing, and Modeling 4. Rendering and Display 5. Physics Based Animation 6. Point Based Video 7. Lessons learned 4

3 Course Material 1. The slides 2. Our book on Point Based Graphics Gross, M.; Pfister, H.: Point Based Graphics, Morgan Kaufmann, Motivation 6

4 Polynomials... Rigorous mathematical concept Robust evaluation of geometric entities Shape control for smooth shapes Require proper parameterization Discontinuity modeling Topological flexibility Reduce p, refine h! 7 Triangles... Simple geometric primitives Hardware to support them Digital processing Explicit topology The widely accepted queen of graphics primitives Separation of geometry and attributes Complex LOD management Compression and streaming is highly non-trivial 8

5 Getting to the point Natural representation for many 3D acquisition systems No separation of geometry and appearance/attributes No separation of surfaces and volumes No connectivity or topology possibly more? 9 History of Points in Graphics Particle systems [Reeves 1983] Points as a display primitive [Whitted, Levoy 1985] Oriented particles [Szeliski, Tonnesen 1992] Particles and implicit surfaces [Witkin, Heckbert 1994] Rendering Architectures [Grossmann, Dally 1998] Digital Michelangelo [Levoy et al. 2000] Surfels [Pfister et al. 2000] QSplat [Rusinkiewicz, Levoy 2000] Point Clouds [Linsen, Prautzsch 2001] Point set surfaces [Alexa et al. 2001] Radial basis functions [Carr et al. 2001] Surface splatting [Zwicker et al. 2001] Randomized z-buffer [Wand et al. 2001] Sampling [Stamminger, Drettakis 2001] Pointshop3D [Zwicker, Pauly, Knoll, Gross 2002] Raytracing [Alexa et al. 2003] Boolean Operations [Adams et al. 2003] Modeling [Pauly et al. 2003] blue-c [Gross et al. 2003] Meshless Physics [Muller et al. 2004] Spherical MLS [Guennebaud, Gross 2007]...many more 10

6 Points A Motivation 3D content creation pipeline Points generalize Pixels! Representation 12

7 Surface Model Compute continuous surface from a set of discrete point samples discrete set of point samples P = { p i, c i, m i,... } continuous surface interpolating or approximating P 13 Surface Model Moving least squares (MLS) approximation Surface defined as stationary set of projection operator P implicit surface model Weighted least squares optimization Gaussian kernel function local, smooth mesh-less, adaptive (Alexa, Levin, Amenta, et al.) 14

8 Point Set Surfaces (PSS) [Levin 2003], [Alexa et al. 2001,2003] 2D-example: smooth curve from a set of points using moving least squares (MLS) approximation 15 Simple PSS Definition n i p i x 16

9 Simple PSS definition x 17 Simple PSS definition 18

10 Issues Loss of detail, tight fits, stability 19 Issues Loss of detail, tight fits, stability 20

11 Projection onto algebraic sphere Improved Stability Curvature for free Very fast on GPU Spherical MLS (Guennebaud, Gross, Siggraph 2007) PointGraphics Algebraic Point Set Surfaces Key idea: plane fit sphere fit Benefits: low sampling density tight approximation efficiency 22

12 Issue How to fit a sphere onto a set of points?? minimize some distance metric! 23 Algebraic fit Algebraic distance: plane equation! the sphere is described by Algebraic fit:! add constraint(s) 24

13 Low Sampling Rates planar fit spherical fit 25 Stability APSS vs 26 SPSS

14 Progressive Downsampling from 150k to 5 pts Differential Operators => e.g., accessibility shading 28

15 Local Kernel Regression LKR for implicit function to surface: (Oztireli, Gunnebaud, Gross, Eurographics 2009) 29 Surface Definition Surface definition becomes 30

16 Our Surface Definition LKR Formulation of [Kolluri] Robust error function Iteratively Reweighted Least Squares (IRLS) 31 Our Surface Definition Surface definition becomes 32

17 Edges and Corners Results 33 Results A tough one! 34

18 Results s 35 Results Stability 36

19 Noise & Outliers Results Processing, Editing, Modeling 38

20 Local Surface Analysis local neighborhood (e.g. k-nearest) eigenvectors span covariance ellipsoid smallest eigenvector is leastsquares normal surface variation (Linsen, Prautzsch, Garland) measures deviation from tangent plane curvature 39 (Pauly, Gross, Kobbelt, IEEE Vis 2002) Local Surface Analysis

21 Resampling - Particles Resample surface by distributing particles Relaxation - Adjust repulsion radius Upsampling possible (Heckbert) original model 296,850 points uniform repulsion 2,000 points 41 adaptive repulsion 3,000 points Particle Simulation 42

22 Resampling- Simplification Iteratively contracts point pairs (Hoppe) Uses quadric error metric (Heckbert) Similar to QSlim 296,850 points 2,000 points remaining contraction pairs 43 3D Image Editing Interactive 3D painting Cleaning Carving Textureing and antialiasing Modeling Spectral processing...of point sampled geometry From 2D Pixels to 3D Points 44

23 Pointshop 3D Interactive system for point-based surface editing Generalize 2D photo editing concepts and functionality to 3D point-sampled surfaces Use 3D surface pixels (surfels) as versatile display and modeling primitive Input irregular pointsampled model Surface editing Pointshop 3D Output irregular pointsampled model Does not require intermediate triangulation (Zwicker, Pauly, Knoll, Gross, Siggraph 2002) 45 Concept Parameterization v u Resampling Editing Operator 46

24 Parametrization Constrained minimum distortion parameterization (Levy, Siggraph 2001) C(X) = j 2 { X(u j ) x j } 2 X + u du 2! = min. feature points distortion Extension to irregular point clouds Multigrid solver for resulting sparse linear least squares problem 47 Parametrization Landmarks set interactively by the user (Levy) landmarks parametrization 48

25 Painting textures Examples 49 Examples Engraving surface detail 50

26 Examples Filtering appearance and geometry 51 Examples Multiscale feature extraction (Pauly, Keiser, Gross, Eurographics 2003) 52

27 3D Haptic Painting Mass-spring skeleton Physically-based deformation Force feedback Point-sampled surface Geometric deformation Paint transfer (Adams, Wicke, et al. PBG 2004) 53 Paint Transfer Sample Collection Paint Buffer Construction Object Sample Projection Brush Sample Projection Paint Model Evaluation Reprojection No separation of geometry and texture, no connectivity! 54

28 Brush Splitting 55 Paint Transfer PointGraphics

29 Results PointGraphics Painted Bunnies Nemo Day n Night Flower No separation of geometry and texture No charting and parametrization 58

30 Surface Processing Toolbox Overview 59 Manual Hole Filling 1) Scan with holes 2) Application of MLS spray can (Weyrich et al. 2004) 3) Cont. use of MLS spray can 4) Point relaxation 60

31 Manual Artifact Removal 1) Initial geometry 2) Eraser 3) MLS Spray Can 4) Point Relaxation 5) MLS Smoother 61 Multi-Scale Modeling Scale spaces: levels of smoothness same degrees of freedom on smoothest level 62

32 Multi-Scale Modeling Discretization: base domain + orthogonal diff. discretization in scale discretization in space (Pauly, Kobbelt, Gross, TOG 2006) 63 Filtering 64

33 Interactive Editing editing metaphor continuous free-form deformation function smooth transition between deformed and un-deformed region deformation function composed of simple translation and rotation components 65 Example Large Deformations, Dynamic Resampling (Pauly, Keiser, Kobbelt, Gross Siggraph 2003) 10,000 points 271,743 points 66

34 Dynamic Sampling Interpolate scalar attributes 67 Free-form Deformation 68

35 Boolean Operations Signed distance function by MLS operator Download: graphics.ethz.ch/pointshop3d 70

36 4. Rendering 71 Surfels Framebuffer resolutions stay roughly the same Size of typical triangles in complex 3D models project to < 1 pixel Point or pixel based rendering methods become more and more attractive Points store several surface attributes (surfels) To render, forward project each point separately 72

37 Surfels 1. View independent sampling (preprocessing) Geometry Sampling and Texture Prefiltering Surfel LDC Tree 3-to-1 Reduction Optional Reduced LDC Tree 2. Rendering (runtime) Block Culling Forward Warping Visibility Splatting Texture Filtering Deferred Shading Image Reconstruction and Antialiasing (Pfister, Zwicker, Baar, Gross, Siggraph 2000) 73 LDC Tree Level 2 Level 1 Level 0 Store only pointers to surfels at level 0. Do not store empty blocks. 74

38 Problems After projection the image may contain holes Texture and edge aliasing 75 Visibility Splatting Object space Z-Buffer Image Buffer 76

39 Image Reconstruction Image Buffer Output Image 2D filtering operations in screen space 77 Forward Warping 347 K points 204 K points 78

40 Sampling Artifacts screen space pixel sampling A Closer Look onto Sampling Unified approach to reconstruction and aliasing EWA, Heckbert 86 minification aliasing magnification 128 x 192 holes 80

41 Splatting screen space splatting 81 Warping Reconstruction Kernels forward projection screen space object space reconstruction kernel warped reconstruction kernel 82

42 Signal Processing View Object Space Screen Space Warp Screen Space Filter Sample Screen Space 83 Gaussian Kernels Gaussian reconstruction kernel Gaussian low-pass filter screen space screen space 84

43 Gaussian Kernels Closed under affine mappings and convolution Gaussian resampling filter screen space EWA Analytic expression of the resampling filter can be computed efficiently 85 Irregular Textures pixel sampling sampling pattern optimized screen space EWA (Zwicker, Pfister, Baar, Gross Siggraph 2001) 86

44 Examples Combine reconstruction and bandlimitation PointGraphics Improvements: Phong Shading + Shadow Maps (Botsch at al. PBG 2005) 88

45 Ray-Tracing Compute ray intersection with MLS surface (Adams, Keiser, Dutre, Pauly, Gross, Guibas, EG 05) 89 Examples (Alexa et al. 2004, Adams et al. 2005) PointGraphics

46 Point Rendering Hardware ASIC and FPGA design 30 Mio. EWA splats/s Lean architecture FP units for geometry 0.25 mu, 500KG 91 Integration Into Poly Pipelines Re-use of existing GPU units Novel Units Splat rasterizer Ternary depth test Accumulation Surface reconstruction buffer Normalize (Weyrich et al., Siggraph 2007) 92

47 Video 93 Point Processing Units Spatial search: knn and enn Common in most point-operations Kd-tree [Bentley 75] and priority queue Heinzle, Gunnebaud, Botsch, Gross Graphics Hardware

48 Coherent Neighbor Cache (enn) Find neighbors in slightly bigger radius. Reuse if. 95 Coherent Neighbor Cache (knn) Find (n+1) neighbors. Reuse result for spatially close query. Re-use if. 96

49 The Architecture 97 Coherent Neighbor Cache Prototype Eight cached neighborhoods Problem: parallel queries in knn module Interleave spatially similar queries 98

50 Kd-Tree Traversal 99 Processing Module Multithreaded quad-port bank of 16 registers 128 threads Programmability using FPGA-technology. 100

51 Results knn and MLS CUDA: x4 CUDA w/o sort: x4.0 MLS CUDA x3.8 CPU: x1.5 CUDA: x2.4 CUDA w/o sort: x3.1 FPGA: x1 FPGA: x1 CPU: x1.4 FPGA: x1 MLS CPU: x0.4 CUDA: x1.6 CPU: x Results Approximation Error (MLS projection) Cache hits Error 102

52 5. Physically-based Animation 103 Concept Point-based approach for physically-based animation Point based volume (physics) physical elements Point based surface (appearance) surface elements (Müller, Keiser, Nealen, Pauly, Gross, Alexa, SCA 2004) surfels {s i } simulation nodes {p j } 104

53 Simulation Loop Start with undeformed object and apply external forces (per physical element) Body Gradient Stress forces Strain Add Time (Hookean of external (from (Greens displacement integration elastic forces material) strain) energy) field We Body start forces with pure + new elasticity, external and forces add = plasticity/flow next integration thereafter step 105 Topological Separation Offline simulation at ca. 5 sec./frame 106

54 Topological Separation Collision detections of penetrating point sets Offline simulation at ca. 5 sec./frame 107 Viscous Fluids Combine Navier-Stokes and solid mechanics (Keiser, Gasser, Gross, PBG 2005) 108

55 Melting 109 Shell Physics- Lena Point shell structures (fibres) connect surface (Wicke, Steinmann, Gross, EG 2005) 110

56 Fracture and Cracks Crack initiation where stress above threshold crack created by inserting 3 crack nodes each carrying 2 opposing surfels connection is crack front Pauly et al. Siggraph 2005 external force one fracture surface crack front 111 external force Example 112

57 Resampling 113 Fracture Dynamic sampling of facture surface (Pauly, Keiser, Adams, Dutre, Gross, Guibas Siggraph 2005) 114

58 Mesh-Points (Steinemann, Otaduy, Gross, SSCA 2006) Point Based Video (Gross et al. Siggraph 2003) 116

59 System Overview Gross et al. Siggraph PointGraphics

60 3D Mirror 119 Displays

61 3D Video Pipeline 121 3D Video Representation 3D Video fragement Generalizes pixels to 3D points 122

62 Differential 3D Video Stream green: new Insert red: expired Delete blue: color changed UpdateCol white: color unchanged UpdatePos possibly black: background 123 3D Video Stream Waschbüsch et al. PG

63 3D Video Recorder 125 MoreExamples Lamboray et al. ICIP 2004 Würmlin et al. VMV

64 Active 3D Video Acquisition Stereo cameras Texture camera Structured light projector 127 Final 3D Video 128

65 Results 129 So Points Points are a useful modeling/graphics primitive Sample based approach to graphics and modeling Signal Processing view Resample without restructuring Local operations on large datasets Do not store topology Complement on other graphics representations Graphics pipelines for points? Consumer electronics? 130

66 Research Themes Illustration and stylistic visualization Implicits and Isosurfaces - MLS Procedural point geometry Add semantics to point samples ( sameness ) Compression and out of core representations Dynamic point clouds More advanced filters for point sampled data Explore combinations of primitives Hardware for point graphics 131 Resources tutorials.php 132

67 Researchers/Groups (Random&Incomplete) Marc Alexa TU Berlin Hanspeter Pfister Harvard Marc Pauly ETH Zurich Matthias Zwicker University of Berne Tim Weyrich UC London Mario Botsch University of Bielefeld Leo Guibas Stanford University Leif Kobbelt RWTH Aachen Marc Stamminger University of Erlangen Gael Guennebaud INRIA Bordeaux 133 Researchers/Groups (Random&Incomplete) Nina Amenta UC Davis Amitabh Varshney University of Maryland Michael Wand MPI Saarbrücken Bart Adams Stanford website Loic Barthe, Mathias Paulin Toulouse Wojciech Matusik Adobe Research Renato Pajarola University of Zurich Marc Levoy Stanford University 134

68 Thank You! 135