MASSIVE TIME-LAPSE POINT CLOUD RENDERING with VR

Transcription

1 April 4-7, 2016 Silicon Valley MASSIVE TIME-LAPSE POINT CLOUD RENDERING with VR Innfarn Yoo, OpenGL Chips and Core Markus Schuetz, Professional Visualization

2 Introduction Previous Work AGENDA Methods Progressive Blue-Noise Point Cloud High-Quality VR Point Cloud Conclusion Demos 2

3 INTRODUCTION Point Cloud A set of points that represents the external surface of an object Our Dataset: Project Endeavor New NVIDIA building under construction Time-Lapse Point Clouds 3

4 INTRODUCTION Point Cloud Representation Advantages Disadvantages Simplicity Scalability Easiness of capturing Visually incomplete Easy to get noise Increasing data size Easiness of data handling 4

5 INTRODUCTION Our Focus Point Cloud Massive Scale (more than 1 TB) Time-Lapse VR Rendering Real-Time Rendering Instant Scalability Efficient Out-of-Core Design Plausible Visualization High-Quality VR Experience 5

6 INTRODUCTION Our Contributions A Novel Approach for Massive Time-Lapse Point Cloud Rendering Adapting Progressive Blue-Noise Point Cloud (PBNPC) Resampling High-Quality Point Cloud VR Experience Our method provides several important features: performance, quality, & navigation 6

7 Introduction Previous Work AGENDA Methods Progressive Blue-Noise Point Cloud High-Quality VR Point Cloud Conclusion Demos 7

8 PREVIOUS WORK Plant growth analysis using time-lapse point cloud Li et al., Analyzing Growing Plants from 4D Point Cloud Data, 2013, Transaction on Graphics Image source: Li et al., Analyzing growing plants from 4D point cloud data, 2013, ToG 8

9 PREVIOUS WORK Hyper-lapse from video (sequence of images) First Person Hyper-lapse Video, Kopf et al., 2014, Transaction on Graphics Image source: Kopf et al., First Person Hyper-lapse Video, 2014, ToG 9

10 Introduction Previous Work AGENDA Methods Progressive Blue-Noise Point Cloud High-Quality VR Point Cloud Conclusion Demos 10

11 PROGRESSIVE BLUE-NOISE POINT CLOUD (PBNPC) 11

12 PROBLEMS DATA SIZE REGISTRATION COLOR MISMATCH Per Day PC: 1.5 GB 2 Years: 1 TB Current: 120 GB GPU Memory: up to 24 GB (NVIDIA Quadro M6000) Out-of-Core Loading Drone captured Capturing is not perfect Actual site changes by daily construction Daily weather changes Capturing time changes Sun position changes Shadows 12

13 1. Data Size: 1.5 GB Per Day 13

14 2. Color Mismatch July 27, 2015 July 30,

15 3. Registration Problem 15

16 PIPELINE OVERVIEW Preprocessing Real-Time Processing Input Point Cloud (LAS files) Registration PBNPC Creation Color Correction Adjusting # of points & Rendering Estimating Time-Lapse Speed Filling Sparse Vertex Buffer Async Loading & Removing Sparse Buffer 16

17 INPUT FILES Input Point Cloud (LAS files) Registration PBNPC Creation Color Correction Drone captures point cloud every 2-3 day Input file format: LAS (liblas) Sampling density is varying per day Boundary is varying per day (Noise) 1.5 GB per capture * 86 := 120 GB Reduced to 50 GB 17

18 MAJOR PROBLEMS Input Point Cloud (LAS files) Registration PBNPC Creation Color Correction Massive Scale Point Cloud Data Handling We need Immediate Scalability & Preserve Visual Quality 18

19 BLUE-NOISE POINT CLOUD Types of Noise, depend on frequency distribution White Noise, Pink Noise, and Blue-Noise Blue-Noise Point Cloud Nearly Poisson Distribution Approximation of Human Retina Cells Distribution Visually Plausible Image source: Recursive Wang Tiles for Real-Time Blue Noise, Kopf et al., 2006, ACM SIGGRAPH 19

20 PROGRESSIVE BLUE-NOISE POINT CLOUD Progressive Blue-Noise Point Cloud (PBNPC) Adding or removing any number of points preserve Blue-Noise Characteristics Using Recursive Wang Tiles for Real-Time Blue Noise, Kopf et al., 2006, ACM SIGGRAPH Image source: Recursive Wang Tiles for Real-Time Blue Noise, Kopf et al., 2006, ACM SIGGRAPH 20

21 VIDEO: PBNPC 21

22 REGISTRATION Input Point Cloud (LAS files) Registration PBNPC Creation Color Correction We tried to use Approximated Nearest Neighbors (ANN) to align Point Clouds - Several problems: too slow and low accuracy Render depth maps in several different camera positions Generate gradient maps from depth maps Octree-based Search + Hill Climbing Algorithm 22

23 TIME-LAPSE COLOR CORRECTION Input Point Cloud (LAS files) Registration PBNPC Creation Color Correction We are not correcting colors YET Instead Time-Lapse Blending between Days Blending alleviates the color mismatching problem Blending cannot solve shadow(sun position) and color distribution problems 23

24 TIME-LAPSE COLOR CORRECTION No Blending 24

25 TIME-LAPSE COLOR CORRECTION Blending 25

26 PIPELINE OVERVIEW Preprocessing Real-Time Processing Input Point Cloud (LAS files) Registration PBNPC Creation Color Correction Adjusting # of points & Rendering Estimating Time-Lapse Speed Filling Sparse Vertex Buffer Async Loading & Removing Sparse Buffer 26

27 OPENGL 4.5, SPARSE BUFFER ARB_sparse_buffer Adjusting # of points & Rendering Estimating Time- Lapse Speed Async Loading & Removing Sparse Buffer Filling Sparse Vertex Buffer Newly introduced OpenGL 4.5 Extension ( Decouple GPU s virtual and physical memory Similar to ARB_sparse_texture extension We are using Sparse Buffer as a Stack (Prepare entire virtual memory per daily PC) 27

28 OPENGL 4.5, SPARSE BUFFER ARB_sparse_buffer We allocate virtual memory for entire time-lapse point clouds glbufferstorage(target, size, data_ptr, GL_SPARSE_STORAGE_BIT_ARB) data_ptr will be ignored, when with sparse storage bit Physical memory can be committed by using glbufferpagecommitmentarb(target, offset, size, commit) GL_TRUE or GL_FALSE for commit or decommit memory Greatly ease our effort to manage GPU memory 28

29 ESTIMATING TIME-LAPSE SPEED Based on Loading & Unloading Probability Adjusting # of points & Rendering Estimate Time- Lapse Speed Async Loading & Removing Sparse Buffer Filling Sparse Vertex Buffer Calculating Time-Lapse Direction & Speed Amount of Async Loading Request is based on Probability Probability adjusted by Time-Lapse Direction & Speed 29

30 ADJUSTING # OF POINTS Adjusting # of points & Rendering Estimate Time- Lapse Speed Async Loading & Removing Sparse Buffer Filling Sparse Vertex Buffer For Real-Time Rendering (Normal > 60 Hz & VR > 90 Hz) Instant Level of Details (LOD) by using Progressive Blue-Noise Point Cloud Simply adjusting LOD percentage until target FPS accomplished 30

31 HIGH-QUALITY VR POINT CLOUD 31

32 VR POINT CLOUDS Performance, Quality, User Interaction 32

33 VR PERFORMANCE Each point cloud at least 30 million points Rendering 2 point clouds during time-slice transitions Render twice (once for each eye) At 90 Frames per Second +some more to account for additional VR overhead 33

34 VR PERFORMANCE Out-Of-Core data structures necessary Multi-Resolution Octree used for Point Cloud VR demo ( Interactions with Gigantic Point Clouds, Claus Scheiblauer) Load and render only visible parts up to desired Level of Detail source: Potree: Rendering Large Point Clouds in Web Browsers, Markus Schuetz 34

35 VR PERFORMANCE High Level-Of-Detail near camera Render only ~3 million points out of billions 1.3 billion points at FPS on Quadro M GB source: Potree: Rendering Large Point Clouds in Web Browsers, Markus Schuetz 35

36 VR POINT CLOUDS Quality Strong aliasing inherent to Point Cloud Rendering Surfaces made up of overlapping points that occlude each other. Closest to camera wins. Aliasing more noticeable in VR due to constant motion and low resolution Perceived as sparkling 36

37 SOURCES OF ALIASING OCCLUSIONS Surface Patches made up of overlapping points Points fighting for visibility LEVEL OF DETAIL Building Multi-Resolution Octree, only considering point coordinates SILHOUETTES Model Silhouettes Point Sprite Silhouettes Like Nearest-Neighbor source: Potree: Rendering Large Point Clouds in Web Browsers, Markus Schuetz 37

38 SOURCES OF ALIASING Occlusions Surface Patches made up of overlapping points that constantly fight for visibility Blend fragments together instead of rendering only the closest ( High-quality surface splatting on today's GPUs, Botsch et al., 2005) Using screen-aligned circles instead of oriented splats source: Potree: Rendering Large Point Clouds in Web Browsers, Markus Schuetz 38

39 SOURCES OF ALIASING Level-of-Detail Additionally store interpolated colors in lower Levels-Of-Detail Like Mip-Mapping for point clouds Interpolated colors partially reduce occlusion-aliasing 39

40 SOURCES OF ALIASING Silhouettes Large amount of silhouettes due to holes from incompletely or sparsely captured 3D data and noise Plus: Each point sprite has its own small silhouette Smallest HMD movements and noise constantly change silhouettes Using MSAA to reduce these aliasing artifacts MSAA partially reduces occlusion-aliasing, too: 40

41 HIGH-QUALITY POINT CLOUDS IN VR MSAA and High-Quality (HQ) Splatting currently not compatible HQ-Splatting solves occlusion MSAA and averaged color LODs partially reduce occlusions MSAA faster and eliminates most sparkling effects MSAA, together with averaged color LODs, better suited for VR 41

42 HIGH-QUALITY POINT CLOUDS IN VR Transitions Render into different framebuffers, then combine Reduce performance impact by rendering one time-slice: at a lower resolution or without MSAA Quality reduction less noticeable during transition 42

43 HIGH-QUALITY POINT CLOUDS IN VR 43

44 VR POINT CLOUDS User Interaction Point clouds rarely dense enough for world-scale first-person navigation Table-Scale navigation -> Treat like a small model on a table or floating in space Grab gesture to move model Pinch-To-Zoom gestures (scale, rotate and move) Measurements 44

45 VR POINT CLOUDS Navigation Grab a point in space to drag and drop the model 45

46 VR POINT CLOUDS Navigation Grab a region-of-interest to enlarge, shrink and rotate it 46

47 VR POINT CLOUDS Controller Assignments Previous / Next Day Measurements 47

48 Introduction Previous Work AGENDA Methods Progressive Blue-Noise Point Cloud High-Quality VR Point Cloud Conclusion Demos 48

49 CONCLUSION Methods to render large point cloud data with hundreds of time-slices High-Quality subsampling and progressive LOD using PBNPC High-Quality Point Cloud Rendering for VR Fast and precise navigation 49

50 LIMITATIONS AND FUTURE WORK PBNPC sampled in 2D -> extend to 3D Correcting color mismatching problem Integrating NVIDIAs VR SLI Extension Eliminate pitfalls in VR navigation Multi-Res PBNPC for arbitrary large Point Clouds 50

51 NVIDIA S NEW BUILD VISUALIZATION 51

52 Introduction Previous Work AGENDA Methods Progressive Blue-Noise Point Cloud High-Quality VR Point Cloud Conclusion Demos 52

53 April 4-7, 2016 Silicon Valley THANK YOU