V-PANE Virtual Perspectives Augmenting Natural Experiences

Transcription

1 V-PANE Virtual Perspectives Augmenting Natural Experiences GTC 2017 The views, opinions and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. Distribution Statement A (Approved for Public Release, Distribution Unlimited). DARPA DISTAR #27938 Kerry Moffitt Scientist Kerry.Moffitt@Raytheon.com

2 Contents Background: DARPA, GXV-T V-PANE Overview and Architecture Geometry Video Ongoing Work

3 DARPA Ground X-Vehicle Technologies (GXV-T) Goal: Improved vehicle survivability and mobility Development Areas: Increased agility Enhanced mobility Crew augmentation Signature management Source:

4 V-PANE Overview Objective: Develop the next generation of ground-vehicle humanmachine interfaces that fuse real-time sensor feeds with video data projected onto a 3D geometric model of the environment. The program aims to develop a fully user-controlled, multipleperspective, live virtual representation of the vehicle s surroundings. Networked IED Report Boomerang Shot Detection IED Synthetic 1 st Person View ATAK Route Plan Popular Press: Wired Magazine National Defense American Security Today Product Design and Development: Secondary View & Touchscreen

5 High-Level V-PANE Vision Boomerang Shot Detection Additional Onboard Sensors Lidar Ground X-Vehicle Video Image Array Color LWIR LWIR Video Gunner View & Controls Point Cloud Future Imagery Projected onto 3D Geometry Driver/Commander View & Controls Live 3D Model 3D View Renderer Static Imagery & Maps SA System (e.g., ATAK) Path Plan 1. The real-time fusion of: A. Lidar point clouds into a 3D model and B. Multiple 2D video streams onto that model and C. Other 2D or 3D threat, position or mapping data 2. The real-time rendering of that model with video into 2D displays from multiple perspectives 3. The real-time control of the multiple perspectives

6 V-PANE User Capabilities People Obstacles Obstructions Real-time Route Analysis Vehicles Object Detection 360 Visualization Arbitrary Visualization Perspectives Multi-Spectral Image Fusion After- Action Reports Slope Reconnaissance Offline Viewing Routing Blue Force Locations Fused Semantic Information Shot Reports Cue, Slew, Track a Location Location Probing Range, Bearing, Elevation Include Pre-existing Data Position Slope (pitch/roll) Dynamic Controls

7 Sensor Array

8 Real-Time Scanning, Modeling, Rendering Lidar + Video + IMU = The World

9 Real-Time Scanning, Modeling, Rendering

10 V-PANE Workstation Architecture Lidar IMU/GPS Cameras 2 x Velodyne HDL-32E 1 NovAtel 1 VectorNav 1 IOI 4KSDI 5 BMD HD 2 LWIR NTSC LAN USB 2 x BMD Q2 Video Grabbers P100 Fusion CPU1 CPU2 K80 Compression P100 Raycasting P100 Video Frusta Projection SSD M6000 Video Cache, Rendering

11 Inter-GPGPU (P2P) CPU-Driven QPI DisplayPort V-PANE Data Pipeline Processing and Recording in a Single Server Maximum-Load Analysis with PEX 8747 PCIe Switches Inter-GPGPU (non-p2p) 1400k pts/s 3 MB/s Lidar IMU/GPS Cameras 2 x Velodyne HDL-32E 1 NovAtel 1 VectorNav Position, Orientation 800 Hz 2 x 40 KB/s 1 IOI 4KSDI 5 BMD HD 2 LWIR NTSC NB: 1 - P2P links require no transfer to or from CPU 2 - Non-P2P links count twice 3 - QPI link hits both CPUs P100 Fusion Voxels 5 GB/s P100 Raycasting Processed Lidar 18 MB/s LAN Raw Lidar 3 MB/s Pixel Positions 3 GB/s Voxels 5 GB/s USB CPU1 Raw IMU 80 KB/s Depths, Indices 4 GB/s Voxels 5 GB/s P100 Video Frusta Projection Voxels 1 GB/s Video, Voxels 600 MB/s Depths, Indices 4 GB/s Voxels 100 MB/s Video, Voxels 1.4 GB/s SSD 2 x BMD Q2 Video Grabbers Live Video 2.9 GB/s CPU2 M6000 Video Cache, Rendering Compressed Video 500 MB/s Video 3.3 GB/s Depths, Indices 4 GB/s Live Video 2.9 GB/s HD Video 500 MB/s HD Video 500 MB/s K80 Compression

12 Lidar Projection: Geometry Fusion Lidar lasers (1) reflect back from surfaces in the world (2), sampling depth During each fusion update, every voxel reverseprojects (3) to lidar focal point to determine distance from voxel to surface Depth samples stored as rectangular array with fixed distance between samples (4) to optimize lookup (this requires resampling fixed-angledelta lidar data)

13 Beam Modeling 100 m m

14 Wobbler Fill in the vertical gaps between lasers

15 Ray Casting Over pixels in current view Project into voxel array Find nearest zero-crossing

16 Raycast: Determine 3D Point per Pixel For every pixel to be rendered on screen (1), project from virtual camera through pixel (2) into voxel space, to find intersection point with nearest surface on that ray (3) Output is a 3-space point per pixel to be rendered

17 Video Projection: Color Index per Pixel For every pixel to be rendered on screen (1), reverse-project from 3D point to real-world camera frame (2), to find intersection point with image captured in that frame (3) Any given scene may involve 100s of frames

18 V-PANE Data Processing Frequency and Latency Geometry Fusion GPGPU 1 < 50 ms Cop y Geometry Fusion Transfer Voxels < 50 ms GPGPU 2... Ray Cast < 16 ms Update Frequency Geometry: 20 Hz Video Capture: 60 Hz Rendering: 60 Hz Latency Geometry: < 148 ms Video: < 132 ms GPGPU 3 Video Project Video Capture Video Capture Grabber GPU 1 Render < 16 ms < 100 ms < 16 ms < 16 ms Video Latency: < 132 ms Geometry Latency: < 148 ms

19 Profiling Geometry Fusion and Ray Casting Geometry Fusion Cycle Time: < 50 ms Ray Casting: < 16 ms

20 V-PANE Workstation Architecture Lidar IMU/GPS Cameras 2 x Velodyne 1 NovAtel 1 IOI 4KSDI HDL-32E 1 VectorNav 5 BMD HD 2 LWIR NTSC LAN USB 2 x BMD Q2 Video Grabbers P100 Fusion CPU1 CPU2 K80 Compression P100 Raycasting P100 Video Frusta Projection SSD M6000 Video Cache, Rendering

21 Image Processing in V-PANE From Live Cameras YUV to Pinned Host Memory Color Convert cudamemcpy [Cache] Undistort Fusion Projection Compression Build Frame Copy Bits Download Copy Frame Compress From Disk DXT5 to Pinned Host Memory Upload Decompress Copy Bits Undistort [Cache] Fusion Host CPU GPU Projection CUDA Build Frame Copy Frame OpenGL R e n d e r

22 Thread Activity (Video I/O) 16.6 ms Main Thread (M6000) Decompress Process Upload Render VBLANK Decompress Dequeue Dequeue Video Load/Receive Threads Enqueue Enqueue Dequeue Compression Thread (K80) Process Upload Compress Store Process

23 Video Latency: Best Observed Case Event in World Captured Signal Output Signal Visible on Display Single 1080p60 Stream Camera Capture Grabber Capture Host Processing Display Hardware Hardware * Software Cycle: 17 ms After Capture, Render Can Upload D M A S w a p Wait for VB U p l d D M A D r a w S w a p Cycle: 17 ms Capture Thread Render Thread Time (milliseconds) * Total latency and host processing latency are observed Hardware latency is inferred

24 LWIR Integration

25 Ongoing Work: Object Detection Image classification

26 Ongoing Work: Level of Detail Add a second voxel array: 10x range, to 1 km radius 1/10 th voxel resolution per dimension Ray caster uses per pixel when no hit in primary (hi-res) voxels Requires only 0.1% compute to update given 100 m lidar range 2,000 m 200 m

27 Ongoing Work and Challenges Voxel cache, Voxel LOD GPS/altitude Occlusion testing for video projection Bandwidth vs. render quality Timestamps GPS from IMU and lidar, but not from cameras Off by 100 ms = 2.5m What if video but no geometry? (Skybox) Voxel precision Image quality vs. geometry update rate (20 cm voxels? 10? 5?)

28 Questions? The End