Open Standards for AR and VR Neil Trevett Khronos President NVIDIA VP Developer January 2018

Transcription

1 Copyright Khronos Group Page 1 Open Standards for AR and Neil Trevett Khronos President NVIDIA VP Developer Ecosystem January 2018

2 Khronos Mission E.g. OpenGL ES provides 3D on almost every smartphone over 4 Billion units shipped Software Silicon Khronos is an International Industry Consortium of over 100 companies creating royalty-free, Open Standards to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing Copyright Khronos Group Page 2

3 Copyright Khronos Group Page 3 Khronos Principles of Organization Any company is welcome to join. One company one vote Strong IP Framework to enable ROYALTY-FREE specifications: all members agree not assert patents against conformant implementations Only invest where there is strong industry momentum to ensure industry relevance let Darwinism rule! Software Silicon Any member can propose new standards initiatives Conformance Tests and Adopters Programs for defining conformance, specification integrity and crossvendor portability Non-profit organization - Membership and Adopters fees cover operating, marketing and engineering expenses

4 Copyright Khronos Group Page 4 Why /AR Needs Acceleration API Standards Without API Standards With API Standards Platform Fragmentation Application Portability Everything runs on CPU Silicon Acceleration Standard Acceleration APIs provide PERFORMANCE, POWER AND PORTABILITY

5 Copyright Khronos Group Page 5 Native Acceleration APIs for /AR Vision sensor(s) Tracking and Positioning Geometric scene reconstruction Semantic scene understanding (Neural Networks) Download 3D augmentation object and scene data /AR Application Generate Low Latency 3D Content and Augmentations Interact with sensor, haptic and display devices

6 Copyright Khronos Group Page 6 Native Acceleration APIs for /AR Vision sensor(s) Tracking and Positioning Geometric scene reconstruction Semantic scene understanding (Neural Networks) Download 3D augmentation object and scene data /AR Application Generate Low Latency 3D Content and Augmentations Interact with sensor, haptic and display devices

7 Copyright Khronos Group Page 7 The Need for Cross-Platform 3D Standards Vulkan Provides Clean, modern architecture Low overhead, explicit GPU access Portable across desktop and mobile Multi-thread / multi-core friendly Primary 3D API on Android Platform Efficient, low-latency, predictable performance ideal for AR/ Access to multiple platforms Vulkan Portable Subset

8 Copyright Khronos Group Page 8 Pervasive Vulkan All Major GPU Companies shipping Vulkan Drivers for Desktop and Mobile Platforms Desktop, Mobile, Embedded and Console Platforms Supporting Vulkan Including phones and tablets from Google, Huawei, Samsung, Sony, Xiaomi - both premium and mid-range devices Desktop Android 7.0+ Nintendo Switch Android TV Embedded Linux Platforms Steam Gear Oculus Rift Google Daydream Game Engines

9 Copyright Khronos Group Page 9 XR = AR + V1.0 - focused on V A After 1.0 equal focus on AR Cross-Platform, Portable /AR DEVICE MANAGEMENT Device discovery Movement tracking Input and haptics Flexible graphics configuration

10 Copyright Khronos Group Page 10 OpenXR Solving AR/ Fragmentation App 1 App 2 App 3 App 4 Proprietary Engine Device Discovery Device Events Sensor tracking Haptics HMD Parameters App 1 App 2 App 3 App 4 Proprietary Engine Application Interface Device Layer Device 1 Device 2 Device 3 Device 4 Device 5 Installable Drivers Device 1 Device 2 Device 3 Device 4 Device 5 Before OpenXR Market Fragmentation After OpenXR Wide Interoperability of Apps and Devices

11 Copyright Khronos Group Page 11 OpenXR Working Group Members Design work started in December 2016 Typically months to develop a V1.0 specification

12 Copyright Khronos Group Page 12 OpenXR and Run-times a Win-Win Access to any OpenXR Application Proprietary APIs Proprietary Driver Interfaces Extensions Application Interface Run-time Extensions Device Layer Any successful standard encourages and enables healthy industry competition OpenXR will not replace run-times or outlaw existing interfaces OpenXR will simply provide cross-vendor APIs that can be exposed by a runtime to access more apps and devices Access to any OpenXR Device OpenXR for portable AR AND apps and devices with initial focus on

13 Copyright Khronos Group Page 13 Neural Network Workflow Vision/AI Applications Datasets Neural Network Architecture Neural Net Training Neural Net Training Neural Neural Frameworks Net Training Frameworks Frameworks Net Training Frameworks Trained Networks Vision and Neural Net Inferencing Runtime Desktop and Cloud Hardware GPU CPU CPU CPU CPU GPU cudnn MIOpen MKL-DNN Diverse Inferencing Acceleration Hardware FPGA Custom Hardware DSP

14 Copyright Khronos Group Page 14 NNEF - Solving Neural Net Fragmentation Before NNEF NN Training and Inferencing Fragmentation NN Authoring Framework 1 NN Authoring Framework 2 NN Authoring Framework 3 Every Tool Needs an Exporter to Every Accelerator Inference Engine 1 Inference Engine 2 Inference Engine 3 With NNEF- NN Training and Inferencing Interoperability NN Authoring Framework 1 NN Authoring Framework 2 Inference Engine 1 Inference Engine 2 NN Authoring Framework 3 Optimization and processing tools Inference Engine 3

15 Copyright Khronos Group Page 15 NNEF 1.0 Provisional Released as Provisional to get industry feedback before finalization - Welcoming comments and feedback on Khronos GitHub Initial focus on passing trained frameworks to embedded inference engines - Authoring interchange, importing NNEF into tools, is also an emerging use case Support deployable range of network topologies - Rapid evolution to encompass new network types as they emerge from research NNEF Working Group Participants

16 Copyright Khronos Group Page 16 NNEF Implementations and Roadmap Active NNEF roadmap development - Track development of new network types - Explicit support for authoring interchange - Support retraining with third party NNEF tools - Address an ever wider range of applications - Increase the expressive power of the format NNEF open source projects - Hosted on Khronos NNEF GitHub repository - Apache 2.0 license - More RFQ projects coming Neural Network Exchange Industry Initiatives TensorFlow and Caffe Exporters Defined Specification Open Source Project Stability for hardware deployment Software stack flexibility Files Syntax Parser/ Validator Multi-company Governance Flat and Compound Ops Initiated by Facebook Flat Ops Only

17 Copyright Khronos Group Page 17 OpenVX Power Efficiency Wide range of vision hardware architectures OpenVX provides a high-level Graph-based abstraction -> Enables Graph-level optimizations! Can be implemented on almost any hardware or processor! -> Portable, Efficient Vision Processing! X100 X10 X1 Dedicated Hardware Vision DSPs GPU Compute Multi-core CPU Computation Flexibility Vision Node Vision Engines GPU Vision Node Vision Node Middleware DSP Vision Processing Graph Applications Hardware Vision Node Software Portability

18 Copyright Khronos Group Page 18 OpenVX - Graph-Level Abstraction OpenVX developers express a graph of image operations ( Nodes ) - Using a C API Nodes can be executed on any hardware or processor coded in any language - Implementers can optimize under the high-level graph abstraction Graphs are the key to run-time power and performance optimizations - E.g. Node fusion, tiled graph processing for cache efficiency etc. Camera Input OpenVX Nodes Pyr t Rendering Output RGB Frame Color Conversion YUV Frame Channel Extract Gray Frame Image Pyramid Optical Flow Array of Keypoints Harris Track OpenVX Graph Feature Extraction Example Graph Ftr t-1 Array of Features

19 Copyright Khronos Group Page 19 OpenVX Evolution Conformant Implementations Conformant Implementations OpenVX 1.0 Spec released October 2014 New Functionality Expanded Nodes Functionality Enhanced Graph Framework AMD OpenVX Tools - Open source, highly optimized for x86 CPU and OpenCL for GPU - Graph Optimizer - Scripting for rapid prototyping, without re-compiling, at production performance levels OpenVX 1.1 Spec released May 2016 New Functionality Conditional node execution Feature detection Classification operators Expanded imaging operations Extensions Neural Network Acceleration Graph Save and Restore 16-bit image operation Safety Critical OpenVX 1.1 SC for safety-certifiable systems OpenVX 1.2 Spec released May 2017 Adopters Program November 2017 OpenVX Roadmap under Discussion NNEF Import Programmable user kernels with accelerator offload

20 Copyright Khronos Group Page 20 OpenVX Neural Net Extension and NNEF Convolution Neural Network topologies can be represented as OpenVX graphs - Layers are represented as OpenVX nodes - Layers connected by multi-dimensional tensors objects - Layer types include convolution, activation, pooling, fully-connected, soft-max - CNN nodes can be mixed with traditional vision nodes Import/Export Extension - Efficient handling of network Weights/Biases or complete networks OpenVX will be able to import NNEF files into OpenVX Neural Nets Native Camera Control Vision Node Vision Node CNN Nodes Vision Node Downstream Application Processing An OpenVX graph mixing CNN nodes with traditional vision nodes

21 Copyright Khronos Group Page 21 gltf Cross-Platform 3D Asset Transmission All gltf spec development on open GitHub: OpenGL Transmission Format Efficient transmission of 3D scenes and assets gltf 1.0 Primarily for WebGL Uses GLSL for materials Released December 2015 gltf Physically Based Rendering! Metallic-Roughness and Specular-Glossiness Materials Rendering API independence Released June 2017

22 Copyright Khronos Group Page 22 COLLADA and gltf 3D Asset Formats COLLADA FOR AUTHORING INTERCHANGE Retains extensive data to enable editable assets to be passed between authoring tools gltf FOR RUN-TIME TRANSMISSION Compact file size and efficient processing/import More tools Collada2gltf Translator COLLADA is intended to be used for intermediate interchange, gltf is designed for run-time delivery gltf can be generated from intermediate formats such as COLLADA, or exported directly from authoring tools to fit individual use cases Many More Apps and Engines Based on any 3D API

23 Copyright Khronos Group Page 23 Strong gltf Industry Support Please let us know if your company logo is not here and you would like it added! Publicly Stated Support for gltf

24 Copyright Khronos Group Page 24 Google Draco gltf Extension Coming Soon Library for compressing and decompressing 3D geometric meshes and point clouds - Google has released Draco encoders and decoders in open source - C++ source code encoder to compress 3D data - C++ and JavaScript decoders for the encoded data Draco gltf extension is in progress and ready for feedback! - Draco designed and built for compression efficiency and speed - great fit with gltf! - Typical Draco compression ratios

25 Copyright Khronos Group Page 25 APIs and File Formats Runtime acceleration APIs typically require a file format to complete their deployment ecosystem 3D Graphics Acceleration 3D Asset Transmission Programmable Parallel Programming Graphics and parallel processing intermediate code format Vision and Inferencing Acceleration Neural Network Exchange Format

26 Copyright Khronos Group Page 26 Layered Software Ecosystems for AR and In Discussion!! Application Native AR Apps 3D Web Apps Web Apps AR Web Apps Middleware Browser Native WebXR (Mozilla) High-level AR APIs similar to proprietary platforms: Android Tango, Apple ARKit, Google ARCore WebVX? General purpose, graphbased vision and Neural Net processing for the Web enables innovation IN the Web Stack GPU Compute for Vision and Machine Learning + 3D Rendering

27 Copyright Khronos Group Page 27 Safety Critical APIs OpenGL SC Fixed function graphics subset Experience and Guidelines OpenGL SC April 2016 Shader programmable pipeline subset New Generation APIs for safety certifiable vision, graphics and compute e.g. ISO and DO-178B/C OpenGL ES Fixed function graphics OpenGL ES Shader programmable pipeline OpenVX SC 1.1 Released May 2017 Restricted deployment implementation executes on the target hardware by reading the binary format and executing the pre-compiled graphs Khronos SCAP Safety Critical Advisory Panel Guidelines for designing APIs that ease system certification. Open to Khronos member AND industry experts OpenCL SC TSG Formed Working on OpenCL SC 1.2 Eliminate Undefined Behavior Eliminate Callback Functions Static Pool of Event Objects

28 Copyright Khronos Group Page 28 Need for Camera Control API? Advanced control of ISP and camera subsystem with cross-platform portability - Generate sophisticated image stream for advanced imaging & vision apps No platform API currently fulfills all developer requirements - Portable access to growing sensor diversity: e.g. depth sensors and sensor arrays - Cross sensor synch: e.g. synch of camera and MEMS sensors - Advanced, high-frequency per-frame burst control of camera/sensor: e.g. ROI - Multiple input, output re-circulating streams with RAW, Bayer or YUV Processing Defines control of Sensor, Color Filter Array Lens, Flash, Focus, Aperture Auto Exposure (AE) Auto White Balance (AWB) Auto Focus (AF) Image Signal Processor (ISP) Stream of Images for Vision Processing OpenKCAM standard is currently on ice do we need to restart?

29 Copyright Khronos Group Page 29 Opportunities for Cooperation All Khronos standards are open and royalty-free - Usually also backed with open source tests, tools and implementations Any company or organization is welcome to join Khronos - Membership fees start at $3,500 for a voice and a vote in any of these standards Khronos welcomes cooperative projects and liaison with ETSI/ARF for /AR! - We hope Khronos standards can help you reach your objectives - Khronos liaison models could enable direct participation by ETSI/ARF representatives Neil Trevett - ntrevett@nvidia.com -