Overview and AR/VR Roadmap

Size: px

Start display at page:

Download "Overview and AR/VR Roadmap"

Jewel Campbell
5 years ago
Views:

1 Khronos Group Inc Page 1 Overview and AR/ Roadmap Neil Trevett Khronos President NVIDIA VP Developer Ecosystems

International Industry Consortium creating royalty-free, open standards to enable

2 Khronos Group Inc Page 2 Khronos Connects Software to Silicon Software Silicon Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing

Khronos Group Inc. 2018 - Page 3 What is An Open Standard?

Software <-> Hardware Bad Standards - Overprescribes implementation

Commoditization Good Standards - Prescribes ONLY interoperability -

A Truly OPEN standard Is not controlled by a single company but by the

payments Has a well defined IP Framework to protect standard AND member s

capabilities and usability Reduce Costs By sharing development between

3 Khronos Group Inc Page 3 What is An Open Standard? An INTEROPERABILITY STANDARD enables two entities to COMMUNICATE. E.g. Software <-> Hardware Bad Standards - Overprescribes implementation details - Forces lowest common denominator - Stifles innovation -> Commoditization Good Standards - Prescribes ONLY interoperability - Enables implementation diversity Encourages innovation -> Differentiation A Truly OPEN standard Is not controlled by a single company but by the whole industry Is freely available to use by any company without royalty payments Has a well defined IP Framework to protect standard AND member s IP Standards Grow Markets By reducing consumer confusion and increasing capabilities and usability Reduce Costs By sharing development between many companies and driving volume Accelerate Time to Market With well-proven functionality, testing and interoperability

Khronos Group Inc. 2018 - Page 4 The Need for AR/ Standards Vertically Integrated One-off Solutions in 1990s High-volume Consumer Platforms Today!

Consumer platforms have insatiable need for content to drive demand - can t afford silo d content!

4 Khronos Group Inc Page 4 The Need for AR/ Standards Vertically Integrated One-off Solutions in 1990s High-volume Consumer Platforms Today! Interoperability standards so content can run on all devices! Compete on adding user value not restricting user choice! Consumer platforms have insatiable need for content to drive demand - can t afford silo d content! Consumer confidence will my device run the content I want? Avoid Betamax vs. VHS Syndrome! /AR will build on a constellation of standards HMD Connectivity Wireless and 5G Verticals and Security Video Formats Acceleration APIs XR in the Browser

including neural network inferencing for machine learning Download 3D

5 Khronos Group Inc Page 5 Standards for AR and Vision and sensor processing - including neural network inferencing for machine learning Download 3D object and scene data High-performance, low-latency 3D Graphics Portable interaction with /AR sensor, haptic and display devices

6 Khronos Group Inc Page 6 Vulkan and New Generation GPU APIs Modern architecture Low overhead Multi-thread friendly EXPLICIT GPU access for EFFICIENT, LOW-LATENCY, PREDICTABLE performance Vulkan Porting Tools Non-proprietary, royalty-free open standard By the industry for the industry Portable across multiple platforms - desktop and mobile

7 Khronos Group Inc Page 7 Vulkan Now Available Everywhere -Related Features NOW Multi-GPU support Multiview Rendering Context priority Front buffer rendering IN DISCUSSION Variable Rate Rendering Tiled rendering (beam racing)

8 Khronos Group Inc Page 8 The Metaverse will be the Web!

9 Khronos Group Inc Page 9 WebGL Stack Content downloaded from the Web Content JavaScript, HTML, CSS,... Middleware provides accessibility for non-expert programmers E.g. three.js library JavaScript Middleware Low-level WebGL API provides a powerful foundation for a rich JavaScript middleware ecosystem Browser provides WebGL 3D engine alongside other HTML5 technologies - no plug-in required OS Provided Drivers WebGL uses native OpenGL or OpenGL ES or Angle = OpenGL ES over DX JavaScript CSS HTML5 Reliable WebGL relies on work by both GPU and Browser Vendors -> Khronos has the right membership to enable that cooperation

Pervasive, portable access to OpenGL ES 2.

10 Khronos Group Inc Page 10 WebGL Deployment WebGL 2.0 is Here! Pervasive, portable access to OpenGL ES 2.0-class mobile graphics WebGL % Globally WebGL 2.0 OpenGL ES 3.0-based desktopclass graphics comes to the Web! 65.77% Globally WebGL 2.0 will reach WebGL 1.0 levels of availability when Safari and Edge ship enabling a new class of Web-based AR/ 3D Apps!

11 Khronos Group Inc Page 11 Khronos Standards for AR and Vision and sensor processing - including neural network inferencing for machine learning Download 3D object and scene data High-performance, low-latency 3D Graphics Portable interaction with /AR sensor, haptic and display devices

Khronos Group Inc. 2018 - Page 12 gltf The JPEG of 3D! gltf spec development on open GitHub get involved! https://github.

Efficient, reliable transmission Bring 3D assets into 1000s of apps and engines gltf 1.

12 Khronos Group Inc Page 12 gltf The JPEG of 3D! gltf spec development on open GitHub get involved! Compact to Transmit Simple and Fast to Load Describes Full Scenes Runtime Neutral Open and Extensible Efficient, reliable transmission Bring 3D assets into 1000s of apps and engines gltf 1.0 December 2015 Primarily for WebGL Uses GLSL for materials gltf 2.0 June 2017 Native AND Web APIs Physically Based Rendering Metallic-Roughness and Specular-Glossiness

gltf Ecosystem Creation Tools Repositories Discover Sony 3D Creator Oculus Create Experience Windows Mixed Reality Home Modo Paint 3D Drive Demand Collada2gltf

13 gltf Ecosystem Creation Tools Repositories Discover Sony 3D Creator Oculus Create Experience Windows Mixed Reality Home Modo Paint 3D Drive Demand Collada2gltf gltf-validator gltf-vscode FBX2glTF gltf-asset-generator Users 3D Builder Prep for 3D printing Mixed Reality Viewer Apps and Engines Khronos Group Inc Page 13

14 Khronos Group Inc Page 14 gltf Roadmap gltf manages its roadmap very carefully complexity is the enemy - Mission #1: ensure widespread, consistent, reliable usage Optional Draco Mesh Compression Extension Shipping Now - Adoption enabled with open source encoders and decoders Extensions and projects in discussion - Draco compressed Point clouds, Universal compressed textures, LODs and streaming - Nexgen PBR materials, Advanced animations (e.g. Avatars and Face emoji), Metadata Developing reference viewer for visual consistency across web and native engines Stable Core Spec Domain-specific extensions stay as extensions New widely needed functionality ships first as extensions Mesh Compression Ratios Integrate extensions into new core spec only when: 1) Widespread need is confirmed by the industry 2) Widespread reliable implementation is enabled (e.g. open source)

15 Khronos Group Inc Page 15 Khronos Standards for AR and Vision and sensor processing - including neural network inferencing for machine learning Download 3D object and scene data High-performance, low-latency 3D Graphics Portable interaction with /AR sensor, haptic and display devices

16 Khronos Group Inc Page 16 Machine Learning Acceleration Training on Desktop and Cloud Deployment on Embedded Devices Training Data Sets Neural Neural Neural Net Net Training Training Training Frameworks Frameworks Net Training Frameworks Frameworks Trained Networks Optimization Vision and Inferencing Applications OR Vision and Inferencing Runtimes Live Data Desktop and Cloud GPU/TPU Acceleration Diverse Inferencing Acceleration Hardware

17 Khronos Group Inc Page 17 OpenVX Vision and Inferencing Acceleration Wide range of vision and inferencing hardware architectures so OpenVX provides a high-level Graph-based abstraction for portability -> Enables Graph-level optimizations! Can be implemented on almost any hardware or processor! -> Portable, Efficient Vision and Inference Processing! Shipping Implementations Native Camera Control Vision Node Vision Node CNN Nodes Vision Node Downstream Application Processing NNEF Translator converts NNEF representation into OpenVX Node Graphs An OpenVX graph mixing CNN nodes with traditional vision nodes

18 Khronos Group Inc Page 18 OpenCL Unique Heterogeneous Runtime OpenCL is the only industry standard for low-level heterogeneous compute Portable control over memory and parallel task execution The closest you can be to your embedded accelerator and still be portable Application or Inferencing Run-time Application or Inferencing Run-time GPU APIs Growing number of optimized OpenCL vision and inferencing libraries Vision: OpenCV, Halide, Visioncpp Machine Learning: Xiaomi MACE, Arm Compute Library Linear Algebra: cldnn, clblast, ViennaCL GPU CPU FPGA GPU GPU DSP Custom Hardware

19 Khronos Group Inc Page 19 Khronos Standards for AR and Vision and sensor processing - including neural network inferencing for machine learning Download 3D object and scene data High-performance, low-latency 3D Graphics Portable interaction with /AR sensor, haptic and display devices

Khronos Group Inc. 2018 - Page 20 XR = AR + V1.

20 Khronos Group Inc Page 20 XR = AR + V1.0 - focused on V A After 1.0 equal focus on AR

21 Khronos Group Inc Page 21 OpenXR Solving /AR Fragmentation App 1 AR App 2 App 3 AR App 4 App 1 AR App 2 App 3 AR App 4 WebXR Proprietary Engine WebXR Proprietary Engine Application Interface Device Integration Layer Device 1 Device 2 Device 3 Device 4 Device 5 Device 1 Device 2 Device 3 Device 4 Device 5 Before OpenXR XR Market Fragmentation After OpenXR Wide interoperability of XR apps and devices

Multiple Sensor Tracking Device Events Pose Normalization Haptics Control

22 Khronos Group Inc Page 22 OpenXR Architecture Portable AR/ Input Device Discovery Multiple Sensor Tracking Device Events Pose Normalization Haptics Control Optical Corrections OpenXR doesn t replace AR/ runtimes! It enables those runtimes to use PORTABLE APIs to expose their functionality

23 Khronos Group Inc Page 23 Companies Publicly Supporting OpenXR is a collaborative design Integrating many lessons from proprietary first-generation API designs

24 Khronos Group Inc Page 24 Epic Showdown Demo at SIGGRAPH Demo runs portably across Star and Microsoft Windows Mixed Reality headsets through the OpenXR APIs via an Unreal Engine 4 plugin

25 Khronos Group Inc Page 25 Mobile Augmented Reality Libraries Encapsulated Vision-based Functionality Also leveraging motion sensors ARKit Pose Tracking Yes Yes Plane Detection and Tracking (Hori. & Vert.) Yes Yes Image Recognition and Tracking Yes Yes 3D Object Scanning, Recognition and Tracking Yes No Ambient light level and temperature Yes Yes Environment Light Probes Yes No Environment Reflective Texturing Yes No Link to Neural Net-based Object Detection Yes Yes Access to Point Cloud Yes Yes Camera Intrinsics Yes Yes 3 rd party AR Libraries typically use ARKit/ARCore if available or implement own tracking and functionality if not Multi-user and persistent cloud-based anchors Yes Yes Multi-user Viewing Yes No Face tracking (iphone X) Yes No OS Availability ios Android and ios

26 Khronos Group Inc Page 26 Bringing and AR to the Web Native XR Apps 3D Engines Future versions of OpenXR will include cross-platform extended AR functionality System-exposed AR Capabilities WebXR Web XR Apps 3D Engines Close ongoing collaboration between WebXR and OpenXR Khronos providing the foundation for 3D and XR in the Web and native stacks

27 Khronos Group Inc Page 27 Summary Khronos is creating cutting-edge royalty-free open standards - For 3D, /AR, Compute, Vision and machine learning Join Khronos any company is welcome! - Influence the direction of industry standards - Get early access to draft specifications More Information ntrevett@nvidia.com

Open Standards for Building Virtual and Augmented Realities. Neil Trevett Khronos President NVIDIA VP Developer Ecosystems

Open Standards for Building Virtual and Augmented Realities Neil Trevett Khronos President NVIDIA VP Developer Ecosystems Khronos Mission Asian Members Software Silicon Khronos is an International Industry