Open Standards for Building Virtual and Augmented Realities. Neil Trevett Khronos President NVIDIA VP Developer Ecosystems

Transcription

1 Open Standards for Building Virtual and Augmented Realities Neil Trevett Khronos President NVIDIA VP Developer Ecosystems

2 Khronos Mission Asian Members Software Silicon Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing 2

3 Why Open Standards? An INTEROPERABILITY STANDARD enables two interoperating entities to COMMUNICATE E.g. Software <-> Hardware Bad Standards - Overprescribes implementation details - Forces everyone to implement a lowest common denominator - Stifles innovation -> Commoditization Good Standards - Prescribes only interoperability - Enables implementation diversity Encourages innovation -> Differentiation A Truly OPEN standard Is not controlled by a single company but by the whole industry Is freely available to use by any company without royalty payments Has well defined IP Framework to protect standard AND member s IP Standards Grow Markets By reducing consumer confusion and increasing capabilities and usability Reduce Costs By sharing development between many companies and driving volume Accelerate Time to Market With well-proven testing and interoperability 3

4 Khronos Standards for AR and Vision and sensor processing - including neural network inferencing for machine learning Download 3D object scene data High-performance, low-latency 3D Graphics Portable interaction with /AR sensor, haptic and display devices 4

5 Vulkan New Generation 3D Graphics Non-proprietary, royalty-free open standard By the industry for the industry Portable across multiple platforms both desktop and mobile -Specific Features NOW Multi-GPU support Multiview Rendering Context priority Front buffer rendering Evolving 3D API features for displays and compositors IN DISCUSSION Variable Rate Rendering Tiled rendering (beam racing) Modern architecture Low overhead Multi-thread friendly EXPLICIT GPU access for EFFICIENT, LOW-LATENCY, PREDICTABLE performance 5

6 XR = AR + V1.0 - focused on V A After 1.0 equal focus on AR 6

7 OpenXR Solving /AR Fragmentation App 1 AR App 2 App 3 AR App 4 App 1 AR App 2 App 3 AR App 4 WebXR Proprietary Engine WebXR Proprietary Engine Application Interface Integration Layer Before OpenXR XR Market Fragmentation After OpenXR Wide interoperability of XR apps and devices 7

8 OpenXR Architecture Portable AR/ Input Discovery Multiple Sensor Tracking Events Pose Normalization Haptics Control Optical Corrections 8

9 Companies Publicly Supporting OpenXR OpenXR is a collaborative design Integrating many lessons from proprietary first-generation API designs 9

10 Input and Haptics Abstraction Apps use abstracted Input Actions - E.g. Move, Jump, Teleport Events mapped to Input Actions - For the specific system in use Many advantages - Content can use new devices with no code changes - Can mix-and-match multiple input sources to create a unified UI - Easy optional feature support (e.g. eye and body tracking) - Future-proofing for innovation in input devices and form factors 10

11 OpenXR Viewport Configuration Flexibility Applications can: - Query for runtime supported Viewport Configurations - Applications can then set the Viewport Configurations that they plan to use - Select and change their active configuration over the lifetime of the session Camera Passthrough AR Stereoscopic / AR Projection CAVE-like One Viewport Two Viewports (one per eye) Twelve Viewports (six per eye) /viewport_configuration/ar_mono/magic_window /viewport_configuration/vr/hmd /viewport_configuration/vr_cube/cave_vr Photo Credit: Dave Pape 11

12 OpenXR Development Process Call for Participation / Exploratory Group Formation Fall F2F, October 2016: Korea Statement of Work / Working Group Formation Winter F2F, January 2017: Vancouver Specification Work Spring F2F, April 2017: Amsterdam Interim F2F, July 2017: Washington Defining the MVP Fall F2F, September 2017: Chicago Much more detailed specification overview and SIGGRAPH session videos: Present Day Coming Soon Resolving Implementation Issues Winterim F2F, November 2017: Washington Winter F2F, January 2018: Taipei First Public Information GDC, March 2018 First Public Demonstrations! SIGGRAPH, August 2018 Release Provisional Specification Multiple implementations Underway! Final specification will incorporate implementation feedback Conformance Tests and Adopters Program Feedback Finalize Implementations Ratify and release Final Specification and Enable Conformant Implementations to Ship 12

13 Epic Showdown Demo at SIGGRAPH Demo runs portably across Star and Microsoft Windows Mixed Reality headsets through the OpenXR APIs via an Unreal Engine 4 plugin 13

14 Bringing and AR to the Web Native XR Apps 3D Engines Future versions of OpenXR will include cross-platform extended AR functionality System-exposed AR Capabilities WebXR Web XR Apps 3D Engines Close ongoing collaboration between WebXR and OpenXR Khronos providing the foundation for 3D and XR in the Web and native applications 14

15 gltf The JPEG of 3D! gltf spec development on open GitHub get involved! Compact to Transmit Simple and Fast to Load Describes Full Scenes Runtime Neutral Open and Extensible Efficient, reliable transmission Bring 3D assets into 1000s of apps and engines gltf 1.0 December 2015 Primarily for WebGL Uses GLSL for materials gltf 2.0 June 2017 Native AND Web APIs Physically Based Rendering Metallic-Roughness and Specular-Glossiness 15

16 gltf Ecosystem Creation Tools Repositories Discover Sony 3D Creator Oculus Create Experience Windows Mixed Reality Home Modo Paint 3D Drive Demand Collada2gltf gltf-validator gltf-vscode FBX2glTF gltf-asset-generator Users 3D Builder Prep for 3D printing Mixed Reality Viewer Apps and Engines 16

17 gltf Roadmap gltf manages its roadmap very carefully complexity is the enemy - Mission #1: ensure widespread, consistent, reliable usage Rollout new functionality first as extensions: e.g. Draco Mesh Compression - Open source encoders and decoders are available - Extension is now shipping in more and more tools and engines New extensions are being developed - E.g. Universal Compressed Texture Transmission Format Stable Core Spec Domain-specific extensions stay as extensions New widely needed functionality ships first as extensions Mesh Compression Ratios Integrate extensions into new core spec only when: 1) Widespread need is confirmed by the industry 2) Widespread reliable implementation is enabled (e.g. open source) 17

18 Summary Khronos is creating cutting-edge royalty-free open standards for AR/ - For both native and Web applications Please join Khronos! - We welcome members from China and Asia - Influence the direction of industry standards - Get early access to draft specifications More Information ntrevett@nvidia.com WeChat: neiltrevett 18