Copyright Khronos Group Page 1

Transcription

1 Open Standards and Open Source Together How Khronos APIs Accelerate Fast and Cool Applications Neil Trevett Khronos President NVIDIA Vice President Mobile Ecosystem Copyright Khronos Group Page 1

2 Copyright Khronos Group Page 2 Khronos Connects Software to Silicon Industry Consortium creating OPEN STANDARD APIs for hardware acceleration Any company is welcome one company one vote ROYALTY-FREE specifications State-of-the art IP framework protects members AND the standards Software Conformance Tests and Adopters Programs for specification integrity and cross-vendor portability Low-level silicon APIs needed on almost every platform: graphics, parallel compute, rich media, vision, sensor and camera processing Silicon International, non-profit organization Membership and Adopters fees cover operating and engineering expenses Strong industry momentum 100s of man years invested by industry experts Well over a BILLION people use Khronos APIs Every Day

3 Copyright Khronos Group Page 3 BOARD OF PROMOTERS Over 100 members worldwide any company is welcome to join

4 Copyright Khronos Group Page 4

5 Copyright Khronos Group Page 5 Khronos Cooperative Framework API Working Groups (Industry and Academic members) $ $ Conformance Tests and Adopters Program Royalty-Free Specifications Documentation, Tools SDKs, Code Samples, Educator Guidelines Courseware Materials Members Wider Community Adopters Build conformant implementation and products Developers Develop applications using the APIs Educators / Certifiers Create Courses Training and Certification

6 Copyright Khronos Group Page 6 Khronos Relevance to Linaro Mobile Enterprise Digital Home and IOT 3D Graphics Heterogeneous Parallel Computation Low Power Vision Acceleration High efficiency graphics and compute Increased use of open source as a tool to build new generation ecosystems

7 OpenGL ES Fixed function Pipeline Programmable Vertex and fragment shaders 32-bit integers and floats NPOT, 3D/depth textures Texture arrays Multiple Render Targets Compute Shaders Tessellation and geometry shaders ASTC Texture Compression Floating point render targets Debug and robustness for security Epic s Rivalry demo using full Unreal Engine Driver Update Silicon Update Silicon Update Driver Update Android L Silicon Update AEP (Android Extension Pack) OpenGL ES 2.0 OpenGL ES 3.x The most widely deployed 3D graphics API in history Industry will ship >1.7 billion OpenGL ES-enabled devices in Copyright Khronos Group Page 7

8 Power Efficiency Copyright Khronos Group Page 8 OpenVX Vision Acceleration Targeted at low-power, real-time applications - Mobile and embedded platforms Portability across diverse heterogeneous processors - Multi-core CPUs, GPUs, DSPs and DSP arrays ISPs, Dedicated hardware Doesn t require high-power CPU/GPU Complex - Low-power host can setup and manage frame-rate vision processing pipeline X100 X10 X1 Dedicated Hardware Vision DSPs Vision Processing Efficiency GPU Compute Multi-core CPU Computation Flexibility Application Accelerator Application Application Accelerator Application Accelerator Accelerator OpenVX extends easily re-usable vision acceleration to very low power domains

9 Copyright Khronos Group Page 9 OpenVX Graphs The Key to Efficiency OpenVX developers express a graph of image operations ( Nodes ) - Nodes can be on any hardware or processor coded in any language - E.g. on GPU nodes may implemented in OpenCL Graph enables implementations to optimize for power and performance - E.g. Nodes may be fused by the implementation to eliminate memory transfers - E.g. Processing can be tiled to keep data entirely in local memory/cache Minimizes host interaction during frame-rate graph execution - Host processor can setup graph which can then execute almost autonomously Example OpenVX Graph Native Camera Control OpenVX Node OpenVX Node OpenVX Node OpenVX Node Downstream Application Processing

10 Copyright Khronos Group Page 10 OpenVX and OpenCV are Complementary Implementation Conformance Consistency Scope Efficiency Typical Use Case Embedded Deployment Community driven open source library Extensive OpenCV Test Suite but no formal Adopters program Available functions can vary depending on implementation / platform Very wide 1000s of imaging and vision functions Multiple camera APIs/interfaces Memory-based architecture Each operation reads and writes to memory Rapid experimentation and prototyping - especially on desktop Re-usable code Open standard API designed to be implemented by hardware vendors Implementations must pass defined conformance test suite to use trademark All core functions must be available in all conformant implementations Tight focus on core hardware accelerated functions for mobile vision but extensible Uses external/native camera API Graph-based execution Optimizable computation and data transfer Production development & deployment on mobile and embedded devices Callable library

11 Copyright Khronos Group Page 11 OpenVX Status Finalized OpenVX 1.0 specification released October OpenVX spec maintenance update released June Khronos open source sample implementation of OpenVX 1.0 released - Full conformance test suite and Adopters Program available - Test suite exercises graph framework and functionality of each OpenVX 1.0 node Commercial conformant products - NVIDIA, Synopsis, Vivante and many more coming

12 OpenCL Portable Heterogeneous Computing OpenCL = Two APIs and Two Kernel languages - C Platform Layer API to query, select and initialize compute devices - OpenCL C and OpenCL C++ kernel languages to write parallel code - C Runtime API to build and execute kernels across multiple devices OpenCL Kernel OpenCL Code Kernel OpenCL Code Kernel OpenCL Code Kernel Code Kernel code compiled for devices GPU DSP FPGA HW CPU CPU Runtime API loads and executes kernels across devices CPU Host Devices Copyright Khronos Group Page 12

13 Copyright Khronos Group Page 13 OpenCL 2.1 Provisional March 2015 New OpenCL C++ kernel language based on a subset of C Significantly enhanced programmer productivity and code performance Support for the new Khronos SPIR-V intermediate language in core - SPIR-V used to ingest from C++ front-end no C++ compiler in driver - OpenCL C ingestion still supported to preserve kernel code investment Runs on any OpenCL 2.0-capable hardware - Only driver update required 3-component vectors Additional image formats Multiple hosts and devices Buffer region operations Enhanced event-driven execution Additional OpenCL C built-ins Improved OpenGL data/event interop Device partitioning Separate compilation and linking Enhanced image support Built-in kernels / custom devices Enhanced DX and OpenGL Interop Shared Virtual Memory On-device dispatch Generic Address Space Enhanced Image Support C11 Atomics Pipes Android ICD OpenCL C++ Shading language SPIR-V in Core Subgroups into core Subgroup query operations clclonekernel Low-latency device timer queries Dec08 OpenCL 1.0 Specification Jun10 OpenCL 1.1 Specification Nov11 OpenCL 1.2 Specification 18 months 18 months 24 months Nov13 OpenCL 2.0 Specification 16 months Mar15 OpenCL 2.1 Specification (Provisional)

14 OpenCL Implementations 1.0 May Jul Jun Aug Aug May Dec May Feb Mar Dec Jul14 Desktop 1.0 May Jun May Aug Feb Nov Sep Apr14 Mobile 1.1 Apr Dec Sep Jan May13 Embedded 1.2 May Jul13 FPGA 1.0 Dec Aug15 Vendor timelines are first implementation of each spec generation Dec08 OpenCL 1.0 Specification Jun10 OpenCL 1.1 Specification Nov11 OpenCL 1.2 Specification Nov13 OpenCL 2.0 Specification Mar15 OpenCL 2.1 Specification Copyright Khronos Group Page 14

15 Copyright Khronos Group Page 15 OpenCL as Parallel Language Backend JavaScript binding for initiation of OpenCL C kernels Language for image processing and computational photography MulticoreWare open source project on Bitbucket Single Source C++ Programming for OpenCL Java language extensions for parallelism River Trail Language extensions to JavaScript Compiler directives for Fortran, C and C++ PyOpenCL Python wrapper around OpenCL Harlan High level language for GPU programming Approaching 200 languages, frameworks and projects using OpenCL as a compiler target to access vendor optimized, heterogeneous compute runtimes

16 Copyright Khronos Group Page 16 SPIR-V Transforms the Language Ecosystem First multi-api, intermediate language for parallel compute and graphics - Native representation for Vulkan shader and OpenCL kernel source languages - Cross vendor intermediate representation - Language front-ends can easily access multiple hardware run-times - Acceleration hardware can leverage multiple language front-ends - Encourages tools for program analysis and optimization in SPIR form Multiple Developer Advantages Same front-end compiler for multiple platforms Reduces runtime kernel compilation time Don t have to ship shader/kernel source code Drivers are simpler and more reliable Tools for analysis and optimization Diverse Languages and Frameworks Hardware runtimes on multiple architectures Standard Portable Intermediate Representation

17 Copyright Khronos Group Page 17 Driving the SPIR-V Open Source Ecosystem Khronos will open source these tools and translators GLSL Third party kernel and shader Languages OpenCL C OpenCL C++ SPIR-V Tools SPIR-V Validator SPIR-V (Dis)Assembler LLVM LLVM to SPIR-V Bi-directional Translator Other Intermediate Forms SPIR-V 32-bit Word Stream Extensible and easily parsed Retains data object and control flow information for effective code generation and translation IHV Driver Runtimes

18 Copyright Khronos Group Page 18 SPIR-V Open Source Community Activity Python byte code to SPIR-V Convertor - Write shaders or kernels in Python, Encode and decode SPIR-V in Python - Dis(Assembler) with high level human readable assembler syntax.net IL to SPIR-V Convertor - Write and debug shaders or kernels using C#, SPIR-V interpreter Shade SPIR-V virtual machine - Test and debug SPIR-V binaries for binary correctness in human readable format Otherside SPIR-V virtual machine - Academic software rasterizer project to produce C code from SPIR-V Rust (Dis)Assembler - Encode and decode SPIR-V binaries in Rust Go (Dis)Assembler - Encode and decode SPIR-V in Go, SPIR-V represented in Go data structures Haskell EDSL - SPIR-V like language embedded in Haskell with significantly relaxed layout constraints Lisp SPIR-V Specification - Lisp readable SPIR-V specification JSON SPIR-V specification - Conversion of HTML SPIR-V specification to JSON format This is just the start.

19 Next Generation GPU APIs Only Windows 10 Only Apple Cross Platform Vulkan Target Availability Vulkan Committed Platform Adoption Copyright Khronos Group Page 19

20 Driver Copyright Khronos Group Page 20 Vulkan Explicit GPU Control Complex drivers lead to driver overhead and cross vendor unpredictability Error management is always active Driver processes full shading language source Application Traditional graphics drivers include significant context, memory and error management Application responsible for memory allocation and thread management to generate command buffers Direct GPU Control Driver Simpler drivers for low-overhead efficiency and cross vendor consistency Layered architecture so validation and debug layers can be unloaded when not needed Run-time only has to ingest SPIR-V intermediate language Separate APIs for desktop and mobile markets GPU GPU Unified API for mobile, desktop, console and embedded platforms Vulkan delivers the maximized performance and cross platform portability needed by sophisticated engines, middleware and apps

21 Copyright Khronos Group Page 21 Vulkan Multi-threading Efficiency CPU Thread Command Buffer 1. Multiple threads can construct Command Buffers in parallel Application is responsible for thread management and synch Command Buffer CPU Thread CPU Thread CPU Thread Command Buffer Command Queue GPU Command Buffer CPU Thread 2. Command Buffers placed in Command Queue by separate submission thread CPU Thread Command Buffer Can create graphics, compute and DMA command buffers with a general queue model that can be extended to more heterogeneous processing in the future

22 The Power of a Three Layer Ecosystem Applications can use Vulkan directly for maximum flexibility and control Application uses utility libraries to speed development Utility libraries and layers Application Games Engines fully optimized over Vulkan The industry s leading games and engine vendors are participating in the Vulkan working group Developers can choose at which level to use the Vulkan Ecosystem Rich Area for Innovation Many utilities and layers will be in open source Layers to ease transition from OpenGL Domain specific flexibility The same ecosystem dynamic as OpenCL A widely pervasive, powerful, flexible foundation layer enables diverse middleware tools and libraries What open source layers and libraries does Linaro community need? Copyright Khronos Group Page 22

23 Copyright Khronos Group Page 23 Vulkan Tools Architecture What tools would Linaro developers value most? Layered design for cross-vendor tools innovation and flexibility - IHVs plug into a common, extensible architecture for code validation, debugging and profiling during development without impacting production performance Khronos Open Source Loader enables use of tools layers during debug - Finds and load drivers, dispatches API calls to correct driver and layers Tools layers can be open sourced Khronos, Vendors and Community - Valve/LunarG is helping drive first round tools mainly for desktop Production Path (Performance) Vulkan-based Title Debug Layers can be installed during Development Interactive Debugger Validation Layers Vulkan s Common Loader Debug Layers IHV s Installable Client Driver Debug information via standardized API calls

24 Copyright Khronos Group Page 24 Vulkan Feature Sets Vulkan supports hardware with a wide range of hardware capabilities - Mobile OpenGL ES 3.1 up to desktop OpenGL 4.5 and beyond One unified API framework for desktop, mobile, console, and embedded - No "Vulkan ES" or "Vulkan Desktop" Vulkan precisely defines a set of "fine-grained features" - Features are specifically enabled at device creation time (similar to extensions) Platform owners define a Feature Set for their platform - Vulkan provides the mechanism but does not mandate policy - Khronos will define Feature Sets for platforms where owner is not engaged Cab define and maintain Feature Sets definitions for its platforms

25 Copyright Khronos Group Page 25 Vulkan Window System Integration (WSI) Explicit control for acquisition and presentation of images - Designed to fit the Vulkan API and today s compositing window systems - Cleanly separates device creation from window system Platform provides an array of persistent presentable images = Vulkan Swapchain - Device exposes which queues support presentation - Application explicitly controls which image to render and present Standardized extensions - unified API for multiple window systems - Works across Android, Mir, Windows (Vista and up), Wayland and X (with DRI3) - Platforms can extend functionality, define custom WSI stack, or have no display at all Linaro Windows System integration requirements? VkQueue Transition to Present Present Transition to Render Transition to Present Present Transition to Render Time Swapchain Extensions Platform WSI Extension Explicit control for acquisition and presentation of images Platform WSI Extension Custom WSI Extension Image X Image Y Image Y Compositor or Display Engine Image X

26 Copyright Khronos Group Page 26 Vulkan Status Rapid progress since project start in June Significant proposals and IP contributions received from members Participants come from all segments of the graphics industry - Including an unprecedented level of participation from game engine ISVs Initial specs and implementations expected this year - Will work on any GPU hardware that supports OpenGL ES 3.1/OpenGL 4.X and up - Can ship on any OS including Windows XP/7/8/10 Working Group Participants

27 Copyright Khronos Group Page 27 Developing Ecosystem and Spec in Parallel Open sourcing Vulkan test suite to enable developer feedback and contributions Khronos supplied open source loader and layered tools architecture Open source layered tools Flexible Windows System Integration working with platform vendors Example code, documentation and course notes SPIR-V for language innovation Opportunity to leverage open source Vulkan conformance tests with LAVA?

28 Safety Critical Working Group 2005 OpenGL SC 1.0 Fixed function graphics subset 2016 (planned) OpenGL SC 2.0 Programmable shader pipeline subset New Generation API for safety certifiable graphics AND compute Many future safety critical use cases involve vision and compute acceleration (e.g. neural nets) 2003 OpenGL ES 1.0 Fixed function graphics 2007 OpenGL ES 2/3 Programmable shader pipeline Linaro security and certification requirements? Copyright Khronos Group Page 28

29 Copyright Khronos Group Page 29 Khronos Open Standards for Graphics and Compute 1990 s Workhorse cross-platform professional 3D apps & gaming LATEST STATUS New Extensions to enable latest desktop graphics capabilities 2000 s Ubiquitous mobile gaming & graphics apps OpenGL ES 3.2 released to bring AEP functionality to core 2005 Safety Critical Graphics New Safety Critical Working Group Call for Participation 2008 Heterogeneous parallel compute OpenCL 2.0 specification update and C++ Headers released Portable intermediate representation for graphics and parallel compute 2014 Provisional Spec Update and significant open source activity High-efficiency GPU graphics and compute for performance critical apps 2015 Adopted by Android and other platforms. Building ecosystem

30 Copyright Khronos Group Page 30 Opportunities for Closer Cooperation! Khronos invites Linaro to Define Vulkan feature sets for Linaro platforms Guide Linaro platform Vulkan Windows System Integration Leverage open source Vulkan conformance tests for LAVA Joint language and framework innovation using LLVM and SPIR-V Drive open source Vulkan layers and tools for Linaro use cases Supply requirements for API security and safety certifications Any company is welcome to join Khronos - $15K annual membership fee for access to all Khronos API working groups More Information -