Copyright Khronos Group Page 1

Transcription

1 Update on Khronos Standards for Vision and Machine Learning December 2017 Neil Trevett Khronos President NVIDIA VP Developer Copyright Khronos Group Page 1

2 Khronos Mission Software Silicon Khronos is an International Industry Consortium of over 100 companies creating royalty-free, Open Standard APIs to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing Copyright Khronos Group Page 2

3 Khronos Open Standards Tracking and Positioning Vision sensor(s) Download 3D augmentation object and scene data Geometric scene reconstruction Semantic scene understanding (Neural Networks) Generate Low Latency 3D Content and Augmentations VR/AR Application Interact with sensor, haptic and display devices Copyright Khronos Group Page 3

4 Standards for Vision and Neural Networks Vision/AI Applications NeuralNet NetTraining Training Neural Neural Net Training Frameworks NeuralFrameworks Net Training Frameworks Frameworks Desktop and Cloud Hardware Trained Networks Vision and Neural Net Inferencing Runtime Embedded/Mobile Embedded/Mobile Embedded/Mobile Vision/Inferencing Hardware Embedded/Mobile Vision/Inferencing Hardware Vision/InferencingHardware Hardware Vision/Inferencing Copyright Khronos Group Page 4

5 Copyright Khronos Group Page 5 NNEF - Solving Neural Net Fragmentation Before NNEF NN Training and Inferencing Fragmentation NN Authoring Framework 1 NN Authoring Framework 2 NN Authoring Framework 3 Every Tool Needs an Exporter to Every Accelerator Inference Engine 1 Inference Engine 2 Inference Engine 3 With NNEF- NN Training and Inferencing Interoperability NN Authoring Framework 1 NN Authoring Framework 2 Inference Engine 1 Inference Engine 2 NN Authoring Framework 3 Optimization and processing tools NNEF is a Cross-vendor Neural Net file format Encapsulates network formal semantics, structure, data formats, commonly-used operations (such as convolution, pooling, normalization, etc.) Inference Engine 3

6 NNEF Status and Roadmap NNEF V1.0 provisional release targeting late December Focus on passing trained frameworks to embedded inference engines - Support First cut range of network types NNEF Roadmap - Track development of new network types - Allow authoring and retraining (3rd party tools) - Address a wider range of applications (outside vision apps) - Increase the expressive power of the format Baseline Importer Validator Khronos Open Source Projects File Format Baseline Exporter Working Group Participants Copyright Khronos Group Page 6

7 Copyright Khronos Group Page 7 OpenVX Power Efficiency Wide range of vision hardware architectures OpenVX provides a high-level Graph-based abstraction -> Enables Graph-level optimizations! Can be implemented on almost any hardware or processor! -> Portable, Efficient Vision Processing! X100 X10 X1 Dedicated Hardware Vision DSPs GPU Compute Multi-core CPU Computation Flexibility Vision Node Vision Engines GPU Vision Node Vision Node Middleware DSP Vision Processing Graph Applications Hardware Vision Node Software Portability

8 Copyright Khronos Group Page 8 Simple Edge Detector in OpenVX vx_image input = vxcreateimage(1920, 1080); vx_image output = vxcreateimage(0, 0); vx_image horiz = vxcreatevirtualimage(); vx_image vert = vxcreatevirtualimage(); vx_image mag = vxcreatevirtualimage(); vx_graph g = vxcreategraph(); vxsobel3x3node(g, input, horiz, vert); vxmagnitudenode(g, horiz, vert, mag); vxthresholdnode(g, mag, THRESH, output); status = vxverifygraph(g); status = vxprocessgraph(g); Declare Input and Output Images Declare Intermediate Images Construct the Graph topology Compile the Graph Execute the Graph i h S M m T v o

9 Copyright Khronos Group Page 9 OpenVX Evolution Conformant Implementations Conformant Implementations OpenVX 1.0 Spec released October 2014 New Functionality Expanded Nodes Functionality Enhanced Graph Framework AMD OpenVX Tools - Open source, highly optimized for x86 CPU and OpenCL for GPU - Graph Optimizer - Scripting for rapid prototyping, without re-compiling, at production performance levels OpenVX 1.1 Spec released May 2016 New Functionality Conditional node execution Feature detection Classification operators Expanded imaging operations Extensions Neural Network Acceleration Graph Save and Restore 16-bit image operation Safety Critical OpenVX 1.1 SC for safety-certifiable systems OpenVX 1.2 Spec released May 2017 Adopters Program November 2017 OpenVX Roadmap under Discussion NNEF Import Programmable user kernels with accelerator offload

10 Copyright Khronos Group Page 10 New OpenVX 1.2 Functions Feature detection: find features useful for object detection and recognition - Histogram of gradients HOG Template matching - Local binary patterns LBP Line finding Classification: detect and recognize objects in an image based on a set of features - Import a classifier model trained offline - Classify objects based on a set of input features Image Processing: transform an image - Generalized nonlinear filter: Dilate, erode, median with arbitrary kernel shapes - Non maximum suppression: Find local maximum values in an image - Edge-preserving noise reduction Conditional execution & node predication - Selectively execute portions of a graph based on a true/false predicate Many, many minor improvements New Extensions - Import/export: compile a graph; save and run later - 16-bit support: signed 16-bit image data - Neural networks: Layers are represented as OpenVX nodes B A S Condition C If A then S B else S C

11 Copyright Khronos Group Page 11 OpenVX 1.2 and Neural Net Extension Convolution Neural Network topologies can be represented as OpenVX graphs - Layers are represented as OpenVX nodes - Layers connected by multi-dimensional tensors objects - Layer types include convolution, activation, pooling, fully-connected, soft-max - CNN nodes can be mixed with traditional vision nodes Import/Export Extension - Efficient handling of network Weights/Biases or complete networks OpenVX will be able to import NNEF files into OpenVX Neural Nets Native Camera Control Vision Node Vision Node CNN Nodes Vision Node Downstream Application Processing An OpenVX graph mixing CNN nodes with traditional vision nodes

12 Copyright Khronos Group Page 12 OpenVX SC - Safety Critical Vision Processing OpenVX based on OpenVX 1.1 main specification - Enhanced determinism - Specification identifies and numbers requirements MISRA C clean per KlocWorks v10 Divides functionality into development and deployment feature sets - Adds requirement to support import/export extension OpenVX SC Development Feature Set (Create Graph) Verify Export Binary format Import OpenVX SC Deployment Feature Set (Execute Graph) Entire graph creation API Implementation dependent format No graph creation API

13 Safety Critical APIs Experience and Guidelines OpenGL SC OpenGL SC April 2016 Fixed function graphics subset OpenGL ES Fixed function graphics New Generation APIs for safety certifiable vision, graphics and compute e.g. ISO and DO-178B/C Shader programmable pipeline subset OpenGL ES Shader programmable pipeline OpenVX SC 1.1 Released May 2017 Restricted deployment implementation executes on the target hardware by reading the binary format and executing the pre-compiled graphs Khronos SCAP Safety Critical Advisory Panel Guidelines for designing APIs that ease system certification. Open to Khronos member AND industry experts OpenCL SC TSG Formed Working on OpenCL SC 1.2 Eliminate Undefined Behavior Eliminate Callback Functions Static Pool of Event Objects Copyright Khronos Group Page 13

14 Standards for Vision and Neural Networks Vision/AI Applications NeuralNet NetTraining Training Neural Neural Net Training Frameworks NeuralFrameworks Net Training Frameworks Frameworks Desktop and Cloud Hardware Trained Networks Vision and Neural Net Inferencing Runtime Embedded/Mobile Embedded/Mobile Embedded/Mobile Vision/Inferencing Hardware Embedded/Mobile Vision/Inferencing Hardware Vision/InferencingHardware Hardware Vision/Inferencing Copyright Khronos Group Page 14

15 OpenCL Low-level Parallel Programing Low-level, explicit programming of heterogeneous parallel compute resources - One code tree can be executed on CPUs, GPUs, DSPs and FPGA OpenCL C or C++ language to write kernel programs to execute on any compute device - Platform Layer API - to query, select and initialize compute devices - Runtime API - to build and execute kernels programs on multiple devices The programmer gets to control: - What programs execute on what device - Where data is stored in various speed and size memories in the system - When programs are run, and what operations are dependent on earlier operations OpenCL Kernel OpenCL Code Kernel OpenCL Code Kernel OpenCL Code Kernel Code GPU Kernel code compiled for devices DSP FPGA CPU CPU Devices Runtime API loads and executes kernels across devices CPU Host Copyright Khronos Group Page 15

16 OpenCL 2.2 Released in May OpenCL 1.2 OpenCL 2.0 Becomes industry baseline for heterogeneous parallel computing Enables new class of hardware OpenCL 2.1 SPIR-V 1.0 OpenCL 2.2 SPIR-V 1.2 SPIR-V in Core OpenCL C++ Kernel Language Static subset of C++14 Templates and Lambdas SVM Generic Addresses On-device dispatch Kernel Language Flexibility SPIR-V 1.2 OpenCL C++ support Pipes Efficient device-scope communication between kernels Code Generation Optimizations - Specialization constants at SPIR-V compilation time - Constructors and destructors of program scope global objects - User callbacks can be set at program release time Copyright Khronos Group Page 16

17 OpenCL Conformant Implementations 1.0 May Jul Jun Aug Aug May Dec May Feb Mar Dec Jul14 Desktop 2.1 Jun May Jun May Feb Aug Sep13 Mobile 1.2 Mar Apr Apr Nov Apr Dec Sep Nov Jan May13 Embedded 1.2 May Jul13 FPGA 1.0 Dec Aug15 Vendor timelines are first conformant submission for each spec generation Dec08 OpenCL 1.0 Specification Jun10 OpenCL 1.1 Specification Nov11 OpenCL 1.2 Specification Nov13 OpenCL 2.0 Specification Nov15 OpenCL 2.1 Specification Copyright Khronos Group Page 17

18 OpenCL as Language/Library Backend C++ based Neural network framework Language for image processing and computational photography MulticoreWare Single open source Source C++ project on Programming Bitbucket for OpenCL Hundreds of languages, frameworks and projects using OpenCL to access vendor-optimized, heterogeneous compute runtimes Java language extensions for parallelism Vision processing open source project Compiler directives for Fortran, C and C++ Open source software library for machine learning Over 4,000 GitHub repositories using OpenCL: tools, applications, libraries, languages - up from 2,000 two years ago Copyright Khronos Group Page 18

19 SYCL Ecosystem Single-source heterogeneous programming using STANDARD C++ - Use C++ templates and lambda functions for host & device code - Layered over OpenCL Fast and powerful path for bring C++ apps and libraries to OpenCL - C++ Kernel Fusion - better performance on complex software than hand-coding - Halide, Eigen, Boost.Compute, SYCLBLAS, SYCL Eigen, SYCL TensorFlow, SYCL GTX - trisycl, ComputeCpp, VisionCpp, ComputeCpp SDK More information at Copyright Khronos Group Page 19

20 SYCL Publicly Released Today! Major update representing two and a half years of work by Khronos members - Incorporates significant experience gained from three separate implementations - Feedback from developers of machine learning frameworks such as TensorFlow - TensorFlow now supports SYCL alongside the original CUDA accelerator back-end - Layers over OpenCL 1.2 Developer Choice This is a significant release update for SYCL with an enhanced ecosystem that matches our intention to support machine learning and align with modern C++17. SYCL continues to help us to drive the C++ standard towards heterogeneous support. Our intention is to move forward quickly with the SYCL roadmap to bring even more emphasis to machine learning and Safety Critical support, as well as continued alignment with future ISO C++, Michael Wong, SYCL working group chair The development of the two specifications are aligned so code can be easily shared between the two approaches C++ Kernel Language Low Level Control GPGPU -style separation of device-side kernel source code and host code Single-source C++ Programmer Familiarity Approach also taken by C++ AMP and OpenMP Copyright Khronos Group Page 20

21 OpenCL Roadmap Single source C++ programming. Great for supporting C++ apps, libraries and frameworks Industry working to bring Heterogeneous compute to standard ISO C++ SYCL 1.2 C++11 Single source programming SYCL 2.2 C++14 Single source programming OpenCL 1.2 OpenCL C Kernel Language OpenCL 2.1 SPIR-V in Core OpenCL 2.2 C++ Kernel Language C++17 Parallel STL hosted by Khronos Executors for scheduling work Managed pointers or channels for sharing data Help bring OpenCLclass compute to Vulkan for GPUs OpenCL for DSPs - Embedded imaging, vision and inferencing - Flexible reduced precision - Conformance without IEEE 32 Floating Point - Explicit DMA Copyright Khronos Group Page 21

22 Standards for Vision and Neural Networks Vision/AI Applications NeuralNet NetTraining Training Neural Neural Net Training Frameworks NeuralFrameworks Net Training Frameworks Frameworks Desktop and Cloud Hardware Trained Networks Vision and Neural Net Inferencing Runtime Embedded/Mobile Embedded/Mobile Embedded/Mobile Vision/Inferencing Hardware Embedded/Mobile Vision/Inferencing Hardware Vision/InferencingHardware Hardware Vision/Inferencing Copyright Khronos Group Page 22

23 Pervasive Vulkan All Major GPU Companies shipping Vulkan Drivers for Desktop and Mobile Platforms Desktop, Mobile, Embedded and Console Platforms Supporting Vulkan Including phones and tablets from Google, Huawei, Samsung, Sony, Xiaomi - both premium and mid-range devices Desktop Android 7.0+ Nintendo Switch Android TV Embedded Linux VR Platforms SteamVR GearVR Oculus Rift Google Daydream Game Engines Copyright Khronos Group Page 23

24 Copyright Khronos Group Page 24 Market Demand for Universal 3D Portability Games Engines Native 3D Apps Browser Engines Games Engines Native 3D Apps Browser Engines JavaScript and WebAssembly Native bindings for nexgen WebGL Vulkan Universally Portable Subset Community Outreach at GDC 2017 Create a hybrid Portability API? MAP Vulkan to Metal and DX12 Tools Layers API Libraries Shader Translators Feedback - AVOID CREATING A FOURTH API!!! Would need new specification, CTS, Documentation. Additional developer learning curve. A whole new specification to name, brand, promote. Would INCREASE industry fragmentation

25 Copyright Khronos Group Page 25 Vulkan Portability TSG Process Open source project with similar goals Vulkan on ios and macos Metal Shading Language HLSL Expand/test existing open source SPIRV-Cross Tool Vulkan Portability Deliverables 1. Vulkan Subset Diff Spec 2. Vulkan Subset Development Layer 3. Vulkan Subset API Library over DX12/Metal 4. SPIRV-Cross Translator 5. Vulkan Subset Conformance Tests Layers, APIs, Translators and Tests all to be developed and released in open source API Overlap Analysis Identify Vulkan features not directly mappable to DX12 and Metal Possible proposals for Vulkan extensions for enhanced portability (and possibly Web robustness) sent to Vulkan WG New Vulkan functionality may affect the overlap analysis

26 Copyright Khronos Group Page 26 Vulkan Evolution Feb16 Vulkan 1.0 Explicit Access to GPU Acceleration Strengthened Ecosystem and SDK Enhanced developer and debugging tools Regression testing for SDK stability Enhanced Conformance Testing (API now has 198K test cases - up from 107K last year) Compiler robustness - including HLSL support Vulkan Extensions Maintenance updates plus additional functionality Explicit Building Blocks for VR Explicit Building Blocks for Homogeneous Multi-GPU Enhanced Windows System Integration Increased Shader Language Flexibility Enhanced Cross-Process and Cross-API Sharing Widened Platform Support Through Vulkan Portability Initiative Including Vulkan on macos and ios Vulkan Roadmap Regular Vulkan Core Releases Integrates proven KHR extensions Roadmap Discussions Pushes forward the envelope of GPU Acceleration: Enhanced Compute and Language Flexibility Optimized Vision and Inferencing Acceleration Ray Tracing

27 Clspv OpenCL C to Vulkan Compiler Experimental collaboration between Google, Codeplay, and Adobe - Successfully tested on over 200K lines of Adobe OpenCL C production code - Released in open source - Tracks top-of-tree LLVM and clang, not a fork Uses new Vulkan extensions to support OpenCL C compute operations - VK_KHR_16bit_storage/SPV_KHR_16bit_storage - VK_KHR_variable_pointers/SPV_KHR_variable_pointers Compiles OpenCL C s programming model to Vulkan s SPIR-V execution environment - Proof-of-concept that OpenCL compute can be brought seamlessly to Vulkan Vulkan Universal Portability Universally Portable Graphics and Advanced Compute OpenCL-class compute in Vulkan Copyright Khronos Group Page 27

28 Layered Vision/ Neural Net Ecosystem Implementers may use OpenCL or Vulkan to implement OpenVX nodes on programmable processors OpenVX enables the graph to be extended to include hardware architectures that don t support programmable APIs Application Vulkan Roadmap Enhanced compute vision and inferencing on any platforms with GPU acceleration OpenCL Roadmap Flexible precision for widespread deployment on lowcost embedded processors Programmable Vision Processors Dedicated Vision Hardware And then developers can use OpenVX to enable a developer to easily connect those nodes into a graph The OpenVX graph abstraction enables implementers to optimize execution across diverse hardware architectures for optimal power and performance Copyright Khronos Group Page 28

29 Copyright Khronos Group Page 29 Khronos Cooperative Framework $$ Khronos Board Strategy, budget and oversight $ API Working Groups (Industry, Academic and Associate members) One vote per industry member Advisory Panels By invitation, no fee. Provide requirements and draft spec feedback $ Adopters Programs Conformance Tests Royalty-Free Specifications Implementations, Tools Documentation, Samples Educator Guidelines Courseware Materials Open and royalty-free. Often open sourced Industry and Community Feedback and contributions on released materials Adopters Build conformant implementation and products Developers Develop applications using the APIs Educators / Certifiers Create Courses Training and Certification Membership Fees Academic $1,000 Small companies $175/employee (min $3,500) Non-profits $7,500 Larger companies - $18,000

30 Copyright Khronos Group Page 30 New Open Source Engagement Model Khronos is open sourcing specification sources, conformance tests, tools - Merge requests welcome from the community (subject to review by OpenCL working group) Deeper Community Enablement - Mix your own documentation! - Contribute and fix conformance tests - Fix the specification, headers, ICD etc. - Contribute new features (carefully) Conformance Test Suite Source Contributions and Distribution under Apache 2.0 Khronos Adopters Program Source Materials for Specifications and Reference Documentation CONTRIBUTED Under Khronos IP Framework (you won t assert patents against conformant implementations, and license copyright for Khronos use) Contributions and Distribution under Apache 2.0 Spec Build System and Scripts Khronos builds and Ratifies Canonical Specification under Khronos IP Framework. No changes or re-hosting allowed Spec and Ref Language Source Redistribution under CC-BY 4.0 Spec and Ref Language Source and derivative materials. Re-mixable under CC-BY by the industry and community Anyone can test any implementation at any time Conformant Implementations can use trademark and are covered by Khronos IP Framework Community built documentation and tools

31 Copyright Khronos Group Page 31 Khronos Advisory Panels Specification drafts and invitations for requirements and feedback Requirements and feedback on specification drafts Working Group Khronos Members Any company can join Membership Fee Sign NDA and IP Framework Shared list and Repository Hosted by Khronos. Under Khronos NDA Advisory Panel Khronos Advisors Invited industry experts $0 Cost Sign NDA and IP Framework Advisory Panels Active for Vulkan, OpenCL/SYCL and NNEF

32 Copyright Khronos Group Page 32 Need for Camera Control API? Advanced control of ISP and camera subsystem with cross-platform portability - Generate sophisticated image stream for advanced imaging & vision apps No platform API currently fulfills all developer requirements - Portable access to growing sensor diversity: e.g. depth sensors and sensor arrays - Cross sensor synch: e.g. synch of camera and MEMS sensors - Advanced, high-frequency per-frame burst control of camera/sensor: e.g. ROI - Multiple input, output re-circulating streams with RAW, Bayer or YUV Processing Defines control of Sensor, Color Filter Array Lens, Flash, Focus, Aperture Auto Exposure (AE) Auto White Balance (AWB) Auto Focus (AF) Image Signal Processor (ISP) Stream of Images for Vision Processing OpenKCAM standard is currently on ice do we need to restart?

33 Copyright Khronos Group Page 33 Please Get Involved! Khronos is creating cutting-edge royalty-free open standards - For embedded vision and neural network acceleration Khronos standards are key to many emerging markets such as 3D rendering and advanced gaming, Augmented and Virtual Reality, Embedded Vision and machine learning - Advanced next generation capabilities for ALL platforms Khronos encourages your participation ntrevett@nvidia.com

34 Copyright Khronos Group Page 34 OpenVX - Graph-Level Abstraction OpenVX developers express a graph of image operations ( Nodes ) - Using a C API Nodes can be executed on any hardware or processor coded in any language - Implementers can optimize under the high-level graph abstraction Graphs are the key to run-time power and performance optimizations - E.g. Node fusion, tiled graph processing for cache efficiency etc. Camera Input OpenVX Nodes Pyr t Rendering Output RGB Frame Color Conversion YUV Frame Channel Extract Gray Frame Image Pyramid Optical Flow Array of Keypoints Harris Track OpenVX Graph Feature Extraction Example Graph Ftr t-1 Array of Features

35 Copyright Khronos Group Page 35 OpenVX Efficiency through Graphs.. Graph Scheduling Memory Management Kernel Fusion Data Tiling Split the graph execution across the whole system: CPU / GPU / dedicated HW Reuse pre-allocated memory for multiple intermediate data Replace a subgraph with a single faster node Execute a subgraph at tile granularity instead of image granularity Faster execution or lower power consumption Less allocation overhead, more memory for other applications Better memory locality, less kernel launch overhead Better use of data cache and local memory

36 SPIR-V Ecosystem Khronos open source tools and translators Third party kernel and shader languages GLSL HLSL SPIR-V Khronos defined cross-api IR Native graphics and parallel compute Easily parsed/extended 32-bit stream Data object/control flow retained for effective code generation/translation glslang MSL HLSL GLSL SPIRV-Cross OpenCL C OpenCL C++ Front-end Front-end SPIR-V (Dis)Assembler LLVM SPIR-V Validator SPIRV-opt SPIRV-remap SPIR-V Optimizations Inlining (exhaustive) Store/Load Elimination Dead Code Elimination Dead Branch Elimination Common Uniform Elimination Coming Loop Unrolling and Constant Folding Common Subexpression Elimination LLVM to SPIR-V Bi-directional Translator Khronos liaising with Clang/LLVM Community E.g. discussing SPIR-V as supported Clang target Additional Intermediate Forms IHV Driver Runtimes Copyright Khronos Group Page 36

37 Copyright Khronos Group Page 37 Vulkan and New Generation 3D APIs Only Windows 10 Only Apple Cross Platform Clean, modern architecture Low overhead, explicit GPU access Portable across desktop and mobile Multi-thread / multi-core friendly Efficient, low-latency, predictable performance

38 Copyright Khronos Group Page 38 Vulkan Explicit GPU Control Vulkan = high performance and low latency 3D and GPU Compute. Ideal for VR/AR applications Complex drivers cause overhead and inconsistent behavior across vendors Always active error handling Full GLSL preprocessor and compiler in driver OpenGL vs. OpenGL ES Application Single thread per context High-level Driver Abstraction Layered GPU Control Context management Memory allocation Full GLSL compiler Error detection GPU Application Memory allocation Thread management Synchronization Multi-threaded generation of command buffers Thin Driver Explicit GPU Control GPU Vulkan 1.0 provides access to OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality but with increased performance and flexibility Multiple Front-end Compilers GLSL, HLSL etc. SPIR-V pre-compiled shaders Loadable debug and validation layers Resource management offloaded to app: low-overhead, low-latency driver Consistent behavior: no fighting with driver heuristics Validation and debug layers loaded only when needed SPIR-V intermediate language: shading language flexibility Multi-threaded command creation. Multiple graphics, command and DMA queues Unified API across all platforms with feature set flexibility

39 OpenCL Ecosystem Hardware Implementers OpenCL 2.2 Top to Bottom C++ Desktop/Mobile/Embedded/FPGA 100s of applications using OpenCL acceleration Rendering, visualization, video editing, simulation, image processing, vision and neural network inferencing Single Source C++ Programming Core API and Language Specs Portable Kernel Intermediate Language Copyright Khronos Group Page 39