Silicon Acceleration APIs

Transcription

1 Copyright Khronos Group Page 1 Silicon Acceleration APIs Embedded Technology 2016, Yokohama Neil Trevett Vice President Developer Ecosystem, NVIDIA President, Khronos November 2016

2 Copyright Khronos Group Page 2 Khronos and Open Standards Software Silicon Khronos is an Industry Consortium of over 100 companies We create royalty-free, open standard APIs for hardware acceleration of Graphics, Parallel Compute, Neural Networks and Vision

3 Copyright Khronos Group Page 3 Accelerated API Landscape Vision Frameworks Neural Net Libraries OpenVX Neural Net Extension High-level Language-based Acceleration Frameworks 3D Graphics Explicit Kernels GPU FPGA DSP Dedicated Hardware

4 Copyright Khronos Group Page 4 OpenCL Low-level Parallel Programing Low level programming of heterogeneous parallel compute resources - One code tree can be executed on CPUs, GPUs, DSPs and FPGA OpenCL C language to write kernel programs to execute on any compute device - Platform Layer API - to query, select and initialize compute devices - Runtime API - to build and execute kernels programs on multiple devices New in OpenCL OpenCL C++ kernel language - a static subset of C Adaptable and elegant sharable code great for building libraries - Templates enable meta-programming for highly adaptive software - Lambdas used to implement nested/dynamic parallelism OpenCL Kernel OpenCL Code Kernel OpenCL Code Kernel OpenCL Code Kernel Code Kernel code compiled for devices GPU DSP FPGA CPU CPU Devices Runtime API loads and executes kernels across devices CPU Host

5 OpenCL Conformant Implementations 1.0 May Jul Jun Aug Aug May Dec May Feb11 Desktop 1.1 Mar Dec Jul Jun May Jun May Aug Mar Feb Sep13 Mobile 1.1 Nov Apr Nov Apr Dec Jan May Jul Sep14 FPGA 1.0 Dec14 Embedded 1.2 May Aug15 Vendor timelines are first implementation of each spec generation Dec08 OpenCL 1.0 Specification Jun10 OpenCL 1.1 Specification Nov11 OpenCL 1.2 Specification Nov13 OpenCL 2.0 Specification Nov15 OpenCL 2.1 Specification Copyright Khronos Group Page 5

6 Copyright Khronos Group Page 6 SYCL for OpenCL Single-source heterogeneous programming using STANDARD C++ - Use C++ templates and lambda functions for host & device code Aligns the hardware acceleration of OpenCL with direction of the C++ standard - C++14 with open source C++17 Parallel STL hosted by Khronos Developer Choice The development of the two specifications are aligned so code can be easily shared between the two approaches C++ Kernel Language Low Level Control GPGPU -style separation of device-side kernel source code and host code Single-source C++ Programmer Familiarity Approach also taken by C++ AMP and OpenMP

7 Copyright Khronos Group Page 7 Embedded Vision Acceleration Embedded Technology, Yokohama Shorin Kyo, Huawei

8 Power Efficiency Copyright Khronos Group Page 8 OpenVX Low Power Vision Acceleration Targeted at vision acceleration in real-time, mobile and embedded platforms - Precisely defined API for production deployment Higher abstraction than OpenCL for performance portability across diverse architectures - Multi-core CPUs, GPUs, DSPs and DSP arrays, ISPs, Dedicated hardware Extends portable vision acceleration to very low power domains - Doesn t require high-power CPU/GPU Complex or OpenCL precision X100 X10 X1 Dedicated Hardware Vision DSPs Vision Processing Efficiency GPU Compute Multi-core CPU Computation Flexibility Vision Engine Middleware Application GPU DSP Hardware

9 Copyright Khronos Group Page 9 OpenVX Graphs OpenVX developers express a graph of image operations ( Nodes ) - Nodes can be on any hardware or processor coded in any language Graphs can execute almost autonomously - Possible to Minimize host interaction during frame-rate graph execution Graphs are the key to run-time optimization opportunities Camera Input OpenVX Nodes Pyr t Rendering Output RGB Frame Color Conversion YUV Frame Channel Extract Gray Frame Image Pyramid Optical Flow Array of Keypoints Harris Track OpenVX Graph Feature Extraction Example Graph Ftr t-1 Array of Features

10 Copyright Khronos Group Page 10 OpenVX Efficiency through Graphs.. Graph Scheduling Memory Management Kernel Merge Data Tiling Split the graph execution across the whole system: CPU / GPU / dedicated HW Reuse pre-allocated memory for multiple intermediate data Replace a subgraph with a single faster node Execute a subgraph at tile granularity instead of image granularity Faster execution or lower power consumption Less allocation overhead, more memory for other applications Better memory locality, less kernel launch overhead Better use of data cache and local memory

11 Copyright Khronos Group Page 11 Example Relative Performance 10 9 Relative Performance OpenCV (GPU accelerated) OpenVX (GPU accelerated) 2.5 NVIDIA implementation experience. Geometric mean of >2200 primitives, grouped into each categories, running at different image sizes and parameter settings Arithmetic Analysis Filter Geometric Overall

12 Copyright Khronos Group Page 12 Layered Vision Processing Ecosystem Implementers may use OpenCL or Compute Shaders to implement OpenVX nodes on programmable processors And then developers can use OpenVX to enable a developer to easily connect those nodes into a graph Programmable Vision Processors Application C/C++ Dedicated Vision Hardware AMD OpenVX - Open source, highly optimized for x86 CPU and OpenCL for GPU - Graph Optimizer looks at entire processing pipeline and removes/replaces/merges functions to improve performance and bandwidth - Scripting for rapid prototyping, without re-compiling, at production performance levels OpenVX enables the graph to be extended to include hardware architectures that don t support programmable APIs The OpenVX graph enables implementers to optimize execution across diverse hardware architectures an drive to lower power implementations

13 Copyright Khronos Group Page 13 OpenVX 1.0 Shipping, OpenVX 1.1 Released! Multiple OpenVX 1.0 Implementations shipping spec in October Open source sample implementation and conformance tests available OpenVX 1.1 Specification released 2nd May Expands node functionality AND enhances graph framework OpenVX is EXTENSIBLE - Implementers can add their own nodes at any time to meet customer and market needs = shipping implementations

14 Copyright Khronos Group Page 14 OpenVX Neural Net Extension Convolution Neural Network topologies can be represented as OpenVX graphs - Layers are represented as OpenVX nodes - Layers connected by multi-dimensional tensors objects - Layer types include convolution, activation, pooling, fully-connected, soft-max - CNN nodes can be mixed with traditional vision nodes Import/Export Extension - Efficient handling of network Weights/Biases or complete networks The specification is provisional - Welcome feedback from the deep learning community Native Camera Control Vision Node Vision Node CNN Nodes Vision Node Downstream Application Processing An OpenVX graph mixing CNN nodes with traditional vision nodes

15 Copyright Khronos Group Page 15 NNEF - Neural Network Exchange Format NN Authoring Framework 1 NN Authoring Framework 2 NN Authoring Framework 3 NN Authoring Framework 1 NN Authoring Framework 2 NN Authoring Framework 3 Neural Net Exchange Format Inference Engine 1 Inference Engine 2 Inference Engine 3 Inference Engine 1 Inference Engine 2 Inference Engine 3 NNEF encapsulates neural network structure, data formats, commonly used operations (such as convolution, pooling, normalization, etc.) and formal network semantics NNEF 1.0 is currently being defined OpenVX will import NNEF files

16 OpenGL ES Fixed function Pipeline Vertex and fragment shaders 32-bit integers and floats NPOT, 3D/depth textures Texture arrays Multiple Render Targets Compute Shaders Tessellation and geometry shaders ASTC Texture Compression Floating point render targets Debug and robustness for security Epic s Rivalry demo using full Unreal Engine ES 1.0 Driver Update 2004 ES 1.1 Silicon Update 2007 ES 2.0 Close to 2 Billion OpenGL ES devices shipped in 2015 Silicon Update 2012 ES 3.0 Driver Update 2014 ES 3.1 L Silicon Update AEP 2015 ES 3.2 Driver Update 2016 N OpenGL ES 2.0 OpenGL ES 3.x Vertex and Fragment Shaders Compute Shaders Copyright Khronos Group Page 16

17 Copyright Khronos Group Page 17 The Key Principle of Vulkan: Explicit Control Application tells the driver what it is going to do - In enough detail that driver doesn t have to guess - When the driver needs to know it You have to supply a LOT of information! And, you have to supply it early In return, driver promises to do - What the application asks for - When it asks for it - Very quickly Efficient, predictable performance No driver magic no surprises

18 Copyright Khronos Group Page 18 Vulkan Explicit GPU Control Application Single thread per context High-level Driver Abstraction Context management Memory allocation Full GLSL compiler Error detection Layered GPU Control GPU Application Memory allocation Thread management Synchronization Multi-threaded generation of command buffers Thin Driver Explicit GPU Control GPU Vulkan 1.0 provides access to OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality but with increased performance and flexibility Language Front-end Compilers Initially GLSL SPIR-V pre-compiled shaders Loadable debug and validation layers Vulkan Benefits Resource management in app code: Less driver hitches and surprises Simpler drivers: Improved efficiency/performance Reduced CPU bottlenecks Lower latency Increased portability Multi-threaded Command Buffers: Command creation can be multi-threaded Multiple CPU cores increase performance Graphics, compute and DMA queues: Work dispatch flexibility SPIR-V Pre-compiled Shaders: No front-end compiler in driver Future shading language flexibility Loadable Layers No error handling overhead in production code

19 Copyright Khronos Group Page 19 Vulkan Multi-threading Efficiency CPU Thread Command Buffer 1. Multiple threads can construct Command Buffers in parallel Application is responsible for thread management and synch Command Buffer CPU Thread CPU Thread CPU Thread Command Buffer Command Queue GPU Command Buffer CPU Thread 2. Command Buffers placed in Command Queue by separate submission thread CPU Thread Command Buffer Applications can create graphics, compute and DMA command buffers with a general queue model that can be extended to more heterogeneous processing in the future

20 Khronos Safety Critical APIs DO-178B/C ISO OpenGL SC Fixed function graphics subset Experience and Guidelines OpenGL SC April 2016 Shader programmable pipeline subset New Generation APIs for safety certifiable vision, graphics and compute OpenGL ES Fixed function graphics OpenGL ES Shader programmable pipeline Safety Critical vision processing Safety Critical heterogeneous compute Small driver size Advanced functionality Graphics and compute Copyright Khronos Group Page 20

21 Copyright Khronos Group Page 21 Please Consider Joining Khronos! Understand early industry requirements! Influence how standards evolve! Ship products that conform to international standards! Access draft specs to build products faster! Khronos is proven to RAPIDLY generate hardware API standards that create significant market opportunities Any company or organization is welcome to join Khronos for a voice and a vote in any of its standards

22 Copyright Khronos Group Page 22 VeriSilicon Computer Vision solution with OpenVX over VIP Gaku Ogura Simon Jones 18 November 2016

23 Copyright Khronos Group Page 23 VeriSilicon Who We Are Founded in 2001 Over 650 employees 70% dedicated to R&D

24 Copyright Khronos Group Page 24 VeriSilicon What we can do? Idea to Spec Spec to RTL IP Development & Licensing System Architecture Silicon Design Shipping Test RTL to Netlist Package Netlist to GDS Manufacture

25 Movement of Intelligent Devices Multi-Media Graphics, Video, Audio, Voice Natural User Interface - VISION - Natural Language Processing - Sensors Many applications need flexibility 1000s of algorithms new one every day Compute intensive need hardware acceleration Algorithms keep changing need programmability Copyright Khronos Group Page 25

26 Copyright Khronos Group Page 26 Deep Learning: Neural Networks Return with Vengeance With big data and high density compute, deep neural networks can be trained to solve impossible computer vision problems % Teams using Deep Learning

27 Copyright Khronos Group Page 27 The Basics of Deep Neural Networks (DNN) Training: MACs/dataset Offline (mostly) Analogous to compiling forward Pug =? Rottweiler Inference: MACs/image On-the-fly Analogous to running s/w Embedded systems use this backward 100K-100M Network Coefficients forward error Rottweiler

28 Challenges of Real-Time Inference Classification Detection Segmentation 100G MAC/s 1T MAC/s 10T MAC/s Majority of deep neural net computation is Multiply-Accumulate (MAC) for convolutions and inner products Memory bandwidth is a bigger challenge - Convolution in deep neural nets works with not just 2D images, but high-dimensional arrays (i.e. tensor ) - Example: AlexNet (classification) has 60M parameters (240MB FP32), DDR BW with brute force implementation: 35GB/s Copyright Khronos Group Page 28

29 Copyright Khronos Group Page 29 Processor Architectures Programmability CPU OpenCL OpenVX OpenCV VIP VeriSilicon GPU VeriSilicon ARM Imagination Custom RTL DSP VeriSilicon CEVA Videantis Cadence Synopsys Energy

30 Copyright Khronos Group Page 30 Efficient Processor Architecture to Enable DNN for Embedded Vision Programmable engine to cope with new CV algorithms Future new NN layers can be implemented here Scalable architecture to support different PPA (Performance, Power, Area) requirements High utilization MACs for DNN Scalable architecture to support different PPA (Performance, Power, Area) requirements Handles other DNN functions (normalization, pooling, ) Handles pruning, compression, batching to increase MAC utilization and decrease DDR BW Vision Framework API Programmable Programmable Engine Engine Engine Vision Applications Scalable Scalable MACs MACs Intelligent Data Fabrics Local Memory / Cache Deep Learning Frameworks Mature CV Block HW Accelerator Mature CV Block HW Accelerator Customer Defined HW Accelerator Custom RTL to accelerate mature CV algorithms for 24/7 low power operation I/F to extend HW acceleration for application specific purpose For data synchronization between threads/blocks and minimize DDR BW

31 Copyright Khronos Group Page 31 Use Case Study: Faster RCNN One of state-of-the-art object detection CNNs - Network configuration region proposals - 60M parameters, 30G MACs/image Collaborative computing in VIP - NN Engine: number crunching - Convolution layers, full connected layers - Tensor Processing Fabric: data rearrangement - LRN, max pooling, ROI pooling - Programmable Engine: precision compute - RPN, NMS, softmax Programmabl e Engine NN Cores TP Fabric Local Memory / Cache Output of a layer is queued in DDR if next layer is on different processing engine Real-time performance (30fps HD) under 28nm HPM

32 Copyright Khronos Group Page 32 Scalable Performance from VIP8000 VIP8000 Series VIP Nano VIP8000UL VIP8000L VIP8000 Number of Execution cores Clock Frequency (MHz) HD Performance@800MHz Perspective Warping 60 fps 120 fps 240 fps 480 fps Optical Flow LK 30 fps 60 fps 120 fps 240 fps Pedestrian Detection 30 fps 60 fps 120 fps 240 fps Convolutional Neural Network (AlexNet) 125 fps 250 fps 500 fps 1000 fps Scalable performance with SAME application code on different processor variants

33 Comprehensive Multi-Media/Pixel Subsystems CPU VPUCore (Video) GC Core (GPU) VIP Core (Vision/Image) ZSP (Audio/Voice) ISP AXI Bus VeriSilicon Universal Compression Local Bus IO Controller DMA Engine DC (Display) Memory Controller Wireless Connectivity Mixed Signal IPs Company Proprietary and Copyright Khronos Group Page 33

34 Copyright Khronos Group Page 34