What's Next in Graphics APIs? SIGGRAPH Asia December 2014 Neil Trevett Khronos President NVIDIA VP Mobile

Transcription

1 What's Next in Graphics APIs? SIGGRAPH Asia December 2014 Neil Trevett Khronos President NVIDIA VP Mobile Copyright Khronos Group Page 1

2 Copyright Khronos Group Page 2 Khronos Connects Software to Silicon Open Consortium creating ROYALTY-FREE, OPEN STANDARD APIs for hardware acceleration Defining the roadmap for low-level silicon interfaces needed on every platform Graphics, compute, rich media, vision, sensor and camera processing Rigorous specifications AND conformance tests for crossvendor portability Acceleration APIs BY the Industry FOR the Industry Well over a BILLION people use Khronos APIs Every Day

3 Copyright Khronos Group Page 3

4 Copyright Khronos Group Page 4 Khronos Standards Visual Computing - 3D Graphics - Heterogeneous Parallel Computing 3D Asset Handling - 3D authoring asset interchange - 3D asset transmission format with compression Over 100 companies defining royalty-free APIs to connect software to silicon Sensor Processing - Vision Acceleration - Camera Control - Sensor Fusion Acceleration in HTML5-3D in browser no Plug-in - Heterogeneous computing for JavaScript

5 Access to 3D on Over 2 BILLION Devices 1.9B Mobiles / year 300M Desktops / year Windows, Mac, Linux 1B Browsers / year Source: Gartner (December 2013) Copyright Khronos Group Page 5

6 Copyright Khronos Group Page 6 OpenGL 3D API Family Tree Fixed function 3D Pipeline Programmable vertex and fragment shaders ES3 is backward compatible so new features can be added incrementally OpenGL ES 1.1 Content OpenGL ES 2.0 Content WebGL 1.0 OpenGL ES 3.0 Content WebGL 2.0 OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 OpenGL ES 1.0 OpenGL ES 3.1 ES-Next OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 3.0 OpenGL 3.2 OpenGL 4.0 OpenGL 4.1 OpenGL 4.2 OpenGL 4.4 OpenGL 4.3 OpenGL 4.5 GL-Next OpenGL 4.4 is a superset of DX

7 Copyright Khronos Group Page 7 OpenGL ES and WebGL Roadmap Fixed function Pipeline Programmable Shaders 32-bit integers and floats NPOT, 3D/depth textures Texture arrays Multiple Render Targets Compute Shaders Driver Update Silicon Update Silicon Update Driver Update Spec at GDC March 2014 Standard in Android L WebGL 1.0 WebGL 2.0 Under Development WebGL Open Review

8 Copyright Khronos Group Page 8 OpenGL ES 3.1 Goals Bring developer requested features from desktop OpenGL 4 to mobile - Advanced features, modern programming styles - Higher performance with lower overhead Headline features - Compute Shaders and Draw-Indirect - Compute shaders can create geometry or other rendering data - and also the draw commands needed to render them - Offload work from CPU to GPU critical for mobile perf and power Run on OpenGL ES 3.0 hardware expose hidden capabilities of shipping devices - Enable very rapid adoption across the industry Better looking, faster performing apps!

9 Copyright Khronos Group Page 9 OpenGL ES 3.1 Adoption Momentum Widespread industry participation to release specification in March Tool and Game Engine Developers, GPU Designers, SoC Vendors - Platform Owners, End Equipment Makers, Middleware ISVs Khronos launched the OpenGL ES 3.1 Adopters program in June Broad set of conformance tests to ensure reliable cross-vendor operation Google announced that OpenGL ES 3.1 is standard in Android L - At Google IO June 2014 First wave of GPU vendors conformant in July ARM, Imagination Technologies, Intel, NVIDIA, Qualcomm, Vivante -

10 Copyright Khronos Group Page 10 Google Android Extension Pack (AEP) Set of extensions for OpenGL ES Accessible through a single query - Functionality to support AAA games Functionality from desktop OpenGL - Tessellation - Improves the detail of geometry rendered - Geometry shaders - Add details and shadows - ASTC Texture Compression - High quality texture compression Enables premium graphics effects - Deferred rendering - Physically-based shading - High Dynamic Range tone mapping - Global Illumination and reflection - Smoke and particle effects Epic s Rivalry demo using full Unreal Engine 4 Running in real-time on NVIDIA Tegra K1 with OpenGL ES AEP

11 Copyright Khronos Group Page 11 Next Generation OpenGL Initiative Ground up re-design of API for high-efficiency access to graphics and compute on modern GPUs and platforms Design from first principles even if means breaking compatibility with traditional OpenGL An open-standard, cross-platform 3D+compute API for the modern era Platform Diversity and need for cross-platform API standards increasing After twenty two years the architecture of GPUs and platforms has radically changed

12 Copyright Khronos Group Page 12 Ground-up Explicit API Redesign Traditional OpenGL Originally architected for graphics workstations with direct renderers and split memory Driver does lots of work: state validation, dependency tracking, error checking. Limits and randomizes performance Threading model doesn t enable generation of graphics commands in parallel to command execution Syntax evolved over twenty years complex API choices can obscure optimal performance path Shader language compiler built into driver. Only GLSL supported. Have to ship shader source Despite conformance testing developers must often handle implementation variability between vendors Next Generation OpenGL Matches architecture of modern platforms including mobile platforms with unified memory, tiled rendering Explicit API the application has direct, predictable control over the operation of the GPU Multi-core friendly with multiple command queues that can be created in parallel Removing legacy requirements simplifies API design, reduces specification size and enables clear usage guidance Standard Intermediate Language as compiler target simplifies driver and enables front-end language flexibility and reliability Simpler API, common language front-ends, more rigorous testing increase cross vendor functional/performance portability

13 Copyright Khronos Group Page 13 Cross Platform Challenge One family of GPUs One OS One GPU on one OS All Modern Platforms and GPUs Participation of key players Proven IP Framework Battle-tested cooperative model The drive to not let the 3D industry fragment

14 Copyright Khronos Group Page 14 Portability Streamlined API is easier to implement and test Standard intermediate language improves shader program portability and reduces driver complexity Crossvendor Portability Enhanced conformance testing methodology WebGL doubles conformance tests over ~21200 vs. ~ suite will contain ~20% more tests Most contributed by open source community

15 Copyright Khronos Group Page 15 Status Organized as a joint project of ARB and OpenGL ES working groups - Likely to become standalone working group soon - Working at very high intensity since June - Making rapid progress - Very significant proposals and IP contributions received from members Participants come from all segments of the graphics industry - Including an unprecedented level of participation from game engine ISVs

16 OpenCL Portable Heterogeneous Computing Portable Heterogeneous programming of diverse compute resources - Targeting supercomputers -> embedded systems -> mobile devices One code tree can be executed on CPUs, GPUs, DSPs, FPGA and hardware - Dynamically interrogate system load and balance work across available processors OpenCL = Two APIs and C-based Kernel language - Platform Layer API to query, select and initialize compute devices - Kernel language - Subset of ISO C99 + language extensions - C Runtime API to build and execute kernels OpenCL across multiple devices OpenCL 2.0 Updated November 2014 OpenCL 2.0 Update Clarifications for support for Blocks in OpenCL C; Refinements to the precision requirements for math functions in fast math mode; Clarification of flags that can be applied to pipes; A new extension, cl_khr_device_enqueue_local_arg_types, for enqueueing device kernels to use arguments that are a pointer to a user defined type in local memory; Clarification of the CL_MEM_KERNEL_READ_AND_WRITE flag to enable filtering of image formats that can be passed to a single kernel instance as read_write GPU DSP FPGA Kernel OpenCL Code Kernel OpenCL Code Kernel OpenCL Code Kernel Code HW CPU CPU Copyright Khronos Group Page 16

17 Copyright Khronos Group Page 17 OpenCL Roadmap What markets has OpenCL been aimed at? What problems is OpenCL solving? How will OpenCL need to adapt in the future? Discussion Focus for New Capabilities HPC Desktop Mobile HPC Desktop Mobile Web HPC Desktop Mobile Web FPGA HPC Desktop Mobile Web FPGA Embedded Safety Critical Dec08 OpenCL 1.0 Specification 3-component vectors Additional image formats Multiple hosts and devices Buffer region operations Enhanced event-driven execution Additional OpenCL C built-ins Improved OpenGL data/event interop Jun10 OpenCL 1.1 Specification Device partitioning Separate compilation and linking Enhanced image support Built-in kernels / custom devices Enhanced DX and OpenGL Interop Nov11 18 months 18 months 24 months OpenCL 1.2 Specification Shared Virtual Memory On-device dispatch Generic Address Space Enhanced Image Support C11 Atomics Pipes Android ICD Nov13 OpenCL 2.0 Specification Roadmap Discussions Binning/Triaging SW and HW features Will use Provisional Specs Some common requests: - C++ Programming - SPIR in Core - Refine and evolve Memory and Execution Models - Better debug and profiling - Trans-API Interop

18 OpenCL Implementations Desktop 1.0 May Jul Jun Aug Aug May Sep May Feb Mar Dec Jul May Jun Aug12 Mobile 1.0 Feb Sep Aug Nov Apr Jan Apr Dec May Jul13 FPGA 1.0 Dec14 Dec08 OpenCL 1.0 Jun10 OpenCL 1.1 Nov11 Nov13 OpenCL 1.2 OpenCL 2.0 Copyright Khronos Group Page 18

19 Copyright Khronos Group Page 19 OpenCL Applications Broad commercial uptake of OpenCL - Mainly imaging, video and vision processing - Adobe, Apple, Corel, ArcSoft Etc. Etc. OpenCL on Sourceforge, Github, Google Code, Bitbucket finds over 2,000 projects - OpenCL implementations - Beignet, pocl - VLC, X264, FFMPEG, Handbrake - GIMP, ImageMagick, IrfanView - Hadoop, Memcached - WinZip, Crypto++ Etc. Etc. Desktop benchmarks use OpenCL - PCMark 8 video chat and edit - Basemark CL, CompuBench Desktop Basemark CL

20 Copyright Khronos Group Page 20 OpenCL as Parallel Language Backend JavaScript binding for initiation of OpenCL C kernels Language for image processing and computational photography MulticoreWare open source project on Bitbucket Embedded array language for Haskell Java language extensions for parallelism River Trail Language extensions to JavaScript Compiler directives for Fortran, C and C++ PyOpenCL Python wrapper around OpenCL Harlan High level language for GPU programming OpenCL provides vendor optimized, cross-platform, cross-vendor access to heterogeneous compute resources

21 Copyright Khronos Group Page 21 Libraries and Languages using OpenCL Library Name Overview Website Accelerate accelerate: An embedded language for accelerated array processing amgcl Simple and generic algebraic multigrid framework Aparapi API for data parallel Java. Allows suitable code to be executed on GPU via OpenCL. ArrayFire Array-based function library Bolt Bolt C++ Template Library Boost.Compute Boost.Compute is a GPU/parallel-computing library for C++ based on OpenCL. Bullet Physics Bullet Physic OpenCL accelerated Rigid Body Pipeline C++ AMP CLANG/LLVM based C++AMP 1.2 standard and transforms it into OpenCL-C clblas cl BLAS implementation clfft OpenCL FFT Libarary clmagma clmagma 1.1 is an OpenCL port of MAGMA clpp OpenCL Data Parallel Primitives Library clspmv Sparse Matrix Solver Clyther Python just-in-time specialization engine for OpenCL Codeplay Math Lib OpenCL 1.2 Math library Concord C++ Hetrogenous Programing Framework ( Support OpenCL 1.2 ) TBB like COPRTHR CO-PRocessing THReads (COPRTHR) SDK DL- Data Layout DL Enables Optimized Data Layout Across Heterogeneous Processors ForOpenCL Fortran to OpenCL tool fortrancl FortranCL is an OpenCL interface for Fortran FSCL.Compiler FSharp to OpenCL Compiler GATLAS GPU Automatically Tuned Linear Algebra Software ( Project looks stalled) GMAC Global Memory for Accelerators GPULib Iterative sparse solvers gpumatrix A matrix and array library on GPU with interface compatible with Eigen. GPUVerify GPUVerify is a tool for formal analysis of GPU kernels written in OpenCL Halide Halide Programming language for high-performance image processing Harlan Harlan: A Scheme-Based GPU Programming Language HOpenCL Haskell OpenCL Wrapper API libcl C++ Generic parallel algorithms library Libra SDK Cross Platform Acceleration API M³ Platform Parallel Framework and Primitive Libraries MUMPS Direct Sparse solver Octave Octave acceleration via OpenCL

22 Copyright Khronos Group Page 22 Libraries and Languages using OpenCL #2 Open Fortran Parser ANTLR-based parsing tools that support the Fortran 2008 standard OpenACC to OpenCL Compiler Rose based OpenACC to OpenCL Compiler. OpenCL.jl Julia OpenCL 1.2 bindings OpenCLIPP OpenCL Integrated Performance Primitives - A library of optimized OpenCL image processing functions OpenCLLink Mathematica to use the OpenCL parallel computing language OpenClooVision Computer vision framework based on OpenCL and C# OpenCV-CL OpenCL accelerated OpenCV OpenHMPP Directive-based OpenACC and OpenHMPP Source to OpenCL compiler Paralution C++ sparse iterative solvers and preconditioners library with OpenCL support Pardiso Direct Sparse solver Pencil PENCIL to be a suitable target language for the compilation of domain-specific languages (DSLs). PETSc Portable, Extensible Toolkit for Scientific Computation PyOpenCL OpenCL parallel computation API from Python QT with OpenCL Using OpenCL with QT RaijinCL library for matrix operations for OpenCL Rivertrail JavaScript which supports Data Parallelism via OpenCL RNG Random number generation for parallelcomputations ROpenCL Parallel Computing for R Using OpenCL Rose Compiler Rose Compiler with OpenCL Support Rust-OpenCl OpenCL bindings for Rust. ScalaCL Scala support of OpenCL SkelCL SkelCL is a library providing high-level abstractions for alleviated programming of modern parallel heterogeneohttps://github.com/skelcl/skelcl SnuCL SnuCL naturally extends the original OpenCL semantics to the heterogeneous cluster SpeedIT 2.4 OpenCl based OpenFoam acceleration library streamscan StreamScan: Fast Scan Algorithms for GPUs without Global Barrier Synchronization- SuperLU Direct Sparse solver TM-Task Management Heterogeneous Task Scheduling and Management Trilinos Building blocks for the development of scientific applications; constructing and using sparse and dense matriceshttp://trilinos.sandia.gov/ VexCL VexCL is a C++ vector expression template library for OpenCL/CUDA ViennaCL open-source linear algebra library for computations on many-core architectures (GPUs, MIC) and multi-core CPUhttp://viennacl.sourceforge.net/ VirtualCL VirtualCL (VCL) cluster platform is a wrapper for OpenCL VOBLA Vehicle for Optimized Basic Linear Algebra - Optimized Basic Linear Algebra DSL VOCL Virtualized OpenCL enviornment VSI/Pro VSIPL implementation in OpenCL WAMS Algebraic Multigrid Solver using state-of-the-art wavelet preconditioners- solver for sparse linear equations

23 Widening OpenCL Ecosystem OpenCL C Kernel Source Alternative Language Diverse, Alternative Language for domainspecific for Languages, Kernels Kernels frameworks and tools High-level High-level Single Frameworks source Frameworks file applications SPIR Generator (e.g. patched Clang) SPIR (Standard Portable Intermediate Representation) First portable IR that includes support for parallel computation Created in close cooperation with LLVM community SPIR 2.0 Provisional Released August 2014 (uses LLVM 3.4) SPIR is easier compiler target than C OpenCL run-time can consume SPIR OpenCL C Runtime Device X Device Y Device Z SYCL Programming abstraction that combines portability and efficiency of OpenCL with ease of use and flexibility of C++ Single source file programming SYCL 1.2 Provisional Updated November 2014 Copyright Khronos Group Page 23

24 APIs for Mobile Compute GPU Compute Shaders (OpenGL 4.4 and OpenGL ES 3.1) Pervasively available on almost any mobile device or OS Easy integration into graphics apps no API interop needed Program in GLSL not C Limited to acceleration on a single GPU General Purpose Heterogeneous Programming Framework Flexible, low-level access to any devices with OpenCL compiler Open standard for any device or OS being used as backend by many languages and frameworks Single programming and run-time framework for CPUs, GPUs, DSPs, hardware Needs full compiler stack and IEEE precision Metal Integrated Graphics and Compute Subset of a mix of OpenGL and OpenCL functionality C++11-based kernel language Apple only (ios 8 only, A7 and later hardware), GPU only C/C++ Language Integrated GPU Compute Easy programmability and low level access to GPU: Unified Memory, Virtual Addressing, Mature and optimized tools and performance Extensive compute and imaging libraries available (NPP, cufft, cublas, cuda-gdb, nvprof etc.) NVIDIA only, GPU only RS RenderScript - Easy, High-level Compute Offload from Java C99 based kernel language for simple offload from Java apps to CPU and GPU JIT Compilation provide host and device portability Android only Limited control over acceleration configuration Copyright Khronos Group Page 24

25 Copyright Khronos Group Page 25 Mobile Vision Acceleration = New Experiences Need for advanced sensors and the acceleration to process them Computational Photography and Videography Face, Body and Gesture Tracking 3D Scene/Object Reconstruction Augmented Reality

26 Visual Computing = Graphics AND Vision Graphics Processing New mobile visual sensors for MORE DATA Data Advanced mobile hardware for MORE PROCESSING Imagery Enables closer intertwining of real and virtual worlds High-Quality Reflections, Refractions, and Caustics in Augmented Reality and their Contribution to Visual Coherence P. Kán, H. Kaufmann, Institute of Software Technology and Interactive Systems, Vienna University of Technology, Vienna, Austria Vision Processing Copyright Khronos Group Page 26

27 Copyright Khronos Group Page 27 Vision Pipeline Challenges and Opportunities Growing Camera Diversity Capturing color, range and lightfields Diverse Vision Processors Driving for high performance and low power Sensor Proliferation Diverse sensor awareness of the user and surroundings Light / Proximity 2 cameras 3 microphones Touch Camera sensors >20MPix Novel sensor configurations Stereo pairs Plenoptic Arrays Active Structured Light Active TOF Multi-core CPUs Programmable GPUs DSPs and DSP arrays Camera ISPs Dedicated vision IP blocks 19 Position - GPS - WiFi (fingerprint) - Cellular trilateration - NFC/Bluetooth Beacons Accelerometer Magnetometer Gyroscope Pressure / Temp / Humidity Flexible sensor and camera control to generate required image stream Use best processing available for image stream processing with code portability Control/fuse vision data by/with all other sensor data on device

28 Copyright Khronos Group Page 28 OpenVX Power Efficient Vision Acceleration Out-of-the-Box vision acceleration framework - Enables low-power, real-time applications - Targeted at mobile and embedded platforms Functional Portability - Tightly defined specification - Full conformance tests Performance portability across diverse HW - Higher-level abstraction hides hardware details - ISPs, Dedicated hardware, DSPs and DSP arrays, GPUs, Multi-core CPUs Enables low-power, always-on acceleration - Can run solely on dedicated vision hardware - Does not require full SOC CPU/GPU complex to be powered on Application Application Application Application Vision Accelerator Vision Accelerator Vision Accelerator Vision Accelerator

29 Copyright Khronos Group Page 29 OpenVX Graphs The Key to Efficiency Vision processing directed graphs for power and performance efficiency - Each Node can be implemented in software or accelerated hardware - Nodes may be fused by the implementation to eliminate memory transfers - Processing can be tiled to keep data entirely in local memory/cache VXU Utility Library for access to single nodes - Easy way to start using OpenVX by calling each node independently EGLStreams can provide data and event interop with other Khronos APIs - BUT use of other Khronos APIs are not mandated Native Camera Control OpenVX Node OpenVX Node OpenVX Node OpenVX Node Downstream Application Processing Example OpenVX Graph

30 Copyright Khronos Group Page 30 OpenVX 1.0 Function Overview Core data structures - Images and Image Pyramids - Processing Graphs, Kernels, Parameters Image Processing - Arithmetic, Logical, and statistical operations - Multichannel Color and BitDepth Extraction and Conversion - 2D Filtering and Morphological operations - Image Resizing and Warping Core Computer Vision - Pyramid computation - Integral Image computation Feature Extraction and Tracking - Histogram Computation and Equalization - Canny Edge Detection - Harris and FAST Corner detection - Sparse Optical Flow Widely used extensions adopted into future versions of the core OpenVX Specification Is Extensible Khronos maintains extension registry OpenVX 1.0 defines framework for creating, managing and executing graphs Focused set of widely used functions that are readily accelerated Implementers can add functions as extensions

31 Copyright Khronos Group Page 31 OpenVX and OpenCV are Complementary Governance Conformance Community driven open source with no formal specification No conformance tests for consistency and every vendor implements different subset Formal specification defined and implemented by hardware vendors Full conformance test suite / process creates a reliable acceleration platform Portability APIs can vary depending on processor Hardware abstracted for portability Scope Efficiency Very wide 1000s of imaging and vision functions Multiple camera APIs/interfaces Memory-based architecture Each operation reads and writes memory Tight focus on hardware accelerated functions for mobile vision Use external camera API Graph-based execution Optimizable computation and data transfer Use Case Rapid experimentation and Prototyping Production development & deployment

32 Copyright Khronos Group Page 32 OpenVX 1.0 Launched! Finalized OpenVX 1.0 specification released October Full conformance test suite and Adopters Program immediately available - $20K Adopters fee ($15K for members) working group reviews submitted results - Test suite exercises graph framework and functionality of each OpenVX 1.0 node - Approved Conformant implementations can use the OpenVX trademark Khronos working on open source sample implementation of OpenVX Expected release on GitHub by end of 2014

33 Copyright Khronos Group Page 33 Khronos APIs for Vision Processing Any compute API can be used for vision acceleration - OpenCL, OpenGL Compute Shaders OpenVX is the only vision API that does not NEED a CPU/GPU complex - Can use any processor from high-end GPU, through DSPs to hardware blocks Regardless of the underlying hardware the application remains portable - The higher abstraction level of OpenVX protects app from hardware differences App portability to dedicated vision hardware and graph-based optimizations are the keys to achieving very lower power vision processing Many implementers may choose to use OpenCL or OpenGL Compute Shaders to implement OpenVX nodes and OpenVX to enable a developer to connect those nodes into a graph Programmable Vision Processors Dedicated Vision Hardware

34 Copyright Khronos Group Page 34 Need for Camera Control API - OpenKCAM Advanced control of ISP and camera subsystem with cross-platform portability - Generate sophisticated image stream for advanced imaging & vision apps No platform API currently fulfills all developer requirements - Portable access to growing sensor diversity: e.g. depth sensors and sensor arrays - Cross sensor synch: e.g. synch of camera and MEMS sensors - Advanced, high-frequency per-frame burst control of camera/sensor: e.g. ROI - Multiple input, output re-circulating streams with RAW, Bayer or YUV Processing Defines control of Sensor, Color Filter Array Lens, Flash, Focus, Aperture Auto Exposure (AE) Auto White Balance (AWB) Auto Focus (AF) Image Signal Processor (ISP) EGLStreams Image/Vision Applications

35 Copyright Khronos Group Page 35 Low-level Sensor Abstraction API Apps request semantic sensor information StreamInput defines possible requests, e.g. Read Physical or Virtual Sensors e.g. Game Quaternion Context detection e.g. Am I in an elevator? Apps Need Sophisticated Access to Sensor Data Without coding to specific sensor hardware Sensor Discoverability Sensor Code Portability Advanced Sensors Everywhere Multi-axis motion/position, quaternions, context-awareness, gestures, activity monitoring, health and environmental sensors StreamInput processing graph provides optimized sensor data stream High-value, smart sensor fusion middleware can connect to apps in a portable way Apps can gain magical situational awareness

36 Copyright Khronos Group Page 36 Khronos APIs for Augmented Reality AR needs not just advanced sensor processing, vision acceleration, computation and rendering - but also for all these subsystems to work efficiently together Audio Rendering MEMS Sensors Sensor Fusion Application on CPUs, GPUs and DSPs Precision timestamps on all sensor samples Vision Processing Advanced Camera Control and stream generation EGLStream - stream data between APIs 3D Rendering and Video Composition On GPU

37 Copyright Khronos Group Page 37 Questions? Khronos members are building the future of mobile processing - Graphics, compute, vision and sensor processing Any company is welcome to join Khronos to have a voice in these developments! - Well-defined IP framework protects your IP and conformant implementations - ntrevett@nvidia.com