Open Standards for Vision and AI Peter McGuinness NNEF WG Chair CEO, Highwai, Inc May 2018

Transcription

1 Copyright Khronos Group Page 1 Open Standards for Vision and AI Peter McGuinness NNEF WG Chair CEO, Highwai, Inc peter.mcguinness@gobrach.com May 2018

2 Khronos Mission E.g. OpenGL ES provides 3D on almost every smartphone over 4 Billion units shipped Software Silicon Khronos is an International Industry Consortium of over 100 companies creating royalty-free, Open Standards to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing Copyright Khronos Group Page 2

3 Khronos Principles of Organization Any company is welcome to join. One company one vote Strong IP Framework to enable ROYALTY-FREE specifications: all members agree not assert patents against conformant implementations Only invest where there is strong industry momentum to ensure industry relevance let Darwinism rule! Any member can propose new standards initiatives Software Silicon Conformance Tests and Adopters Programs for defining conformance, specification integrity and crossvendor portability Non-profit organization Membership and Adopters fees cover operating, marketing and engineering expenses Copyright Khronos Group Page 3

4 Copyright Khronos Group Page 4 Why AI and Vision Need Standards Without Standards With Standards Platform Fragmentation Application Portability Everything runs on CPU Silicon Acceleration Standard Acceleration APIs provide PERFORMANCE, POWER AND PORTABILITY

5 Native Acceleration for Vision and AI Geometric scene reconstruction Tracking and Positioning Import trained networks Semantic scene understanding (Neural Networks) Accelerate on embedded Inferencing hardware Vision sensor(s) Vision Application Copyright Khronos Group Page 5

6 Neural Network Workflow Applications Using Embedded Vision and Inferencing WinML Authoring interchange NeuralNet NetTraining Training Neural NeuralNet NetTraining Training Frameworks Neural Frameworks Frameworks Frameworks Framework Specific Formats Trained Networks Compilation and Optimization Vision and Neural Net Inferencing Runtimes Embedded Deployment Desktop and Cloud Hardware cudnn MIOpen GPU MKL-DNN TPU Training = Desktop / Cloud Diverse Inferencing Acceleration Hardware CPU CPU CPU GPU FPGA DSP Custom Hardware Deployment on Embedded Devices Copyright Khronos Group Page 6

7 Copyright Khronos Group Page 7 NNEF - Solving Neural Net Fragmentation Before NNEF NN Training and Inferencing Fragmentation NN Authoring Framework 1 NN Authoring Framework 2 NN Authoring Framework 3 Every Tool Needs an Exporter to Every Accelerator Inference Engine 1 Inference Engine 2 Inference Engine 3 With NNEF- NN Training and Inferencing Interoperability NN Authoring Framework 1 NN Authoring Framework 2 Inference Engine 1 Inference Engine 2 NN Authoring Framework 3 Optimization and processing tools Inference Engine 3

8 Copyright Khronos Group Page 8 Platform Neural Network Stacks Consistent Fundamental Steps 1. Import trained NN model file 2. Build optimized version of graph 3. Accelerate on GPU or other processor using available low-level API Core ML Model Microsoft Windows Windows Machine Learning (WinML) Google Android Neural Network API (NNAPI) Apple MacOS and ios CoreML

9 Copyright Khronos Group Page 9 NNEF Ecosystem Live Imminent TensorFlow and Caffe2 Importer / Exporters TensorFlow and Caffe Exporters Files Syntax Parser/ Validator NNEF 1.0 Provisional Released for industry feedback before finalization Google NNAPI Convertor OpenVX Ingestion & Execution Comparing Neural Network Exchange Industry Initiatives NNEF open source projects hosted on Khronos NNEF GitHub repository Apache 2.0 license Defined Specification Stability for hardware deployment Multi-company Governance Flat and Compound Ops Open Source Project Software stack flexibility Initiated by Facebook Flat Ops Only

10 Copyright Khronos Group Page 10 NNEF File Structure Network Structure File Distilled, platform independent network description Human readable, syntactical elements from Python Standardized Operations Rigorously defined semantics Linear, convolution, pooling, normalization, activation, unary/binary Supports fully connected, convolutional, recurrent architectures Two Levels of Expressiveness Flat Basic transfer of computation graphs with standardized operations Simple to parse and translate to vendor specific formats Compositional Define custom compound operations Higher-level graph descriptions More complex to parse but offers more optimization hints Can associate multiple Data Files with one Network Structure File e.g. the same data in multiple formats Network Data File Network Data File Network Data File Binary format contains parameter tensors Binary Supports format float contains and quantized parameter (integer) tensors data Binary Supports Flexible format bit float widths contains and quantized and parameter quantization (integer) tensors algorithms data Supports Flexible Quantization bit float widths and algorithms quantized and quantization expressed (integer) algorithms as data extensible Flexible Quantization bit widths algorithms compound and quantization expressed operations algorithms as extensible Quantization Quantization algorithms compound info provided expressed operations as hints as extensible for execution Quantization compound info provided operations as hints for execution Quantization info provided as hints for execution Split Structure and Data files Easy independent access to network structure or individual parameter data Set of files can use a container such as tar or zip with optional compression and encryption

11 Copyright Khronos Group Page 11 NNEF Execution within OpenVX Convolution Neural Network topologies can be represented as OpenVX graphs - Can also combine traditional vision and neural network operations OpenVX Neural Network Extension - Defines OpenVX nodes to represent many common NN layer types - Layer types include convolution, activation, pooling, fully-connected, soft-max - Defines multi-dimensional tensors objects to connect layers Kernel Import Extension - Enables loading of external program representations into OpenVX graphs Native Camera Control Vision Node Vision Node CNN Nodes Vision Node Downstream Application Processing An OpenVX graph mixing CNN nodes with traditional vision nodes NNEF importer converts NNEF representation into OpenVX Graphs using Kernel Import Extension

12 Copyright Khronos Group Page 12 NNEF 1.0 Provisional Released as Provisional to get industry feedback before finalization - Welcoming comments and feedback on Khronos GitHub Initial focus on passing trained frameworks to embedded inference engines - Authoring interchange, importing NNEF into tools, is also an emerging use case Support deployable range of network topologies - Rapid evolution to encompass new network types as they emerge from research NNEF Working Group Participants

13 APIs and File Formats Runtime acceleration APIs typically require a file format to complete their deployment ecosystem 3D Graphics Acceleration Programmable Parallel Programming Vision and Inferencing Acceleration 3D Asset Transmission Graphics and parallel processing intermediate code format Neural Network Exchange Format Copyright Khronos Group Page 13

14 Copyright Khronos Group Page 14 Opportunities for Cooperation All Khronos standards are open and royalty-free - Usually also backed with open source tests, tools and implementations Any company or organization is welcome to join Khronos - Membership fees start at $3,500 for a voice and a vote in any of these standards Khronos welcomes cooperative projects and liaison with ETSI/ARF for VR/AR! - We hope Khronos standards can help you reach your objectives - Khronos liaison models could enable direct participation by ETSI/ARF representatives Neil Trevett - ntrevett@nvidia.com -