Snapdragon NPE Overview

Transcription

1 March 2018 Linaro Connect Hong Kong Snapdragon NPE Overview Mark Charlebois Director, Engineering Qualcomm Technologies, Inc.

2 Caffe2 Snapdragon Neural Processing Engine Efficient execution on Snapdragon Model framework/ network support Developer Tools Software accelerated runtime for the execution of deep neural networks on device What s new? Available at: developer.qualcomm.com Fixed and floating point optimizations Supports Caffe2, CNTK, MxNet New optimizations for networks TensorFlow, the TensorFlow logo and any related marks are trademarks of Google Inc 2

3 Elements of Snapdragon NPE SDK API C++ library in binary form and header files Java library for Android integration C++ and Python API support for interacting with DLC DLC Snapdragon NPE DNN model format Network is a collection of connected layers DNN models are stored in DLC files Snapdragon NPE SDK Tools Model converters to create Snapdragon NPE compatible DNN models from popular training framework formats Optimization and debugging support tools Support Assets Development host (x86 Ubuntu 14.04) User and reference documentation Tutorials and examples Benchmarking 3

4 OS Snapdragon NPE SW Diagram SDK Productivity Components 3 rd Party Apps Benchmarking Network Debug Tools Tutorial Samples User & Reference Docs OS Drivers Core Runtime Profiling Logging Compute Networks Runtime Engine Model Debug User Defined Layers (UDL) API CPU GPU libopencl.so DSP DL Container Model loader libsnpe_[a,c]dsp*.so libsnpe_dsp_*skel.so SDK API DNN Model Conversion Tools Caffe/2 -> DLC fixed TensorFlow -> DLC fixed Caffe/2 -> DLC Float TensorFlow -> DLC Float UDL Plugin Android & Linux (x86_64, Armv7, Armv7hf, AArch64) QuRT HW CPU Adreno GPU Hexagon DSP 4

5 Snapdragon NPE SDK SDK can be downloaded from Qualcomm Developer Network The NPE SDK supports Qualcomm Snapdragon 845, 820, 835, 625, 626, 650, 652, 653, 660, 630, 636, and 450 as well as the Qualcomm Snapdragon 820Am automotive platform and Qualcomm Snapdragon Flight. For Qualcomm Adreno GPU support, libopencl.so must be present on device. Toolchains: Android (armv7, aarch64) - GCC and Clang toolchains Linux (armv7, armv7hf, aarch64, x86_64*) - GCC on ARM, Clang on x86_64 * CPU only 5

6 NPE SDK Developer Tools snpe-net-run snpe-caffe-to-dlc snpe-caffe2-to-dlc snpe-tensorflow-to-dlc snpe-onnx-to-dlc* snpe-diagview snpe-dlc-info snpe-dlc-quantize snpe_bench.py *Coming soon 6

7 Using the Snapdragon NPE 7

8 Snapdragon NPE Workflow GoogleNet Inception SSD Alexnet ResNet User Defined Layer (UDL) enables prototyping of layers not yet supported MobileNet Caffe2 SqueezeNet Faster RCNN TensorFlow, the TensorFlow logo and any related marks are trademarks of Google Inc 8

9 Input Image Formatting In the Snapdragon NPE, images must be presented as a tensor of shape (height x width x channel), where channel is the fastest-changing dimension. (1, 1) (1, 1) (1, 1) (1,2) (1,2) (1,2) (H,W) See $SNPE_ROOT/models/alexnet/scripts/create_alexnet_raws.py in the SDK NCHW NCHW Mean Image Convert NHWC Caffe Format SNPE Format For current Snapdragon NPE SDK release, N=1. Batch support coming in future release. 9

10 Quantized vs Non-Quantized Models Non-quantized DLC files use 32 bit floating point representations of network parameters. Quantized DLC files use 8 bit fixed point representations of network parameters and are smaller..dlc File DSP Runtime (8-bit) CPU/GPU Runtime (32 bit) Quantized.dlc File 10

11 Making a Snapdragon NPE Enabled Application App setup bool useusersuppliedbuffers = false; // Set the Runtime static zdl::dlsystem::runtime_t runtime = zdl::snpe::snpefactory::isruntimeavailable(zdl::dlsystem::runtime_t::gpu))? zdl::dlsystem::runtime_t::gpu : zdl::dlsystem::runtime_t::cpu; // Load DLC Container std::unique_ptr<zdl::dlcontainer::idlcontainer> container = zdl::dlcontainer::idlcontainer::open(dlcpath); // Build SNPE instance zdl::snpe::snpebuilder snpebuilder(container); std::unique_ptr<zdl::snpe::snpe> snpe = snpebuilder.setoutputlayers({}).setruntimeprocessor(runtime).setudlbundle(udlbundle).setuseusersuppliedbuffers(useusersuppliedbuffers).build(); 11

12 Making a Snapdragon NPE Enabled Application Running the network (ITensor) // Load the inputs std::unique_ptr<zdl::dlsystem::itensor> inputtensor = loadinputtensor(snpe, fileline); // See SDK docs static zdl::dlsystem::tensormap outputtensormap; // Run the network snpe.execute(inputtensor, outputtensormap); zdl::dlsystem::stringlist tensornames = outputtensormap.gettensornames(); // Access the results std::for_each( tensornames.begin(), tensornames.end(), [&](const char* name){ } auto tensorptr = outputtensormap.gettensor(name); for ( auto it = tensorptr->cbegin(); it!= tensorptr->cend(); ++it ){ }); float f = *it;... 12

13 Making a SNPE Enabled Application Running the network (UserBuffer) // Load the Inputs loadinputuserbuffer(applicationinputbuffers, snpe, fileline); // See SDK Docs // Run the Network snpe.execute(inputmap, outputmap); const zdl::dlsystem::stringlist& outputbuffernames = outputmap.getuserbuffernames(); // Access the results std::for_each(outputbuffernames.begin(), outputbuffernames.end(), [&](const char* name) { auto buffer = applicationoutputbuffers.at(name).data(); }); float *f; for (auto i=0; i< buffer.size(); i+=sizeof(float)) { } f = reinterpret_cast<float *>(&buffer[i]);... 13

14 Thank you! Follow us on: For more information, visit us at: & Nothing in these materials is an offer to sell any of the components or devices referenced herein Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm is a trademark of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to Qualcomm may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.