Deploying Deep Learning Networks to Embedded GPUs and CPUs

Transcription

1 Deploying Deep Learning Networks to Embedded GPUs and CPUs Rishu Gupta, PhD Senior Application Engineer, Computer Vision 2015 The MathWorks, Inc. 1

2 MATLAB Deep Learning Framework Access Data Design + Train Deploy Manage large image sets Automate image labeling Easy access to models Acceleration with GPU s Scale to clusters 2

3 Multi-Platform Deep Learning Deployment Desktop Data-center Nvidia TX1, TX2, TK1 Raspberry pi Mobile Beagle bone Embedded 3

4 Multi-Platform Deep Learning Deployment Need code that takes advantage of: NVIDIA CUDA libraries, including cudnn and TensorRT Intel Math Kernel Library for Deep Neural Networks (MKL-DNN) for Intel processors ARM Compute libraries for ARM processors 4

5 Multi-Platform Deep Learning Deployment Need code that takes advantage of: NVIDIA CUDA libraries, including cudnn and TensorRT NVIDIA Jetson TX1 board Intel Math Kernel Library for Deep Neural Networks (MKL-DNN) for Intel processors Android Phone ARM Compute libraries for ARM processors Intel Xeon Desktop PC Raspberry Pi Board 5

6 Algorithm Design to Embedded Deployment Workflow Conventional Approach Desktop GPU C++ Challenges Integrating multiple libraries and packages Verifying and maintaining multiple implementations Algorithm & vendor lock-in 1 High-level language Deep learning framework Large, complex software stack 2 C/C++ Low-level APIs Application-specific libraries Embedded GPU C++ 3 C/C++ Target-optimized libraries Optimize for memory & speed 6

7 Solution- GPU Coder for Deep Learning Deployment Target Libraries Application logic Intel MKL-DNN Library GPU Coder NVIDIA TensorRT & cudnn Libraries ARM Compute Library 7

8 Deep Learning Deployment Workflows INFERENCE ENGINE DEPLOYMENT INTEGRATED APPLICATION DEPLOYMENT Trained DNN Preprocessing Postprocessing cnncodegen codegen Portable target code Portable target code 8

9 Workflow for Inference Engine Deployment INFERENCE ENGINE DEPLOYMENT Trained DNN Steps for inference engine deployment 1. Generate the code for trained model >> cnncodegen(net, 'targetlib', 'cudnn ) 2. Copy the generated code onto target board cnncodegen Portable target code 3. Build the code for the inference engine >> make C./codegen f mk 4. Use hand written main function to call inference engine 5. Generate the exe and test the executable >> make C./ 9

10 How to get a Trained DNN into MATLAB? Model importer Train in MATLAB Trained DNN Reference model Transfer learning 10

11 Deep Learning Inference Deployment Target Libraries Model importer Intel MKL-DNN Library Train in MATLAB Trained DNN NVIDIA TensorRT & cudnn Libraries Reference model Transfer learning ARM Compute Library 11

12 Building DNN from Scratch Load Training Data Build Layer Architecture Set Training Options Train Network %% Create a datastore imds = imagedatastore( Data,... 'IncludeSubfolders',true,'LabelSource','foldernames'); num_classes = numel(unique(imds.labels)); %% Build layer architecture layers = [imageinputlayer([ ]) convolution2dlayer(5,20) relulayer() maxpooling2dlayer(2,'stride',2) fullyconnectedlayer(512) fullyconnectedlayer(2) softmaxlayer() classificationlayer()]; %% Set Training Options trainopts = trainingoptions( 'sgdm',... 'MiniBatchSize', minibatchsize,... 'Plots', 'training-progress ); %% Train Network net = trainnetwork(imds, layers, trainopts); 12

13 Pedestrian Detection DNN Deployment on ARM Processor layers = [imageinputlayer([ ]) convolution2dlayer(5,20) relulayer() maxpooling2dlayer(2,'stride',2) CrossChannelNormalizationLayer(5,'K',1); convolution2dlayer(5,20) relulayer() maxpooling2dlayer(2,'stride',2) fullyconnectedlayer(512) fullyconnectedlayer(2) softmaxlayer() classificationlayer()]; 13

14 Pedestrian Detection DNN Deployment on ARM Processor ARM Neon instruction set architecture Example: ARM Cortex A ARM Compute Library Low-level Software functions Computer vision, machine learning etc Pedestrian detection on Raspberry pi 14

15 Deep Learning Inference Deployment Model importer Target Libraries NVIDIA TensorRT & cudnn Libraries Train in MATLAB Trained DNN Intel MKL-DNN Library Reference model Transfer learning ARM Compute Library 15

16 Importing DNN from Open Source Framework Caffe Model Importer (including Caffe Model Zoo) importcaffelayers importcaffenetwork network = importcaffenetwork(protofile, 'yolo.caffemodel'); TensorFlow-Keras Model Importer importkeraslayers importkerasnetwork 16

17 Deep Learning Inference Deployment Model importer Target Libraries NVIDIA TensorRT & cudnn Libraries Object Detection Train in MATLAB Trained DNN Intel MKL-DNN Library Reference model Transfer learning ARM Compute Library 17

18 Deep Learning Inference Deployment Model importer Target Libraries NVIDIA TensorRT & cudnn Libraries Train in MATLAB Trained DNN Intel MKL-DNN Library Reference model Transfer learning ARM Compute Library 18

19 Deep Learning Inference Deployment Model importer Target Libraries NVIDIA TensorRT & cudnn Libraries Train in MATLAB Trained DNN Intel MKL-DNN Library Reference model Transfer learning ARM Compute Library 19

20 Layered Architecture for Segnet- Semantic Segmentation DAG Network Total number of layers: 91 20

21 NVIDIA TensorRT PROGRAMMABLE INFERENCE ACCELERATOR TESLA P4 TensorRT JETSON TX2 DRIVE PX 2 NVIDIA DLA TESLA V100 21

22 Performance Summary (VGG-16) on TitanXP GPU Coder (TensorRT int8) GPU Coder (TensorRT fp32) GPU Coder (cudnn fp32) MATLAB (cudnn fp32)

23 How Good is Generated Code Performance? Performance of CNN inference (Alexnet) on Titan XP GPU Performance of CNN inference (Alexnet) on Jetson (Tegra) TX2 23

24 Frames per second Alexnet Inference on NVIDIA Titan Xp GPU Coder + TensorRT (3.0.1, int8) GPU Coder + TensorRT (3.0.1) GPU Coder + cudnn MXNet (1.1.0) TensorFlow (1.6.0) Batch Size CPU Intel(R) Xeon(R) CPU E GHz Testing platform GPU cudnn Pascal Titan Xp v7 24

25 Frames per second VGG-16 Inference on NVIDIA Titan Xp GPU Coder + TensorRT (3.0.1, int8) GPU Coder + TensorRT (3.0.1) GPU Coder + cudnn MXNet (1.1.0) TensorFlow (1.6.0) Batch Size CPU Intel(R) Xeon(R) CPU E GHz Testing platform GPU cudnn Pascal Titan Xp v7 25

26 Frames per second Alexnet Inference on Jetson TX2: Frame-Rate Performance GPU Coder + TensorRT GPU Coder + cudnn C++ Caffe (1.0.0-rc5) Batch Size 26

27 Brief Summary DNN libraries are great for inference, GPU coder generates code that takes advantage of: NVIDIA CUDA libraries, including cudnn, and TensorRT Intel Math Kernel Library for Deep Neural Networks (MKL-DNN) ARM Compute libraries for mobile platforms 27

28 Brief Summary DNN libraries are great for inference, GPU coder generates code that takes advantage of: NVIDIA CUDA libraries, including cudnn, and TensorRT but, applications require more than just inference Intel Math Kernel Library for Deep Neural Networks (MKL-DNN) ARM Compute libraries for mobile platforms 28

29 Deep learning Workflows- Integrated Application Deployment INTEGRATED APPLICATION DEPLOYMENT Preprocessing Postprocessing codegen Portable target code 29

30 Traffic sign detection and recognition YOLO Recognition net Object detection DNN Strongest Bounding Box Classifier DNN 30

31 Traffic sign detection and recognition 31

32 Traffic sign detection and recognition 32

33 GPU Coder Helps You Deploy Applications to GPUs Faster GPU Coder CUDA Kernel creation Memory allocation Data transfer minimization Library function mapping Loop optimizations Dependence analysis Data locality analysis GPU memory allocation Data-dependence analysis Dynamic memcpy reduction 33

34 CUDA Code Generation from GPU Coder app Integrated editor and simplified workflow for code generation 34

35 Summary- GPU Coder MATLAB algorithm (functional reference) Build type Call CUDA from MATLAB directly Desktop GPU Call CUDA from (C++) handcoded main() Desktop GPU Embedded GPU Call CUDA from (C++) hand-coded main()..mex.lib Cross-compiled.lib C++ C++ 1 Functional test 2 Deployment unit-test 3 Deployment integration-test 4 Real-time test 35

36 MATLAB Deep Learning Framework DEPLOYMENT Intel MKL-DNN Library Access Data Manage large image sets Automate image labeling Easy access to models Design + Train Acceleration with GPU s Scale to clusters NVIDIA TensorRT & cudnn Libraries ARM Compute Library 36

37 Speaker Details LinkedIn: Contact MathWorks India Products/Training Enquiry Booth Call: Share your experience with MATLAB & Simulink on Social Media Use #MATLABEXPO I use #MATLAB because Attending #MATLABEXPO Examples I use #MATLAB because it helps me be a data scientist! Attending #MATLABEXPO Learning new capabilities in #MATLAB and #Simulink at #MATLABEXPO. Share your session feedback: Please fill in your feedback for this session in the feedback form 37