GPU Coder: Automatic CUDA and TensorRT code generation from MATLAB

Transcription

1 GPU Coder: Automatic CUDA and TensorRT code generation from MATLAB Ram Kokku 2018 The MathWorks, Inc. 1

2 GPUs and CUDA programming faster Performance CUDA OpenCL C/C++ GPU Coder MATLAB Python Ease of programming (expressivity, algorithmic, ) easier GPUs are hardware on steroids, but, programming them is hard 2

3 Consider an example: saxpy Scalarized MATLAB Vectorized MATLAB Automatic compilation from an expressive language to a high-performance language 3

4 Deep Learning applications TensorRT Deep learning Traffic Sign Recognition TensorRTis great for inference, but, applications require more than inference 4

5 GPU Coder is new technology released in September 2017 Accelerated implementation of parallel algorithms on GPUs Neural Networks Deep Learning, machine learning Image Processing and Computer Vision Image filtering, feature detection/extraction Signal Processing and Communications FFT, filtering, cross correlation, 5x faster than TensorFlow 2x faster than mxnet 60x faster than CPUs for stereo disparity 20x faster than CPUs for FFTs 5

6 Example: Lidar semantic segmentation 6

7 Talk outline Introduction GPU Coder internals Automatic parallelization Memory optimization Deep learning compilation Application demo: Lidar processing in MATLAB using deep learning 7

8 GPU Coder automatically extracts parallelism from MATLAB 1. Scalarized MATLAB ( for-all loops) Infer CUDA kernels from MATLAB loops 2. Vectorized MATLAB (math operators and library functions) 3. Composite functions in MATLAB (maps to cublas, cufft, cusolver, cudnn, TensorRT) Library replacement 8

9 From a loop to a CUDA kernel for k = 1:n t = A(k).* X(k); C(k) = t + Y(k); { } Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions mykernel<<< f(n) >>>(A, X, Y, C, n); static global mykernel(a, X, Y, C, n) { int k = getthreadindex(n); } int t = A[k] * X[k]; C[k] = t + Y[k]; Is this loop parallel? Y Compute kernel size Classify kernel variables (input, output, local) Create kernel from loop body Depence analysis to understand the iteration space f(n) Ins: A, X, Y, n Outs: C Local: t, k 9

10 From a loop-nesting to CUDA kernels Imperfect Loops for i = 1:p (outer prologue code) for j = 1:m for k = 1:n (inner loop) (outer epilogue code) Perfect Loops for i = 1:p for j = 1:m for k = 1:n (inner loop) Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions Fundamental requirement: Loops need to be contiguous for parallelization 10

11 From a loop-nesting to CUDA kernels Imperfect Loops Perfect Loops Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions for i = 1:p (outer prologue code) for j = 1:m for k = 1:n (inner loop) (outer epilogue code) Loop Perfectization (if possible) for i = 1:p for j = 1:m for k = 1:n (outer prologue code) (inner loop) if k == n (outer epilogue code) Fundamental requirement: Loops need to be contiguous for parallelization 11

12 From a loop-nesting to CUDA kernels Example 1 Example 2 Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions (M x N) (K x P) for a = 1:M for b = 1:N (outer prologue code) for c = 1:K for d = 1:P (inner loop) (P) (MxN + KxL) for i = 1:P for a = 1:M for b = 1:N (inner loop) for x = 1:K for y = 1:L (inner loop) Find parallel loops Partition loop nesting Depence analysis Heuristic may favor larger iteration space Fundamental requirement: Loops need to be contiguous for parallelization 12

13 From a loop-nesting to CUDA kernels Example 1 Example 2 Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions (M x N) (K x P) for a = 1:M for b = 1:N (outer prologue code) for c = 1:K for d = 1:P (inner loop) (P) (MxN + KxL) for i = 1:P for a = 1:M for b = 1:N (inner loop) for x = 1:K for y = 1:L (inner loop) Find parallel loops Partition loop nesting Create kernel for each partition Depence analysis Heuristic may favor larger iteration space Use process from single loop conversion Fundamental requirement: Loops need to be contiguous for parallelization 13

14 From vectorized MATLAB to CUDA kernels output(:, 1) = (input(:, 1) x_im).* factor; Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions Assume the following sizes: output : M x 3 input : M x 3 x_im : M x 1 factor : M x 1 for i = 1:M diff(i) = input(i, 1) x_im(i); for a = 1:M output(i, 1) = diff(i) * factor(i); for i = 1:M diff(i) = input(i, 1) x_im(i); output(i, 1) = diff(i) * factor(i); Scalarization Loop Fusion Reduce to for-loops Create larger parallel loops (and hence CUDA kernels) 14

15 From vectorized MATLAB to CUDA kernels output(:, 1) = (input(:, 1) x_im).* factor; Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions Assume the following sizes: output : M x 3 input : M x 3 x_im : M x 1 factor : M x 1 for i = 1:M diff(i) = input(i, 1) x_im(i); for a = 1:M output(i, 1) = diff(i) * factor(i); for i = 1:M diff(i) = input(i, 1) x_im(i); output(i, 1) = diff(i) * factor(i); for i = 1:M tmp = input(i, 1) x_im(i); output(i, 1) = tmp * factor(i); Scalarization Loop Fusion Scalar Replacement Reduce to for-loops Create larger parallel loops (and hence CUDA kernels) Reduce temp matrices to temp scalars 15

16 From composite functions to optimized CUDA Extracting parallelism in MATLAB 1. Scalarized MATLAB (for loops) 2. Vectorized MATLAB 3. Composite functions Core math Matrix multiply (cublas) Linear algebra (cusolver) FFT functions (cufft) Convolution Image processing Computer vision imfilter imresize imerode imdilate bwlabel imwarp Neural Networks Deep learning inference (cudnn, TensorRT) Over 300+ MATLAB functions are optimized for CUDA code generation 16

17 Talk outline Introduction GPU Coder internals Automatic parallelization Memory optimization Deep learning compilation Application demo: Lidar processing in MATLAB using deep learning 17

18 Optimizing CPU-GPU data movement is a challenge A = for i = 1:N A(i) imfilter Where is the ideal placement of memcpy? A = cudamemcpyhtod(ga, a); kernel1<<< >>>(ga) cudamemcpydtoh( ) cudamemcpyhtod( ) imfilter_kernel( ) cudamemcpydtoh( ) Naïve placement Optimized placement K1 K2 K3 K1 K2 K3 18

19 GPU Coder optimizes memcpy placement cudamemcpy *not* needed A(:) =. C(:) =. for i = 1:N. gb = kernel1(ga); ga = kernel2(gb); if (some_condition) gc = kernel3(ga, gb);.. = C; cudamemcpy *definitely* needed cudamemcpy *may be* needed Assume ga, gb and gc are mapped to GPU memory Observations Equivalent to Partial redundancy elimination (PRE) Dynamic strategy track memory location with a status flag per variable Use-Def to determine where to insert memcpy Generated (pseudo) code A(:) = A_isDirtyOnCpu = true; for i = 1:N if (A_isDirtyOnCpu) cudamemcpy(ga, A); A_isDirtyOnCpu = false; gb = kernel1(ga); ga = kernel2(gb); if (somecondition) gc = kernel3(ga, gb); C_isDirtyOnGpu = true; if (C_isDirtyOnGpu) cudamemcpy(c, gc); C_isDirtyOnGpu = false; = C; 19

20 GPU memory hierarchy is deep Memory Visibility Heuristics/notes GPU Coder support Global memory CPU + GPU Share data b/w CPU and GPU Yes Local memory/registers per GPU thread Thread local data Yes Shared memory per GPU block Shared data between threads Yes Texture memory CPU + GPU Shared read-only data with 2D alignment Future Constant memory GPU Read-only constants Yes 20

21 GPU Coder automatically maps data to shared memory Input image Conv. kernel Output image cols kh kw rows Dotprod GPU Coder automatically creates shared memory for many MATLAB image processing functions: imfilter, imerode, imdilate, conv2, 21

22

23

24 NVIDIA Hardware Support Package (HSP) Simple out-of-box targeting to NVIDIA boards: Jetson Drive platform 24

25 Talk outline Introduction GPU Coder internals Automatic parallelization Memory optimization Deep learning compilation Application demo: Lidar processing in MATLAB using deep learning 25

26 Deep learning workflow in MATLAB DNN design + training Model importer Train in MATLAB Model importer Trained DNN Design in MATLAB Manage large image sets Automate data labeling Easy access to models Training in MATLAB Acceleration with GPU s Scale to clusters 26

27 Deep learning workflow in MATLAB DNN design + training Model importer Application design Application logic Train in MATLAB Trained DNN Model importer 27

28 Deep learning workflow in MATLAB DNN design + training Application design Standalone Deployment Model importer Application logic C++/CUDA + TensorRT Train in MATLAB Trained DNN Model importer C++/CUDA + cudnn 28

29 Deep learning workflow in MATLAB DNN design + training Application design Standalone Deployment Model importer Application logic C++/CUDA + TensorRT Train in MATLAB Trained DNN Model importer C++/CUDA + cudnn 29

30 Traffic sign detection and recognition Object detection DNN Strongest Bounding Box Classifier DNN 30

31 GPU Coder allows for choice in deployment (cudnn, TensorRT) Application logic C++/CUDA + cudnn C++/CUDA + TensorRT C++/CUDA + TensorRT (int8) 31

32 Performance summary (VGG-16) on TitanXP GPU Coder (TensorRT int8) GPU Coder (TensorRT fp32) GPU Coder (cudnn fp32) MATLAB (cudnn fp32)

33 MATLAB and GPU Coder support state-of-art classification networks GoogLeNet ResNet50 Alexnet vs Squeezenet Network # Layers Size Frame-rate (GPU Coder) Alexnet MB 500 Fps ResNet MB 160 Fps GoogLeNet MB 190 Fps Squeezenet 68 5 MB 615 Fps 33

34 ResNet-50 Inference on NVIDIA Titan V Frames per second MATLAB GPU Coder + TensorRT 4 (int8) MATLAB GPU Coder + TensorRT 4 MATLAB GPU Coder + cudnn PyTorch Tensorflow Batch Size Testing platform CPU: Intel Xeon CPU E GHz GPU: NVIDIA Titan-V 34

35 ResNet-50 Inference on NVIDIA Titan V MATLAB GPU Coder + TensorRT 4 (int8) Frames per second Tensorflow + TensorRT 4 (int8) MATLAB GPU Coder + TensorRT 4 (fp32) Tensorflow + TensorRT 4 (fp32) Batch Size Testing platform CPU: Intel Xeon CPU E GHz GPU: NVIDIA Titan-V 35

36 Lidar processing application design is easy in MATLAB DNN design + training Model importer Application design Application logic Train in MATLAB Trained DNN Model importer 36

37 Lidar processing application design is easy in MATLAB DNN design + training Model importer Application design Application logic Train in MATLAB Trained DNN Model importer 37

38 Talk outline Introduction GPU Coder internals Automatic parallelization Memory optimization Deep learning compilation Application demo: Lidar processing in MATLAB using deep learning 38

39 Why Use Lidar and Deep Learning for Autonomous Vehicles? Why use Lidar? Provides accurate 3-D structure of scene Required sensor for autonomous driving Why use deep learning? Best accuracy High-performance inference with GPU Coder Classifying Point Cloud Clusters Ground Cars Lidar Sematic Segmentation 39

40 What Does Lidar Data Look Like? 40

41 Lidar processing application design is easy in MATLAB DNN design + training Application design Standalone Deployment Application logic C++/CUDA + TensorRT Data prep, labeling Training Trained DNN C++/CUDA + cudnn 41

42 Data preparation and labeling of Lidar is a challenge DNN design + training Accessing lidar data Lidar preprocessing Labeling lidar data Training Trained DNN 42

43 Access and Visualize Lidar Data Access Stored Lidar Data Velodyne file I/O (pcap) Individual point clouds (.pcd,ply) Visualize Lidar Data Streaming Lidar player Static point cloud display Point cloud differences 43

44 Lidar Preprocessing Remove Ground Fit plane using RANSAC Cluster Segment clusters using Euclidean distance 44

45 Ground Truth Labeling of Lidar Data 45

46 Ground Truth Labeling of Lidar Data 46

47 Ground Truth Labeling of Lidar Data 47

48 Organize Data for Training Raw Point Cloud Data Ground Truth Labels Transformed to Label Mask Project to 2D 48

49 Create Network Architecture Easy MATLAB API to create network 49

50 Deployment using GPU Coder C++/CUDA + TensorRT C++/CUDA + cudnn 50

51 Lidar semantic segmentation 51

52 Deep learning workflow in MATLAB DNN design + training Application design Standalone Deployment Model importer Application logic C++/CUDA + TensorRT Train in MATLAB Trained DNN Model importer C++/CUDA + cudnn 52

53 Thank you 53

54 Backup 54

55 Outline #1 (without lidar demo) Parallelism and GPUs (why GPU Coder, use-cases etc) [4 mins] Compiler overview [15 mins] Targeting loops, Memory optimizations, Library replacements Demo: Workflow demo using stereo disparity (ext to run on HW) Benchmarks: IPT functions, stereo disparity Workflow and hardware targeting [10 mins] From algorithm to embedded Demo: Ext stereo-disparity demo to run on Jetson or DrivePX2 Deep learning compiler overview [15 mins] Design philosophy, Dataflow graph optimizations, Library plugins Demo: Applications using DNN like traffic-signs, age-detection (both cudnn + TensorRT) Demo: Quantization with TensorRT/int8 (logo detection) Benchmarks: alexnet, resnet, segnet 55

56 Outline #2 (with lidar demo) Parallelism and GPUs (why GPU Coder, use-cases etc) [4 mins] Compiler and workflow [15 mins] Loops, memory, library mapping and optimizations Rapid prototyping and deployment with HSP Demo: Workflow demo using stereo disparity (ext to run on HW) Benchmarks: IPT functions, stereo disparity Deep learning compiler overview [15 mins] Design philosophy, dataflow graph optimizations, library plugins Demo: Applications using DNN like traffic-signs, age-detection (both cudnn + TensorRT) Demo: Quantization with TensorRT/int8 (logo detection) Benchmarks: alexnet, resnet, segnet End-to-End Workflow demo (lidar) [15 mins] Labeling, training Deployment on DrivePX2 56

57 Demos for talk Workflow demo (alexnet, resnet, traffic signs, stereo disparity) Board demo (Jetson, DrivePX2) TensorRT demos (traffic signs, logonet + int8) Arvind s lidar demo (-to-) Performance benchmarks IPT functions Deep learning inference (GPC (cudnn, TensorRT) vs Caffe2, TF, MxNet) Alexnet Resnet GoogleNet Segnet 57

58 Localization Planning Autonomous systems Perception Controls 58

59 Consider an example: saxpy Scalarized MATLAB Vectorized MATLAB 59

60 Deep learning workflow in MATLAB Model importer Application logic C++/CUDA + TensorRT Train in MATLAB Trained DNN Model importer C++/CUDA + cudnn 60

61 Re-targetability: A target-agnostic interface layer Pre-trained DL network Front-End Cross-layer optimizations C back- GPU Back DL structure Auto-generated Target-agnostic code malloc, scheduling ARM Neon interface ARM Mali interface MKL-DNN interface cudnn interface Hand-written interface layer code TensorRT interface Algorithm only ARM CL ARM CL MKL-DNN library Vor library cudnn library TensorRT 61

62 Deep learning workflow in MATLAB DNN design + training Model importer Application logic C++/CUDA + TensorRT Train in MATLAB Trained DNN Model importer C++/CUDA + cudnn 62

63 Design pattern: Stencil kernel Problem: how to auto-generate hand-coded version from: 1. for x=1:orow Key insight: this part always the same 2. for y=1:ocol 3. C(x,y) = f ( In(x:x+window_x, y:y+window_y)); Only need user to specify this part f() takes in an input window and returns a scalar output. e.g. convolution : x = sum( input(:).* kernel(:)) 63

64 Design pattern: Stencil kernel n m kernel myfunc im out 64

65 GPU Coder automatically maps data to shared memory Input image Conv. kernel Output image cols kh kw rows GPU Coder automatically infers data locality Creates GPU shared memory Collaborative loading Re-write algorithm to use shared memory Dotprod Optimized image processing toolbox functions: conv2 imfilter imerode Imdilate 65

66 Memory Management Modes are Configurable Standard (explicit copy) cudamalloc Separate host and GPU memory allocations (and variables) CUDA 7 or earlier for desktop GPUs Optimal dynamic memcpy minimization Unified memory (implicit copy) cudamallocmanaged No need for separate variables CUDA 8 or embedded GPUs Optimal dynamic device-sync 66

67 Design pattern: Stencil kernel n m kernel myfunc im Optimized image processing toolbox functions: conv2 imfilter imerode Imdilate out 67

68 Example generated code: 1-D FIR filter Shared memory decl Collaborative load Sync point Algorithm with translated accesses to shared memory 68

69 Inference benchmarking on TitanXP Alexnet VGG TensorFlow (cudnn-fp32)

70 Inference benchmarking on TitanXP MATLAB GPU Coder (TensorRT-int8) MATLAB GPU Coder (TensorRT-fp32) MATLAB GPU Coder (cudnn-fp32) mxnet (cudnn-fp32) TensorFlow (cudnn-fp32) Images / sec Batch size Batch size Alexnet VGG-16 70

71 Deep Learning Workflow Access Data Preprocess and Label Train Network Inference Raw Point Clouds Label Data Create Deep Network Train on Multi- GPU 71

72 Access and Visualize Lidar Data Access Data Access Stored Lidar Data Velodyne file I/O (pcap) Individual point clouds (.pcd,ply) Visualize Lidar Data Streaming Lidar player Static point cloud display Point cloud differences 72

73 Lidar Preprocessing Preprocess and Label ROI + Remove Ground Fit plane using RANSAC Cluster Segment clusters using Euclidean distance 73

74 Ground Truth Labeling of Lidar Data Preprocess and Label 74

75 Ground Truth Labeling of Lidar Data Preprocess and Label 75

76 Ground Truth Labeling of Lidar Data Preprocess and Label 76

77 Organize Data for Training Train Network Raw Point Cloud Data Ground Truth Labels Transformed to Label Mask Project to 2D 77

78 Create Network Architecture Train Network Easy API to create network structure 78

79 Compile Model to CUDA Code Inference GPU Coder 79