IMPLEMENTING DEEP LEARNING USING CUDNN 이예하 VUNO INC.

Transcription

1 IMPLEMENTING DEEP LEARNING USING CUDNN 이예하 VUNO INC.

2 CONTENTS Deep Learning Review Implementation on GPU using cudnn Optimization Issues Introduction to VUNO-Net

3 DEEP LEARNING REVIEW

4 BRIEF HISTORY OF NEURAL NETWORK ADALINE XOR Problem Multi-layered Perceptron (Backpropagation) SVM Deep Neural Network (Pretraining) Perceptron Golden Age Dark Age ( AI Winter ) Electronic Brain S. McCulloch - W. Pitts F. Rosenblatt B. Widrow - M. Hoff M. Minsky - S. Papert D. Rumelhart - G. Hinton - R. Wiliams V. Vapnik - C. Cortes G. Hinton - S. Ruslan Adjustable Weights Weights are not Learned Learnable Weights and Threshold XOR Problem Solution to nonlinearly separable problems Big computation, local optima and overfitting Limitations of learning prior knowledge Kernel function: Human Intervention Hierarchical feature Learning

5 MACHINE/DEEP LEARNING IS EATING THE WORLD!

6 BUILDING BLOCKS Restricted Boltzmann machine Auto-encoder Deep belief Network Deep Boltzmann machine Generative stochastic networks Recurrent neural networks Convolutional neural netwoks

7 CONVOLUTIONAL NEURAL NETWORKS LeNet-5 (Yann LeCun, 1998)

8 CONVOLUTIONAL NEURAL NETWORKS Alex Net (Alex Krizhevsky et. al., 2012) GoogleNet (Szegedy et. Al., 2015)

9 CONVOLUTIONAL NEURAL NETWORKS Network Softmax Layer (Output) Forward Pass Forward Pass Backward Pass Layer Fully Connected Layer Pooling Layer Convoluti on Layer Input / Output Weights Neuron activation

10 FULLY CONNECTED LAYER - FORWARD Matrix calculation is very fast on GPU cublas library

11 FULLY CONNECTED LAYER - BACKWARD Matrix calculation is very fast on GPU Element-wise multiplication can be done efficiently using GPU thread

12 CONVOLUTION LAYER - FORWARD w1 w3 x1 x4 x7 w1 w1 w3 w3 w2 w2 y1 y3 w2 w4 x2 x5 x8 w4 w1 w4 w1 w3 w3 w2 w2 y2 y4 x3 x6 x9 w4 w4

13 CONVOLUTION LAYER - BACKWARD w1 w3 x1 x4 x7 w1 w1 w3 w3 w2 w2 y1 y3 w2 w4 x2 x5 x8 w4 w1 w4 w1 w3 w3 w2 w2 y2 y4 x3 x6 x9 w4 w4

14 CONVOLUTION LAYER - BACKWARD Error x1 x4 x7 L L Gradient x2 x5 x8 w 1 L w 3 L w 2 w 4 x3 x6 x9

15 HOW TO EVALUATE THE CONVOLUTION LAYER EFFICIENTLY? Both Forward and Backward passes can be computed with convolution scheme Lower the convolutions into a matrix multiplication (cudnn) There are several ways to implement convolutions efficiently Fast Fourier Transform to compute the convolution (cudnn_v3) Computing the convolutions directly (cuda-convnet)

16 IMPLEMENTATION ON GPU USING CUDNN

17 INTRODUCTION TO CUDNN cudnn is a GPU-accelerated library of primitives for deep neural networks Convolution forward and backward Pooling forward and backward Softmax forward and backward Neuron activations forward and backward: Rectified linear (ReLU) Sigmoid Hyperbolic tangent (TANH) Tensor transformation functions

18 INTRODUCTION TO CUDNN (VERSION 2) cudnn's convolution routines aim for performance competitive with the fastest GEMM Lowering the convolutions into a matrix multiplication (Sharan Chetlur et. al., 2015)

19 INTRODUCTION TO CUDNN Benchmarks

20 LEARNING VGG MODEL USING CUDNN Data Layer Convolution Layer Pooling Layer Fully Connected Layer Softmax Layer

21 COMMON DATA STRUCTURE FOR LAYER Device memory & tensor description for input/output data & error Tensor Description defines dimensions of data float cudnntensordescriptor_t cudnntensordescriptor_t *d_input, *d_output, *d_inputdelta, *d_outputdelta inputdesc; outputdesc;

22 DATA LAYER

23 CONVOLUTION LAYER Initailization

24 CONVOLUTION LAYER

25 CONVOLUTION LAYER

26 CONVOLUTION LAYER

27 CONVOLUTION LAYER

28 CONVOLUTION LAYER

29 CONVOLUTION LAYER

30 CONVOLUTION LAYER

31 CONVOLUTION LAYER

32 CONVOLUTION LAYER

33 CONVOLUTION LAYER

34 CONVOLUTION LAYER

35 POOLING LAYER / SOFTMAX LAYER

36 OPTIMIZATION ISSUES

37 OPTIMIZATION

38 OPTIMIZATION

39 OPTIMIZATION

40 OPTIMIZATION

41 SPEED

42 SPEED

43 PARALLELISM

44 PARALLELISM

45 PARALLELISM

46 INTRODUCING VUNO-NET

47 THE TEAM

48 VUNO-NET

49 VUNO-NET

50 PERFORMANCE

51 APPLICATION

52 APPLICATION

53 APPLICATION

54 VISUALIZATION

55 VISUALIZATION

56 VISUALIZATION

57

58 THANK YOU