CNN Basics. Chongruo Wu

Transcription

1 CNN Basics Chongruo Wu

2 Overview Forward: compute the output of each layer Back propagation: compute gradient Updating: update the parameters with computed gradient

3 Agenda 1. Forward Conv, Fully Connected, Pooing, Non-linear Function Loss functions 2. Back Propagation, Computing Gradient Chain rule 3. Updating Parameters SGD 4. Training

4 Agenda 1. Forward Conv, Fully Connected, Pooing, non-linear Function Loss functions 2. Backward, Computing Gradient Chain rule 3. Updating Parameters SGD 4. Training

5 Before Deep Learning Slide credit: Kristen Grauman

6 Before Deep Learning Slide credit: Kristen Grauman

7 Convolution ( One Channel )

8 Convolution, One Channel Gif Animation, Kernels are learned. Here we show the forward process After each iteration, parameters of kernels are changed.

9 Convolution, Padding Gif Animation, Pad = 1 output size = input size

10 Convolution, Stride Gif Animation, Stride > 1

11 Dilated Convolution Gif Animation, Dilated Stride > 1

12 Convolution ( Multiple Channels )

13 Convolution, Multiple Channels Representation for Feature Map: ( batch, channels, height, width ) For input image ( batch, 3, height, width ) R,G,B

14 Andrej Karpathy, Bay Area Deep Learning School, 2016

15 Andrej Karpathy, Bay Area Deep Learning School, 2016

16 Andrej Karpathy, Bay Area Deep Learning School, 2016

17 Andrej Karpathy, Bay Area Deep Learning School, 2016

18 Andrej Karpathy, Bay Area Deep Learning School, 2016

19 Andrej Karpathy, Bay Area Deep Learning School, 2016

20 In pytorch, conv2 = nn.conv2d(3, 6, kernel_size=5) Andrej Karpathy, Bay Area Deep Learning School, 2016

21 Andrej Karpathy, Bay Area Deep Learning School, 2016

22 Andrej Karpathy, Bay Area Deep Learning School, 2016

23 Andrej Karpathy, Bay Area Deep Learning School, 2016

24 Andrej Karpathy, Bay Area Deep Learning School, 2016

25 Convolution, Extended Work Depthwise Convolution Model compression MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

26 Convolution, Extended Work Input Feature Map MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

27 Convolution, Extended Work Deformable Convolution Deformable Convolutional Networks

28 Convolution, Extended Work Deformable Convolution Deformable Convolutional Networks

29 Non-Linear Function

30 Non-Linear Function Conv -> ReLU -> Conv Why we need Non-Linearity Function? Conv, linear operation Two consecutive convolutional layers (no ReLU) are considered as one convolutional layer. Rectified Linear Unit (ReLU)

31 Extended Work, PReLU Parametric Rectified Linear Unit (PReLU) Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

32 Andrej Karpathy, Bay Area Deep Learning School, 2016

33 Andrej Karpathy, Bay Area Deep Learning School, 2016

34 Pooling

35 Andrej Karpathy, Bay Area Deep Learning School, 2016

36 Andrej Karpathy, Bay Area Deep Learning School, 2016

37 Pooling Max pooling Average pooling

38 Fully Connected Layer

39 Fully Connected Layer Representation for Feature Map: ( batch, channels, height, width ) Fully Connected Layer: ( batch, channels, 1, 1) or ( batch, channels ) #parameters: input_channel * output_channel Stanford cs231n.

40 Fully Connected Layer Usually we apply reshape operation for input feature map ( n, c, h, w ) -> ( n, c*h*w, 1, 1) or ( n, c*h*w ) #parameters: input_channel * output_channel Stanford cs231n.

41 Convolutional Vs. Fully Connected Layer Convolutional Layer Local information Kernels are shared Fully Connected Layer Global information #parameters large Stanford cs231n.

42 Agenda 1. Forward Conv, Fully Connected, Pooing, non-linear Function Loss functions 2. Back Propagation, Computing Gradient Chain rule 3. Updating Parameters SGD 4. Training

43 Overview Forward: compute output of each layer Back propagation: compute gradient Updating: update the parameters with computed gradient

44 Loss Functions Softmax Regression Contrastive/Triplet Loss

45 Stanford cs231n.

46 Andrej Karpathy, Bay Area Deep Learning School, 2016

47 Stanford cs231n.

48 Stanford cs231n.

49 Stanford cs231n.

50 y_i is the groundtruth Stanford cs231n.

51 y_i is the groundtruth Stanford cs231n.

52 Stanford cs231n.

53 Groundtruth (one hot vector) [ 1, 0, 0] Stanford cs231n.

54 Groundtruth (one hot vector) [ 1, 0, 0] Stanford cs231n.

55 Stanford cs231n.

56 Stanford cs231n.

57 Loss Functions Softmax Regression Bounding box regression Contrastive/Triplelet Loss Object(face) recognition, clustering

58 Andrej Karpathy, Bay Area Deep Learning School, 2016

59 Define Network

60 Network Case Study (optional)

61 Andrej Karpathy, Bay Area Deep Learning School, 2016

62 Andrej Karpathy, Bay Area Deep Learning School, 2016

63 VGG Network Andrej Karpathy, Bay Area Deep Learning School, 2016

64 VGG Network Andrej Karpathy, Bay Area Deep Learning School, 2016

65 Agenda 1. Forward Conv, Fully Connected, Pooing, non-linear Function Loss functions 2. Back Propagation, Computing Gradient Chain rule 3. Updating Parameters SGD 4. Training

66 Overview Forward: compute output of each layer Backward: compute gradient Update: update the parameters with computed gradient

67 Back Propagation Parameters: Convolutional Layer, Fully Connected Layer PReLU, Batch Normalization No Parameters: Pooling, ReLU, Dropout

68 Stanford cs231n.

69 Stanford cs231n.

70 Stanford cs231n.

71 Stanford cs231n.

72 Stanford cs231n.

73 Stanford cs231n.

74 Stanford cs231n.

75 Stanford cs231n.

76 Stanford cs231n.

77 Stanford cs231n.

78 Stanford cs231n.

79 Stanford cs231n.

80 Back Propagation Convolutional Layer Fully Connected Layer

81 Back Propagation Once you finish your computation you can call.backward() and have all the gradients computed automatically. Stanford cs231n. Pytorch Tutorial.

82 Agenda 1. Forward Conv, Fully Connected, Pooing, non-linear Function Loss functions 2. Backward, Computing Gradient Chain rule 3. Updating Parameters SGD 4. Training

83 Overview Forward: compute output of each layer Backward: compute gradient Update: update the parameters with computed gradient

84 Updating Gradient Stanford cs231n.

85 Updating Gradient Andrew Ng s Machine Learning course, Coursera

86 Updating Gradient decrease the learning rate after several epochs Deep Residual Learning for Image Recognition

87 Stanford cs231n.

88 Updating Gradient Momentum Stanford cs231n.

89 Stanford cs231n.

90 Other updating Strategies SGD Momentum Rmsprop Adagrad Adam Gif Animation, Stanford cs231n.

91 Overview Forward: compute output of each layer Backward: compute gradient Update: update the parameters with computed gradient

92 Agenda 1. Forward Conv, Fully Connected, Pooing, non-linear Function Loss functions 2. Back Propagation, Computing Gradient Chain rule 3. Updating Parameters SGD 4. Training

93 Pytorch, zero to all. HKUST

94 Learning Rate

95 Resources

96 Resources Course: Stanford CS231n

97 Resources Bay Area Deep Learning School

98 Thank You