Fei-Fei Li & Justin Johnson & Serena Yeung
|
|
|
- Edwina Stevens
- 5 years ago
- Views:
Transcription
1 Lecture 9-1
2 Administrative A2 due Wed May 2 Midterm: In-class Tue May 8. Covers material through Lecture 10 (Thu May 3). Sample midterm released on piazza. Midterm review session: Fri May 4 discussion section Lecture 9-2
3 Last time: Deep learning frameworks PaddlePaddle Chainer (Baidu) Caffe Caffe2 (UC Berkeley) (Facebook) MXNet CNTK (Amazon) Torch PyTorch (NYU / Facebook) (Facebook) Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of choice at AWS (Microsoft) Deeplearning4j Theano TensorFlow (U Montreal) (Google) And others... Lecture 9-3
4 Last time: Deep learning frameworks (1) Easily build big computational graphs (2) Easily compute gradients in computational graphs (3) Run it all efficiently on GPU (wrap cudnn, cublas, etc) Lecture 9-4
5 Today: CNN Architectures Case Studies - AlexNet VGG GoogLeNet ResNet Also... - NiN (Network in Network) Wide ResNet ResNeXT Stochastic Depth Squeeze-and-Excitation Network - DenseNet FractalNet SqueezeNet NASNet Lecture 9-5
6 Review: LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride 2 i.e. architecture is [CONV-POOL-CONV-POOL-FC-FC] Lecture 9-6
7 Case Study: AlexNet [Krizhevsky et al. 2012] Architecture: CONV1 MAX POOL1 NORM1 CONV2 MAX POOL2 NORM2 CONV3 CONV4 CONV5 Max POOL3 FC6 FC7 FC8 Lecture 9-7
8 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11 filters applied at stride 4 => Q: what is the output volume size? Hint: (227-11)/4+1 = 55 Lecture 9-8
9 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11 filters applied at stride 4 => Output volume [55x55x96] Q: What is the total number of parameters in this layer? Lecture 9-9
10 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11 filters applied at stride 4 => Output volume [55x55x96] Parameters: (11*11*3)*96 = 35K Lecture 9-10
11 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images After CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Q: what is the output volume size? Hint: (55-3)/2+1 = 27 Lecture 9-11
12 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images After CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Output volume: 27x27x96 Q: what is the number of parameters in this layer? Lecture 9-12
13 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images After CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Output volume: 27x27x96 Parameters: 0! Lecture 9-13
14 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images After CONV1: 55x55x96 After POOL1: 27x27x96... Lecture 9-14
15 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet architecture: [227x227x3] INPUT [55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0 [27x27x96] MAX POOL1: 3x3 filters at stride 2 [27x27x96] NORM1: Normalization layer [27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2 [13x13x256] NORM2: Normalization layer [13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 [13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 [13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1 [6x6x256] MAX POOL3: 3x3 filters at stride 2 [4096] FC6: 4096 neurons [4096] FC7: 4096 neurons [1000] FC8: 1000 neurons (class scores) Lecture 9-15
16 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet architecture: [227x227x3] INPUT [55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0 [27x27x96] MAX POOL1: 3x3 filters at stride 2 [27x27x96] NORM1: Normalization layer [27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2 [13x13x256] NORM2: Normalization layer [13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 [13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 [13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1 [6x6x256] MAX POOL3: 3x3 filters at stride 2 [4096] FC6: 4096 neurons [4096] FC7: 4096 neurons [1000] FC8: 1000 neurons (class scores) Details/Retrospectives: - first use of ReLU - used Norm layers (not common anymore) - heavy data augmentation - dropout batch size SGD Momentum Learning rate 1e-2, reduced by 10 manually when val accuracy plateaus - L2 weight decay 5e-4-7 CNN ensemble: 18.2% -> 15.4% Lecture 9-16
17 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet architecture: [227x227x3] INPUT [55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0 [27x27x96] MAX POOL1: 3x3 filters at stride 2 [27x27x96] NORM1: Normalization layer [27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2 [13x13x256] NORM2: Normalization layer [13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 [13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 [13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1 [6x6x256] MAX POOL3: 3x3 filters at stride 2 [4096] FC6: 4096 neurons [4096] FC7: 4096 neurons [1000] FC8: 1000 neurons (class scores) [55x55x48] x 2 Historical note: Trained on GTX 580 GPU with only 3 GB of memory. Network spread across 2 GPUs, half the neurons (feature maps) on each GPU. Lecture 9-17
18 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet architecture: [227x227x3] INPUT [55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0 [27x27x96] MAX POOL1: 3x3 filters at stride 2 [27x27x96] NORM1: Normalization layer [27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2 [13x13x256] NORM2: Normalization layer [13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 [13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 [13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1 [6x6x256] MAX POOL3: 3x3 filters at stride 2 [4096] FC6: 4096 neurons [4096] FC7: 4096 neurons [1000] FC8: 1000 neurons (class scores) CONV1, CONV2, CONV4, CONV5: Connections only with feature maps on same GPU Lecture 9-18
19 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet architecture: [227x227x3] INPUT [55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0 [27x27x96] MAX POOL1: 3x3 filters at stride 2 [27x27x96] NORM1: Normalization layer [27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2 [13x13x256] MAX POOL2: 3x3 filters at stride 2 [13x13x256] NORM2: Normalization layer [13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1 [13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1 [13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1 [6x6x256] MAX POOL3: 3x3 filters at stride 2 [4096] FC6: 4096 neurons [4096] FC7: 4096 neurons [1000] FC8: 1000 neurons (class scores) CONV3, FC6, FC7, FC8: Connections with all feature maps in preceding layer, communication across GPUs Lecture 9-19
20 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners 152 layers 152 layers 152 layers 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-20
21 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners 152 layers 152 layers 152 layers First CNN-based winner 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-21
22 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners ZFNet: Improved hyperparameters over AlexNet 19 layers shallow 8 layers 152 layers 152 layers 152 layers 22 layers 8 layers Lecture 9-22
23 ZFNet [Zeiler and Fergus, 2013] AlexNet but: CONV1: change from (11x11 stride 4) to (7x7 stride 2) CONV3,4,5: instead of 384, 384, 256 filters use 512, 1024, 512 ImageNet top 5 error: 16.4% -> 11.7% Lecture 9-23
24 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners 152 layers 152 layers 152 layers Deeper Networks 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-24
25 Case Study: VGGNet [Simonyan and Zisserman, 2014] Small filters, Deeper networks 8 layers (AlexNet) -> layers (VGG16Net) Only 3x3 CONV stride 1, pad 1 and 2x2 MAX POOL stride % top 5 error in ILSVRC 13 (ZFNet) -> 7.3% top 5 error in ILSVRC 14 AlexNet VGG16 Lecture 9-25 VGG19
26 Case Study: VGGNet [Simonyan and Zisserman, 2014] Q: Why use smaller filters? (3x3 conv) AlexNet VGG16 Lecture 9-26 VGG19
27 Case Study: VGGNet [Simonyan and Zisserman, 2014] Q: Why use smaller filters? (3x3 conv) Stack of three 3x3 conv (stride 1) layers has same effective receptive field as one 7x7 conv layer Q: What is the effective receptive field of three 3x3 conv (stride 1) layers? AlexNet VGG16 Lecture 9-27 VGG19
28 Case Study: VGGNet [Simonyan and Zisserman, 2014] Q: Why use smaller filters? (3x3 conv) Stack of three 3x3 conv (stride 1) layers has same effective receptive field as one 7x7 conv layer [7x7] AlexNet VGG16 Lecture 9-28 VGG19
29 Case Study: VGGNet [Simonyan and Zisserman, 2014] Q: Why use smaller filters? (3x3 conv) Stack of three 3x3 conv (stride 1) layers has same effective receptive field as one 7x7 conv layer But deeper, more non-linearities And fewer parameters: 3 * (32C2) vs. 72C2 for C channels per layer AlexNet VGG16 Lecture 9-29 VGG19
30 (not counting biases) INPUT: [224x224x3] memory: 224*224*3=150K params: 0 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864 POOL2: [112x112x64] memory: 112*112*64=800K params: 0 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456 POOL2: [56x56x128] memory: 56*56*128=400K params: 0 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 POOL2: [28x28x256] memory: 28*28*256=200K params: 0 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 POOL2: [14x14x512] memory: 14*14*512=100K params: 0 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 POOL2: [7x7x512] memory: 7*7*512=25K params: 0 FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448 FC: [1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216 FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000 Lecture 9-30 VGG16
31 (not counting biases) INPUT: [224x224x3] memory: 224*224*3=150K params: 0 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864 POOL2: [112x112x64] memory: 112*112*64=800K params: 0 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456 POOL2: [56x56x128] memory: 56*56*128=400K params: 0 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 POOL2: [28x28x256] memory: 28*28*256=200K params: 0 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 POOL2: [14x14x512] memory: 14*14*512=100K params: 0 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 POOL2: [7x7x512] memory: 7*7*512=25K params: 0 FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448 FC: [1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216 FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000 VGG16 TOTAL memory: 24M * 4 bytes ~= 96MB / image (for a forward pass) TOTAL params: 138M parameters Lecture 9-31
32 (not counting biases) INPUT: [224x224x3] memory: 224*224*3=150K params: 0 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864 POOL2: [112x112x64] memory: 112*112*64=800K params: 0 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456 POOL2: [56x56x128] memory: 56*56*128=400K params: 0 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 POOL2: [28x28x256] memory: 28*28*256=200K params: 0 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 POOL2: [14x14x512] memory: 14*14*512=100K params: 0 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 POOL2: [7x7x512] memory: 7*7*512=25K params: 0 FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448 FC: [1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216 FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000 Note: Most memory is in early CONV Most params are in late FC TOTAL memory: 24M * 4 bytes ~= 96MB / image (only forward! ~*2 for bwd) TOTAL params: 138M parameters Lecture 9-32
33 (not counting biases) INPUT: [224x224x3] memory: 224*224*3=150K params: 0 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*3)*64 = 1,728 CONV3-64: [224x224x64] memory: 224*224*64=3.2M params: (3*3*64)*64 = 36,864 POOL2: [112x112x64] memory: 112*112*64=800K params: 0 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*64)*128 = 73,728 CONV3-128: [112x112x128] memory: 112*112*128=1.6M params: (3*3*128)*128 = 147,456 POOL2: [56x56x128] memory: 56*56*128=400K params: 0 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*128)*256 = 294,912 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 CONV3-256: [56x56x256] memory: 56*56*256=800K params: (3*3*256)*256 = 589,824 POOL2: [28x28x256] memory: 28*28*256=200K params: 0 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*256)*512 = 1,179,648 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 CONV3-512: [28x28x512] memory: 28*28*512=400K params: (3*3*512)*512 = 2,359,296 POOL2: [14x14x512] memory: 14*14*512=100K params: 0 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 CONV3-512: [14x14x512] memory: 14*14*512=100K params: (3*3*512)*512 = 2,359,296 POOL2: [7x7x512] memory: 7*7*512=25K params: 0 FC: [1x1x4096] memory: 4096 params: 7*7*512*4096 = 102,760,448 FC: [1x1x4096] memory: 4096 params: 4096*4096 = 16,777,216 FC: [1x1x1000] memory: 1000 params: 4096*1000 = 4,096,000 TOTAL memory: 24M * 4 bytes ~= 96MB / image (only forward! ~*2 for bwd) TOTAL params: 138M parameters Lecture 9-33 VGG16 Common names
34 Case Study: VGGNet [Simonyan and Zisserman, 2014] Details: - ILSVRC 14 2nd in classification, 1st in localization - Similar training procedure as Krizhevsky No Local Response Normalisation (LRN) - Use VGG16 or VGG19 (VGG19 only slightly better, more memory) - Use ensembles for best results - FC7 features generalize well to other tasks AlexNet VGG16 Lecture 9-34 VGG19
35 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners 152 layers 152 layers 152 layers Deeper Networks 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-35
36 Case Study: GoogLeNet [Szegedy et al., 2014] Deeper networks, with computational efficiency - 22 layers Efficient Inception module No FC layers Only 5 million parameters! 12x less than AlexNet ILSVRC 14 classification winner (6.7% top 5 error) Inception module Lecture 9-36
37 Case Study: GoogLeNet [Szegedy et al., 2014] Inception module : design a good local network topology (network within a network) and then stack these modules on top of each other Inception module Lecture 9-37
38 Case Study: GoogLeNet [Szegedy et al., 2014] Apply parallel filter operations on the input from previous layer: - Multiple receptive field sizes for convolution (1x1, 3x3, 5x5) - Pooling operation (3x3) Naive Inception module Concatenate all filter outputs together depth-wise Lecture 9-38
39 Case Study: GoogLeNet [Szegedy et al., 2014] Apply parallel filter operations on the input from previous layer: - Multiple receptive field sizes for convolution (1x1, 3x3, 5x5) - Pooling operation (3x3) Naive Inception module Concatenate all filter outputs together depth-wise Q: What is the problem with this? [Hint: Computational complexity] Lecture 9-39
40 Case Study: GoogLeNet [Szegedy et al., 2014] Q: What is the problem with this? [Hint: Computational complexity] Example: Module input: 28x28x256 Naive Inception module Lecture 9-40
41 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q: What is the problem with this? [Hint: Computational complexity] Q1: What is the output size of the 1x1 conv, with 128 filters? Module input: 28x28x256 Naive Inception module Lecture 9-41
42 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q: What is the problem with this? [Hint: Computational complexity] Q1: What is the output size of the 1x1 conv, with 128 filters? 28x28x128 Module input: 28x28x256 Naive Inception module Lecture 9-42
43 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q: What is the problem with this? [Hint: Computational complexity] Q2: What are the output sizes of all different filter operations? 28x28x128 Module input: 28x28x256 Naive Inception module Lecture 9-43
44 Case Study: GoogLeNet [Szegedy et al., 2014] Example: 28x28x128 Q: What is the problem with this? [Hint: Computational complexity] Q2: What are the output sizes of all different filter operations? 28x28x192 28x28x96 28x28x256 Module input: 28x28x256 Naive Inception module Lecture 9-44
45 Case Study: GoogLeNet [Szegedy et al., 2014] Example: 28x28x128 Q: What is the problem with this? [Hint: Computational complexity] Q3:What is output size after filter concatenation? 28x28x192 28x28x96 28x28x256 Module input: 28x28x256 Naive Inception module Lecture 9-45
46 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q: What is the problem with this? [Hint: Computational complexity] Q3:What is output size after filter concatenation? 28x28x( ) = 28x28x672 28x28x128 28x28x192 28x28x96 28x28x256 Module input: 28x28x256 Naive Inception module Lecture 9-46
47 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q3:What is output size after filter concatenation? 28x28x( ) = 28x28x672 28x28x128 Q: What is the problem with this? [Hint: Computational complexity] 28x28x192 28x28x96 28x28x256 Conv Ops: [1x1 conv, 128] 28x28x128x1x1x256 [3x3 conv, 192] 28x28x192x3x3x256 [5x5 conv, 96] 28x28x96x5x5x256 Total: 854M ops Module input: 28x28x256 Naive Inception module Lecture 9-47
48 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q3:What is output size after filter concatenation? 28x28x( ) = 28x28x672 28x28x128 Q: What is the problem with this? [Hint: Computational complexity] 28x28x192 28x28x96 28x28x256 Conv Ops: [1x1 conv, 128] 28x28x128x1x1x256 [3x3 conv, 192] 28x28x192x3x3x256 [5x5 conv, 96] 28x28x96x5x5x256 Total: 854M ops Very expensive compute Module input: 28x28x256 Naive Inception module Pooling layer also preserves feature depth, which means total depth after concatenation can only grow at every layer! Lecture 9-48
49 Case Study: GoogLeNet [Szegedy et al., 2014] Example: Q3:What is output size after filter concatenation? 28x28x( ) = 529k 28x28x128 Q: What is the problem with this? [Hint: Computational complexity] 28x28x192 28x28x96 28x28x256 Solution: bottleneck layers that use 1x1 convolutions to reduce feature depth Module input: 28x28x256 Naive Inception module Lecture 9-49
50 Reminder: 1x1 convolutions x1 CONV with 32 filters 56 (each filter has size 1x1x64, and performs a 64-dimensional dot product) Lecture 9-50
51 Reminder: 1x1 convolutions 56 1x1 CONV with 32 filters 56 preserves spatial dimensions, reduces depth! Projects depth to lower dimension (combination of feature maps) Lecture 9-51
52 Case Study: GoogLeNet [Szegedy et al., 2014] Naive Inception module Inception module with dimension reduction Lecture 9-52
53 Case Study: GoogLeNet [Szegedy et al., 2014] 1x1 conv bottleneck layers Naive Inception module Inception module with dimension reduction Lecture 9-53
54 Case Study: GoogLeNet [Szegedy et al., 2014] Using same parallel layers as naive example, and adding 1x1 conv, 64 filter bottlenecks: 28x28x480 28x28x128 28x28x192 28x28x64 28x28x96 28x28x64 28x28x64 28x28x256 Module input: 28x28x256 Conv Ops: [1x1 conv, 64] 28x28x64x1x1x256 [1x1 conv, 64] 28x28x64x1x1x256 [1x1 conv, 128] 28x28x128x1x1x256 [3x3 conv, 192] 28x28x192x3x3x64 [5x5 conv, 96] 28x28x96x5x5x64 [1x1 conv, 64] 28x28x64x1x1x256 Total: 358M ops Compared to 854M ops for naive version Bottleneck can also reduce depth after pooling layer Inception module with dimension reduction Lecture 9-54
55 Case Study: GoogLeNet [Szegedy et al., 2014] Stack Inception modules with dimension reduction on top of each other Inception module Lecture 9-55
56 Case Study: GoogLeNet [Szegedy et al., 2014] Full GoogLeNet architecture Stem Network: Conv-Pool2x Conv-Pool Lecture 9-56
57 Case Study: GoogLeNet [Szegedy et al., 2014] Full GoogLeNet architecture Stacked Inception Modules Lecture 9-57
58 Case Study: GoogLeNet [Szegedy et al., 2014] Full GoogLeNet architecture Classifier output Lecture 9-58
59 Case Study: GoogLeNet [Szegedy et al., 2014] Full GoogLeNet architecture Classifier output (removed expensive FC layers!) Lecture 9-59
60 Case Study: GoogLeNet [Szegedy et al., 2014] Full GoogLeNet architecture Auxiliary classification outputs to inject additional gradient at lower layers (AvgPool-1x1Conv-FC-FC-Softmax) Lecture 9-60
61 Case Study: GoogLeNet [Szegedy et al., 2014] Full GoogLeNet architecture 22 total layers with weights (parallel layers count as 1 layer => 2 layers per Inception module. Don t count auxiliary output layers) Lecture 9-61
62 Case Study: GoogLeNet [Szegedy et al., 2014] Deeper networks, with computational efficiency - 22 layers Efficient Inception module No FC layers 12x less params than AlexNet ILSVRC 14 classification winner (6.7% top 5 error) Inception module Lecture 9-62
63 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners Revolution of Depth 152 layers 152 layers 152 layers 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-63
64 Case Study: ResNet [He et al., 2015] relu Very deep networks using residual connections F(x) + x layer model for ImageNet ILSVRC 15 classification winner (3.57% top 5 error) Swept all classification and detection competitions in ILSVRC 15 and COCO 15! F(x) relu X identity X Residual block Lecture 9-64
65 Case Study: ResNet [He et al., 2015] What happens when we continue stacking deeper layers on a plain convolutional neural network? Lecture 9-65
66 Case Study: ResNet [He et al., 2015] What happens when we continue stacking deeper layers on a plain convolutional neural network? Q: What s strange about these training and test curves? [Hint: look at the order of the curves] Lecture 9-66
67 Case Study: ResNet [He et al., 2015] What happens when we continue stacking deeper layers on a plain convolutional neural network? 56-layer model performs worse on both training and test error -> The deeper model performs worse, but it s not caused by overfitting! Lecture 9-67
68 Case Study: ResNet [He et al., 2015] Hypothesis: the problem is an optimization problem, deeper models are harder to optimize Lecture 9-68
69 Case Study: ResNet [He et al., 2015] Hypothesis: the problem is an optimization problem, deeper models are harder to optimize The deeper model should be able to perform at least as well as the shallower model. A solution by construction is copying the learned layers from the shallower model and setting additional layers to identity mapping. Lecture 9-69
70 Case Study: ResNet [He et al., 2015] Solution: Use network layers to fit a residual mapping instead of directly trying to fit a desired underlying mapping relu H(x) relu F(x) + x F(x) X Plain layers relu X identity X Residual block Lecture 9-70
71 Case Study: ResNet [He et al., 2015] Solution: Use network layers to fit a residual mapping instead of directly trying to fit a desired underlying mapping H(x) relu H(x) = F(x) + x relu F(x) + x F(x) X Plain layers relu X identity Use layers to fit residual F(x) = H(x) - x instead of H(x) directly X Residual block Lecture 9-71
72 Case Study: ResNet [He et al., 2015] Full ResNet architecture: - Stack residual blocks - Every residual block has two 3x3 conv layers relu F(x) + x... F(x) relu X identity X Residual block Lecture 9-72
73 Case Study: ResNet [He et al., 2015] Full ResNet architecture: - Stack residual blocks - Every residual block has two 3x3 conv layers - Periodically, double # of filters and downsample spatially using stride 2 (/2 in each dimension) relu F(x) + x... F(x) relu X identity 3x3 conv, 128 filters, /2 spatially with stride 2 3x3 conv, 64 filters X Residual block Lecture 9-73
74 Case Study: ResNet [He et al., 2015] Full ResNet architecture: - Stack residual blocks - Every residual block has two 3x3 conv layers - Periodically, double # of filters and downsample spatially using stride 2 (/2 in each dimension) - Additional conv layer at the beginning relu F(x) + x... F(x) relu X identity X Residual block Beginning conv layer Lecture 9-74
75 Case Study: ResNet No FC layers besides FC 1000 to output classes [He et al., 2015] Full ResNet architecture: - Stack residual blocks - Every residual block has two 3x3 conv layers - Periodically, double # of filters and downsample spatially using stride 2 (/2 in each dimension) - Additional conv layer at the beginning - No FC layers at the end (only FC 1000 to output classes) relu F(x) + x... F(x) relu Global average pooling layer after last conv layer X identity X Residual block Lecture 9-75
76 Case Study: ResNet [He et al., 2015] Total depths of 34, 50, 101, or 152 layers for ImageNet... Lecture 9-76
77 Case Study: ResNet [He et al., 2015] For deeper networks (ResNet-50+), use bottleneck layer to improve efficiency (similar to GoogLeNet) Lecture 9-77
78 Case Study: ResNet [He et al., 2015] For deeper networks (ResNet-50+), use bottleneck layer to improve efficiency (similar to GoogLeNet) 1x1 conv, 256 filters projects back to 256 feature maps (28x28x256) 3x3 conv operates over only 64 feature maps 1x1 conv, 64 filters to project to 28x28x64 Lecture 9-78
79 Case Study: ResNet [He et al., 2015] Training ResNet in practice: - Batch Normalization after every CONV layer Xavier 2/ initialization from He et al. SGD + Momentum (0.9) Learning rate: 0.1, divided by 10 when validation error plateaus Mini-batch size 256 Weight decay of 1e-5 No dropout used Lecture 9-79
80 Case Study: ResNet [He et al., 2015] Experimental Results - Able to train very deep networks without degrading (152 layers on ImageNet, 1202 on Cifar) - Deeper networks now achieve lowing training error as expected - Swept 1st place in all ILSVRC and COCO 2015 competitions Lecture 9-80
81 Case Study: ResNet [He et al., 2015] Experimental Results - Able to train very deep networks without degrading (152 layers on ImageNet, 1202 on Cifar) - Deeper networks now achieve lowing training error as expected - Swept 1st place in all ILSVRC and COCO 2015 competitions ILSVRC 2015 classification winner (3.6% top 5 error) -- better than human performance! (Russakovsky 2014) Lecture 9-81
82 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners 152 layers 152 layers 152 layers 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-82
83 Comparing complexity... An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-83
84 Comparing complexity... Inception-v4: Resnet + Inception! An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-84
85 VGG: Highest memory, most operations Comparing complexity... An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-85
86 Comparing complexity... GoogLeNet: most efficient An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-86
87 Comparing complexity... AlexNet: Smaller compute, still memory heavy, lower accuracy An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-87
88 Comparing complexity... ResNet: Moderate efficiency depending on model, highest accuracy An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-88
89 Forward pass time and power consumption An Analysis of Deep Neural Network Models for Practical Applications, Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, Reproduced with permission. Lecture 9-89
90 Other architectures to know... Lecture 9-90
91 Network in Network (NiN) [Lin et al. 2014] Mlpconv layer with micronetwork within each conv layer to compute more abstract features for local patches Micronetwork uses multilayer perceptron (FC, i.e. 1x1 conv layers) Precursor to GoogLeNet and ResNet bottleneck layers Philosophical inspiration for GoogLeNet Figures copyright Lin et al., Reproduced with permission. Lecture 9-91
92 Improving ResNets... Identity Mappings in Deep Residual Networks [He et al. 2016] - - Improved ResNet block design from creators of ResNet Creates a more direct path for propagating information throughout network (moves activation to residual mapping pathway) Gives better performance Lecture 9-92
93 Improving ResNets... Wide Residual Networks [Zagoruyko et al. 2016] - Argues that residuals are the important factor, not depth User wider residual blocks (F x k filters instead of F filters in each layer) 50-layer wide ResNet outperforms 152-layer original ResNet Increasing width instead of depth more computationally efficient (parallelizable) Basic residual block Wide residual block Lecture 9-93
94 Improving ResNets... Aggregated Residual Transformations for Deep Neural Networks (ResNeXt) [Xie et al. 2016] - - Also from creators of ResNet Increases width of residual block through multiple parallel pathways ( cardinality ) Parallel pathways similar in spirit to Inception module Lecture 9-94
95 Improving ResNets... Deep Networks with Stochastic Depth [Huang et al. 2016] - - Motivation: reduce vanishing gradients and training time through short networks during training Randomly drop a subset of layers during each training pass Bypass with identity function Use full deep network at test time Lecture 9-95
96 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners Network ensembling 152 layers 152 layers 152 layers 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-96
97 Improving ResNets... Good Practices for Deep Feature Fusion [Shao et al. 2016] - Multi-scale ensembling of Inception, Inception-Resnet, Resnet, Wide Resnet models ILSVRC 16 classification winner Lecture 9-97
98 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners Adaptive feature map reweighting 152 layers 152 layers 152 layers 19 layers shallow 8 layers 22 layers 8 layers Lecture 9-98
99 Improving ResNets... Squeeze-and-Excitation Networks (SENet) [Hu et al. 2017] - - Add a feature recalibration module that learns to adaptively reweight feature maps Global information (global avg. pooling layer) + 2 FC layers used to determine feature map weights ILSVRC 17 classification winner (using ResNeXt-152 as a base architecture) Lecture 9-99
100 Beyond ResNets... FractalNet: Ultra-Deep Neural Networks without Residuals [Larsson et al. 2017] - - Argues that key is transitioning effectively from shallow to deep and residual representations are not necessary Fractal architecture with both shallow and deep paths to output Trained with dropping out sub-paths Full network at test time Figures copyright Larsson et al., Reproduced with permission. Lecture
101 Beyond ResNets... Densely Connected Convolutional Networks [Huang et al. 2017] - - Dense blocks where each layer is connected to every other layer in feedforward fashion Alleviates vanishing gradient, strengthens feature propagation, encourages feature reuse Lecture 9 -
102 Efficient networks... SqueezeNet: AlexNet-level Accuracy With 50x Fewer Parameters and <0.5Mb Model Size [Iandola et al. 2017] Fire modules consisting of a squeeze layer with 1x1 filters feeding an expand layer with 1x1 and 3x3 filters AlexNet level accuracy on ImageNet with 50x fewer parameters Can compress to 510x smaller than AlexNet (0.5Mb) Figure copyright Iandola, Han, Moskewicz, Ashraf, Dally, Keutzer, Reproduced with permission. Lecture 9 -
103 Meta-learning: Learning to learn network architectures... Neural Architecture Search with Reinforcement Learning (NAS) [Zoph et al. 2016] - - Controller network that learns to design a good network architecture (output a string corresponding to network design) Iterate: 1) Sample an architecture from search space 2) Train the architecture to get a reward R corresponding to accuracy 3) Compute gradient of sample probability, and scale by R to perform controller parameter update (i.e. increase likelihood of good architecture being sampled, decrease likelihood of bad architecture) Lecture
104 Meta-learning: Learning to learn network architectures... Learning Transferable Architectures for Scalable Image Recognition [Zoph et al. 2017] - Applying neural architecture search (NAS) to a large dataset like ImageNet is expensive Design a search space of building blocks ( cells ) that can be flexibly stacked NASNet: Use NAS to find best cell structure on smaller CIFAR-10 dataset, then transfer architecture to ImageNet Lecture
105 Summary: CNN Architectures Case Studies - AlexNet VGG GoogLeNet ResNet Also... - NiN (Network in Network) Wide ResNet ResNeXT Stochastic Depth Squeeze-and-Excitation Network - DenseNet FractalNet SqueezeNet NASNet Lecture
106 Summary: CNN Architectures - VGG, GoogLeNet, ResNet all in wide use, available in model zoos ResNet current best default, also consider SENet when available Trend towards extremely deep networks Significant research centers around design of layer / skip connections and improving gradient flow - Efforts to investigate necessity of depth vs. width and residual connections - Even more recent trend towards meta-learning - Next time: Recurrent neural networks Lecture 9-106
Intro to Deep Learning. Slides Credit: Andrej Karapathy, Derek Hoiem, Marc Aurelio, Yann LeCunn
Intro to Deep Learning Slides Credit: Andrej Karapathy, Derek Hoiem, Marc Aurelio, Yann LeCunn Why this class? Deep Features Have been able to harness the big data in the most efficient and effective
CENG 783. Special topics in. Deep Learning. AlchemyAPI. Week 11. Sinan Kalkan
CENG 783 Special topics in Deep Learning AlchemyAPI Week 11 Sinan Kalkan TRAINING A CNN Fig: http://www.robots.ox.ac.uk/~vgg/practicals/cnn/ Feed-forward pass Note that this is written in terms of the
Inception and Residual Networks. Hantao Zhang. Deep Learning with Python.
Inception and Residual Networks Hantao Zhang Deep Learning with Python https://en.wikipedia.org/wiki/residual_neural_network Deep Neural Network Progress from Large Scale Visual Recognition Challenge (ILSVRC)
Know your data - many types of networks
Architectures Know your data - many types of networks Fixed length representation Variable length representation Online video sequences, or samples of different sizes Images Specific architectures for
Inception Network Overview. David White CS793
Inception Network Overview David White CS793 So, Leonardo DiCaprio dreams about dreaming... https://m.media-amazon.com/images/m/mv5bmjaxmzy3njcxnf5bml5banbnxkftztcwnti5otm0mw@@._v1_sy1000_cr0,0,675,1 000_AL_.jpg
Convolu'onal Neural Networks
Convolu'onal Neural Networks Dr. Kira Radinsky CTO SalesPredict Visi8ng Professor/Scien8st Technion Slides were adapted from Fei-Fei Li & Andrej Karpathy & Jus8n Johnson A bit of history: Hubel & Wiesel,
Convolutional Neural Networks
NPFL114, Lecture 4 Convolutional Neural Networks Milan Straka March 25, 2019 Charles University in Prague Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise
CNNS FROM THE BASICS TO RECENT ADVANCES. Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague
CNNS FROM THE BASICS TO RECENT ADVANCES Dmytro Mishkin Center for Machine Perception Czech Technical University in Prague ducha.aiki@gmail.com OUTLINE Short review of the CNN design Architecture progress
Deep Learning with Tensorflow AlexNet
Machine Learning and Computer Vision Group Deep Learning with Tensorflow http://cvml.ist.ac.at/courses/dlwt_w17/ AlexNet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "Imagenet classification
COMP9444 Neural Networks and Deep Learning 7. Image Processing. COMP9444 c Alan Blair, 2017
COMP9444 Neural Networks and Deep Learning 7. Image Processing COMP9444 17s2 Image Processing 1 Outline Image Datasets and Tasks Convolution in Detail AlexNet Weight Initialization Batch Normalization
HENet: A Highly Efficient Convolutional Neural. Networks Optimized for Accuracy, Speed and Storage
HENet: A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage Qiuyu Zhu Shanghai University zhuqiuyu@staff.shu.edu.cn Ruixin Zhang Shanghai University chriszhang96@shu.edu.cn
Fuzzy Set Theory in Computer Vision: Example 3
Fuzzy Set Theory in Computer Vision: Example 3 Derek T. Anderson and James M. Keller FUZZ-IEEE, July 2017 Overview Purpose of these slides are to make you aware of a few of the different CNN architectures
Convolutional Neural Networks. Computer Vision Jia-Bin Huang, Virginia Tech
Convolutional Neural Networks Computer Vision Jia-Bin Huang, Virginia Tech Today s class Overview Convolutional Neural Network (CNN) Training CNN Understanding and Visualizing CNN Image Categorization:
Study of Residual Networks for Image Recognition
Study of Residual Networks for Image Recognition Mohammad Sadegh Ebrahimi Stanford University sadegh@stanford.edu Hossein Karkeh Abadi Stanford University hosseink@stanford.edu Abstract Deep neural networks
INTRODUCTION TO DEEP LEARNING
INTRODUCTION TO DEEP LEARNING CONTENTS Introduction to deep learning Contents 1. Examples 2. Machine learning 3. Neural networks 4. Deep learning 5. Convolutional neural networks 6. Conclusion 7. Additional
Computer Vision Lecture 16
Computer Vision Lecture 16 Deep Learning for Object Categorization 14.01.2016 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar registration period
Deep Learning for Computer Vision II
IIIT Hyderabad Deep Learning for Computer Vision II C. V. Jawahar Paradigm Shift Feature Extraction (SIFT, HoG, ) Part Models / Encoding Classifier Sparrow Feature Learning Classifier Sparrow L 1 L 2 L
Machine Learning. Deep Learning. Eric Xing (and Pengtao Xie) , Fall Lecture 8, October 6, Eric CMU,
Machine Learning 10-701, Fall 2015 Deep Learning Eric Xing (and Pengtao Xie) Lecture 8, October 6, 2015 Eric Xing @ CMU, 2015 1 A perennial challenge in computer vision: feature engineering SIFT Spin image
Advanced Video Analysis & Imaging
Advanced Video Analysis & Imaging (5LSH0), Module 09B Machine Learning with Convolutional Neural Networks (CNNs) - Workout Farhad G. Zanjani, Clint Sebastian, Egor Bondarev, Peter H.N. de With ( p.h.n.de.with@tue.nl
Residual Networks And Attention Models. cs273b Recitation 11/11/2016. Anna Shcherbina
Residual Networks And Attention Models cs273b Recitation 11/11/2016 Anna Shcherbina Introduction to ResNets Introduced in 2015 by Microsoft Research Deep Residual Learning for Image Recognition (He, Zhang,
Spatial Localization and Detection. Lecture 8-1
Lecture 8: Spatial Localization and Detection Lecture 8-1 Administrative - Project Proposals were due on Saturday Homework 2 due Friday 2/5 Homework 1 grades out this week Midterm will be in-class on Wednesday
Channel Locality Block: A Variant of Squeeze-and-Excitation
Channel Locality Block: A Variant of Squeeze-and-Excitation 1 st Huayu Li Northern Arizona University Flagstaff, United State Northern Arizona University hl459@nau.edu arxiv:1901.01493v1 [cs.lg] 6 Jan
Lecture 37: ConvNets (Cont d) and Training
Lecture 37: ConvNets (Cont d) and Training CS 4670/5670 Sean Bell [http://bbabenko.tumblr.com/post/83319141207/convolutional-learnings-things-i-learned-by] (Unrelated) Dog vs Food [Karen Zack, @teenybiscuit]
Elastic Neural Networks for Classification
Elastic Neural Networks for Classification Yi Zhou 1, Yue Bai 1, Shuvra S. Bhattacharyya 1, 2 and Heikki Huttunen 1 1 Tampere University of Technology, Finland, 2 University of Maryland, USA arxiv:1810.00589v3
Real-time convolutional networks for sonar image classification in low-power embedded systems
Real-time convolutional networks for sonar image classification in low-power embedded systems Matias Valdenegro-Toro Ocean Systems Laboratory - School of Engineering & Physical Sciences Heriot-Watt University,
Learning Deep Representations for Visual Recognition
Learning Deep Representations for Visual Recognition CVPR 2018 Tutorial Kaiming He Facebook AI Research (FAIR) Deep Learning is Representation Learning Representation Learning: worth a conference name
Convolutional Neural Networks
Lecturer: Barnabas Poczos Introduction to Machine Learning (Lecture Notes) Convolutional Neural Networks Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
Deep Learning Explained Module 4: Convolution Neural Networks (CNN or Conv Nets)
Deep Learning Explained Module 4: Convolution Neural Networks (CNN or Conv Nets) Sayan D. Pathak, Ph.D., Principal ML Scientist, Microsoft Roland Fernandez, Senior Researcher, Microsoft Module Outline
Deep Learning in Visual Recognition. Thanks Da Zhang for the slides
Deep Learning in Visual Recognition Thanks Da Zhang for the slides Deep Learning is Everywhere 2 Roadmap Introduction Convolutional Neural Network Application Image Classification Object Detection Object
Learning Deep Features for Visual Recognition
7x7 conv, 64, /2, pool/2 1x1 conv, 64 3x3 conv, 64 1x1 conv, 64 3x3 conv, 64 1x1 conv, 64 3x3 conv, 64 1x1 conv, 128, /2 3x3 conv, 128 1x1 conv, 512 1x1 conv, 128 3x3 conv, 128 1x1 conv, 512 1x1 conv,
Como funciona o Deep Learning
Como funciona o Deep Learning Moacir Ponti (com ajuda de Gabriel Paranhos da Costa) ICMC, Universidade de São Paulo Contact: www.icmc.usp.br/~moacir moacir@icmc.usp.br Uberlandia-MG/Brazil October, 2017
Supplementary material for Analyzing Filters Toward Efficient ConvNet
Supplementary material for Analyzing Filters Toward Efficient Net Takumi Kobayashi National Institute of Advanced Industrial Science and Technology, Japan takumi.kobayashi@aist.go.jp A. Orthonormal Steerable
Deep Learning and Its Applications
Convolutional Neural Network and Its Application in Image Recognition Oct 28, 2016 Outline 1 A Motivating Example 2 The Convolutional Neural Network (CNN) Model 3 Training the CNN Model 4 Issues and Recent
All You Want To Know About CNNs. Yukun Zhu
All You Want To Know About CNNs Yukun Zhu Deep Learning Deep Learning Image from http://imgur.com/ Deep Learning Image from http://imgur.com/ Deep Learning Image from http://imgur.com/ Deep Learning Image
Convolutional Neural Networks and Supervised Learning
Convolutional Neural Networks and Supervised Learning Eilif Solberg August 30, 2018 Outline Convolutional Architectures Convolutional neural networks Training Loss Optimization Regularization Hyperparameter
Deep Residual Learning
Deep Residual Learning MSRA @ ILSVRC & COCO 2015 competitions Kaiming He with Xiangyu Zhang, Shaoqing Ren, Jifeng Dai, & Jian Sun Microsoft Research Asia (MSRA) MSRA @ ILSVRC & COCO 2015 Competitions 1st
Introduction to Neural Networks
Introduction to Neural Networks Jakob Verbeek 2017-2018 Biological motivation Neuron is basic computational unit of the brain about 10^11 neurons in human brain Simplified neuron model as linear threshold
Convolutional Neural Networks: Applications and a short timeline. 7th Deep Learning Meetup Kornel Kis Vienna,
Convolutional Neural Networks: Applications and a short timeline 7th Deep Learning Meetup Kornel Kis Vienna, 1.12.2016. Introduction Currently a master student Master thesis at BME SmartLab Started deep
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius
MIXED PRECISION TRAINING: THEORY AND PRACTICE Paulius Micikevicius What is Mixed Precision Training? Reduced precision tensor math with FP32 accumulation, FP16 storage Successfully used to train a variety
Tutorial on Keras CAP ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY
Tutorial on Keras CAP 6412 - ADVANCED COMPUTER VISION SPRING 2018 KISHAN S ATHREY Deep learning packages TensorFlow Google PyTorch Facebook AI research Keras Francois Chollet (now at Google) Chainer Company
Deconvolutions in Convolutional Neural Networks
Overview Deconvolutions in Convolutional Neural Networks Bohyung Han bhhan@postech.ac.kr Computer Vision Lab. Convolutional Neural Networks (CNNs) Deconvolutions in CNNs Applications Network visualization
CS 2750: Machine Learning. Neural Networks. Prof. Adriana Kovashka University of Pittsburgh April 13, 2016
CS 2750: Machine Learning Neural Networks Prof. Adriana Kovashka University of Pittsburgh April 13, 2016 Plan for today Neural network definition and examples Training neural networks (backprop) Convolutional
Convolutional Neural Networks + Neural Style Transfer. Justin Johnson 2/1/2017
Convolutional Neural Networks + Neural Style Transfer Justin Johnson 2/1/2017 Outline Convolutional Neural Networks Convolution Pooling Feature Visualization Neural Style Transfer Feature Inversion Texture
Application of Convolutional Neural Network for Image Classification on Pascal VOC Challenge 2012 dataset
Application of Convolutional Neural Network for Image Classification on Pascal VOC Challenge 2012 dataset Suyash Shetty Manipal Institute of Technology suyash.shashikant@learner.manipal.edu Abstract In
CSE 559A: Computer Vision
CSE 559A: Computer Vision Fall 2018: T-R: 11:30-1pm @ Lopata 101 Instructor: Ayan Chakrabarti (ayan@wustl.edu). Course Staff: Zhihao Xia, Charlie Wu, Han Liu http://www.cse.wustl.edu/~ayan/courses/cse559a/
Structured Prediction using Convolutional Neural Networks
Overview Structured Prediction using Convolutional Neural Networks Bohyung Han bhhan@postech.ac.kr Computer Vision Lab. Convolutional Neural Networks (CNNs) Structured predictions for low level computer
Computer Vision Lecture 16
Announcements Computer Vision Lecture 16 Deep Learning Applications 11.01.2017 Seminar registration period starts on Friday We will offer a lab course in the summer semester Deep Robot Learning Topic:
Computer Vision Lecture 16
Computer Vision Lecture 16 Deep Learning Applications 11.01.2017 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Announcements Seminar registration period starts
CMU Lecture 18: Deep learning and Vision: Convolutional neural networks. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 18: Deep learning and Vision: Convolutional neural networks Teacher: Gianni A. Di Caro DEEP, SHALLOW, CONNECTED, SPARSE? Fully connected multi-layer feed-forward perceptrons: More powerful
Perceptron: This is convolution!
Perceptron: This is convolution! v v v Shared weights v Filter = local perceptron. Also called kernel. By pooling responses at different locations, we gain robustness to the exact spatial location of image
arxiv: v1 [cs.cv] 18 Jan 2018
![arxiv: v1 [cs.cv] 18 Jan 2018](/thumbs/74/70143455.jpg "arxiv: v1 [cs.cv] 18 Jan 2018")
Sparsely Connected Convolutional Networks arxiv:1.595v1 [cs.cv] 1 Jan 1 Ligeng Zhu* lykenz@sfu.ca Greg Mori Abstract mori@sfu.ca Residual learning [] with skip connections permits training ultra-deep neural
CrescendoNet: A New Deep Convolutional Neural Network with Ensemble Behavior
CrescendoNet: A New Deep Convolutional Neural Network with Ensemble Behavior Xiang Zhang, Nishant Vishwamitra, Hongxin Hu, Feng Luo School of Computing Clemson University xzhang7@clemson.edu Abstract We
Exploration of the Effect of Residual Connection on top of SqueezeNet A Combination study of Inception Model and Bypass Layers
Exploration of the Effect of Residual Connection on top of SqueezeNet A Combination study of Inception Model and Layers Abstract Two of the most popular model as of now is the Inception module of GoogLeNet
CNN optimization. Rassadin A
CNN optimization Rassadin A. 01.2017-02.2017 What to optimize? Training stage time consumption (CPU / GPU) Inference stage time consumption (CPU / GPU) Training stage memory consumption Inference stage
In-Place Activated BatchNorm for Memory- Optimized Training of DNNs
In-Place Activated BatchNorm for Memory- Optimized Training of DNNs Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder Mapillary Research Paper: https://arxiv.org/abs/1712.02616 Code: https://github.com/mapillary/inplace_abn
Throughput-Optimized OpenCL-based FPGA Accelerator for Large-Scale Convolutional Neural Networks
Throughput-Optimized OpenCL-based FPGA Accelerator for Large-Scale Convolutional Neural Networks Naveen Suda, Vikas Chandra *, Ganesh Dasika *, Abinash Mohanty, Yufei Ma, Sarma Vrudhula, Jae-sun Seo, Yu
Progressive Neural Architecture Search
Progressive Neural Architecture Search Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy 09/10/2018 @ECCV 1 Outline Introduction
Deep learning for dense per-pixel prediction. Chunhua Shen The University of Adelaide, Australia
Deep learning for dense per-pixel prediction Chunhua Shen The University of Adelaide, Australia Image understanding Classification error Convolution Neural Networks 0.3 0.2 0.1 Image Classification [Krizhevsky
Deep Convolutional Neural Networks. Nov. 20th, 2015 Bruce Draper
Deep Convolutional Neural Networks Nov. 20th, 2015 Bruce Draper Background: Fully-connected single layer neural networks Feed-forward classification Trained through back-propagation Example Computer Vision
Deep Learning Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD.
Deep Learning 861.061 Basic Lecture - Complex Systems & Artificial Intelligence 2017/18 (VO) Asan Agibetov, PhD asan.agibetov@meduniwien.ac.at Medical University of Vienna Center for Medical Statistics,
arxiv: v1 [cs.cv] 4 Dec 2014
![arxiv: v1 [cs.cv] 4 Dec 2014](/thumbs/83/87999872.jpg "arxiv: v1 [cs.cv] 4 Dec 2014")
Convolutional Neural Networks at Constrained Time Cost Kaiming He Jian Sun Microsoft Research {kahe,jiansun}@microsoft.com arxiv:1412.1710v1 [cs.cv] 4 Dec 2014 Abstract Though recent advanced convolutional
Index. Springer Nature Switzerland AG 2019 B. Moons et al., Embedded Deep Learning,
Index A Algorithmic noise tolerance (ANT), 93 94 Application specific instruction set processors (ASIPs), 115 116 Approximate computing application level, 95 circuits-levels, 93 94 DAS and DVAS, 107 110
Deep Neural Networks:
Deep Neural Networks: Part II Convolutional Neural Network (CNN) Yuan-Kai Wang, 2016 Web site of this course: http://pattern-recognition.weebly.com source: CNN for ImageClassification, by S. Lazebnik,
A Novel Weight-Shared Multi-Stage Network Architecture of CNNs for Scale Invariance
JOURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 A Novel Weight-Shared Multi-Stage Network Architecture of CNNs for Scale Invariance Ryo Takahashi, Takashi Matsubara, Member, IEEE, and Kuniaki
arxiv: v2 [cs.cv] 26 Jan 2018
![arxiv: v2 [cs.cv] 26 Jan 2018](/thumbs/76/73206564.jpg "arxiv: v2 [cs.cv] 26 Jan 2018")
DIRACNETS: TRAINING VERY DEEP NEURAL NET- WORKS WITHOUT SKIP-CONNECTIONS Sergey Zagoruyko, Nikos Komodakis Université Paris-Est, École des Ponts ParisTech Paris, France {sergey.zagoruyko,nikos.komodakis}@enpc.fr
A performance comparison of Deep Learning frameworks on KNL
A performance comparison of Deep Learning frameworks on KNL R. Zanella, G. Fiameni, M. Rorro Middleware, Data Management - SCAI - CINECA IXPUG Bologna, March 5, 2018 Table of Contents 1. Problem description
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset By Joa õ Carreira and Andrew Zisserman Presenter: Zhisheng Huang 03/02/2018 Outline: Introduction Action classification architectures
Deep Learning. Visualizing and Understanding Convolutional Networks. Christopher Funk. Pennsylvania State University.
Visualizing and Understanding Convolutional Networks Christopher Pennsylvania State University February 23, 2015 Some Slide Information taken from Pierre Sermanet (Google) presentation on and Computer
Two-Stream Convolutional Networks for Action Recognition in Videos
Two-Stream Convolutional Networks for Action Recognition in Videos Karen Simonyan Andrew Zisserman Cemil Zalluhoğlu Introduction Aim Extend deep Convolution Networks to action recognition in video. Motivation
Deep Learning for Computer Vision with MATLAB By Jon Cherrie
Deep Learning for Computer Vision with MATLAB By Jon Cherrie 2015 The MathWorks, Inc. 1 Deep learning is getting a lot of attention "Dahl and his colleagues won $22,000 with a deeplearning system. 'We
Real Time Monitoring of CCTV Camera Images Using Object Detectors and Scene Classification for Retail and Surveillance Applications
Real Time Monitoring of CCTV Camera Images Using Object Detectors and Scene Classification for Retail and Surveillance Applications Anand Joshi CS229-Machine Learning, Computer Science, Stanford University,
Deep Learning & Neural Networks
Deep Learning & Neural Networks Machine Learning CSE4546 Sham Kakade University of Washington November 29, 2016 Sham Kakade 1 Announcements: HW4 posted Poster Session Thurs, Dec 8 Today: Review: EM Neural
Machine Learning 13. week
Machine Learning 13. week Deep Learning Convolutional Neural Network Recurrent Neural Network 1 Why Deep Learning is so Popular? 1. Increase in the amount of data Thanks to the Internet, huge amount of
Everything you wanted to know about Deep Learning for Computer Vision but were afraid to ask
Everything you wanted to know about Deep Learning for Computer Vision but were afraid to ask Moacir A. Ponti, Leonardo S. F. Ribeiro, Tiago S. Nazare ICMC University of São Paulo São Carlos/SP, 13566-590,
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability Janis Keuper Itwm.fraunhofer.de/ml Competence Center High Performance Computing Fraunhofer ITWM, Kaiserslautern,
Hardware and Software. Fei-Fei Li & Justin Johnson & Serena Yeung. Lecture 6-1
Lecture 6: Hardware and Software Lecture 6-1 Administrative Assignment 1 was due yesterday. Assignment 2 is out, due Wed May 1. Project proposal due Wed April 24. Project-only office hours leading up to
3D Densely Convolutional Networks for Volumetric Segmentation. Toan Duc Bui, Jitae Shin, and Taesup Moon
3D Densely Convolutional Networks for Volumetric Segmentation Toan Duc Bui, Jitae Shin, and Taesup Moon School of Electronic and Electrical Engineering, Sungkyunkwan University, Republic of Korea arxiv:1709.03199v2
Predicting Goal-Scoring Opportunities in Soccer by Using Deep Convolutional Neural Networks. Master s Thesis
Predicting Goal-Scoring Opportunities in Soccer by Using Deep Convolutional Neural Networks Martijn Wagenaar 6 November 26 Master s Thesis Department of Artificial Intelligence, University of Groningen,
MoonRiver: Deep Neural Network in C++
MoonRiver: Deep Neural Network in C++ Chung-Yi Weng Computer Science & Engineering University of Washington chungyi@cs.washington.edu Abstract Artificial intelligence resurges with its dramatic improvement
Training Deep Neural Networks (in parallel)
Lecture 9: Training Deep Neural Networks (in parallel) Visual Computing Systems How would you describe this professor? Easy? Mean? Boring? Nerdy? Professor classification task Classifies professors as
Fusion of Mini-Deep Nets
Rochester Institute of Technology RIT Scholar Works Theses Thesis/Dissertation Collections 5-2016 Fusion of Mini-Deep Nets Sai Prasad Nooka spn8235@rit.edu Follow this and additional works at: http://scholarworks.rit.edu/theses
POINT CLOUD DEEP LEARNING
POINT CLOUD DEEP LEARNING Innfarn Yoo, 3/29/28 / 57 Introduction AGENDA Previous Work Method Result Conclusion 2 / 57 INTRODUCTION 3 / 57 2D OBJECT CLASSIFICATION Deep Learning for 2D Object Classification
High Performance Computing
High Performance Computing 9th Lecture 2016/10/28 YUKI ITO 1 Selected Paper: vdnn: Virtualized Deep Neural Networks for Scalable, MemoryEfficient Neural Network Design Minsoo Rhu, Natalia Gimelshein, Jason
Aggregated Residual Transformations for Deep Neural Networks
Aggregated Residual Transformations for Deep Neural Networks Saining Xie 1 Ross Girshick 2 Piotr Dollár 2 Zhuowen Tu 1 Kaiming He 2 1 UC San Diego 2 Facebook AI Research {s9xie,ztu}@ucsd.edu {rbg,pdollar,kaiminghe}@fb.com
Encoder-Decoder Networks for Semantic Segmentation. Sachin Mehta
Encoder-Decoder Networks for Semantic Segmentation Sachin Mehta Outline > Overview of Semantic Segmentation > Encoder-Decoder Networks > Results What is Semantic Segmentation? Input: RGB Image Output:
A Survey on Deep Learning Methods for Robot Vision
A Survey on Deep Learning Methods for Robot Vision Javier Ruiz-del-Solar, Patricio Loncomilla, Naiomi Soto Advanced Mining Technology Center & Dept. of Electrical Engineering Universidad de Chile, Chile
Tiny ImageNet Challenge
Tiny ImageNet Challenge Jiayu Wu Stanford University jiayuwu@stanford.edu Qixiang Zhang Stanford University qixiang@stanford.edu Guoxi Xu Stanford University guoxixu@stanford.edu Abstract We present image
Architecting new deep neural networks for embedded applications
Architecting new deep neural networks for embedded applications Forrest Iandola 1 Machine Learning in 2012 Sentiment Analysis LDA Object Detection Deformable Parts Model Word Prediction Linear Interpolation
COMP 551 Applied Machine Learning Lecture 16: Deep Learning
COMP 551 Applied Machine Learning Lecture 16: Deep Learning Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted, all
Using Machine Learning for Classification of Cancer Cells
Using Machine Learning for Classification of Cancer Cells Camille Biscarrat University of California, Berkeley I Introduction Cell screening is a commonly used technique in the development of new drugs.
Fuzzy Set Theory in Computer Vision: Example 3, Part II
Fuzzy Set Theory in Computer Vision: Example 3, Part II Derek T. Anderson and James M. Keller FUZZ-IEEE, July 2017 Overview Resource; CS231n: Convolutional Neural Networks for Visual Recognition https://github.com/tuanavu/stanford-
Deep Learning For Video Classification. Presented by Natalie Carlebach & Gil Sharon
Deep Learning For Video Classification Presented by Natalie Carlebach & Gil Sharon Overview Of Presentation Motivation Challenges of video classification Common datasets 4 different methods presented in
Restricted Connectivity in Deep Neural Networks
Restricted Connectivity in Deep Neural Networks Yani Ioannou University of Cambridge April 7, 207 Overview Introduction Research Overview PhD Research Motivation Structural Priors Spatial Structural Priors
WHILE increasingly deeper networks are proposed in the
LDS-Inspired Residual Networks Anastasios Dimou, Dimitrios Ataloglou, Kosmas Dimitropoulos, Member, IEEE, Federico Alvarez, Member, IEEE, and Petros Daras, Senior Member, IEEE Abstract Residual Networks
Mocha.jl. Deep Learning in Julia. Chiyuan Zhang CSAIL, MIT
Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning with multi-layer (3~30) neural networks, on a huge training set. State-of-the-art on many AI tasks Computer Vision:
arxiv: v5 [cs.cv] 11 Dec 2018
![arxiv: v5 [cs.cv] 11 Dec 2018](/thumbs/93/111261947.jpg "arxiv: v5 [cs.cv] 11 Dec 2018")
Fire SSD: Wide Fire Modules based Single Shot Detector on Edge Device HengFui, Liau, 1, Nimmagadda, Yamini, 2 and YengLiong, Wong 1 {heng.hui.liau,yamini.nimmagadda,yeng.liong.wong}@intel.com, arxiv:1806.05363v5
BHNN: a Memory-Efficient Accelerator for Compressing Deep Neural Network with Blocked Hashing Techniques
BHNN: a Memory-Efficient Accelerator for Compressing Deep Neural Network with Blocked Hashing Techniques Jingyang Zhu 1, Zhiliang Qian 2*, and Chi-Ying Tsui 1 1 The Hong Kong University of Science and
Yelp Restaurant Photo Classification
Yelp Restaurant Photo Classification Rajarshi Roy Stanford University rroy@stanford.edu Abstract The Yelp Restaurant Photo Classification challenge is a Kaggle challenge that focuses on the problem predicting
Tiny ImageNet Visual Recognition Challenge
Tiny ImageNet Visual Recognition Challenge Ya Le Department of Statistics Stanford University yle@stanford.edu Xuan Yang Department of Electrical Engineering Stanford University xuany@stanford.edu Abstract
Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture
The 51st Annual IEEE/ACM International Symposium on Microarchitecture Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture Byungchul Hong Yeonju Ro John Kim FuriosaAI Samsung