Deep Residual Learning

Transcription

1 Deep Residual Learning ILSVRC & COCO 2015 competitions Kaiming He with Xiangyu Zhang, Shaoqing Ren, Jifeng Dai, & Jian Sun Microsoft Research Asia (MSRA)

2 ILSVRC & COCO 2015 Competitions 1st places in all five main tracks ImageNet Classification: Ultra-deep (quote Yann) 152-layer nets ImageNet Detection: 16% better than 2nd ImageNet Localization: 27% better than 2nd COCO Detection: 11% better than 2nd COCO Segmentation: 12% better than 2nd *improvements are relative numbers

3 Revolution of Depth 152 layers layers 19 layers layers 8 layers shallow ILSVRC'15 ResNet ILSVRC'14 GoogleNet ILSVRC'14 VGG ILSVRC'13 ILSVRC'12 AlexNet ImageNet Classification top-5 error (%) ILSVRC'11 ILSVRC'10

4 Revolution of Depth 101 layers Engines of visual recognition shallow 8 layers 16 layers HOG, DPM AlexNet (RCNN) VGG (RCNN) ResNet (Faster RCNN)* PASCAL VOC 2007 Object Detection map (%) *w/ other improvements & more data

5 Revolution of Depth AlexNet, 8 layers (ILSVRC 2012) 11x11 conv, 96, /4, pool/2 5x5 conv, 256, pool/2 3x3 conv, 384 3x3 conv, 384, pool/2 fc, 4096 fc, 4096 fc, 1000

6 Revolution of Depth soft max2 Soft maxact ivat ion FC AlexNet, 8 layers (ILSVRC 2012) 11x11 conv, 96, /4, pool/2 5x5 conv, 256, pool/2 3x3 conv, 384 VGG, 19 layers (ILSVRC 2014), pool/2 GoogleNet, 22 layers (ILSVRC 2014) AveragePool 7x7+ 1(V) Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) soft max1 3x3 conv, 384, pool/2 Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) MaxPool 3x3+ 2(S) Soft maxact ivat ion FC, pool/2 Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) FC Conv 1x1+ 1(S) fc, 4096 Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) Dept hconcat AveragePool 5x5+ 3(V) fc, 4096 Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) fc, 1000, pool/2 Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) soft max0 Soft maxact ivat ion Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) Dept hconcat FC FC Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) Conv 1x1+ 1(S) AveragePool 5x5+ 3(V), pool/2, pool/2 fc, 4096 fc, 4096 Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) MaxPool 3x3+ 2(S) Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) Dept hconcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S) Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S) MaxPool 3x3+ 2(S) LocalRespNorm Conv 3x3+ 1(S) Conv 1x1+ 1(V) LocalRespNorm MaxPool 3x3+ 2(S) fc, 1000 input Conv 7x7+ 2(S)

7 11x11 conv, 96, /4, pool/2 5x5 conv, 256, pool/2 3x3 conv, 384 3x3 conv, 384, pool/2 fc, 4096 fc, 4096 fc, 1000, pool/2, pool/2, pool/2, pool/2, pool/2 fc, 4096 fc, 4096 fc, x7 conv, 64, /2, po ol /2 1x1 conv, 64 1x1 conv, 64 1x1 conv, 64 1x1 conv, 128, /2 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128, /2, /2 1x1 conv, x1 conv, x1 conv, 2048 ave pool, fc 1000 Revolution of Depth AlexNet, 8 layers (ILSVRC 2012) VGG, 19 layers (ILSVRC 2014) ResNet, 152 layers (ILSVRC 2015)

8 7x7 conv, 64, /2, po ol /2 1x1 conv, 64 Revolution of Depth 1x1 conv, 64 ResNet, 152 layers 1x1 conv, 64 1x1 conv, 128, /2 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 1x1 conv, 128 (there was an animation here) 1x1 conv, 128, /2

9 1x1 conv, 128 Revolution of Depth, /2 ResNet, 152 layers (there was an animation here) Kaiming He, Xiangyu Zhang, 1x1 conv, Shaoqing 256 Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arxiv 2015.

10 Revolution of Depth ResNet, 152 layers (there was an animation here) Kaiming He, Xiangyu Zhang, 3x3 conv, Shaoqing 256 Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arxiv 2015.

11 Revolution of Depth ResNet, 152 layers, /2 1x1 conv, x1 conv, 2048 (there was an animation here) 1x1 conv, 2048 ave pool, fc 1000

12 Is learning better networks as simple as stacking more layers?

13 Simply stacking layers? 20 train error (%) 56-layer CIFAR test error (%) 56-layer layer layer iter. (1e4) iter. (1e4) Plain nets: stacking 3x3 conv layers 56-layer net has higher training error and test error than 20-layer net

14 Simply stacking layers? error (%) CIFAR layer 44-layer 32-layer 20-layer error (%) ImageNet layer 5 plain-20 plain-32 plain plain iter. (1e4) solid: test/val dashed: train 30 plain-18 plain iter. (1e4) 18-layer Overly deep plain nets have higher training error A general phenomenon, observed in many datasets

15 a shallower model (18 layers) 7x7 conv, 64, /2 7x7 conv, 64, /2 a deeper counterpart (34 layers), /2, /2 A deeper model should not have higher training error, /2 extra layers, /2 A solution by construction: original layers: copied from a learned shallower model extra layers: set as identity at least the same training error, /2 fc 1000, /2 fc 1000 Optimization difficulties: solvers cannot find the solution when going deeper

16 Deep Residual Learning Plaint net xx HH xx is any desired mapping, hope the 2 weight layers fit HH(xx) any two stacked layers weight layer weight layer HH(xx) relu relu

17 Deep Residual Learning Residual net xx HH xx is any desired mapping, hope the 2 weight layers fit HH(xx) weight layer hope the 2 weight layers fit FF(xx) FF(xx) relu weight layer identity xx let HH xx = FF xx + xx HH xx = FF xx + xx relu

18 Deep Residual Learning FF xx is a residual mapping w.r.t. identity xx FF(xx) weight layer relu weight layer identity xx If identity were optimal, easy to set weights as 0 If optimal mapping is closer to identity, easier to find small fluctuations HH xx = FF xx + xx relu

19 Related Works Residual Representations VLAD & Fisher Vector [Jegou et al 2010], [Perronnin et al 2007] Encoding residual vectors; powerful shallower representations. Product Quantization (IVF-ADC) [Jegou et al 2011] Quantizing residual vectors; efficient nearest-neighbor search. MultiGrid & Hierarchical Precondition [Briggs, et al 2000], [Szeliski 1990, 2006] Solving residual sub-problems; efficient PDE solvers.

20 7x7 conv, 64, /2 7x7 conv, 64, /2 pool, /2 pool, /2 Network Design plain net ResNet, /2, /2 Keep it simple Our basic design (VGG-style) all 3x3 conv (almost) spatial size /2 => # filters x2 Simple design; just deep!, /2, /2 Other remarks: no max pooling (almost) no hidden fc no dropout, /2, /2 avg pool avg pool fc 1000 fc 1000

21 Training All plain/residual nets are trained from scratch All plain/residual nets use Batch Normalization Standard hyper-parameters & augmentation

22 CIFAR-10 experiments error (%) CIFAR-10 plain nets 5 plain-20 plain-32 plain plain iter. (1e4) layer 44-layer 32-layer 20-layer solid: test dashed: train error (%) CIFAR-10 ResNets iter. (1e4) ResNet-20 ResNet-32 ResNet-44 ResNet-56 ResNet layer 32-layer 44-layer 56-layer 110-layer Deep ResNets can be trained without difficulties Deeper ResNets have lower training error, and also lower test error

23 ImageNet experiments ImageNet plain nets ImageNet ResNets error (%) layer error (%) layer 30 solid: test plain-18 dashed: train plain iter. (1e4) 18-layer 30 ResNet-18 ResNet iter. (1e4) layer Deep ResNets can be trained without difficulties Deeper ResNets have lower training error, and also lower test error

24 ImageNet experiments A practical design of going deeper 64-d 256-d 3x3, 64 relu 3x3, 64 1x1, 64 relu 3x3, 64 relu 1x1, 256 relu all-3x3 similar complexity relu bottleneck (for ResNet-50/101/152)

25 ImageNet experiments this model has lower time complexity than VGG-16/19 Deeper ResNets have lower error ResNet-152 ResNet-101 ResNet-50 ResNet crop testing, top-5 val error (%)

26 ImageNet experiments 152 layers layers 19 layers layers 8 layers shallow ILSVRC'15 ResNet ILSVRC'14 GoogleNet ILSVRC'14 VGG ILSVRC'13 ILSVRC'12 AlexNet ImageNet Classification top-5 error (%) ILSVRC'11 ILSVRC'10

27 Just classification? A treasure from ImageNet is on learning features.

28 Features matter. (quote [Girshick et al. 2014], the R-CNN paper) task 2nd-place winner MSRA margin (relative) ImageNet Localization (top-5 error) % absolute 8.5% better! ImageNet Detection % COCO Detection % COCO Segmentation % Our results are all based on ResNet-101 Our features are well transferrable

29 Object Detection (brief) Simply Faster R-CNN + ResNet proposals classifier RoI pooling Faster R-CNN baseline map@.5 map@.5:.95 VGG ResNet Region Proposal Net feature map COCO detection results (ResNet has 28% relative gain) CNN image Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015.

30 Object Detection (brief) RPN learns proposals by extremely deep nets We use only 300 proposals (no SS/EB/MCG!) Add what is just missing in Faster R-CNN Iterative localization Context modeling Multi-scale testing All are based on CNN features; all are end-to-end (train and/or inference) All benefit more from deeper features cumulative gains! Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015.

31 Our results on COCO too many objects, let s check carefully! *the original image is from the COCO dataset Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015.

32 *the original image is from the COCO dataset Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015.

33 *the original image is from the COCO dataset Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS 2015.

34 Instance Segmentation (brief) CONVs box instances (RoIs) Solely CNN-based ( features matter ) Differentiable RoI warping layer (w.r.t box coord.) Multi-task cascades, exact end-to-end training CONVs FCs mask instances RoI warping, pooling categorized instances for each RoI FCs conv feature map masking person person horse person for each RoI Jifeng Dai, Kaiming He, & Jian Sun. Instance-aware Semantic Segmentation via Multi-task Network Cascades. arxiv 2015.

35 input *the original image is from the COCO dataset Jifeng Dai, Kaiming He, & Jian Sun. Instance-aware Semantic Segmentation via Multi-task Network Cascades. arxiv 2015.

36 Conclusions MSRA team Deeper is still better Features matter! Kaiming He Xiangyu Zhang Shaoqing Ren Faster R-CNN is just amazing Jifeng Dai Jian Sun Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS Jifeng Dai, Kaiming He, & Jian Sun. Instance-aware Semantic Segmentation via Multi-task Network Cascades. arxiv 2015.