Object detection with CNNs

Transcription

1 Object detection with CNNs 80% PASCAL VOC mean0average0precision0(map) 70% 60% 50% 40% 30% 20% 10% Before CNNs After CNNs 0% year

2 Region proposals As an alternative to sliding window search, evaluate a few hundred region proposals Can use slower but more powerful features and classifiers Take advantage of low-level perceptual organization cues Proposal mechanism can be category-independent Proposal mechanism can be trained

3 Selective search Use hierarchical segmentation: start with small superpixels and merge based on diverse cues J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, Selective Search for Object Recognition, IJCV 2013

4 Evaluation of region proposals J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, Selective Search for Object Recognition, IJCV 2013

5 Selective search detection pipeline Feature extraction: color SIFT, codebook of size 4K, spatial pyramid with four levels = 360K dimensions J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, Selective Search for Object Recognition, IJCV 2013

6 Another proposal method: EdgeBoxes Box score: number of edges in the box minus number of edges that overlap the box boundary Uses a trained edge detector Uses efficient data structures (incl. integral images) for fast evaluation Gets 75% recall with 800 boxes (vs for Selective Search), is 40 times faster C. Zitnick and P. Dollar, Edge Boxes: Locating Object Proposals from Edges, ECCV 2014.

7 R-CNN: Region proposals + CNN features Source: R. Girshick SVMs SVMs ConvNet SVMs ConvNet Classify regions with SVMs Forward each region through ConvNet ConvNet Warped image regions Region proposals Input image R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, CVPR 2014.

8 R-CNN details Regions: ~2000 Selective Search proposals Network: AlexNet pre-trained on ImageNet (1000 classes), fine-tuned on PASCAL (21 classes) Final detector: warp proposal regions, extract fc7 network activations (4096 dimensions), classify with linear SVM Bounding box regression to refine box locations Performance: map of 53.7% on PASCAL 2010 (vs. 35.1% for Selective Search and 33.4% for DPM). R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, CVPR 2014.

9 R-CNN pros and cons Pros Accurate! Any deep architecture can immediately be plugged in Cons Ad hoc training objectives Fine-tune network with softmax classifier (log loss) Train post-hoc linear SVMs (hinge loss) Train post-hoc bounding-box regressions (least squares) Training is slow (84h), takes a lot of disk space 2000 CNN passes per image Inference (detection) is slow (47s / image with VGG16)

10 Fast R-CNN Softmax classifier Linear + softmax Linear Bounding-box regressors FCs Fully-connected layers RoI Pooling layer Region proposals conv5 feature map of image Forward whole image through ConvNet ConvNet Source: R. Girshick R. Girshick, Fast R-CNN, ICCV 2015

11 Fast R-CNN training Log loss + smooth L1 loss Multi-task loss Linear + softmax Linear FCs Trainable ConvNet Source: R. Girshick R. Girshick, Fast R-CNN, ICCV 2015

12 Fast R-CNN: Another view R. Girshick, Fast R-CNN, ICCV 2015

13 ROI pooling closeup (actual output feature map size: 7 x 7) Image source

14 Fast R-CNN results Fast R-CNN Train time (h) Speedup 8.8x 1x Test time / image R-CNN 0.32s 47.0s Test speedup 146x 1x map 66.9% 66.0% Timings exclude object proposal time, which is equal for all methods. All methods use VGG16 from Simonyan and Zisserman. Source: R. Girshick

15 Faster R-CNN Region proposals Region Proposal Network feature map feature map share features CNN CNN S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015

16 Region proposal network (RPN) Slide a small window over the conv5 layer Predict object/no object Regress bounding box coordinates Box regression is with reference to anchors (3 scales x 3 aspect ratios)

17 Faster R-CNN results

18 Object detection progress mean0average0precision0(map) 80% 70% 60% 50% 40% 30% 20% 10% Before deep convnets Faster R-CNN Fast R-CNN R-CNNv1 Using deep convnets 0% year

19 YOLO 1. Take conv feature maps at 7x7 resolution 2. Add two FC layers to predict, at each location, a score for each class and 2 bboxes w/ confidences (output is 7x7x30 for PASCAL) 7x speedup over Faster RCNN ( FPS vs FPS) Some loss of accuracy due to lower recall, poor localization J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016

20 YOLO v2 Batch normalization Pre-train on higher resolution ImageNet Use and improve anchor box idea from Faster RCNN Train at multiple resolutions Very good accuracy, very fast J. Redmon and A. Farhadi, YOLO9000: Better, Faster, Stronger, CVPR 2017

21 SSD W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. Berg, SSD: Single Shot MultiBox Detector, ECCV 2016.

22 Feature pyramid networks Improve predictive power of lower-level feature maps by adding contextual information from higherlevel feature maps Predict different sizes of bounding boxes from different levels of the pyramid (but share parameters of predictors) T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, Feature pyramid networks for object detection, CVPR 2017.

23 New detection benchmark: COCO (2014) 80 categories instead of PASCAL s 20 Current best map: 52%

24 New detection benchmark: COCO (2014) J. Huang et al., Speed/accuracy trade-offs for modern convolutional object detectors, CVPR 2017

25 Review: Object detection with CNNs

26 Review: R-CNN SVMs Classify regions with SVMs SVMs ConvNet SVMs ConvNet ConvNet Forward each region through ConvNet Warped image regions Region proposals Input image R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, CVPR 2014.

27 Review: Fast R-CNN Softmax classifier Linear + softmax Linear Bounding-box regressors FCs Fully-connected layers RoI Pooling layer Region proposals conv5 feature map of image Forward whole image through ConvNet ConvNet R. Girshick, Fast R-CNN, ICCV 2015

28 Review: Faster R-CNN Region proposals Region Proposal Network feature map feature map share features CNN CNN S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015

29 Summary: Object detection with CNNs R-CNN: region proposals + CNN on cropped, resampled regions Fast R-CNN: region proposals + RoI pooling on top of a conv feature map Faster R-CNN: RPN + RoI pooling Next generation of detectors Direct prediction of BB offsets, class scores on top of conv feature maps Get better context by combining feature maps at multiple resolutions