Object Recognition II

Size: px

Start display at page:

Download "Object Recognition II"

Lester Goodwin
5 years ago
Views:

1 Object Recognition II Linda Shapiro EE/CSE 576 with CNN slides from Ross Girshick 1

2 Outline Object detection the task, evaluation, datasets Convolutional Neural Networks (CNNs) overview and history Region-based Convolutional Networks (R-CNNs) 2

3 Image classification K classes Task: assign correct class label to the whole image Digit classification (MNIST) Object recognition (Caltech-101) 3

4 Classification vs. Detection Dog Dog Dog 4

5 Problem formulation { airplane, bird, motorbike, person, sofa } person motorbike Input Desired output 5

6 Evaluating a detector Test image (previously unseen)

7 First detection person detector predictions

8 Second detection person detector predictions

9 Third detection person detector predictions

10 Compare to ground truth person detector predictions ground truth person boxes

11 Sort by confidence true positive (high overlap) X X X false positive (no overlap, low overlap, or duplicate)

12 Evaluation metric X X X t ppppppppppp = #tttt ppppppppppp #tttt ppppppppppp + #fffff ppppppppppp + X rrrrrrrr = #tttt ppppppppppp #gggggg ttttt ooooooo

Evaluation metric 0.9 0.8 0.6 0.5 0.2 0.1.

13 Evaluation metric X X X Average Precision (AP) 0% is worst 100% is best mean AP over classes (map)

14 Pedestrians Histograms of Oriented Gradients for Human Detection, Dalal and Triggs, CVPR 2005 AP ~77% More sophisticated methods: AP ~90% (a) average gradient image over training examples (b) each pixel shows max positive SVM weight in the block centered on that pixel (c) same as (b) for negative SVM weights (d) test image (e) its R-HOG descriptor (f) R-HOG descriptor weighted by positive SVM weights (g) R-HOG descriptor weighted by negative SVM weights 14

15 Overview of HOG Method 1. Compute gradients in the region to be described 2. Put them in bins according to orientation 3. Group the cells into large blocks 4. Normalize each block 5. Train classifiers to decide if these are parts of a human 15

16 Details Gradients [-1 0 1] and [-1 0 1] T were good enough filters. Cell Histograms Each pixel within the cell casts a weighted vote for an orientation-based histogram channel based on the values found in the gradient computation. (9 channels worked) Blocks Group the cells together into larger blocks, either R-HOG blocks (rectangular) or C-HOG blocks (circular). 16

17 More Details Block Normalization They tried 4 different kinds of normalization. L1-norm sqrt of L1-norm L2 norm L2-norm followed by clipping If you think of the block as a vector v, then the normalized block is v/norm(v) 17

18 Example: Dalal-Triggs pedestrian detector 1. Extract fixed-sized (64x128 pixel) window at each position and scale 2. Compute HOG (histogram of gradient) features within each window 3. Score the window with a linear SVM classifier 4. Perform non-maxima suppression to remove overlapping detections with lower scores 18 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

19 Slides by Pete Barnum 19 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

20 Outperforms centered diagonal uncentered cubic-corrected Sobel Slides by Pete Barnum 20 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

21 Histogram of gradient orientations Orientation: 9 bins (for unsigned angles) Histograms in 8x8 pixel cells Votes weighted by magnitude Bilinear interpolation between cells Slides by Pete Barnum 21 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

22 Normalize with respect to surrounding cells Slides by Pete Barnum 22 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

23 # orientations X= # features = 15 x 7 x 9 x 4 = 3780 # cells # normalizations by neighboring cells Slides by Pete Barnum 23 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

24 Training set 24

25 pos w neg w Slides by Pete Barnum 25 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

26 pedestrian Slides by Pete Barnum 26 Navneet Dalal and Bill Triggs, Histograms of Oriented Gradients for Human Detection, CVPR05

27 Detection examples 27

28 30

29 Deformable Parts Model Takes the idea a little further Instead of one rigid HOG model, we have multiple HOG models in a spatial arrangement One root part to find first and multiple other parts in a tree structure. 31

30 The Idea Articulated parts model Object is configuration of parts Each part is detectable 32 Images from Felzenszwalb

31 Deformable objects Images from Caltech Slide Credit: Duan Tran

32 Deformable objects Images from D. Ramanan s dataset 34 Slide Credit: Duan Tran

33 How to model spatial relations? Tree-shaped model 35

34 36

35 Hybrid template/parts model Detections Template Visualization 37 Felzenszwalb et al. 2008

36 Pictorial Structures Model Appearance likelihood Geometry likelihood 38

37 Results for person matching 39

38 Results for person matching 40

39 41 BMVC 2009

2012 State-of-the-art Detector: Deformable Parts

part-based models (pictorial structures) 3.

40 2012 State-of-the-art Detector: Deformable Parts Model (DPM) Lifetime Achievement 1. Strong low-level features based on HOG 2. Efficient matching algorithms for deformable part-based models (pictorial structures) 3. Discriminative learning with latent variables (latent SVM) Felzenszwalb 42 et al., 2008, 2010, 2011, 2012

41 Why did gradient-based models work? Average gradient image 43

42 Generic categories Can we detect people, chairs, horses, cars, dogs, buses, bottles, sheep? PASCAL Visual Object Categories (VOC) dataset 44

43 Generic categories Why doesn t this work (as well)? Can we detect people, chairs, horses, cars, dogs, buses, bottles, sheep? PASCAL Visual Object Categories (VOC) dataset 45

44 Quiz time (Back to Girshick) 46

45 Warm up This is an average image of which object class? 47

46 Warm up pedestrian 48

47 A little harder? 49

48 A little harder? Hint: airplane, bicycle, bus, car, cat, chair, cow, dog, dining table 50

49 A little harder bicycle (PASCAL) 51

50 A little harder, yet? 52

51 A little harder, yet? Hint: white blob on a green background 53

52 A little harder, yet sheep (PASCAL) 54

53 Impossible?? 55

54 Impossible? dog (PASCAL) 56

55 Impossible? dog (PASCAL) Why does the mean look like this? There s no alignment between the examples! How do we combat this? 57

56 PASCAL VOC detection history 70% mean Average Precision (map) 60% 50% 40% 30% 20% 10% 17% DPM 23% DPM, HOG+ BOW 28% DPM, MKL 37% 41% DPM++ DPM++, MKL, Selective Search 41% Selective Search, DPM++, MKL 0% year

57 Part-based models & multiple features (MKL) 70% mean Average Precision (map) 60% 50% 40% 30% 20% 10% 17% DPM 23% DPM, HOG+ BOW 28% DPM, MKL 37% 41% DPM++ DPM++, MKL, Selective Search 41% Selective Search, DPM++, MKL 0% year

58 Kitchen-sink approaches 70% mean Average Precision (map) 60% 50% 40% 30% 20% 10% 17% DPM 23% DPM, HOG+ BOW 28% DPM, MKL increasing complexity & plateau 37% 41% DPM++ DPM++, MKL, Selective Search 41% Selective Search, DPM++, MKL 0% year

59 Region-based Convolutional Networks (R-CNNs) mean Average Precision (map) 70% 60% 50% 40% 30% 20% 10% 17% DPM 23% DPM, HOG+ BOW 28% DPM, MKL 37% 41% DPM++ DPM++, MKL, Selective Search 41% Selective Search, DPM++, MKL 53% R-CNN v1 62% R-CNN v2 0% year [R-CNN. Girshick et al. CVPR 2014]

60 Region-based Convolutional Networks (R-CNNs) 70% mean Average Precision (map) 60% 50% 40% 30% 20% 10% ~1 year ~5 years 0% year [R-CNN. Girshick et al. CVPR 2014]

61 Convolutional Neural Networks Overview 63

62 Standard Neural Networks Fully connected T x = x 1,, x 784 z j = g(w T j x) g t = e t 64

63 From NNs to Convolutional NNs Local connectivity Shared ( tied ) weights Multiple feature maps Pooling 65

64 Convolutional NNs Local connectivity compare Each green unit is only connected to (3) neighboring blue units 66

65 Convolutional NNs Shared ( tied ) weights w 1 w 2 w 3 w 1 w 2 All green units share the same parameters w Each green unit computes the same function, but with a different input window w 3 67

66 Convolutional NNs Convolution with 1-D filter: [w 3, w 2, w 1 ] w 1 w 2 w 3 All green units share the same parameters w Each green unit computes the same function, but with a different input window 68

67 Convolutional NNs Convolution with 1-D filter: [w 3, w 2, w 1 ] w 1 w 2 w 3 All green units share the same parameters w Each green unit computes the same function, but with a different input window 69

68 Convolutional NNs Convolution with 1-D filter: [w 3, w 2, w 1 ] w 1 w 2 w 3 All green units share the same parameters w Each green unit computes the same function, but with a different input window 70

69 Convolutional NNs Convolution with 1-D filter: [w 3, w 2, w 1 ] All green units share the same parameters w w 1 w 2 w 3 Each green unit computes the same function, but with a different input window 71

70 Convolutional NNs Convolution with 1-D filter: [w 3, w 2, w 1 ] All green units share the same parameters w w 1 Each green unit computes the same function, but with a different input window w 2 w 3 72

71 Convolutional NNs Multiple feature maps ww 1 ww 2 ww 3 All orange units compute the same function but with a different input windows w 1 w 2 w 3 Feature map 1 (array of green units) Orange and green units compute different functions Feature map 2 (array of orange units) 73

72 Convolutional NNs Pooling (max, average) Pooling area: 2 units Pooling stride: 2 units Subsamples feature maps 74

73 2D input Pooling Convolution Image 75

74 1989 Backpropagation applied to handwritten zip code recognition, Lecun et al.,

75 Historical perspective

76 Historical perspective 1980 Hubel and Wiesel 1962 Included basic ingredients of ConvNets, but no supervised learning algorithm 78

77 Supervised learning 1986 Gradient descent training with error backpropagation Early demonstration that error backpropagation can be used for supervised training of neural nets (including ConvNets) 79

78 Supervised learning 1986 T vs. C problem Simple ConvNet 80

79 Practical ConvNets Gradient-Based Learning Applied to Document Recognition, Lecun et al.,

80 Demo demo/mnist.html ConvNetJS by Andrej Karpathy (Ph.D. student at Stanford) Software libraries Caffe (C++, python, matlab) Torch7 (C++, lua) Theano (python) 82

81 The fall of ConvNets The rise of Support Vector Machines (SVMs) Mathematical advantages (theory, convex optimization) Competitive performance on tasks such as digit classification Neural nets became unpopular in the mid 1990s 83

82 The key to SVMs It s all about the features HOG features SVM weights (+) (-) Histograms of Oriented Gradients for Human Detection, Dalal and Triggs, CVPR

83 Core idea of deep learning Input: the raw signal (image, waveform, ) Features: hierarchy of features is learned from the raw input 85

84 If SVMs killed neural nets, how did they come back (in computer vision)? 86

85 What s new since the 1980s? More layers LeNet-3 and LeNet-5 had 3 and 5 learnable layers Current models have ReLU non-linearities (Rectified Linear Unit) g x = max 0, x Gradient doesn t vanish Dropout regularization Fast GPU implementations More data g(x) x 87

ratio, and location Object proposals Fast

86 What else? Object Proposals Sliding window based object detection Image Feature Extraction Classificaiton Iterate over window size, aspect ratio, and location Object proposals Fast execution High recall with low # of candidate boxes Image Object Proposal Feature Extraction Classificaiton 88

87 The number of contours wholly enclosed by a bounding box is indicative of the likelihood of the box containing an object. 89

88 Ross s Own System: Region CNNs

89 Competitive Results

90 Top Regions for Six Object Classes

91 Finale Object recognition has moved rapidly in the last 12 years to becoming very appearance based. The HOG descriptor lead to fast recognition of specific views of generic objects, starting with pedestrians and using SVMs. Deformable parts models extended that to allow more objects with articulated limbs, but still specific views. CNNs have become the method of choice; they learn from huge amounts of data and can learn multiple views of each object class. 93

Modern Object Detection. Most slides from Ali Farhadi

Modern Object Detection. Most slides from Ali Farhadi Modern Object Detection Most slides from Ali Farhadi Comparison of Classifiers assuming x in {0 1} Learning Objective Training Inference Naïve Bayes maximize j i logp + logp ( x y ; θ ) ( y ; θ ) i ij