AK Computer Vision Feature Point Detectors and Descriptors

Transcription

1 AK Computer Vision Feature Point Detectors and Descriptors 1

2 Feature Point Detectors and Descriptors: Motivation 2

3 Step 1: Detect local features should be invariant to scale and rotation, or perspective transformation 3

4 Step 2: Rectify patch 4

5 Step 3: Build a description vector ("descriptor") 5

6 Step 4: Match the description vectors aka descriptors 6

7 7

8 Motivation Global image representations are difficult to handle Alternative: describe and match only local regions around interest points Increased robustness to Occlusions 8

9 Motivation Global image representations are difficult to handle Alternative: describe and match only local regions around interest points Increased robustness to Occlusions Geometric transformations: non-rigid deformation, perspective, etc. Intra-category variations: 9

10 Covariant vs. Invariant When a transformation is applied to an image, an invariant measure remains unchanged. a covariant measure changes in a way consistent with the image transformation IP Detector: Covariant detectors => Invariant descriptors 10

11 An Application Image Mosaicing: How to combine several, overlapping images 11

12 Robust Feature-Based Alignment 12

13 Robust Feature-Based Alignment Extract interest points and descriptors 13

14 Robust feature-based alignment Extract interest points and descriptors Compute putative matches 14

15 Robust feature-based alignment Extract interest points and descriptors Compute putative matches Loop (RANSAC): Hypothesize transformation T 15

16 Robust feature-based alignment Extract interest points Compute putative matches Loop (RANSAC): Hypothesize transformation T Verify transformation 16

17 Robust feature-based alignment Extract interest points and descriptors Compute putative matches Loop (RANSAC): Hypothesize transformation T Verify transformation Source: L. Lazebnik 17

18 Structure from Motion 22

19 Detection of interesting image parts INTEREST POINTS 24

20 History of Interest Points First interest point detector in 1977, the Moravec operator 26

21 Harris corner detector Hessian detector Laplace variants Affine variants MSER SURF Edge Foci FAST... Interest Points 27

22 Interest Points HARRIS CORNERS 28

23 Harris Detector Introduced by Harris and Stephens in 1988 Basic assumption: Shifting a local patch in any direction should give a large change in intensity References: A combined corner and edge detector, Harris and Stephens, Alvey Vision Conference 29

24 Small Motion Assumption Taylor Series expansion of I(x + u, y + v): If the motion (u,v) is small, then the first order approximation is good 30

25 Local Feature Detection: The Math 31

26 E(u, v) Local Feature Detection: The Math X (x,y)2w X (x,y)2w X (x,y)2w [uv] apple [I x I y ] apple Ix apple u v apple u [uv] [I I x I y ] y v apple apple I 2 [uv] x I x I y u I x I y Iy 2 v 1 X apple apple I 2 x I x I y A u I x I y v (x,y)2w 2 I 2 y 32

27 Local Feature Detection: The Math E(u, v) [uv] X (x,y)2w apple I 2 x I x I y I x I y I 2 y 1 A apple u v we are looking for (x, y) images locations such that E(u, v) is large for all directions [u, v] Eigenvalues of Q reveal the amount of intensity change in the two principal orthogonal gradient directions within the patch Q 33

28 Geometric Interpretation of Q (λmax) -1/2 (λmin) -1/2 34

29 Recall: Corners as distinctive interest points edge : λ1 >> λ2 λ2 >> λ1 One way to score the cornerness: corner : λ1 and λ2 are large, λ1 ~ λ2 flat region λ1 and λ2 are small 35

30 Harris corner detector 1) Compute matrix Q for each pixel to get its cornerness score 2) Find points with large corner response (f > threshold) 3) Take the points of local maxima, i.e., perform nonmaximum suppression 36

31 Harris Detector: Steps 37

32 Harris Detector: Steps 38

33 Harris Detector: Steps 39

34 Harris Detector: Steps 40

35 Harris Detector: Steps 41

36 Harris Detector Example 42

37 Harris Properties Rotation invariant? Yes Scale invariant? No All points will be classified as edges Corner! 43

38 Interest Points HESSIAN DETECTOR Hessian ( I) = I I xx xy I I xy yy 44

39 Hessian determinant Hessian Detector Hessian ( I) = I I xx xy I I xy yy 46

40 Problem with Harris/Hessian Detector Not scale invariant To overcome this problem: Automatic scale space detection required Finding characteristic scale 47

41 Automatic scale selection Intuition: Find scale that gives local maxima of some function f in both position and scale. f Image 1 f Image 2 s 1 region size s 2 region size 49

42 Scale Invariant Detection Functions for determining scale Kernels: ( (,, ) (,, )) 2 L= Gxx x y + Gyy x y σ σ σ (Laplacian of Gaussian) DoG = G( x, y, kσ) G( x, y, σ) (Difference of Gaussians) f = Kernel Image where Gaussian 2 2 x + y πσ σ Gxy (,, σ ) = e Note: both kernels are invariant to scale and rotation 51

43 Affine Invariance Need to generalize uniform scale changes Local Estimation of structure à Second moment matrix 55

44 Affine Transformation Estimation Warp by Affine Transformation Q 1/2, where Q is the auto-correlation matrix. 56

45 Interest Points DIFFERENCE OF GAUSSIANS 59

46 Laplacian of Gaussian (G xx + G yy ) for feature point detection Laplacian operator 60

47 DoG Efficient Computation Computation in Gaussian scale pyramid Maxima selection in 3x3x3 neighborhood σ σ σ Original image σ 62

48 Results: Lowe s DoG 64

49 Select Canonical Orientation Create histogram of local gradient directions computed over the image patch; Each gradient contributes for its norm, weighted by its distance to patch center; Assign canonical orientation at peak of smoothed histogram. 0 2π 66

50 Interest Points MAXIMALLY STABLE EXTREMAL REGIONS 68

51 MSER 1. Threshold an image at every intensity level References: Robust wide baseline stereo from maximally stable extremal regions, Matas et al., BMVC

52 MSER 1. Threshold an image at every intensity level 2. Find connected components 3. Build tree structure (nested components) 4. Find regions that are maximally stable w.r.t. their size, analyzing stability criterion MSER+ vs. MSER- References: Robust wide baseline stereo from maximally stable extremal regions, Matas et al., BMVC

53 Example Results: MSER 75

54 Affine Covariant Fit Ellipses to each region 76

55 Interest Points SURF 78

56 Efficient IP detector SURF Based on Hessian Approximation by Box Filters References: SURF: Speeded Up Robust Features, Bay et al., CVIU

57 Methodology Using integral images for major speed up Integral Image (summed area tables) is an intermediate representation for the image and contains the sum of gray scale pixel values of image Sum of values within rectangles is calculated using only a few operations independent of location and size Cost three operations only 80

58 Interest Points FAST 83

59 FAST Use heuristic for identifying corner points: Compare intensities of 16 surrounding pixels to center pixel intensity Relies on tests I(center) > I(p) + t or I(center) > < I(p) - t 84

60 FAST Use machine learning to select the tests for faster rejection of non-corner pixels References: Machine learning for high-speed corner detection, Rosten and Drummond, CVPR

61 Learning Decision Tree (ID3) to learn appearance of corner from training data ID3: supervised machine learning method: learn from training data Deterministic (no randomness) Play Tennis? 86

62 Decision Tree Training data: Corner points detected by heuristic Splitting criterion: compare neighboring pixels to center pixel and make 3 splits Select split that has lowest entropy Repeat until entropy = 0 in each leaf node n = 9 worked best Different non-maximum suppression 87

63 FAST 88

64 Interest Point Detectors X (x,y)2w apple I 2 x I x I y I x I y I 2 y 1 A Harris corner detector Hessian detector MSER SURF FAST... Hessian ( I) = I I xx xy I I xy yy 89

65 TILDE 90

66 Interest Points EVALUATION CRITERIONS 91

67 Evaluation Repeatability: average number of corresponding regions detected in images under different geometric and photometric transformations 92

68 Institute for Computer Graphics and Vision Repeatability 93

69 Analyze Overlap 94

70 Overlap Criterion 95

71 Evaluation Repeatability: average number of corresponding regions detected in images under different geometric and photometric transformations Matching score: ratio between the number of correct matches and the smaller number of detected regions 96

72 Matching Score 97

73 Flat scenes Mikolajczyk & Schmid (2004), Mikolajczyk et al. (2004) MSER has highest repeatability (but lowest number) Harris and Hessian provide the most correspondences 3D objects Moreels & Perona (2006) Features on 3D objects are much more unstable All detectors perform poorly for large viewpoint changes 98

74 Evaluation Repeatability: average number of corresponding regions detected in images under different geometric and photometric transformations Matching score: ratio between the number of correct matches and the smaller number of detected regions Number of detected points 99

75 Evaluation Viewpoint change Zoom+Rotation 100

76 Evaluation Zoom+Rotation Blur JPEG Illumination 101

77 102

78 Code in Matlab VLFEAT 103

79 104

80 vl_covdet 1. load an image impath = fullfile( oxford.jpg ) ; im = imread(impath) ; 2. convert it to single precision gray scale imgs = im2single(rgb2gray(im)) ; 3. run SIFT [frames, descrs] = vl_sift(imgs) ; 4. visualise keypoints imagesc(im) ; colormap gray; hold on ; vl_plotframe(frames) ; 105

81 Invariant local patch description DESCRIPTORS 0 2 π 106

82 SIFT SURF HOG BRIEF BRISK LBP Shape Context Self Similarity... Local Descriptors 107

83 Step 3: Build a description vector ("descriptor") 108

84 Step 4: Match the description vectors aka descriptors 109

85 Interest Point Descriptors SIFT 0 2 π 113

86 SIFT Description Vector Made of local histograms of gradients: In practice: 8 orientations x 4 x 4 histograms = 128 dimensions vector. 115

87 Primary Visual Cortex Institute for Computer Graphics and Vision 116

88 Handling Lighting Changes Gains do not affect gradients; Normalization to unit length removes contrast; Saturation affects magnitudes much more than orientation: magnitudes are thresholded. 117

89 SIFT Feature Representation Descriptor contains histograms of a 4 4 spatial grid around the keypoint Each histogram has 8 orientation bins SIFT vector contains = 128 values Normalized to enhance invariance to illumination changes 118

90 Extraordinarily robust matching technique Can handle changes in viewpoint Up to about 60 degree out of plane rotation Can handle significant changes in illumination Sometimes even day vs. night (below) Lots of code available SIFT descriptor 119

91 Example NASA Mars Rover images 120

92 Example NASA Mars Rover images with SIFT feature matches 121

93 Interest Point Descriptors SURF 123

94 Local Descriptors: SURF Fast approximation of SIFT idea Efficient computation by 2D box filters & integral images 6 times faster than SIFT GPU implementation available Feature 100Hz (detector + descriptor, ) References: SURF: Speeded Up Robust Features, Bay et al., CVIU

95 Description Haar Wavelets: efficient calculation by integral images 125

96 Description Split the interest region up into 4 x 4 square sub-regions Calculate Haar wavelet response d x and d y Weight the response with a Gaussian kernel centered at the interest point Sum the response over each sub-region for d x and d y separately à feature vector of length 32 In order to bring in information about the polarity of the intensity changes, extract the sum of absolute value of the responses à feature vector of length 64 Normalize the vector into unit length 126

97 Interest Point Descriptors HISTOGRAM OF GRADIENTS 127

98 Is an adaption of SIFT HOG Based on gradient magnitudes and orientations Cells vs. Blocks Normalization is important References: Histograms of Oriented Gradients for Human Detection, Dalal and Triggs, CVPR

99 HoG Designed for a specific task: detect upright(!) category instances in images Instead of describing neighborhood around interest point, HoG describes an entire window around object Use ideas of SIFT descriptor: Gradient orientation histograms Binning into cells Normalization 129

100 HOG Gradient calculation First step (as in SIFT): Estimate image gradients in the horizontal and vertical directions: [-1 0 1] and [-1 0 1] T 130

101 HOG Orientation binning Second step (as in SIFT): Window is partitioned into a grid of cells Cells can be rectangular or radial Each pixel casts a weighted vote (based on the gradient magnitude) to an orientationbased histogram for the corresponding cells Histogram with 8 bins between 0 and 360 yields best results 131

102 HOG Descriptor blocks Third step (different normalization): Cells are grouped into larger, spatially connected blocks Blocks are overlapping and are used for normalization HOG feature vector is a concatenation of normalized cell histograms for all blocks Very high-dimensional: ~4000 dimensions 132

103 Institute for Computer Graphics and Vision Histogram of Oriented Gradients (HoG) 135

104 Descriptors in VL Feat VLFEAT SIFT AND HOG 136

105 Interest Point Descriptors BRIEF 137

106 BRIEF: A Fast Binary Local Descriptor BRIEF descriptor 138

107 Evaluation 139

108 Evaluation 140

109 Computation Speed For BRIEF, most of the time is spent in Gaussian smoothing. 141

110 Matching Speed 142

111 Rotation and Scale Invariance 143

112 Rotation and Scale Invariance Duplicate the Descriptors: 18 rotations x 3 scales

113 145

114 Interest Point Descriptors BRISK 146

115 BRISK Idea: make BRIEF scale and rotation invariant IP detector = FAST + scale space search Descriptor: based on fixed sampling pattern References: Binary Robust Invariant Scalable Keypoints, Leutenegger et al., ICCV

116 Descriptor Given set of all sampling-point pairs We define A long range set A short range set Orientation estimation Normalized intensity difference 148

117 Interest Point Descriptors LOCAL BINARY PATTERNS 150

118 Local Binary Patterns Encodes texture Binary pixel test (center pixel vs. neighborhood) 8 bit à value between 0 and 255 (LBP code) Histogram of LBP codes within region of interest Frequency LBP index References: A Comparative Study of Texture Measures with Classification Based on Feature Distributions, Ojala et al, Pattern Recognition,

119 Extensions existing for Multiple scales Rotation invariance LBP Gray scale variance as contrast measures Uniform patterns Arbitrary circular neighborhoods Invariance with respect to monotonic transformations can be achieved 152

120 Interest Point Descriptors SHAPE CONTEXT 153

121 Shape Context Shape based description Descriptor calculated on edge map Local neighborhood description 154

122 Descriptor Edge pixels are assigned to histogram bins for varying distances and angles in log-polar manner (defines pooling) Usually 12 orientations and 5 distance bins 60-dimensional descriptor 155

123 Shape context descriptor 156

124 Learning a Descriptor Training data ( similar (correct) different (incorrect) Training sets incorporate various transformations, e.g.: intensity change affine transforma1on 157

125 Interest Point Descriptors EVALUATION CRITERIONS 158

126 Evaluation Measure matching quality: Precision vs. Recall Depends on IP detector 159

127 Descriptors in VL Feat VLFEAT DESCRIPTORS 160

128 Gradient Magnitude and Orientation LOCAL IMAGE GRADIENTS 161

129 The gradient of an image: The gradient points in the direction of most rapid change in intensity The gradient direction (orientation of edge normal) is given by: The edge strength is given by the gradient magnitude 162

130 The discrete gradient How can we differentiate a digital image f[x,y]? Take discrete derivative (finite difference) (a): Roberts cross operator (b): 3x3 Prewitt operator (c): Sobel operator (d) 4x4 Prewitt operator 163

131 Gradient to Edges Still state-of-the-art: Canny detector 164

132 What causes an edge? Reflectance change: appearance information, texture object boundary Cast shadows Change in surface orientation: shape 165

133 Machine Learning LEARNING GRADIENT DETECTORS 166

134 Goal: Learn from human segmentations what good contours are! Human-marked segment boundaries 167

135 Why learning? 1. Modeling assumptions Minimal 2. Parameters None 3. Multiple sources of information Automatically incorporated 4. Real world conditions Training data References: Learning to Detect Natural Image Boundaries Using Brightness and Texture, Martin et al., NIPS

136 Low-level edges vs. perceived contours image human segmentation gradient magnitudes Training data: Berkeley segmentation database: Source: L. Lazebnik 169

137 Individual Features 1976 CIE L*a*b* colorspace Brightness Gradient BG(x,y,r,θ) Difference of L* distributions Color Gradient CG(x,y,r,θ) Difference of a*b* distributions Texture Gradient TG(x,y,r,θ) Difference of distributions of V1-like filter responses (x,y) r θ All features together define our vector representation x i for each pixel 170

138 Feature comparison Oriented Edges Brightness Gradient Color Gradient Texture Gradient No Boundary Boundary 171

139 Machine Learning for Contour detection Given label data form Berkeley y i Contour pixels of human segmentations Corresponding features x i D-dimensional vectors (per pixel) Supervised Machine Learning problem Regression: y in real numbers (gradient magnitude) Any Regression Method is applicable 172

140 Regression Output densely evaluated on image 173

141 Contour detection ~

142 Contour detection ~

143 Contour detection ~2004 Machine Learning Gap 176

144 Contour detection ~2008 (gray) 177

145 Contour detection ~2008(color) 178

146 Today 179