Computer Vision Feature Extraction

Transcription

1 Feature Extraction

2 PART 1: OUVERTURE

3 Feature matching Goal : efficient i matching for registration, ti correspondences for 3D, tracking, recognition

4 Feature matching

5 Feature matching

6 Feature matching Large scale change, heavy occlusion

7 Feature matching Deformation, illumination change, occlusion

8 Feature matching Large scale change, perspective deformation, extensive clutter

9 Feature matching Extensive clutter, scale, occlusion, blur

10 Feature matching

11 Matching: a challenging task features to deal with large variations in Viewpoint

12 Matching: a challenging task features to deal with large variations in Viewpoint Illumination

13 Matching: a challenging task features to deal with large variations in Viewpoint Illumination Background

14 Matching: a challenging task features to deal with large variations in Viewpoint Illumination Background And with occlusions

15 Features à la carte considerations : 1. Completeness 2. Robustness of extraction 3. Ease of extraction 4. Global vs. local

16 THUS:

17 THUS: GO PATCHY!

18 PART 2: `INTEREST POINTS

19 Additional requirement: Discriminant + good localization

20 Additional requirement: A feature should have something typical and well locallised about the pattern within its local patch Shifting the patch a bit should make a big difference in terms of the underlying pattern

21 Uniqueness of a patch consider the pixel pattern within the green patches: Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

22 Uniqueness of a patch How do the patterns change upon a shift? flat region: edge : corner : no change in all no change along significant change directions the edge direction in all directions Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

23 Uniqueness of a patch Consider shifting the patch or `window W by (u,v) W how do the pixels in W change? compare each pixel before and after by summing up the squared differences (SSD) this defines an SSD error of E(u,v): Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

24 Uniqueness of a patch Taylor Series expansion of I: If the motion (u,v) is small, then first order approx is goo Plugging this into the formula at the top Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

25 Uniqueness of a patch W Then, with Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

26 Uniqueness of a patch This can be rewritten further as: Which directions will result in the largest and smallest E values? We can find these directions by looking at the eigenvectors of H Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

27 The eigenvectors of a matrix A are the vectors x that satisfy: if The scalar is the eigenvalue corresponding to x The eigenvalues are found by solving: In our case, A = H is a 2x2 2matrix, so we have The solution:

28 Uniqueness of a patch This can be rewritten further as: Eigenvalues and eigenvectors of H Define shifts with the smallest and largest change (E value) x + = direction of largest increase in E. λ+ = amount of increase in direction x + x - = direction of smallest increase in E. λ - = amount of increase in direction x + Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

29 Uniqueness of a patch Want E(u,v) to be large for small shifts in all directions the minimum of E(u,v) should be large, over all unit vectors [u v] this minimum is given by the smaller eigenvalue ( λ-) of H Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

30 Uniqueness of a patch Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

31 Uniqueness of a patch IN SUMMARY: Compute the gradient at each point in the image Create the H matrix from the entries in the gradient Compute the eigenvalues. Find points with large response ( - > threshold) Choose those points where - is a local maximum as features Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

32 The Harris corner detector λ - is a variant of the Harris operator for feature detection Determinant - k (Trace) 2 The trace is the sum of the diagonals, i.e., trace(h) = h 11 + h 22 Det(H) = product of eigenvalues, Trace(H) = sum Very similar to λ - but less expensive (no square root) Called the Harris Corner Detector or Harris Operator Lots of other detectors, this is one of the most popular Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

33 The Harris corner detector Slide adapted from Darya Frolova, Denis Simakov, Deva Ramanan

34 The Harris corner detector 2 views of an object are the corners stable?

35 The Harris corner detector

36 The Harris corner detector

37 The Harris corner detector

38 The Harris corner detector

39 Interest points Corners are the most prominent example of so-called `Interest Points, i.e. points that can be well localised in different views of a scene `Blobs are another, as we will see but also Blobs are another, as we will see but also a blob is a region with intensity changes in multiple directions

40 PART 3: A TALE OF INVARIANCE

41 Need for a descriptor: There are many corners coming out of our DETECTOR, but they cannot still be told apart We need to describe their surrounding patch in a way that we can discriminate between then, i.e. we need to build a feature vector for the patch, a so-called DESCRIPTOR During a MATCHING step the descriptors can During a MATCHING step, the descriptors can then be compared

42 Additional requirement on the descriptor: Invariance under geom./phot. change

43 Geometric Invariance: A local patch is small, hence probably rather planar But how do planar patches deform when looking at their image projection?

44 Patterns to be compared

45 Projection : a camera model Reversed center-image pinhole model :

46 Projection : equations x = f X Z y = f Y Z an approximation... special cases : 1. Z constant, or 2. object small compared to its distance to the camera x = kx y = ky = pseudo-perspective

47 Deformations under projection Viewed from abitrary directions, the contour projections will be determined suppose the contours are planar we study 3 cases : 1. viewed from a perpendicular direction 2. viewed from any direction but at sufficient distance to use pseudo - perspective p 3. viewed from any direction and from any distance

48 Deformations under projection In particular, how do different views of the same planar contour differ? we study 3 cases : 1. viewed from a perpendicular direction 2. viewed from any direction but at sufficient distance to use pseudo - perspective p 3. viewed from any direction and from any distance

49 Deformations : case 1 X = Y = cos sin θ X θ X + sin cos θ Y θ Y + t + t Z = Z + ( X ( t ), Y ( t ), Z ( t ) ) ( x ( t ), y ( t ) ) ( X () t, Y () t, Z () t ) ( x () t, y () t ), kx, 1 2 t 3 x = y = ky, x ' = k X, y = k Y, x k cosθ sinθ x t1 = + k θ y k sinθ cos θ y t 2 Image projection undergoes a similarity transformation Euclidean motions if k = k (i.e. Z = Z )

50 Summary case 1 Case of fronto-parallel viewing : SIMILARITY + = 1 sin cos s t x x θ θ + = 2 cos sin s t y y θ θ

51 Deformations : case 2 1 X r11 r12 r13 X t = + Y r21 r22 r23 Y t2 Z r31 r32 r33 Z t3 x x = = kx, y = ky, k X, y = k Y, we easily derive k k x = r x + r12 y + k r13z + k k k 11 t1 and should eliminate Z

52 Deformations : case 2 cont d We use the planarity restriction : ax Hence a b + by + cz + d = x + y + cz + d k k Z = a kc x b kc y d c = 0 and thus k k x = r 1 k x x + ar 11 a+ 12 k r Z + k t x b 11 = 12 y k y a21 a22 y b 2 we find a 2D affine transformation

53 Summary case 2 Viewing from a distance : AFFINITY + = b x a a x + = b y a a y

54 Deformations : case 3 3D Euclidean motion, but perspective projecton x = = = f f f X Z r11x r X r r x = f X Y, y = f, Z Z x = f X Y y = f. Z Z Y Y + + r r Z Z + t + t fx fy fz r11 + r12 + r13 + t Z Z Z fx fy fz r31 + r32 + r33 + t Z Z Z f Z f Z

55 Deformations : case 3 Thus, x = f r11 x + r12 y + r13 f + r x + r y + r f ( ft ) 1 / Z ( ft )/ Z 3 y = f r r x x + + r r y y + + r r f f + + ( ft ) 2 / Z ( ft )/ Z 3

56 Deformations : case 3 cont d Using the planarity restriction :. cf by ax df Z + + = f t b t t f c d t r y b d t r x a d t r f x + + = f c d t r y b d t r x a d t r f c t r y b t r x a t r f c d t r y b d t r x a d t r f c d r y b d r x a d r f y = d d d p y p x p p y p x p x = p y p x p p y p x p y = p y p x p p y p x p + + we find a 2D projective transf. = homography

57 Summary case 3 General viewing conditions: PROJECTIVITY PROJECTIVITY p y p x p p y p x p x = p y p x p p y p x p y =

58 Invariance and groups Invariance w.r.t. group actions T 2 T 1 T 1 T 2 image 1 image 2 image 3 I 1 = I 2 = I 3 Invariance under transformations implies invariance under the smallest encompassing group

59 Remarks There are more groups, but the ones described Seem the most relevant for us

60 Remarks Complexity of the groups goes up with the generality of the viewing conditions, and so does the complexity of the group s invariantsi Fewer invariants are found going from left to right Similarities affinities projectivities Invar. Proj. invar. Aff. invar. Sim.

61 Remarks Many types of invariants can be generated, e.g. for points, for lines, for moments, etc. Examples for points Similarities: X X X X Affinities: X X X X X X X X X 1 X 3 Projectivities: X X X X X X X X X X X X X X X X X X X X Projectivities: X X X X X X X X X X X X

62 Thus: A our local l patches are by definition iti small, we expect that invariance under affine or similarity il it transformations ti will do

63 photometric changes

64 photometric changes A reasonable model lis given by R' s R 0 0 = G' 0 sg 0 B ' 0 0 s B R G + B o o o R G B linear part caters for changes of color and/or intensity of the illumination; the 3 scale factors are the same if the illumination changes intensity only the non-linear part caters for specularities

65 Thus: As our patches cover part of a surface containing corresponding surface texture, we will also have to be invariant i under photometric changes

66 photometric changes Contrasts (intensity differences) let the non-linear part cancel; gradients are good! Moreover the orientation of gradients in the color bands is invariant under their linear changes, as is the intensity gradient in case the scale factors are identical; this is indeed relevant if the illumination changes its intensity, but not its color

67 PART 4: EXAMPLES

68 Our goal D fi d i t t i t i Define good interest points, i.e. DETECTOR + DESCRIPTOR

69 Our goal We have glossed over an important t issue. The shape of the patch should change with the viewpoint i

70 The need for variable patch shape The important thing is to achieve such change in patch shape without having to compare the images, i.e. this should happen on the basis of information in one image only!

71 The need for variable patch shape A test case we will use Note the global perspective/projective distortion!

72 Example 1: edge corners + affine moments

73 Example 1: edge corners + affine moments 1. Harris corner detection

74 Example 1: edge corners + affine moments 2. Canny edge detection

75 Example 1: edge corners + affine moments 3. Evaluation relative affine invariant parameter along two edges l = abs( p ( s ) p p ( s ) ) ds i (1) i i i i i

76 Example 1: edge corners + affine moments 4. Construct 1-dimensional family of parallelogram shaped regions

77 Example 1: edge corners + affine moments 5. Select parallelograms based on invariant extrema of function For instance: extrema of average value of a color band within the patch f Ω(l)

78 Example 1: edge corners + affine moments 5. Select parallelograms based on local extrema of invariant function

79 Example 1: edge corners + affine moments

80 Example 1: edge corners + affine moments \6. Describe the pattern within the parallelogram with affine invariant moments Geometric/photometric t t i moment ti invariants i based on generalised colour moments: M = a, b, c p, q x p y q r a ( ) b( ) c x, y g x, y b ( x, y) dxdy M abc pq are not invariant themselves, need to be combined

81 Example 1: edge corners + affine moments \6. Describe the pattern within the parallelogram with affine invariant moments Example among several moment invariants:

82 Example 1: edge corners + affine moments AR application where an interest point s patch is filled with the texture `BIWI. The stability shows how well the patch adapts to the viewpoint

83 Ex. 2: intensity extrema + affine moments 1. Search intensity extrema 2. Observe intensity profile along rays 3. Search maximum of invariant i function f(t) along each ray 4. Connect local maxima 5. Fit ellipse 6. Double ellipse size 7. Describe elliptical patch with moment invariants f ( t) = max( abs( I 0 abs( I 0 t I) I) dt, d )

84 Ex. 2: intensity extrema + affine moments

85 Ex. 2: intensity extrema + affine moments

86 Remark In practice different types of interest points Are often combined Wide baseline stereo matching example. based on ex.1 and ex.2 interest points

87 MSER MSER = Maximally Stable Extremal Regions Similar to the Intensity-Based Regions we saw already Came later, but is more often used Start with intensity extremum Then move intensity threshold away from its value and watch the super/sub-threshold region grow Take regions at thresholds where the growth is slowest

88 MSER

89 MSER

90 MSER

91 MSER

92 MSER

93 MSER

94 MSER

95 MSER

96 MSER

97 MSER

98 MSER

99 MSER

100 MSER

101 MSER

102 MSER

103 MSER

104 MSER Extremal region: region such that Order regions Q... Q 1... Qi Qi+ 1 Maximally Stable Extremal Region: local minimum of q / Q ( i) = Qi +Δ \ Qi Δ i n

105 MSER

106 SIFT SIFT = Scale-Invariant Feature Transform SIFT, developed by David Lowe (Un. British Columbia, Canada), is a carefully crafted interest point detector + descriptor, based on intensity gradients (cf. our comment on photometric invariance) and invariant under similarities, not affine like so far Our summary is a simplified account!

107 SIFT Descriptor is based on blob detection, at several scales, i.e. local extrema of the Laplacian-of-Gaussian, or LoG σ 5 σ 4 L ( σ ) + L ( σ ) σ 3 xx yy σ 2 σ list of (x, y, σ)

108 SIFT

109 SIFT Dominant orientation selection Compute image gradients Build orientation histogram Find maximum 0 2π

110 SIFT Dominant orientation selection Compute image gradients Build orientation histogram Find maximum 0 2π

111 SIFT Computer Thresholded image gradients are sampled over a grid Create array of orientation histograms within blocks 8 orientations x 4x4 histogram array = 128 dimensions Apply weighting with a Gaussian located at the center Normalized to unit vector

112 SIFT Carefully crafted e.g. why 4 x 4 x 8?

113 SIFT Sensitivity to affine changes quite good!!!

114 SURF SURF = Speeded Up Robust Features Given the advantage of simplification, SURF was introduced as a faster alternative on a GPU, SURF can be extracted for video size 100Hz with ease

115 SURF detector t SURF A Hessian-based detector: blobs are detected not by the Laplacian, but tthe determinant tof fthe Hessian matrix : [ L L xy L xy] yy] H =[ L xx Because of less response on edges, better scale selection L yy L yy LREC 2004, 26 May 2004, Lisbon 115 L L L xx yy xy L xy

116 SURF detector SURF Approximate Gaussian derivatives with block filters: L yy L yy L xy LREC 2004, 26 May 2004, Lisbon 116 L L L xx yy xy

117 SURF detector SURF Scale analysis with constant t image size, but variable filter sizes Sca le Sc cale LREC 2004, 26 May 2004, Lisbon 117

118 SURF Fast due to calculations l with Integral images Viola & Jones S S=A-B+D-C LREC 2004, 26 May 2004, Lisbon 118

119 SURF detector results SURF LREC 2004, 26 May 2004, Lisbon 119

120 SURF detector SURF Computation time: +/- 10 times faster than DoG (~100msec)

121 SURF SURF descriptor Orientation assignment: summed x and y responses in wedge LREC 2004, 26 May 2004, Lisbon 121

122 SURF descriptor SURF Distribution-based based descriptor LREC 2004, 26 May 2004, Lisbon 122

123 SURF SURF descriptor dydy dx LREC 2004, 26 May 2004, Lisbon 123

124 SURF descriptor SURF instead of gradient histograms, summed responses of the Haar masks: dx, dx, dy, dy Extended descriptor dx, dx, dy, dy Dy>0 Dy>0 Dx>0 Dx>0 Rotation invariance is optional (U-SURF) LREC 2004, 26 May 2004, Lisbon 124

125 SURF descriptor SURF role of additional, absolute values:

126 SURF Application: robot navigation

127 SURF Robot equipped with omnidirection camera

128 SURF

129 SURF Robot automatically builds a topological map from SURF features

130 SURF Robot automatically builds a topological map, incl. revisited i places as link kintersectionsi Robot can localize itself in the map and navigate to a goal position go oal st tart

131 SURF goal start Visual targets: Navigation becomes an issue Navigation becomes an issue of going to positions with an image that matches that taken at the next target position during teaching

132 SURF Landmark recognition available as kooaba app for the iphone, Android, and Symbian platforms Also underpins kooaba Shooting Star personal photo organizer,

133 notes on matching Interest points are matched on the basis of their descriptors E.g. nearest neighbour, based on some distance like Euclidean or Mahalanobis; good to compare against 2 nd nearest neighbour: OK ifdifference is big; or fuzzy matching w. multiple neighbours Speed-ups by using lower-din. descriptor space (PCA) or through some coarse-to-fine scheme (very fast schemes exist to date!) Matching of individual points typically followed by some consistency check, e.g. epipolar geometry, homograpy, or topological

134 PART 5: EXTENSIONS

135 Examples: extensions of SURF to 3D - To show that there is more than 2D images to think about T i f l f li ti hi h l i t i - To give a feel for applications, which also exist in the 2D (image) case, but will be discussed in more detail later

136 Video: spatio-temporal features Video as a stack of frames: a volume E.g use of 3D version of SURF Different scales for spatial and temporal dimensions (ie. look at different frequencies) but aligned with the image and time axes

137 Video: spatio-temporal features Can be used for several apps: Near-duplicated video copy detection Action recognition o in video Action localisation in video Video synchronisation

138 Video: spatio-temporal features Copy detection ti Link news stories together via reused shots Detect copyright violations Not a trivial i problem Chirons (added labels) Cropping Scaling Changes in color

139 Video: spatio-temporal features Copy detection Look for matching spatio-temporal SURF

140 Video: spatio-temporal features Copy detection ti in a mash-up

141 Video: spatio-temporal features Exemplar-based action recognition i & localisation, with `Bag-of-Words Here we trained the system to detect drinking Training done on several movies Tested on 2 short movies

142 Video: spatio-temporal features Action recognition With so-called Bag-of-Words technique

143 Video: spatio-temporal features Exemplar-based action recognition i & localisation The top 15 detected Drinking actions Picking up jar looks like drinking Smoke floating upwards confuses system

144 3D shape as the set of features 3D SURF, now with fully free orientation in 3D 14

145 3D model description Can be used for several apps: Find other models of same objects Categorize e model as belonging g to a shape class Find other models of same class

146 3D shape as the set of features 3D SURF, now with fully free orientation in 3D 14

147 Similarity between 3D shapes As in 2D, descriptors can be directly compared, or first be clustered into `visual words Features belonging to same visual word Correspondences between same shapes Correspondences between shapes of same class 14

148 3D shape retrieval Given the query 3D shape, we want to find similar (relevant) shapes in large collection of 3D shapes (BoW). Query Retrieved similar shapes 14

149 3D shape categorisation We want to recognize the class (cat, woman, bike..) of the given 3D shape, e.g using ISM (Implicit Shape Model) Recognize the class Model Create recognition model elephant Query horse person Tiger Labelled database of shapes 14

150 3D shape categorization Examples of recognized classes based on 3D obtained from structure-from-motion (i.e. from images) bk bike person plant 15