Lecture 6: Finding Features (part 1/2)

Transcription

1 Lecture 6: Finding Features (part 1/2) Dr. Juan Carlos Niebles Stanford AI Lab Professor Stanford Vision Lab 1

2 What we will learn today? Local invariant features MoOvaOon Requirements, invariances Keypoint localizaoon Harris corner detector Scale invariant region selecoon AutomaOc scale selecoon Difference-of-Gaussian (DoG) detector SIFT: an image region descriptor Next lecture (#7) 2

3 What we will learn today? Local invariant features MoOvaOon Requirements, invariances Keypoint localizaoon Harris corner detector Scale invariant region selecoon AutomaOc scale selecoon Difference-of-Gaussian (DoG) detector Some SIFT: background an image reading: region descriptor Rick Szeliski, Chapter 4.1.1; David Lowe, IJCV

4 Image matching: a challenging problem 4

5 Image matching: a challenging problem by Diva Sian by swashford Slide credit: Steve Seitz 5

6 Harder Case by Diva Sian by scgbt Slide credit: Steve Seitz 6

7 Harder SOll? NASA Mars Rover images Slide credit: Steve Seitz 7

8 Answer Below (Look for Ony colored squares) NASA Mars Rover images with SIFT feature matches (Figure by Noah Snavely) Slide credit: Steve Seitz 8

9 MoOvaOon for using local features Global representaoons have major limitaoons Instead, describe and match only local regions Increased robustness to Occlusions ArOculaOon d d q Intra-category variaoons θ φ θ q φ 9

10 General Approach 1. Find a set of distinctive keypoints A 1 A 2 A 3 B 3 B 1 B 2 2. Define a region around each keypoint 3. Extract and normalize the region content N pixels N pixels f A e.g. color Similarity measure d( f A, fb) < T f B e.g. color 4. Compute a local descriptor from the normalized region 5. Match local descriptors Slide credit: Bastian Leibe 10

11 Common Requirements Problem 1: Detect the same point independently in both images No chance to match! We need a repeatable detector! Slide credit: Darya Frolova, Denis Simakov 11

12 Common Requirements Problem 1: Detect the same point independently in both images Problem 2: For each point correctly recognize the corresponding one? We need a reliable and distinctive descriptor! Slide credit: Darya Frolova, Denis Simakov 12

13 Invariance: Geometric TransformaOons Slide credit: Steve Seitz 13

14 Levels of Geometric Invariance CS131 CS231a Slide credit: Bastian Leibe 14

15 Invariance: Photometric TransformaOons Ofen modeled as a linear transformaoon: Scaling + Offset Slide credit: Tinne Tuytelaars 15

16 Requirements Region extracoon needs to be repeatable and accurate Invariant to translaoon, rotaoon, scale changes Robust or covariant to out-of-plane ( affine) transformaoons Robust to lighong variaoons, noise, blur, quanozaoon Locality: Features are local, therefore robust to occlusion and cluier. QuanOty: We need a sufficient number of regions to cover the object. DisOncOveness: The regions should contain interesong structure. Efficiency: Close to real-ome performance. Slide credit: Bastian Leibe 16

17 Many ExisOng Detectors Available Hessian & Harris [Beaudet 78], [Harris 88] Laplacian, DoG [Lindeberg 98], [Lowe 99] Harris-/Hessian-Laplace [Mikolajczyk & Schmid 01] Harris-/Hessian-Affine [Mikolajczyk & Schmid 04] EBR and IBR [Tuytelaars & Van Gool 04] MSER [Matas 02] Salient Regions [Kadir & Brady 01] Others Those detectors have become a basic building block for many recent applica8ons in Computer Vision. Slide credit: Bastian Leibe 17

18 Keypoint LocalizaOon Goals: Repeatable detecoon Precise localizaoon InteresOng content Look for two-dimensional signal changes Slide credit: Bastian Leibe 18

19 Finding Corners Key property: In the region around a corner, image gradient has two or more dominant direcoons Corners are repeatable and dis8nc8ve C.Harris and M.Stephens. "A Combined Corner and Edge Detector. Proceedings of the 4th Alvey Vision Conference, Slide credit: Svetlana Lazebnik 19

20 Corners as DisOncOve Interest Points Design criteria We should easily recognize the point by looking through a small window (locality) Shifing the window in any direc8on should give a large change in intensity (good localiza8on) flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions 20 Slide credit: Alyosha Efros

21 Harris Detector FormulaOon Change of intensity for the shif [u,v]: E(u,v) = w(x, y) " # I(x + u, y + v) I(x, y) $ % x,y 2 Window function Shifted intensity Intensity Window function w(x,y) = 1 in window, 0 outside or Gaussian Slide credit: Rick Szeliski 21

22 Harris Detector FormulaOon Slide credit: Rick Szeliski This measure of change can be approximated by: where M is a 2 M M E ( u, v) [ u v] M u v 2 matrix computed from image derivaoves: 2 Ix IxI y = w(, x y) 2 xy, II x y Iy Sum over image region the area we are checking for corner Gradient with respect to x, times gradient with respect to y 22

23 Harris Detector FormulaOon Slide credit: Rick Szeliski Image I where M is a 2 M M I x I y I x I y 2 matrix computed from image derivaoves: 2 Ix IxI y = w(, x y) 2 xy, II x y Iy Sum over image region the area we are checking for corner Gradient with respect to x, times gradient with respect to y 23

24 What Does This Matrix Reveal? First, let s consider an axis-aligned corner: M = I I x 2 x I y I I x y 2 I y = λ1 0 0 λ 2 This means: Dominant gradient direcoons align with x or y axis If either λ is close to 0, then this is not a corner, so look for locaoons where both are large. What if we have a corner that is not aligned with the image axes? Slide credit: David Jacobs 24

25 What Does This Matrix Reveal? First, let s consider an axis-aligned corner: M = I I x 2 x I y I I x y 2 I y = λ1 0 0 λ 2 This means: Dominant gradient direcoons align with x or y axis If either λ is close to 0, then this is not a corner, so look for locaoons where both are large. What if we have a corner that is not aligned with the image axes? Slide credit: David Jacobs 25

26 General Case Since M is symmetric, we have (Eigenvalue decomposition) λ = R R 0 λ2 We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientaoon determined by R Direction of the fastest change M ( max )-1/2 ( min )-1/2 Direction of the slowest change adapted from Darya Frolova, Denis Simakov 26

27 InterpreOng the Eigenvalues ClassificaOon of image points using eigenvalues of M: 2 Edge 2 >> 1 Corner 1 and 2 are large, 1 ~ 2 ; E increases in all direcoons Slide credit: Kristen Grauman 1 and 2 are small; E is almost constant in all direcoons Flat region Edge 1 >>

28 Corner Response FuncOon θ = det(m ) α trace(m ) 2 = λ 1 λ 2 α(λ 1 + λ 2 ) 2 2 Edge θ < 0 Corner θ > 0 Slide credit: Kristen Grauman Fast approximaoon Avoid compuong the eigenvalues α: constant (0.04 to 0.06) Flat region Edge θ <

29 Window FuncOon w(x,y) M OpOon 1: uniform window Sum over square window Problem: not rotaoon invariant 2 Ix IxI y = w(, x y) 2 xy, II x y Iy 2 Ix IxI y M = 2 xy, II x y Iy 1 in window, 0 outside OpOon 2: Smooth with Gaussian Gaussian already performs weighted sum 2 Ix IxIy M = g( σ ) 2 II x y Iy Result is rotaoon invariant Gaussian Slide credit: Bastian Leibe 29

30 Summary: Harris Detector [Harris88] Compute second moment matrix (autocorrelaoon matrix) 2 Ix( σd) IxIy( σ ) D M( σi, σd) = g( σi) 2 II x y( σd) Iy( σd) 2. Square of derivatives 1. Image derivatives I x I y I x 2 I y 2 I x I y Slide credit: Krystian Mikolajczyk 3. Gaussian filter g(s I ) 4. Cornerness function two strong eigenvalues θ = det[m (σ I,σ D )] α[trace(m (σ I,σ D ))] 2 = gi ( ) gi ( ) [ gii ( )] α[ gi ( ) + gi ( )] x y x y x y 5. Perform non-maximum suppression g(i x2 ) g(i y2 ) g(i x I y ) 30 R

31 Harris Detector: Workflow Slide adapted from Darya Frolova, Denis Simakov 31

32 Harris Detector: Workflow - computer corner responses θ Slide adapted from Darya Frolova, Denis Simakov 32

33 Harris Detector: Workflow - Take only the local maxima of θ, where θ>threshold Slide adapted from Darya Frolova, Denis Simakov 33

34 Harris Detector: Workflow - ResulOng Harris points Slide adapted from Darya Frolova, Denis Simakov 34

35 Harris Detector Responses [Harris88] Effect: A very precise corner detector. Slide credit: Krystian Mikolajczyk 35

36 Harris Detector Responses [Harris88] Slide credit: Krystian Mikolajczyk 36

37 Harris Detector Responses [Harris88] Results are well suited for finding stereo correspondences Slide credit: Kristen Grauman 37

38 Harris Detector: ProperOes TranslaOon invariance? Slide credit: Kristen Grauman 38

39 Harris Detector: ProperOes TranslaOon invariance RotaOon invariance? Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response θ is invariant to image rotation Slide credit: Kristen Grauman 39

40 Harris Detector: ProperOes TranslaOon invariance RotaOon invariance Scale invariance? Corner Not invariant to image scale! All points will be classified as edges! Slide credit: Kristen Grauman 40

41 What we have learned today? Local invariant features MoOvaOon Requirements, invariances Keypoint localizaoon Harris corner detector Scale invariant region selecoon AutomaOc scale selecoon Difference-of-Gaussian (DoG) detector SIFT: an image region descriptor Some background reading: Rick Szeliski, Chapter ; David Lowe, IJCV Next lecture (#7)

42 42

43 ApplicaOon: Image SOtching Slide credit: Darya Frolova, Denis Simakov 43

44 ApplicaOon: Image SOtching Procedure: Detect feature points in both images Find corresponding pairs Use these pairs to align the images Slide credit: Darya Frolova, Denis Simakov 44

45 ApplicaOon: Image SOtching Procedure: Detect feature points in both images Find corresponding pairs Use these pairs to align the images Slide credit: Darya Frolova, Denis Simakov 45

46 ApplicaOon: Image SOtching Procedure: Detect feature points in both images Find corresponding pairs Use these pairs to align the images Slide credit: Darya Frolova, Denis Simakov 46