Local features and image matching. Prof. Xin Yang HUST

Transcription

1 Local features and image matching Prof. Xin Yang HUST

2 Last time RANSAC for robust geometric transformation estimation Translation, Affine, Homography Image warping Given a 2D transformation T and a source image, compute a transformed image of source based on T Image mosaics Given two images, compute the transformation between them and blend them based on image warping

3 Today How to detect which features to match? Computing local invariant features Detection of interest points Harris corner detection Scale invariant blob detection: LoG Description of local patches

4 Kristen Grauman Local features: main components 1) Detection: Identify the interest points 2) Description: Extract vector feature descriptor surrounding each interest point. x (1) [ x,, x (1) 1 1 d ] 3) Matching: Determine correspondence between descriptors in two views x (2) [ x,, x (2) 2 1 d ]

5 Local features: desired properties Repeatability The same feature can be found in several images despite geometric and photometric transformations Saliency Each feature has a distinctive description Compactness and efficiency Many fewer features than image pixels Locality A feature occupies a relatively small area of the image; robust to clutter and occlusion

6 Goal: interest operator repeatability We want to detect (at least some of) the same points in both images No chance to find true matches! Yet we have to be able to run the detection procedure independently per image

7 Goal: descriptor distinctiveness We want to be able to reliably determine which point goes with which? Must provide some invariance to geometric and photometric differences between the two views

8 Local features: main components 1) Detection: Identify the interest points 2) Description: Extract vector feature descriptor surrounding each interest point. 3) Matching: Determine correspondence between descriptors in two views

9 What points would you choose?

10 Corners as distinctive interest points We should easily recognize the point by looking through a small window Shifting a window in any direction should give a large change in intensity flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions Slide credit: Alyosha Efros, Darya Frolova, Denis Simakov

11 y y y x y x x x I I I I I I I I y x w M ), ( x I I x y I I y y I x I I I y x Corners as distinctive interest points 2 x 2 matrix of image derivatives (averaged in neighborhood of a point) Notation:

12 What does this matrix reveal? First, consider an axis-aligned corner:

13 y y x y x x I I I I I I M First, consider an axis-aligned corner: This means dominant gradient directions align with x or y axis Look for locations where both λ s are large If either λ is close to 0, then this is not corner-like What if we have a corner that is not aligned with the image axes? What does this matrix reveal?

14 What does this matrix reveal? Since M is symmetric, we have M X X 2 T Mx i x i i The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window

15 Corner response function edge : 1 >> 2 2 >> 1 corner : 1 and 2 are large, 1 ~ 2 ; flat region 1 and 2 are small;

16 Harris corner detector 1) Compute M matrix for each image window to get their cornerness scores. 2) Find points whose surrounding window gave large corner response (f> threshold) 3) Take the points of local maxima, i.e., perform nonmaximum suppression

17 Kristen Grauman Example of Harris application

18 Kristen Grauman Example of Harris application Compute corner response at every pixel.

19 Kristen Grauman Example of Harris application

20 Harris Detector: Steps

21 Harris Detector: Steps Compute corner response f

22 Harris Detector: Steps Find points with large corner response: f > threshold

23 Harris Detector: Steps Take only the points of local maxima of f

24 Harris Detector: Steps

25 Properties of the Harris corner detector Rotation invariant? Yes M X X 2 T Scale invariant?

26 Properties of the Harris corner detector Rotation invariant? Yes Scale invariant? No All points will be classified as edges Corner!

27 Scale invariant interest points How can we independently select interest points in each image, such that the detections are repeatable across different scales?

28 Automatic scale selection Intuition: Find scale that gives local maxima of some function f in both position and scale f Image 1 f Image 2 s 1 region size s 2 region size

29 What can be the signature function?

30 Recall: Edge detection f Edge d dx g Derivative of Gaussian f d dx g Edge = maximum of derivative Source: S. Seitz

31 Recall: Edge detection f Edge d dx 2 2 g Second derivative of Gaussian (Laplacian) d dx 2 f 2 g Edge = zero crossing of second derivative Source: S. Seitz

32 Edge = ripple From edges to blobs Blob = superposition of two ripples maximum Spatial selection: the magnitude of the Laplacian response will achieve a maximum at the center of the blob, provided the scale of the Laplacian is matched to the scale of the blob Slide credit: Lana Lazebnik

33 Blob detection in 2D Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D 2 g 2 x g 2 2 y g 2

34 filter scales Blob detection in 2D: scale selection g g Laplacian-of-Gaussian = blob detector g 2 2 x y Bastian Leibe img1 img2 img3

35 Blob detection in 2D We define the characteristic scale as the scale that produces peak of Laplacian response characteristic scale Slide credit: Lana Lazebnik

36 Example Original image at ¾ the size Kristen Grauman

37 Original image at ¾ the size Kristen Grauman

38 Kristen Grauman

39 Kristen Grauman

40 Kristen Grauman

41 Kristen Grauman

42 Kristen Grauman

43 Scale invariant interest points Interest points are local maxima in both position and scale. 5 4 scale L xx ( ) L ( ) yy 3 2 Squared filter response maps 1 List of (x, y, σ)

44 Scale-space blob detector: Example Image credit: Lana Lazebnik

45 We can approximate the Laplacian with a difference of Gaussians; more efficient to implement 2 L Gxx x y Gyy x y (,, ) (,, ) (Laplacian) DoG G( x, y, k) G( x, y, ) (Difference of Gaussians) Technical detail

46

47 Local features: main components 1) Detection: Identify the interest points 2) Description:Extract vector feature descriptor surrounding each interest point. 3) Matching: Determine correspondence between descriptors in two views x (1) [ x,, x (1) 1 1 d x ] (2) [ x,, x (2) 2 1 d ]

48 Geometric transformations e.g. scale, translation, rotation

49 Photometric transformations Figure from T. Tuytelaars ECCV 2006 tutorial

50 Raw patches as local descriptors The simplest way to describe the neighborhood around an interest point is to write down the list of intensities to form a feature vector. But this is very sensitive to even small shifts, rotations.

51 SIFT descriptor [Lowe 2004] Use histograms to bin pixels within sub-patches according to their orientation. 0 2 p Why subpatches? Why does SIFT have some illumination invariance?

52 Making descriptor rotation invariant CSE 576: Computer Vision Rotate patch according to its dominant gradient orientation This puts the patches into a canonical orientation. Image from Matthew Brown

53 Steve Seitz SIFT descriptor [Lowe 2004] Extraordinarily robust matching technique Can handle changes in viewpoint Up to about 60 degree out of plane rotation Can handle significant changes in illumination Sometimes even day vs. night (below) Fast and efficient can run in real time Lots of code available

54 Example NASA Mars Rover images

55 Example NASA Mars Rover images with SIFT feature matches Figure by Noah Snavely

56 SIFT properties Invariant to Scale Rotation Partially invariant to Illumination changes Camera viewpoint Occlusion, clutter

57 Local features: main components 1) Detection: Identify the interest points 2) Description:Extract vector feature descriptor surrounding each interest point. 3) Matching: Determine correspondence between descriptors in two views

58 Kristen Grauman Matching local features

59 Matching local features? Image 1 Image 2 To generate candidate matches, find patches that have the most similar appearance (e.g., lowest SSD) Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance) Kristen Grauman

60 Ambiguous matches???? At what SSD value do we have a good match? To add robustness to matching, can consider ratio : distance to best match / distance to second best match Kristen Grauman Image 1 Image 2 If low, first match looks good. If high, could be ambiguous match.

61 Matching SIFT Descriptors Nearest neighbor (Euclidean distance) Threshold ratio of nearest to 2 nd nearest descriptor Lowe IJCV 2004

62 Recap: robust feature-based alignment Source: L. Lazebnik

63 Recap: robust feature-based alignment Extract features Source: L. Lazebnik

64 Recap: robust feature-based alignment Extract features Compute putative matches Source: L. Lazebnik

65 Recap: robust feature-based alignment Extract features Compute putative matches Loop: Hypothesize transformation T (small group of putative matches that are related by T) Source: L. Lazebnik

66 Recap: robust feature-based alignment Extract features Compute putative matches Loop: Hypothesize transformation T (small group of putative matches that are related by T) Verify transformation (search for other matches consistent with T) Source: L. Lazebnik

67 Recap: robust feature-based alignment Extract features Compute putative matches Loop: Hypothesize transformation T (small group of putative matches that are related by T) Verify transformation (search for other matches consistent with T) Source: L. Lazebnik

68 Recent Work for Local Features Bay, H., Ess, A., Tuytelaars, T. and Gool, L.V., SURF: Speeded-Up Robust Features. In Proc. of ECCV 06. Calonder, M., Lepetit, V., Strecha, C., and Fua, P., Brief: Binary Robust Independent Elementary Features. In Proc. of ECCV 10. Rublee, E., Rabaud, V., Konolige, K., and Bradski, G., ORB: an Efficient Alternative to SIFT or SURF. In Proc. of ICCV 11, Barcelona, Spain. Leutenegger, S., Chli, M., Siegwart, R., BRISK: Binary Robust Invariant Scalable Keypoints. In Proc. of CVPR 11. Alahi, A., Ortiz, R., and Vandergheynst, P., FREAK: Fast Retinal Keypoint, In Proc. of CVPR 12. Yang, X. and Cheng, K. T., LDB: An Ultra-Fast Feature for Scalable Augmented Reality on Mobile Device. In Proc. of ISMAR 12. Calonder, M., Lepetit, V., Konolige K., Bowman, J., Mihelich, P., and Fua, P., Compact Signatures for High-Speed Interest Point Description and Matching, In Proc. of ICCV 09. Winder, S., and Brown, M., Learning Local Image Descriptors, In Proc. of CVPR 07. Winder, S., Hua, G., and Brown, M., Picking the Best DAISY, In Proc. of CVPR 09. Calonder, M., Lepetit, V., Konolige K., Bowman, J., Mihelich, P., and Fua, P., Compact Signatures for High-Speed Interest Point Description and Matching, In Proc. of ICCV 09. Rosten, E., Porter, R., and Drummond, T., Faster and Better: A machine learning approach to corner detection. IEEE Trans. PAMI, 32: , 2010.

69 Applications of local invariant features Wide baseline stereo Motion tracking Panoramas Mobile robot navigation 3D reconstruction Recognition

70 Automatic mosaicing

71 Wide baseline stereo [Image from T. Tuytelaars ECCV 2006 tutorial]

72 Recognition of specific objects, scenes Schmid and Mohr 1997 Sivic and Zisserman, 2003 Kristen Grauman Rothganger et al Lowe 2002

73 Summary Interest point detection Harris corner detector Laplacian of Gaussian, automatic scale selection Invariant descriptors Rotation according to dominant gradient direction Histograms for robustness to small shifts and translations (SIFT descriptor)