Local features and image matching May 8 th, 2018

Transcription

1 Local features and image matcing May 8 t, 2018 Yong Jae Lee UC Davis Last time RANSAC for robust fitting Lines, translation Image mosaics Fitting a 2D transformation Homograpy 2 Today Mosaics recap: How to warp one image to te oter, given H? How to detect wic features to matc? 3 1

2 Motivation for feature-based alignment: Image mosaics Image from ttp://grapics.cs.cmu.edu/courses/15-463/2010_fall/ 4 Projective Transformations x' a y' d w' g b e Projective transformations: Affine transformations, and Projective warps c x f y i w Parallel lines do not necessarily remain parallel 5 How to stitc togeter a panorama (a.k.a. mosaic)? Basic Procedure Take a sequence of images from te same position Rotate te camera about its optical center Compute transformation between second image and first Transform te second image to overlap wit te first Blend te two togeter to create a mosaic (If tere are more images, repeat) 6 Source: Steve Seitz 2

3 Mosaics... image from S. Seitz Obtain a wider angle view by combining multiple images. 7 How to stitc togeter a panorama (a.k.a. mosaic)? Basic Procedure Take a sequence of images from te same position Rotate te camera about its optical center Compute transformation between second image and first Transform te second image to overlap wit te first Blend te two togeter to create a mosaic (If tere are more images, repeat) but wait, wy sould tis work at all? Wat about te 3D geometry of te scene? Wy aren t we using it? 8 Source: Steve Seitz Image reprojection Basic question How to relate two images from te same camera center? ow to map a pixel from PP1 to PP2 Answer Cast a ray troug eac pixel in PP1 Draw te pixel were tat ray intersects PP2 Observation: Rater tan tinking of tis as a 3D reprojection, tink of it as a 2D image warp from one image to anoter. PP1 PP2 9 Source: Alyosa Efros 3

4 Image reprojection: Homograpy A projective transform is a mapping between any two PPs wit te same center of projection rectangle sould map to arbitrary quadrilateral parallel lines aren t preserved but must preserve straigt lines called Homograpy PP2 wx' * * wy' * * w * * p H * x * y * 1 p PP1 10 Source: Alyosa Efros Homograpy x1, y 1 x1, y 1 x2, y 2 x 2, y 2 x, x, n y n n y n To compute te omograpy given pairs of corresponding points in te images, we need to set up an equation were te parameters of H are te unknowns 11 Solving for omograpies Defined up to a scale factor. Constrain Frobenius norm of H to be 1. Problem to be solved: p = Hp wx' wy' w x 12 y 22 1 min A b 2 s.t. 1 were vector of unknowns = [ 00, 01, 02, 10, 11, 12, 20, 21, 22 ] T 2 Adapted from Devi Parik 12 4

5 Solving for omograpies wx i' wy i' w xi 12 yi 22 1 Tere are 9 variables 00,, 22. Are tere 9 degrees of freedom? No. We can multiply all ij by nonzero scalar k witout canging te equations: 13 Enforcing 8 DOF Impose unit vector constraint Subject to: 14 Solving for omograpies wx i' wy i' w xi 12 yi

6 Solving for omograpies A 0 2n 9 9 2n Defines a least squares problem: Since is only defined up to scale, solve for unit vector ĥ Solution: ĥ = eigenvector of A T A wit smallest eigenvalue Works wit 4 or more points ( i. e., 2 1) 16 Projective: # correspondences? T(x,y) y y x x? How many correspondences needed for projective? Source: Alyosa Efros 17 x, Homograpy y wx wy w, w x, y To apply a given omograpy H Compute p = Hp (regular matrix multiply) Convert p from omogeneous to image coordinates wx' * wy' * w * p * * x * * y * * 1 Slide credit: Kristen Grauman 18 H p 6

7 Image warping T(x,y) y y x f(x,y) x g(x,y ) Given a coordinate transform and a source image f(x,y), ow do we compute a transformed image g(x,y ) = f(t(x,y))? Slide from Alyosa Efros, CMU 19 Forward warping T(x,y) y y x f(x,y) x g(x,y ) Send eac pixel f(x,y) to its corresponding location (x,y ) = T(x,y) in te second image Q: wat if pixel lands between two pixels? Slide from Alyosa Efros, CMU 20 Forward warping T(x,y) y y x f(x,y) x g(x,y ) Send eac pixel f(x,y) to its corresponding location (x,y ) = T(x,y) in te second image Q: wat if pixel lands between two pixels? A: distribute color among neigboring pixels (x,y ) Known as splatting Slide from Alyosa Efros, CMU 21 7

8 Inverse warping y T -1 (x,y) y x f(x,y) x g(x,y ) Get eac pixel g(x,y ) from its corresponding location (x,y) =T -1 (x,y ) in te first image Q: wat if pixel comes from between two pixels? Slide from Alyosa Efros, CMU 22 Inverse warping y T -1 (x,y) y x f(x,y) x g(x,y ) Get eac pixel g(x,y ) from its corresponding location (x,y) =T -1 (x,y ) in te first image Q: wat if pixel comes from between two pixels? A: Interpolate color value from neigbors nearest neigbor, bilinear Slide from Alyosa Efros, CMU 23 Bilinear interpolation Sampling at f(x,y): Slide from Alyosa Efros, CMU 24 8

9 Image warping wit omograpies image plane in front black area were no pixel maps to image plane below 25 Source: Steve Seitz Image rectification p p 26 Analysing patterns and sapes Wat is te sape of te b/w floor pattern? Te floor (enlarged) Slide from Antonio Criminisi Automatically 27 rectified floor 9

10 Analysing patterns and sapes Automatic rectification From Martin Kemp Te Science of Art (manual reconstruction) Slide from Antonio Criminisi 28 Canging camera center Does it still work? syntetic PP PP1 PP2 29 Source: Alyosa Efros Recall: same camera center real camera syntetic camera Can generate syntetic camera view as long as it as te same center of projection. 30 Source: Alyosa Efros 10

11 Or: Planar scene (or far away) PP3 PP1 PP2 PP3 is a projection plane of bot centers of projection, so we are OK! Tis is ow big aerial potograps are made 31 Source: Alyosa Efros 32 RANSAC for estimating omograpy RANSAC loop: 1. Select four feature pairs (at random) 2. Compute omograpy H (exact) 3. Compute inliers were SSD(p i, Hp i )< ε 4. Keep largest set of inliers 5. Re-compute least-squares H estimate on all of te inliers 33 Slide credit: Steve Seitz 11

12 Robust feature-based alignment 34 Source: L. Lazebnik Robust feature-based alignment Extract features 35 Source: L. Lazebnik Robust feature-based alignment Extract features Compute putative matces 36 Source: L. Lazebnik 12

13 Robust feature-based alignment Extract features Compute putative matces Loop: Hypotesize transformation T (small group of putative matces tat are related by T) 37 Source: L. Lazebnik Robust feature-based alignment Extract features Compute putative matces Loop: Hypotesize transformation T (small group of putative matces tat are related by T) Verify transformation (searc for oter matces consistent wit T) 38 Source: L. Lazebnik Robust feature-based alignment Extract features Compute putative matces Loop: Hypotesize transformation T (small group of putative matces tat are related by T) Verify transformation (searc for oter matces consistent wit T) 39 Source: L. Lazebnik 13

14 Summary: alignment & warping Write 2d transformations as matrix-vector multiplication (including translation wen we use omogeneous coordinates) Fitting transformations: solve for unknown parameters given corresponding points from two views (affine, projective (omograpy)). Perform image warping (inverse) Mosaics: uses omograpy and image warping to merge views taken from same center of projection. 40 Creating and Exploring a Large Potorealistic Virtual Space Josef Sivic, Biliana Kaneva, Antonio Torralba, Sai Avidan and William T. Freeman, Internet Vision Worksop, CVPR ttp:// 41 Creating and Exploring a Large Potorealistic Virtual Space Current view, and desired view in green Syntesized view from new camera Induced camera motion 42 14

15 Today Mosaics recap: How to warp one image to te oter, given H? How to detect wic features to matc? 43 Detecting local invariant features Detection of interest points Harris corner detection (Scale invariant blob detection: LoG) (Next time: description of local patces) 44 Local features: main components 1) Detection: Identify te interest points 2) Description: Extract vector feature descriptor surrounding eac interest point. (1) x [ x,, x (1) 1 1 d ] 3) Matcing: Determine correspondence between descriptors in two views (2) x [ x,, x (2) 2 1 d 45 ] Kristen Grauman 15

16 Local features: desired properties Repeatability Te same feature can be found in several images despite geometric and potometric transformations Saliency Eac feature as a distinctive description Compactness and efficiency Many fewer features tan image pixels Locality A feature occupies a relatively small area of te image; robust to clutter and occlusion 46 Applications Local features ave be used for: Image alignment 3D reconstruction Motion tracking Robot navigation Indexing and database retrieval Object recognition Lana Lazebnik 47 A ard feature matcing problem NASA Mars Rover images 48 16

17 Answer below (look for tiny colored squares ) NASA Mars Rover images wit SIFT feature matces Figure by Noa Snavely 49 Goal: interest operator repeatability We want to detect (at least some of) te same points in bot images. No cance to find true matces! Yet we ave to be able to run te detection procedure independently per image. 50 Goal: descriptor distinctiveness We want to be able to reliably determine wic point goes wit wic.? Must provide some invariance to geometric and potometric differences between te two views

18 Local features: main components 1) Detection: Identify te interest points 2) Description:Extract vector feature descriptor surrounding eac interest point. 3) Matcing: Determine correspondence between descriptors in two views 52 Wat points would you coose (for repeatability, distinctiveness)? 53 Corners as distinctive interest points We sould easily recognize te point by looking troug a small window Sifting a window in any direction sould give a large cange in intensity flat region: no cange in all directions Slide credit: Alyosa Efros, Darya Frolova, Denis Simakov edge : no cange along te edge direction corner : significant cange in all directions 54 18

19 Corners as distinctive interest points M I xi x I x I y I x I y I y I y 2 x 2 matrix of image derivatives (averaged in neigborood of a point). Notation: I x I x I y I y I I x y I I x y 55 Wat does tis matrix reveal? First, consider an axis-aligned corner: 56 Wat does tis matrix reveal? First, consider an axis-aligned corner: M 2 I x I xi y I xi y 1 2 I y Tis means dominant gradient directions align wit x or y axis Look for locations were bot λ s are large. If eiter λ is close to 0, ten tis is not corner-like. Wat if we ave a corner tat is not aligned wit te 57 image axes? 19

20 Wat does tis matrix reveal? Since M is symmetric, we ave (Eigenvalue decomposition) 1 M X 0 Mx x i i i 0 X 2 T Te eigenvalues of M reveal te amount of intensity cange in te two principal ortogonal gradient directions in te window. 58 Corner response function edge : 1 >> 2 2 >> 1 corner : 1 and 2 are large, 1 ~ 2 ; flat region 1 and 2 are small; 59 Harris corner detector 1) Compute M matrix for eac image window to get teir cornerness scores. 2) Find points wose surrounding window gave large corner response (f > tresold) 3) Take te points of local maxima, i.e., perform non-maximum suppression 60 20

21 Example of Harris application 61 Kristen Grauman Example of Harris application Compute corner response at every pixel. 62 Kristen Grauman Example of Harris application 63 Kristen Grauman 21

22 Properties of te Harris corner detector Rotation invariant? Yes 64 Properties of te Harris corner detector Rotation invariant? Yes Translation invariant? Yes 65 Properties of te Harris corner detector Rotation invariant? Yes Translation invariant? Yes Scale invariant? No All points will be classified as edges Corner! 66 22

23 Summary Image warping to create mosaic, given omograpy Interest point detection Harris corner detector Next time: Laplacian of Gaussian, automatic scale selection 67 23