Image Alignment CSC 767

Transcription

1 Image Algnment CSC 767

2 Image algnment Image from

3 Image algnment: Applcatons Panorama sttchng

4 Image algnment: Applcatons Recognton of object nstances

5 Image algnment: Challenges Small degree of overlap Intensty changes Occluson, clutter

6 Feature-based algnment Search sets of feature matches that agree n terms of: a) Local appearance b) Geometrc confguraton?

7 Feature-based algnment: Overvew Algnment as fttng Affne transformatons Homographes Robust algnment Descrptor-based feature matchng RANSAC Large-scale algnment Inverted ndeng Vocabulary trees Applcaton: searchng the nght sky

8 Algnment as fttng Prevous lectures: fttng a model to features n one mage M Fnd model M that mnmzes resdual(, M )

9 Algnment as fttng Prevous lectures: fttng a model to features n one mage M Fnd model M that mnmzes resdual(, M ) Algnment: fttng a model to a transformaton between pars of features (matches) n two mages ' Fnd transformaton that mnmzes resdual( ( ), ")

10 2D transformaton models Smlarty (translaton, scale, rotaton) Affne Projectve (homography)

11 Let s start wth affne transformatons Smple fttng procedure (lnear least squares) Appromates vewpont changes for roughly planar objects and roughly orthographc cameras Can be used to ntalze fttng for more comple models

12 Fttng an affne transformaton Assume we know the correspondences, how do we get the transformaton? ), ( y ), ( y " # % & + " # % & " # % & = " # % & ' ' t t y m m m m y t M + = Want to fnd M, t to mnmze = # n 1 2 t M

13 Fttng an affne transformaton Assume we know the correspondences, how do we get the transformaton? ), ( y ), ( y " # % & + " # % & " # % & = " # % & ' ' t t y m m m m y " # % & ' ' = " # % & " # % & y t t m m m m y y

14 Fttng an affne transformaton Lnear system wth s unknowns Each match gves us two lnearly ndependent equatons: need at least three to solve for the transformaton parameters " # % & ' ' = " # % & " # % & y t t m m m m y y

15 Fttng a plane projectve transformaton Homography: plane projectve transformaton (transformaton takng a quad to another arbtrary quad)

16 Homography he transformaton between two vews of a planar surface he transformaton between mages from two cameras that share the same center

17 Applcaton: Panorama sttchng Source: Hartley & Zsserman

18 Fttng a homography Recall: homogeneous coordnates Convertng to homogeneous mage coordnates Convertng from homogeneous mage coordnates

19 Fttng a homography Recall: homogeneous coordnates Convertng to homogeneous mage coordnates Convertng from homogeneous mage coordnates Equaton for homography: λ & % ' y' # " = & % h h h h h h h h h #& " % y 1 # "

20 Fttng a homography Equaton for homography: = H λ " # % & " # % & = " # % & ' ' y h h h h h h h h h y λ = 0 " H " # % & ' ' ' ' = " # % & " # % & ' ' y y y h h h h h h h h h = " # % & ' ' ' ( ) * * * +, h h h y y 3 equatons, only 2 lnearly ndependent

21 Drect lnear transform H has 8 degrees of freedom (9 parameters, but scale s arbtrary) One match gves us two lnearly ndependent equatons Homogeneous least squares: fnd h mnmzng Ah 2 Egenvector of A A correspondng to smallest egenvalue Four matches needed for a mnmal soluton = " # % & ' ' ' ' ' ' ( ) * * * * * * +, h h h n n n n n n y y 0 Ah =

22 Robust feature-based algnment So far, we ve assumed that we are gven a set of ground-truth correspondences between the two mages we want to algn What f we don t know the correspondences? (, y ), y ) (

23 Robust feature-based algnment So far, we ve assumed that we are gven a set of ground-truth correspondences between the two mages we want to algn What f we don t know the correspondences??

24 Robust feature-based algnment

25 Robust feature-based algnment

26 Robust feature-based algnment Etract features Compute putatve matches

27 Robust feature-based algnment Etract features Compute putatve matches Loop: Hypothesze transformaton

28 Robust feature-based algnment Etract features Compute putatve matches Loop: Hypothesze transformaton Verfy transformaton (search for other matches consstent wth )

29 Robust feature-based algnment Etract features Compute putatve matches Loop: Hypothesze transformaton Verfy transformaton (search for other matches consstent wth )

30 Generatng putatve correspondences?

31 Generatng putatve correspondences? ( ) ( ) feature descrptor Need to compare feature descrptors of local patches surroundng nterest ponts =? feature descrptor

32 Feature descrptors Recall: feature detecton and descrpton

33 Feature descrptors Smplest descrptor: vector of raw ntensty values How to compare two such vectors? Sum of squared dfferences (SSD) Not nvarant to ntensty change Normalzed correlaton Invarant to affne ntensty change ( ) = u v 2 SSD( v) u, " # % & " # % & = = j j j j v u v u 2 2 ) ( ) ( ) )( ( ) ( ) ( ) ( v u v u v v v v u u u u v u, ρ

34 Dsadvantage of ntensty vectors as descrptors Small deformatons can affect the matchng score a lot

35 Revew: Algnment 2D algnment models Feature-based algnment outlne Descrptor matchng

36 Feature descrptors: SIF Descrptor computaton: Dvde patch nto 44 sub-patches Compute hstogram of gradent orentatons (8 reference angles) nsde each sub-patch Resultng descrptor: 448 = 128 dmensons Davd G. Lowe. "Dstnctve mage features from scale-nvarant keyponts. IJCV 60 (2), pp , 2004.

37 Feature descrptors: SIF Descrptor computaton: Dvde patch nto 44 sub-patches Compute hstogram of gradent orentatons (8 reference angles) nsde each sub-patch Resultng descrptor: 448 = 128 dmensons Advantage over raw vectors of pel values Gradents less senstve to llumnaton change Poolng of gradents over the sub-patches acheves robustness to small shfts, but stll preserves some spatal nformaton Davd G. Lowe. "Dstnctve mage features from scale-nvarant keyponts. IJCV 60 (2), pp , 2004.

38 Feature matchng Generatng putatve matches: for each patch n one mage, fnd a short lst of patches n the other mage that could match t based solely on appearance?

39 Problem: Ambguous putatve matches Source: Y. Furukawa

40 Rejecton of unrelable matches How can we tell whch putatve matches are more relable? Heurstc: compare dstance of nearest neghbor to that of second nearest neghbor Rato of closest dstance to second-closest dstance wll be hgh for features that are not dstnctve hreshold of 0.8 provdes good separaton Davd G. Lowe. "Dstnctve mage features from scale-nvarant keyponts. IJCV 60 (2), pp , 2004.

41 RANSAC he set of putatve matches contans a very hgh percentage of outlers RANSAC loop: 1. Randomly select a seed group of matches 2. Compute transformaton from seed group 3. Fnd nlers to ths transformaton 4. If the number of nlers s suffcently large, re-compute least-squares estmate of transformaton on all of the nlers Keep the transformaton wth the largest number of nlers

42 RANSAC eample: ranslaton Putatve matches

43 RANSAC eample: ranslaton Select one match, count nlers

44 RANSAC eample: ranslaton Select one match, count nlers

45 RANSAC eample: ranslaton Select translaton wth the most nlers

46 Scalablty: Algnment to large databases What f we need to algn a test mage wth thousands or mllons of mages n a model database? Effcent putatve match generaton Appromate descrptor smlarty search, nverted ndces est mage Model database?

47 Large-scale vsual search Rerankng/ Geometrc verfcaton Inverted ndeng Fgure from: Krsten Grauman and Bastan Lebe, Vsual Object Recognton, Synthess Lectures on Artfcal Intellgence and Machne Learnng, Aprl 2011, Vol. 5, No. 2, Pages 1-181

48 Eample ndeng technque: Vocabulary trees est mage Vocabulary tree wth nverted nde Database D. Nstér and H. Stewénus, Scalable Recognton wth a Vocabulary ree, CVPR 2006

49 Descrptor space Goal: fnd a set of representatve prototypes or cluster centers to whch descrptors can be quantzed

50 K-means clusterng Want to mnmze sum of squared Eucldean dstances between ponts and ther nearest cluster centers m k Algorthm: ( D( X, M ) = m cluster k pont n cluster k k 2 ) Randomly ntalze K cluster centers Iterate untl convergence: Assgn each data pont to the nearest center Recompute each cluster center as the mean of all ponts assgned to t

51 K-means demo Source: Another demo:

52 Recall: Vsual codebook for generalzed Hough transform Appearance codebook Source: B. Lebe

53 Herarchcal k-means clusterng of descrptor space (vocabulary tree) Slde credt: D. Nster

54 Vocabulary tree/nverted nde Slde credt: D. Nster

55 Model mages Populatng the vocabulary tree/nverted nde Slde credt: D. Nster

56 Model mages Populatng the vocabulary tree/nverted nde Slde credt: D. Nster

57 Model mages Populatng the vocabulary tree/nverted nde Slde credt: D. Nster

58 Model mages Populatng the vocabulary tree/nverted nde Slde credt: D. Nster

59 Model mages est mage Lookng up a test mage Slde credt: D. Nster

60 Cool applcaton of large-scale algnment: searchng the nght sky B C D A

61 Credt: Slde set developed by S. Lazebnk, Unversty of Illnos at Urbana-Champagn