Computer Vision Lecture 13

Transcription

1 Computer Vision Lecture 3 Recognition wit Local Features astian Leibe RWTH acen ttp:// leibe@vision.rwt-aacen.de

2 Course Outline Image Processing asics Segmentation & Grouping Object Recognition Object Categorization I Sliding Window based Object Detection Local Features & Matcing Local Features Detection and Description Recognition wit Local Features Indeing & Visual Vocabularies Object Categorization II 3D Reconstruction Motion and Tracking 3

3 N piels Recap: Local Feature Matcing Outline. Find a set of distinctive kepoints Define a region around eac kepoint N piels f e.g. color d( f Similarit measure, f ) T f e.g. color. Leibe 3. Etract and normalize te region content 4. Compute a local descriptor from te normalized region 5. Matc local descriptors 4

4 Recap: Harris-Laplace [Mikolajczk 0]. Initialization: Multiscale Harris corner detection 2. Scale selection based on Laplacian (same procedure wit Hessian Hessian-Laplace) Harris points Slide adapted from Krstian Mikolajczk Harris-Laplace points. Leibe 5

5 Recap: SIFT Feature Descriptor Scale Invariant Feature Transform Descriptor computation: Divide patc into 44 sub-patces: 6 cells Compute istogram of gradient orientations (8 reference angles) for all piels inside eac sub-patc Resulting descriptor: 448 = 28 dimensions David G. Lowe. "Distinctive image features from scale-invariant kepoints. IJCV 60 (2), pp. 9-0, Slide credit: Svetlana Lazebnik. Leibe 6

6 Topics of Tis Lecture Recognition wit Local Features Matcing local features Finding consistent configurations lignment: linear transformations ffine estimation Homograp estimation Dealing wit Outliers RNSC Generalized Houg Transform Indeing wit Local Features Inverted file inde Visual Words Visual Vocabular construction tf-idf weigting. Leibe 8

7 Recognition wit Local Features Image content is transformed into local features tat are invariant to translation, rotation, and scale Goal: Verif if te belong to a consistent configuration Slide credit: David Lowe Local Features, e.g. SIFT. Leibe 0

8 Concepts: Warping vs. lignment T Warping: Given a source image and a transformation, wat does te transformed output look like? T lignment: Given two images wit corresponding features, wat is te transformation between tem? Slide credit: Kristen Grauman. Leibe

9 Parametric (Global) Warping T p = (,) Transformation T is a coordinate-canging macine: p = T(p) Wat does it mean tat T is global? It s te same for an point p It can be described b just a few numbers (parameters) Let s represent T as a matri: Slide credit: leej Efros p = Mp,. Leibe p = (, ) ' ' M 2

10 Wat Can be Represented b a 22 Matri? 2D Scaling? ' s * ' s * 2D Rotation around (0,0)? ' cos * sin * ' sin * cos * 2D Searing? ' s * ' s * ' s 0 ' 0 s ' cos sin ' sin cos ' s ' s Slide credit: leej Efros. Leibe 3

11 Wat Can be Represented b a 22 Matri? 2D Mirror about ais? ' ' 2D Mirror over (0,0)? ' ' 2D Translation? ' t ' t ' 0 ' 0 ' 0 ' 0 NO! Slide credit: leej Efros. Leibe 4

12 2D Linear Transforms ' a b ' c d Onl linear 2D transformations can be represented wit a 22 matri. Linear transformations are combinations of Scale, Rotation, Sear, and Mirror Slide credit: leej Efros. Leibe 5

13 Homogeneous Coordinates Q: How can we represent translation as a 33 matri using omogeneous coordinates? ' ' t t : Using te rigtmost column: 0 t Translation 0 t 0 0 Slide credit: leej Efros. Leibe 6

14 asic 2D Transformations asic 2D transformations as 33 matrices ' 0 t ' 0 t 0 0 Translation ' ' s s 0 Scaling 0 0 ' cos sin 0 ' sin cos Rotation ' s 0 ' s Searing Slide credit: leej Efros. Leibe 7

15 Perceptual and Sensor ugmented Computing Computer Vision WS 5/6 2D ffine Transformations ffine transformations are combinations of Linear transformations, and Translations Parallel lines remain parallel 8. Leibe w f e d c b a w 0 0 ' ' Slide credit: leej Efros

16 Perceptual and Sensor ugmented Computing Computer Vision WS 5/6 Projective Transformations Projective transformations: ffine transformations, and Projective warps Parallel lines do not necessaril remain parallel 9. Leibe w i g f e d c b a w ' ' ' Slide credit: leej Efros

17 lignment Problem We ave previousl considered ow to fit a model to image evidence E.g., a line to edge points In alignment, we will fit te parameters of some transformation according to a set of matcing feature pairs ( correspondences ). i i ' T Slide credit: Kristen Grauman. Leibe 20

18 Let s Start wit ffine Transformations Simple fitting procedure (linear least squares) pproimates viewpoint canges for rougl planar objects and rougl ortograpic cameras Can be used to initialize fitting for more comple models Slide credit: Svetlana Lazebnik. Leibe 2

19 Fitting an ffine Transformation ffine model approimates perspective projection of planar objects Slide credit: Kristen Grauman. Leibe 22 Image source: David Lowe

20 Fitting an ffine Transformation ssuming we know te correspondences, ow do we get te transformation? ( i, i ) 2, ) ( i i i i m m 3 m m 2 4 i i t t 2. Leibe 23

21 Recall: Least Squares Estimation Set of data points: Goal: a linear function to predict X s from Xs: Xa b X We want to find a and b. ' How man ( X, X ) pairs do we need? ' Xa b X X a ' X 2a b X 2 X 2 b Wat if te data is nois? X X X... Slide credit: leej Efros 2 3 ' X ' a X 2 ' b X ' ( X, X ),( X, X ),( X, X ) ' ' ' Overconstrained problem min k k 2 Least-squares minimization. Leibe X X ' ' 2 Solution: = + Matlab: = n 24

22 Perceptual and Sensor ugmented Computing Computer Vision WS 5/6 Fitting an ffine Transformation ssuming we know te correspondences, ow do we get te transformation? 25. Leibe ), ( i i ), ( i i t t m m m m i i i i i i i i i i t t m m m m

23 Perceptual and Sensor ugmented Computing Computer Vision WS 5/6 Fitting an ffine Transformation How man matces (correspondence pairs) do we need to solve for te transformation parameters? Once we ave solved for te parameters, ow do we compute te coordinates of te corresponding point for? 26. Leibe i i i i i i t t m m m m ), ( new new Slide credit: Kristen Grauman

24 Homograp projective transform is a mapping between an two perspective projections wit te same center of projection. I.e. two planes in 3D along te same sigt ra Properties Rectangle sould map to arbitrar quadrilateral Parallel lines aren t but must preserve straigt lines Tis is called a omograp PP2 w' * w' * w * p Slide adapted from leej Efros * * * H * * * p. Leibe PP 27

25 Homograp projective transform is a mapping between an two perspective projections wit te same center of projection. I.e. two planes in 3D along te same sigt ra Properties Rectangle sould map to arbitrar quadrilateral Parallel lines aren t but must preserve straigt lines Tis is called a omograp w' 2 3 w' w 3 32 p Slide adapted from leej Efros H p. Leibe Set scale factor to 8 parameters left.

26 Fitting a Homograp Estimating te transformation Homogenous coordinates ' ' z' Image coordinates ' ' z' z' Matri notation ' '' H z' ' Slide credit: Krstian Mikolajczk. Leibe 29

27 Fitting a Homograp Estimating te transformation Homogenous coordinates ' ' z' Image coordinates ' ' z' z' Matri notation ' '' H z' ' Slide credit: Krstian Mikolajczk. Leibe 30

28 Fitting a Homograp Estimating te transformation ' ' z' 2 3 Slide credit: Krstian Mikolajczk Homogenous coordinates Leibe 2 Image coordinates ' ' z' z' Matri notation ' '' H z' ' 3

29 Fitting a Homograp Estimating te transformation ' ' z' 2 3 Slide credit: Krstian Mikolajczk Homogenous coordinates Leibe Image coordinates ' ' z' z' Matri notation ' '' H z' ' 32

30 Fitting a Homograp Estimating te transformation Homogenous coordinates Image coordinates Slide credit: Krstian Mikolajczk. Leibe 33

31 Fitting a Homograp Estimating te transformation Slide credit: Krstian Mikolajczk Homogenous coordinates Leibe Image coordinates

32 Perceptual and Sensor ugmented Computing Computer Vision WS 5/6 Fitting a Homograp Estimating te transformation 35. Leibe Slide credit: Krstian Mikolajczk

33 Fitting a Homograp Estimating te transformation Solution: Null-space vector of SVD 0 d 0 v v9 T UDV? U 0 d v v T 2 Slide credit: Krstian Mikolajczk. Leibe 36

34 Fitting a Homograp Estimating te transformation Solution: Null-space vector of Corresponds to smallest singular vector SVD Slide credit: Krstian Mikolajczk 0. Leibe 2 3 d 0 v v9 T UDV U 0 d v v v9,, v v T 3 2 Minimizes least square error 37

35 Image Warping wit Homograpies p p Slide credit: Steve Seitz Image plane in front lack area were no piel maps to. Leibe image plane below 38

36 Uses: nalzing Patterns and Sapes Wat is te sape of te b/w floor pattern? Slide credit: ntonio Criminisi Te floor (enlarged). Leibe 39

37 utomatic rectification nalzing Patterns and Sapes From Martin Kemp Te Science of rt (manual reconstruction) Slide credit: ntonio Criminisi. Leibe 40

38 Topics of Tis Lecture Recognition wit Local Features Matcing local features Finding consistent configurations lignment: linear transformations ffine estimation Homograp estimation Dealing wit Outliers RNSC Generalized Houg Transform Indeing wit Local Features Inverted file inde Visual Words Visual Vocabular construction tf-idf weigting. Leibe 4

39 Problem: Outliers Outliers can urt te qualit of our parameter estimates, e.g., n erroneous pair of matcing points from two images feature point tat is noise or doesn t belong to te transformation we are fitting. Slide credit: Kristen Grauman. Leibe 42

40 Eample: Least-Squares Line Fitting ssuming all te points tat belong to a particular line are known 43. Leibe Source: Forst & Ponce

41 Outliers ffect Least-Squares Fit 44. Leibe Source: Forst & Ponce

42 Outliers ffect Least-Squares Fit 45. Leibe Source: Forst & Ponce

43 Strateg : RNSC [Fiscler8] RNdom Smple Consensus pproac: we want to avoid te impact of outliers, so let s look for inliers, and use onl tose. Intuition: if an outlier is cosen to compute te current fit, ten te resulting line won t ave muc support from rest of te points. Slide credit: Kristen Grauman. Leibe 46

44 RNSC RNSC loop:. Randoml select a seed group of points on wic to base transformation estimate (e.g., a group of matces) 2. Compute transformation from seed group 3. Find inliers to tis transformation 4. If te number of inliers is sufficientl large, recompute least-squares estimate of transformation on all of te inliers Keep te transformation wit te largest number of inliers Slide credit: Kristen Grauman. Leibe 47

45 RNSC Line Fitting Eample Task: Estimate te best line How man points do we need to estimate te line? Slide credit: Jiniang Cai. Leibe 48

46 RNSC Line Fitting Eample Task: Estimate te best line Sample two points Slide credit: Jiniang Cai. Leibe 49

47 RNSC Line Fitting Eample Task: Estimate te best line Fit a line to tem Slide credit: Jiniang Cai. Leibe 50

48 RNSC Line Fitting Eample Task: Estimate te best line Slide credit: Jiniang Cai. Leibe Total number of points witin a tresold of line. 5

49 RNSC Line Fitting Eample Task: Estimate te best line 7 inlier points Total number of points witin a tresold of line. Slide credit: Jiniang Cai. Leibe 52

50 RNSC Line Fitting Eample Task: Estimate te best line Repeat, until we get a good result. Slide credit: Jiniang Cai. Leibe 53

51 RNSC Line Fitting Eample Task: Estimate te best line inlier points Repeat, until we get a good result. Slide credit: Jiniang Cai. Leibe 54

52 RNSC: How man samples? How man samples are needed? Suppose w is fraction of inliers (points from line). n points needed to define potesis (2 for lines) k samples cosen. Prob. tat a single sample of n points is correct: Prob. tat all k samples fail is: n w ( n k w ) Coose k ig enoug to keep tis below desired failure rate. Slide credit: David Lowe. Leibe 55

53 RNSC: Computed k (p=0.99) Sample size n Proportion of outliers 5% 0% 20% 25% 30% 40% 50% Slide credit: David Lowe. Leibe 56

54 fter RNSC RNSC divides data into inliers and outliers and ields estimate computed from minimal set of inliers. Improve tis initial estimate wit estimation over all inliers (e.g. wit standard least-squares minimization). ut tis ma cange inliers, so alternate fitting wit reclassification as inlier/outlier. Slide credit: David Lowe. Leibe 57

55 Eample: Finding Feature Matces Find best stereo matc witin a square searc window (ere 300 piels 2 ) Global transformation model: epipolar geometr Slide credit: David Lowe. Leibe Images from Hartle & Zisserman 58

56 Eample: Finding Feature Matces Find best stereo matc witin a square searc window (ere 300 piels 2 ) Global transformation model: epipolar geometr before RNSC after RNSC Slide credit: David Lowe. Leibe Images from Hartle & Zisserman 59

57 Problem wit RNSC In man practical situations, te percentage of outliers (incorrect putative matces) is often ver ig (90% or above). lternative strateg: Generalized Houg Transform Slide credit: Svetlana Lazebnik. Leibe 60

58 Strateg 2: Generalized Houg Transform Suppose our features are scale- and rotation-invariant Ten a single feature matc provides an alignment potesis (translation, scale, orientation). model Slide credit: Svetlana Lazebnik. Leibe 6

59 Strateg 2: Generalized Houg Transform Suppose our features are scale- and rotation-invariant Ten a single feature matc provides an alignment potesis (translation, scale, orientation). Of course, a potesis from a single matc is unreliable. Solution: let eac matc vote for its potesis in a Houg space wit ver coarse bins. model Slide credit: Svetlana Lazebnik. Leibe 62

60 Pose Clustering and Verification wit SIFT To detect instances of objects from a model base:. Inde descriptors Distinctive features narrow down possible matces Slide credit: Kristen Grauman. Leibe 63 Image source: David Lowe

61 Indeing Local Features New image Model base Slide credit: Kristen Grauman Image source: David Lowe

62 Pose Clustering and Verification wit SIFT To detect instances of objects from a model base: Slide credit: Kristen Grauman. Inde descriptors Distinctive features narrow down possible matces 2. Generalized Houg transform to vote for poses Kepoints ave record of parameters relative to model coordinate sstem 3. ffine fit to ceck for agreement between model and image features Fit and verif using features from Houg bins wit 3+ votes. Leibe 65 Image source: David Lowe

63 Object Recognition Results ackground subtract for model boundaries Objects recognized Recognition in spite of occlusion Slide credit: Kristen Grauman. Leibe 66 Image source: David Lowe

64 Location Recognition Training [Lowe, IJCV 04]. Leibe Slide credit: David Lowe 67

65 Recall: Difficulties of Voting Noise/clutter can lead to as man votes as true target. in size for te accumulator arra must be cosen carefull. (Recall Houg Transform) In practice, good idea to make broad bins and spread votes to nearb bins, since verification stage can prune bad vote peaks.. Leibe 68

66 Summar Recognition b alignment: looking for object and pose tat fits well wit image Use good correspondences to designate poteses. Invariant local features offer more reliable matces. Find consistent inlier configurations in clutter Generalized Houg Transform RNSC lignment approac to recognition can be effective if we find reliable features witin clutter. pplication: large-scale image retrieval pplication: recognition of specific (mostl planar) objects Movie posters ooks CD covers. Leibe 70

67 References and Furter Reading detailed description of local feature etraction and recognition can be found in Capters 3-5 of Grauman & Leibe (available on te L2P). K. Grauman,. Leibe Visual Object Recognition Morgan & Clapool publisers, 20 R. Hartle,. Zisserman Multiple View Geometr in Computer Vision 2nd Ed., Cambridge Univ. Press, 2004 More details on RNSC can also be found in Capter 4.7 of Hartle & Zisserman.