Matching. Brandon Jennings January 20, 2015

Transcription

1 Matching Brandon Jennings January 20, 2015

2 Video Google: A Text Retrieval Approach to Object Matching in Videos Josef Sivic and Andrew Zisserman

3 Video Google The problem: Desire to match objects in a scene as quickly and efficiently as Google retrieves text documents The idea: query objects as one would query text via Google, using descriptors like vocabulary words 3

4 Outline Background/Related Works Viewpoint Invariant Visual Vocabulary WeighRng Scene Matching EvaluaRon Object Retrieval Results Summary 4

5 Background/Related Work Baeza- Yates & Ribeiro- Neto Text Retrieval steps Wi[en, Moffat, & Bell Inverted file 5

6 h[ps://developer.apple.com/library/mac/ documentaron/userexperience/conceptual/ SearchKitConcepts/searchKit_basics/ searchkit_basics.html h[p://people.cs.pi[.edu/~znar/aos.html

7 Viewpoint Invariant DescripRon Shape Adapted Region Scale determined by local extremum of a Laplacian Shape determined by max intensity gradient isotropy Centered on corner like features Maximally Stable Region StaRonary areas from intensity watershed image segmentaron High contrast with respect to surroundings 7

8 h[p:// 8

9 Figure 1: Top row: Two frames showing the same scene from very different camera viewpoints (from the film Run Lola Run ). Middle row: frames with detected affine invariant regions superimposed. Maximally Stable (MS) regions are in yellow. Shape Adapted (SA) regions are in cyan. Bo[om row: Final matched regions amer indexing and sparal consensus. Note that the correspondences define the scene overlap between the two frames. 9

10 Visual Vocabulary Cluster descriptors together Visual words Mahalanobis distance funcron 10

11 h[p:// 11

12 WeighRng Term frequency- inverse document frequency num of word i in doc d Word frequency num of docs in database num of words in doc d Inverse document freq num of docs in database 12

13 Scene Matching EvaluaRon 164 frames 48 shots 19 locarons 4-9 frames per locaron Whole frame used as query region Average normalized rank of relevant images (0 to 1) 13

14 num of relevant images size of image set rank of ith relevant image 14

15 Each row shows a frame from three different shots of the same locaron in the ground truth data set. 15

16 Shows that the combinaron of shape adopted features and max stable features is be[er than the two individually. 16

17 Average Precision- Recall curve for locaron matching on the ground truth set. 17

18 Compares the average rank measure for different retrieval weighrng methods. 18

19 h[p:// 19

20 Object Retreival Effect of Using Stop List Shows the reducron of the number of visual terms in the inverted index using a stop list. 20

21 21

22 Matching stages. Top row: (lem) Query region and (right) its close- up. Second row: Original word matches. Third row: matches amer using stop- list, Last row: Final set of matches amer filtering on sparal consistency. 22

23 Results 23

24 Summary Pros Immediate run- Rme object retrieval Visual vocabulary Allows for other affine- covariant regions to be added Cons Not dynamic Low ranks in some scenes Demo 24

25 The Pyramid Match Kernel: DiscriminaRve ClassificaRon with Sets of Image Features Kristen Grauman and Trevor Darrell

26 The Pyramid Match Kernel The Problem: ConvenRonal kernel based algorithms are designed to operate on fixed- length vector inputs The Idea: pyramid match kernel over unordered feature sets, mapping sets to a mulrresoluron histogram 26

27 Outline Background Comparison to other approaches Results 27

28 Background/Related Work Previous approaches had many drawbacks: ComputaRonal complexires with large sets LimitaRons to parametric distriburons Kernels not posirve- definite LimitaRons to sets of equal size Dependencies within sets Local features are shown to be invariant to transforms (SIFT) Most approaches use nearest neighbor or vorng, impracrcal for large sets 28

29 29

30 30

31 Shows method resembles oprmal soluron 31

32 Summary Approximates oprmal matching Approxiimates matching for unequal cardinality as well 32