Large-scale Image Search and location recognition Geometric Min-Hashing. Jonathan Bidwell

Transcription

1 Large-scale Image Search and location recognition Geometric Min-Hashing Jonathan Bidwell Nov 3rd 2009 UNC Chapel Hill

2 Large scale + Location

3 Short story...

4 Finding similar photos See: Microsoft PhotoSynth & X. Li, C. Wu, Modeling and Recognition

5 Looking forward...

6 Internet enabled Much more is possible

7 iphone, Mosaica Much more is possible

8 Finding similar photos

9 Large scale search

10 How large? Photo thickness =.0065 inches

11 Large scale image search Flicker ~ 4 billion photos ~410 miles ~220 miles

12 Large scale image search Flicker ~ 15 billion photos ~1,539 miles ~410 miles ~220 miles

13 Large scale image search Flicker ~ 675 billion video frames ~69,247 miles Youtube: 150 million video x 2.5 min each avg

14 Large scale image search Flicker ~ 675 billion video frames ~69,247 miles Youtube: 150 million video x 2.5 min each avg

15 Large scale image search Speed matters Flickr search results unc Brutlag, SpeedMatters, Google Inc.

16 Location recognition

17 Location recognition Direct: Geo-tagged photos 35 54' 39.92" N 79 3' 10.45" W Google Maps, UNC Campus

18 Location recognition Indirect: Image similarity J. Sivic. Video Google: A Text Retrieval Approach to Object Matching in Videos.

19 Min-hash

20 Main idea Use a hash table to lookup image similarity Key Value

21 Min-hash Outline Image representation Feature extraction Bag-of-features Min-hash algorithm Building hash-functions Sorting min-hash values Hash table, image search

22 Image representation Measure distance using image features

23 Image representation Feature extraction - naive approach

24 Image representation Feature extraction - naive approach

25 Image representation Feature extraction - naive approach

26 Image representation Feature extraction - naive approach

27 Image representation Feature extraction - Problems

28 Image representation Scale Invariant Feature Transform (SIFT) Better for matching scale invariant rotation invariant translation invariant

29 Image representation Extract SIFT descriptors 128 x dimensional D. Lowe L. Lazebnik. Bag-Of-Features

30 Bag of Features Images are represented as a collection or bag of features from a known codebook Image Codebook Histogram L. Lazebnik. Bag-Of-Features Slides

31 Bag of Features Images are represented as a collection or bag of features from a known codebook Image Codebook Histogram L. Lazebnik. Bag-Of-Features Slides

32 Bag-of-features Build a visual word codebook

33 Bag-of-features Build a visual word codebook L. Lazebnik. Bag-Of-Features

34 Bag-of-features Build a visual word codebook L. Lazebnik. Bag-Of-Features

35 Bag-of-features Build a visual word codebook Codebook L. Lazebnik. Bag-Of-Features

36 Bag-of-features Build a visual word codebook Codebook L. Lazebnik. Bag-Of-Features

37 Bag-of-features Build a visual word codebook Codebook L. Lazebnik. Bag-Of-Features

38 Bag-of-features Build a visual word codebook Codebook L. Lazebnik. Bag-Of-Features

39 Bag-of-features Build a visual word codebook Codebook L. Lazebnik. Bag-Of-Features

40 Bag-of-features Find the closest codebook feature Codebook Visual word

41 Bag-of-features Find the closest codebook feature Codebook Visual word

42 Bag-of-features Find the closest codebook feature Codebook Visual word

43 Bag-of-features Find the closest codebook feature Codebook Visual word

44 Bag-of-features Find the closest codebook feature Codebook Visual word

45 Bag-of-features Min-hash binary simplification Histogram Binary simplification

46 Bag of Features Book analogy: Books have words Books can be compared by looking at word frequency

47 Bag of Features Book analogy: Books have words Books can be compared by looking at word frequency Book = codebook Word = visual word

48 Min-hash algorithm

49 Set similarity Binary visual word representation A

50 Set similarity Set intersection A1 A

51 Set similarity Set union A1 A

52 Set similarity Probability of set similarity O. Chum, Near Duplicate Image Detection: min-hash and tf-idf Weighting

53 Set similarity Probability of set similarity A A

54 Set similarity Probability of set similarity sims(a1,a2) = = 2/5 = 0.4

55 ~2000 features per image Codebook... Min-Hash for Images

56 Random hash functions Apply random hash functions to each word Random Hash Function f1 A1 A2 Visual word f2 A1 A f3 A1 A

57 Random hash functions Find first 1 value, given random column ordering Random Hash Function f1 A1 A2 Visual word f2 A1 A f3 A1 A

58 Random hash functions Find first 1 value, given random column ordering Random Hash Function f1 A1 A2 Visual word Same function, same response for each visual word element f2 A1 A f3 A A

59 Random hash functions Make hash-keys from hash functions Random Hash Function Visual word f1 f2 f3 f1 A1 A f2 A1 A f3 A1 A Hash-key

60 Set similarity revisited Probability of set similarity Random Hash Function f1 A1 A2 Visual word O. Chum, Near Duplicate Image Detection: min-hash and tf-idf Weighting

61 Min-hash approach Instead of random column permutations, each visual word gets a random number 0,1 Visual word f

62 Min-hash approach Instead of random column permutations, each visual word gets a random number 0,1 Visual word f min-hash value

63 Build hash table keys Build a key of min-hash values for each fj f1 f1 Visual word s f f Note: 0.1 is not the min hash value because the visual word entry at that index is 0

64 Build hash table keys Build a hash-key for a second visual word f1 f1 Visual word s s f1 f

65 Min-Hash Search Build sketch Key - sketch Value - image

66 Min-Hash Search Look for collisions with other hash keys Build sketch Key - sketch Value - image

67 Min-Hash Results TrecVid ,588 frames Vocabulary size 64k Lookup in constant time O. Chum, Near Duplicate Image Detection: min-hash and tf-idf Weighting

68 Geometric Min-Hash

69 Main Idea Min-hash + spatial constraint information O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

70 Spatial Constraints Select a set of F images features Codebook O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

71 Spatial Constraints Select a set of F images features Each feature must Codebook O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

72 Spatial Constraints Select a set of F images features Each feature must Have a unique visual word Codebook O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

73 Spatial Constraints Select a set of F images features Each feature must Have a unique visual word Codebook O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

74 Spatial Constraints Select a set of F images features Each feature must Have a unique visual word Have three neighbors at at the same scale Codebook O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

75 Spatial Constraints Select a set of F images features Each feature must Have a unique visual word Have three neighbors at at the same scale Codebook O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

76 Spatial Constraints Select a central feature from F O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

77 Spatial Constraints Select a central feature from F O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

78 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

79 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

80 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale f O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

81 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale f O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

82 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale f O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

83 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale f O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

84 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale f O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

85 Spatial Constraints Select a central feature from F Select a set of features within a prescribed distance & scale f O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

86 Spatial Constraints Select a set of features from N Sketch O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

87 Spatial Constraints Select a set of features from N Sketch Select using independent random hash functions from N O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

88 Results Small object recognition Input: Image Output: Image Matches Oxford 5k Landmark dataset Processing time Database 15 min 54 sec Per image ~.008 sec O. Chum, Geometric min-hashing: Finding a (Thick) Needle in a Haystack

89 References O. Chum, J. Philbin, and A. Zisserman, Near Duplicate Image Detection: min- Hash and tf-idf Weighting, Proceedings of the British Machine Vision Conference, O. Chum, M. Perd'och, and J. Matas, Geometric min-hashing: Finding a (Thick) Needle in a Haystack, Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2009 J. Sivic and A. Zisserman. Video Google: A Text Retrieval Approach to Object Matching in Videos. Proceedings of the International Conference on Computer Vision (2003)

90

91 Time-scales