WISE: Large Scale Content Based Web Image Search. Michael Isard Joint with: Qifa Ke, Jian Sun, Zhong Wu Microsoft Research Silicon Valley

Transcription

1 WISE: Large Scale Content Based Web Image Search Michael Isard Joint with: Qifa Ke, Jian Sun, Zhong Wu Microsoft Research Silicon Valley 1

2 A picture is worth a thousand words. Query by Images What leaf? Artist? Higher resolution? : Who else using this? Bank web site? ? Ad. in e bay? 2

3 Partial Duplicate Image Search Given a query image find its partial duplicates Given a query image, find its partial duplicates from a database of web images

4 Two Major Challenges How to represent images No text annotations or labels Noise and modification How to efficiently index and query images Large number of images (millions)? Index 4

5 Image Representation: Bag of Words [CVPR 09, ICCV 09] 1: Feature extraction: Bundle Features 2: Quantization 3: Representation Detection [Lowe 2004, Matas et al 2002, Winder et al 2007] Code book Descriptor ID Bag of words [Sivic&Zisserman 2003] Normalization , Description [Lowe 2004, Winder et al 2007] 999,999 1,000,000 5

6 6

7 Matching query to database Use an index Each visual word has a posting list Lists every imagecontaining the word At query time Look up the posting list for each query word Merge lists to find candidate images Partial match: don t need every word to be present 7

8 How much work to query? Disk based index, bottleneck is random reads One seek per posting list Also one seek per matching image To fetch thumbnail etc Keep as little information as possible in posting lists, to keep index size small 8

9 Index Pipeline Implemented in a large computer cluster 256 nodes, using Dryad/DryadLINQ y Image Crawler Content Chunk Feature Extractor Bundled Features Feature Quantizer Visual Words Indexer Media DB (Thumbnail, URL) Inverted Index Crawler Local lfeatures Visual Word Index 9

10 Query Pipeline Query Image Feature Extractor Bundled Features Feature Quantizer Visual Words GUI Results Media DB (Thumbnail, URL) Index Server Inverted Index Search Results (chunkid, imgid) 10

11 Bag of Words: Limitations Quantization Lost discriminative power Sensitive to image variations and noises Soft quantization Quantization Descriptor [Philbin et al, CVPR 2008] 999,999 Hamming embedding [Jegou et al, ECCV 2008] ID ,000

12 Geometric verification In practice, bag of words is too weak Does not exploit any geometry Post process to check spatial illayout of matching features Requires a disk seek per image Only used as a re ranking step to shortlist of matched images 12

13 Geometric Re ranking map Re rank top images baseline (bag-of-words) baseline + reranking Number of images

14 Geometry in the index Previous works: Jegou et al ECCV 2008 Try to match similar orientations andscales Perdoch et al CVPR 2009 Match oriented features more effectively Still feature by feature Globalgeometric consitency applied at the end 14

15 Single Feature is Weak + +

16 Neighboring Features?

17 Define Neighboring Features Previous works knn voting [Sivic&Zisserman 2003] Higher order spatial features [Liu et al][yuan et al][tirilly et al][quack et al] Post geometric spatial verification [Lowe 2004][Chum et al 2007][Nister 2006][Philbin et al 2007] Geometric Min Hash [Chum et al 2009] Challenges Repeatable Partial matching Scalable: simple enough to build into index

18 Define Neighboring Features DoG Features [Lowe 2004] MSER Features [Matas et al 2002] point features region features repeatable repeatable

19 Define Neighboring Features DoG Features [Lowe 2004] MSER Features [Matas et al 2002] point features region features repeatable repeatable region groups points?

20 Bundled Feature: Definition Bundled Feature = A set of DOG features bundled by a MSER region MSER region DoG interest points

21 Bundled Feature: Definition Bundled Features

22 Matching Bundles: Membership Query bundle q = {q j } = { } Matched bundle p = { p i } Membership score: M ( ; ) 4 m q p q p Voting weight: vq ( ) M ( qp ; ) 4 j m Sim( I1, I2) v( q j ) 4 16 qj qj

23 Matching Bundles: Membership Query bundle q = {q j } = { } Matched bundles p 1, p 2, p 3 Membership score: M ( ; ) 2 m q p q p 1 1 p 1 M ( m q ; p ) 2 q p 2 1 M ( m q ; p ) 2 3 q p3 p 3 Sim( I1, I2) v( q j ) 8 q j p 3 p 2 vq ( ) max M ( qp ; ) q j m k j p v ( q ) 2 2 v q1 k ( ) 2 v ( q ) max(1, 2) 2 3 v( q ) 2 4 q

24 Matching Bundles: Geometric Constraint y query candidate query candidate order in query img: 1 < 2 < 3 < 4 order in query img: 1 < 2 < 3 < 4 order in target img: 1 < 3 < 4 < 5 matching order: 5 > 2 > 1 < 3 order inconsistency: = 0 inconsistency: = 2 Penalize inconsistent relative orders: M ( qp ; ) O p O p g q i q i 1

25 Matching Bundles: Formulation Bundle matching score: M( q; p) M ( qp ; ) M ( qp ; ) Image matching score: m membership geometric constraint g vq ( ) max M( qp ; ) q j p k Sim( I1, I2) v( q j ) { q } j k j q Repeatable Partial matching Scalable?

26 Inverted Index (without Bundles) Visual word Posting Image ID Image ID =

27 Inverted Index with Bundles Visual word Posting Image ID Bundle Bits Image ID = 27 1 p 1 p bits 5 bits 5 bits Bundle ID X-Order Y-Order 27, [3,1,1] 2 p 3 27, [1,2,1] 27, [2,2,2]

28 Retrieval Query Image I q Inverted index with bundle bits 1,1, [2,5,9] 3,1, [3,4,5] 9,2, [3,2,5] 10, 1, [1,1,2] 12,1, [1.1.2] 10, 1, [2,2,1] Top candidate images p 1 p 2 p 3

29 Experimental Settings Image database: 1M web images from query click log Ground truth partial duplicates 780 known partial duplicate images in 19 groups Baseline bag of words Visual word vocabulary size = 1 M Soft quantization factor = features per image

30 Partial Duplicate Example

31 Partial Duplicate Example

32 Example Query Results Query Challenging cases

33 Evaluation: Precision Recall A query returns N images T : correct matches A : expected matches Precision = T N Recall = T A Precision Recall

34 Comparison: Precision Recall Query image: Baseline bag of words (started from 13 th ) Bundled features (started from 13 th )

35 More Precision Recall Comparisons

36 Evaluation: map Average Precision (AP) for one query: Area under Precision Recall curve AP map: mean of AP s from all testing queries

37 map: Baseline Bag of Words map baseline HE bundled(membership) bundled bundled + HE Number of images

38 map: Hamming Embedding (HE) map baseline HE bundled(membership) bundled bundled + HE Number of images

39 map: Bundle (Membership) map baseline HE bundled(membership) bundled bundled + HE Number of images

40 map: Bundle (both terms) map baseline HE bundled(membership) bundled bundled + HE Number of images 26% 40%

41 map: Bundle + HE map baseline HE bundled(membership) bundled bundled + HE Number of images 49%

42 Bundle VS. Geometric Re ranking map Re rank top images baseline (bag-of-words) bundle baseline + reranking bundle + reranking Number of images

43 Bundle + Geometric Re ranking Re rank top images 0.65 map % 77% baseline (bag-of-words) bundle baseline + reranking bundle + reranking Number of images

44 More Results

45 Failure Case

46 Demo Setup Client Web Server Index Servers Document Server 6 million images 46

47 Demo Query image Results 47

48 Conclusion Bundle feature More discriminative Enforce spatial constraints while traversing index Partial match Scalable: built into index 9 bits 5 bits 5 bits Bundle ID X-Order Y-Order

49 Thanks!