Approximate Nearest Neighbor Search. Deng Cai Zhejiang University

Transcription

1 Approximate Nearest Neighbor Search Deng Cai Zhejiang University

2 The Era of Big Data

3 How to Find Things Quickly? Web 1.0 Text Search Sparse feature Inverted Index

4 How to Find Things Quickly? Web 2.0, 3.0 Multimedia Search Dense feature GIST 特征 960 维 HoG 特征 3360 维 SIFT 特征 128 维

5 How to Find Things Quickly? Dense feature Inverted index can no longer be used Exhaustive search is inefficient Indexing the large scale dataset Tree based index (KD-Tree, R-Tree, )

6 KD-Tree Spatial partitions and recursive hyper plane decomposition 2-dimensional KD-Tree

7 KD-Tree The worst case time for range search in a d-dimensional KD-Tree containing n nodes is given by the following equation It performs worse than a linear scan when the dimensionality of the space is high! D. T. Lee, C. K. Wong. Worst case analysis for region and partial region searches in multidimensional binary search trees and balanced quad trees, ACTA Informatica, 1977.

8 Approximate Nearest Neighbor (ANN) Search ε-approximate nearest neighbor search is a special case of the nearest neighbor search problem. The solution to the ε- approximate nearest neighbor search is a point or multiple points within distance (1+ε) R from a query point, where R is the distance between the query point and its true nearest neighbor.

9 Approximate Nearest Neighbor (ANN) Search Search nearest neighbors of a query An algorithm returns points Not all the points are the true nearest neighbors of the query We need to examine how many points in this returned set are among the true nearest neighbors of the query. Naturally defines the evaluation metric

10 Hashing for ANN Search Coding Locating Linear scan

11 Hashing for Nearest Neighbor Search Hash Function

12 History LSH (VLDB 1999) Spectral Hashing (NIPS 2009) NIPS(14), ICML(10), AAAI(15), IJCAI(23), CVPR(38), ICCV(15), ACM Multimedia(32), SIGKDD(7),SIGIR(17) Total 171 papers 24 papers per year

13 Unclear questions What are the proper metric for evaluating a ANN search method? How is the performance of hashing methods compared to tree-based methods (e.g. KD-tree)? Which (group of) hashing algorithm is the best? How can we invent a better ANNS algorithm?

14 Question 1: How to evaluate a ANNS method

15 Common Evaluation Mean Average Precision

16 Common Evaluation Hash Look Up Hamming Radius 2

17 Common Evaluation Missing the most important aspect Time! ANNS method Sacrifice the accuracy to achieve the efficiency Accuracy & Recall : The returned set of points given a query : The true nearest neighbors of the query The recall of is the same as the accuracy of Time-Accuracy (Time-Recall) curve

18 Evaluation Time-Accuracy (Time-Recall) curve Index size (memory size) Critical for some applications Indexing time Less important

19 Question 2: Hashing vs. KD-tree How is the performance of hashing methods compared to tree-based methods (e.g. KD-tree)? Datasets SIFT1M: [0~255] integer GIST1M: [0~1.5] float

20 Compared Algorithms flann (KD-tree) VISAPP'09, TPAMI 14 LSH (Locality Sensitive Hashing) VLDB 99 32bit ITQ (Iterative Quantization) CVPR 11 32bit

21 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

22 GIST1M (960 dim) GIST 1NN GIST 100NN

23 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

24 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

25 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

26 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

27 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

28 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

29 GIST1M (960 dim) GIST 1NN GIST 100NN

30 GIST1M (960 dim) GIST 1NN GIST 100NN

31 GIST1M (960 dim) GIST 1NN GIST 100NN

32 GIST1M (960 dim) GIST 1NN GIST 100NN

33 Index Memory & Indexing Time

34 Question 2: Hashing vs. KD-tree Answer: A carefully designed hashing algorithm (e.g. ITQ) is better than KD-tree

35 Question 3: The best hashing algorithm

36 Compared Algorithms LSH (Locality Sensitive Hashing) VLDB 99 32bit Based on random projection ITQ (Iterative Quantization) CVPR 11 32bit AGH (Anchor Graph Hashing) ICML 11 32bit Based on graph embedding

37 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

38 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

39 GIST1M (960 dim) GIST 1NN GIST 100NN

40 GIST1M (960 dim) GIST 1NN GIST 100NN

41 AGH results in the original paper

42 AGH results in the original paper MNIST 10 classes, each class has 7,000 images NUS-WIDE 21 classes, each class has 5,000 to 30,000 images

43 Supervised hashing methods?

44 Supervised hashing methods?

45 Properties of good hashing algorithm 1. Nearby points are mapped into same (nearby) buckets 2. The points are evenly distributed among all the buckets Each bucket is associated with a binary code!!!

46 Question 4: Better ANNS method How can we invent a better ANNS algorithm? Existing algorithms seem not good enough.

47 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

48 GIST1M (960 dim) GIST 1NN GIST 100NN

49 Idea

50 Graph-base Nearest Neighbor Search Additional Indexing structure An offline build knn graph Online search Given a query Use hashing methods to generate a small number of candidates Check the neighbors of these candidates for closer ones iteratively Z. Jin, D. Zhang, Y. Hu, S. Lin, D. Cai, X. He: Fast and Accurate Hashing Via Iterative Nearest Neighbors Expansion. IEEE Trans. Cybernetics 44(11): (2014)

51 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

52 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

53 SIFT1M (128 dim) SIFT 1NN SIFT 100NN

54 GIST1M (960 dim) GIST 1NN GIST 100NN

55 Efanna Extremely Fast Approximate Nearest Neighbor Search Algorithm Based on knn Graph (arxiv) Use truncated KD-tree for initialization Extremely Fast Approximate Nearest Neighbor Graph construction Algorithm C. Fu, D. Cai: EFANNA : An Extremely Fast Approximate Nearest Neighbor Search Algorithm Based on knn Graph. CoRR abs/ (2016)

56 How to build the knn graph efficiently SIFT1M 19 hours; GIST1M 157 hours (a week) No discussions on IEH (IEEE TCB 2014) papers

57 How to build the knn graph efficiently Efanna: Extremely Fast Approximate Nearest Neighbor graph construction Algorithm SIFT1M (128): 659s, 19 hours, to construct a 10NN-graph 188s, % accuracy, 362 times speedup! GIST1M (960): s, 157 hours, to construct a 10NN-graph 2300s, 92% accuracy, 245 times speedup!

58 SIFT1M Accuracy (%) SGraph FastKNN LargeVis brute-force Efanna Kgraph NN-descent NN-expansion

59 GIST1M Accuracy (%) SGraph FastKNN LargeVis brute-force Efanna Kgraph NN-descent NN-expansion

60 Index Memory & Indexing Time

61