The digital copy of this thesis is protected by the Copyright Act 1994 (New Zealand).

Transcription

1 Research Commons at the University of Waikato Copyright Statement: The igital copy of this thesis is protecte by the Copyright Act 99 (New Zealan). The thesis may be consulte by you, provie you comply with the provisions of the Act an the following conitions of use: Any use you make of these ocuments or images must be for research or private stuy purposes only, an you may not make them available to any other person. Authors control the copyright of their thesis. You will recognise the author s right to be ientifie as the author of the thesis, an ue acknowlegement will be mae to the author where appropriate. You will obtain the author s permission before publishing any material from the thesis.

2 Department of Computer Science Hamilton, NewZealan Fast Algorithms for Nearest Neighbour Search Ashraf Masoo Kibriya This thesis is submitte in partial fulfilment of the requirements for the egree of Master of Science at The University of Waikato. March 007 c 007 Ashraf Masoo Kibriya

3

4 Abstract The nearest neighbour problem is of practical significance in a number of fiels. Often we are intereste in fining an object near to a given query object. The problem is ol, an a large number of solutions have been propose for it in the literature. However, it remains the case that even the most popular of the techniques propose for its solution have not been compare against each other. Also, many techniques, incluing the ol an popular ones, can be implemente in a number of ways, an often the ifferent implementations of a technique have not been thoroughly compare either. This research presents a etaile investigation of ifferent implementations of two popular nearest neighbour search ata structures, KDTrees an Metric Trees, an compares the ifferent implementations of each of the two structures against each other. The best implementations of these structures are then compare against each other an against two other techniques, Metho an Cover Trees. Metho is an ol technique that was reiscovere uring the research for this thesis. Cover Trees are one of the most novel an promising ata structures for nearest neighbour search that have been propose in the literature. i

5

6 Acknowlegments The continue support of Department of Computer Science s Machine Learning group, an particularly my supervisor Dr. Eibe Frank, is greatly appreciate, without which this thesis woul not have been possible. iii

7

8 Contents Abstract Acknowlegments i iii Introuction: The Nearest Neighbour Problem. Basic Definition Extensions an Variations of the Problem Applications of the NN Problem Characteristics Common to NN Applications Objectives an Scope of the Thesis Thesis Overview Towars Solving the NN problem 7. Techniques Propose as Solution Partial Distance Search (PDS) k-dimensional Trees (KDTrees), BBF-Trees an Variants Metric (Ball) Trees, vp-trees an Hybri Sp-Trees Voronoi Diagrams Orchar s Metho Metho AESA an LAESA R-Trees, X-Trees, M-Trees, TV-Trees, SR-Trees an VA-Files Locality Sensitive Hashing (LSH) Navigating Nets an Cover Trees Techniques Selecte for Evaluation Evaluate Techniques in epth 9 3. KDTrees Basic Construction Basic query proceure Construction in etail v

9 3.. Query in etail Implementation etails Metric Trees Basic Construction Basic Query Construction in Detail Query in Detail Implementation Details Metho Metho s Preprocessing Metho s Query Proceure s Basic Construction Basic Query Structures in etail Implementation etails Experimental Evaluation 69. Evaluation Proceure Evaluation Datasets Results KDTrees Construction Methos Metric Trees Construction Methos NN Methos Conclusion 3 A Aitional Results for KDTrees 5 B Aitional Results for Metric Trees 5 C Aitional Results for NN Methos 33 vi

10 List of Figures. ɛ-approximate Nearest Neighbour Typical KDTree/BBFTree ecomposition A (a) KDTree vs a (b) BBD-Tree on the same ata A typical metric tree ecomposition Illustration of a vp-tree ecomposition An orer- Voronoi space ecomposition Orchar s metho metho Graphical illustration of metho AESA s elimination AESA s approximation VA-File structure (Weber et al., 99) Illustration of KDTree construction Illustration of KDTree query Different KDTree construction methos Sproull s metho of overlap etection for KDTrees Incremental Distance Calculation Top Down construction metho for Metric Trees Mile Out construction metho of Metric Trees Illustration of Metric Trees query proceure Different construction methos of Metric Trees Metric Trees pruning criterion Different reference points for Metho Illustration of Cover Trees construction process Illustration of Cover Tree query Pruning uring a Cover Tree s query proceure Evaluate Point Distributions KDTrees construction time for increasing n vii

11 .3 KDTrees construction time for increasing Avg query points visite by KDTrees for increasing n on non-uniform query. 7.5 Avg query points visite by KDTrees for increasing n on uniform query....6 Avg query points visite by KDTrees for increasing on non-uniform query. 9.7 Avg query points visite by KDTrees for increasing on uniform query Metric Trees construction time for increasing n Metric Trees construction time for increasing Avg points visite by Metric Trees for increasing n on non-uniform query Avg points visite by Metric Trees for increasing n on uniform query Avg points visite by Metric Trees for increasing on non-uniform query Avg points visite by Metric Trees for increasing on uniform query Preprocessing time of NN methos for increasing n Preprocessing time of NN methos for increasing Avg points visite by NN Methos for increasing n on non-uniform query Avg points visite by NN Methos for increasing n on uniform query..... Avg points visite by NN Methos for increasing on non-uniform query Avg points visite by NN Methos for increasing on uniform query CPU query time of NN Methos for increasing n on non-uniform query.... CPU query time of NN Methos for increasing n on uniform query CPU query time of NN Methos for increasing on non-uniform query....3 CPU query time of NN Methos for increasing on uniform query A. KDTrees construction time for increasing n A. KDTrees construction time for increasing A.3 O(.5 ) construction time A. Degraation of KDTrees towars n at higher s A.5 CPU query time of KDTrees for increasing n on non-uniform query A.6 CPU query time of KDTrees for increasing n on uniform query A.7 CPU query time of KDTrees for increasing on non-uniform query..... A. CPU query time of KDTrees for increasing on uniform query B. Metric Trees construction time for increasing n with = B. Metric Trees construction time for increasing with n=k B.3 CPU query time of Metric Trees for increasing n on non-uniform query... B. CPU query time of Metric Trees for increasing n on uniform query B.5 CPU query time of Metric Trees for increasing on non-uniform query viii

12 B.6 CPU query time of Metric Trees for increasing on uniform query C. Preprocessing time of NN methos for increasing n with = C. Preprocessing time of NN methos for increasing with n=6k ix

13

14 List of Tables. Exact an approx. NN techniques Evaluate methos Measurements mae for each NN metho on each generate ataset Aitional measurements mae for tree base NN methos Distributions on which the NN methos were evaluate xi

15 Department of Computer Science Hamilton, NewZealan Fast Algorithms for Nearest Neighbour Search Ashraf Masoo Kibriya This thesis is submitte in partial fulfilment of the requirements for the egree of Master of Science at The University of Waikato. March 007 c 007 Ashraf Masoo Kibriya

16

17 Abstract The nearest neighbour problem is of practical significance in a number of fiels. Often we are intereste in fining an object near to a given query object. The problem is ol, an a large number of solutions have been propose for it in the literature. However, it remains the case that even the most popular of the techniques propose for its solution have not been compare against each other. Also, many techniques, incluing the ol an popular ones, can be implemente in a number of ways, an often the ifferent implementations of a technique have not been thoroughly compare either. This research presents a etaile investigation of ifferent implementations of two popular nearest neighbour search ata structures, KDTrees an Metric Trees, an compares the ifferent implementations of each of the two structures against each other. The best implementations of these structures are then compare against each other an against two other techniques, Metho an Cover Trees. Metho is an ol technique that was reiscovere uring the research for this thesis. Cover Trees are one of the most novel an promising ata structures for nearest neighbour search that have been propose in the literature. i

18

19 Acknowlegments The continue support of Department of Computer Science s Machine Learning group, an particularly my supervisor Dr. Eibe Frank, is greatly appreciate, without which this thesis woul not have been possible. iii

20

21 Contents Abstract Acknowlegments i iii Introuction: The Nearest Neighbour Problem. Basic Definition Extensions an Variations of the Problem Applications of the NN Problem Characteristics Common to NN Applications Objectives an Scope of the Thesis Thesis Overview Towars Solving the NN problem 7. Techniques Propose as Solution Partial Distance Search (PDS) k-dimensional Trees (KDTrees), BBF-Trees an Variants Metric (Ball) Trees, vp-trees an Hybri Sp-Trees Voronoi Diagrams Orchar s Metho Metho AESA an LAESA R-Trees, X-Trees, M-Trees, TV-Trees, SR-Trees an VA-Files Locality Sensitive Hashing (LSH) Navigating Nets an Cover Trees Techniques Selecte for Evaluation Evaluate Techniques in epth 9 3. KDTrees Basic Construction Basic query proceure Construction in etail v

22 3.. Query in etail Implementation etails Metric Trees Basic Construction Basic Query Construction in Detail Query in Detail Implementation Details Metho Metho s Preprocessing Metho s Query Proceure s Basic Construction Basic Query Structures in etail Implementation etails Experimental Evaluation 69. Evaluation Proceure Evaluation Datasets Results KDTrees Construction Methos Metric Trees Construction Methos NN Methos Conclusion 3 A Aitional Results for KDTrees 5 B Aitional Results for Metric Trees 5 C Aitional Results for NN Methos 33 vi

23 List of Figures. ɛ-approximate Nearest Neighbour Typical KDTree/BBFTree ecomposition A (a) KDTree vs a (b) BBD-Tree on the same ata A typical metric tree ecomposition Illustration of a vp-tree ecomposition An orer- Voronoi space ecomposition Orchar s metho metho Graphical illustration of metho AESA s elimination AESA s approximation VA-File structure (Weber et al., 99) Illustration of KDTree construction Illustration of KDTree query Different KDTree construction methos Sproull s metho of overlap etection for KDTrees Incremental Distance Calculation Top Down construction metho for Metric Trees Mile Out construction metho of Metric Trees Illustration of Metric Trees query proceure Different construction methos of Metric Trees Metric Trees pruning criterion Different reference points for Metho Illustration of Cover Trees construction process Illustration of Cover Tree query Pruning uring a Cover Tree s query proceure Evaluate Point Distributions KDTrees construction time for increasing n vii

24 .3 KDTrees construction time for increasing Avg query points visite by KDTrees for increasing n on non-uniform query. 7.5 Avg query points visite by KDTrees for increasing n on uniform query....6 Avg query points visite by KDTrees for increasing on non-uniform query. 9.7 Avg query points visite by KDTrees for increasing on uniform query Metric Trees construction time for increasing n Metric Trees construction time for increasing Avg points visite by Metric Trees for increasing n on non-uniform query Avg points visite by Metric Trees for increasing n on uniform query Avg points visite by Metric Trees for increasing on non-uniform query Avg points visite by Metric Trees for increasing on uniform query Preprocessing time of NN methos for increasing n Preprocessing time of NN methos for increasing Avg points visite by NN Methos for increasing n on non-uniform query Avg points visite by NN Methos for increasing n on uniform query..... Avg points visite by NN Methos for increasing on non-uniform query Avg points visite by NN Methos for increasing on uniform query CPU query time of NN Methos for increasing n on non-uniform query.... CPU query time of NN Methos for increasing n on uniform query CPU query time of NN Methos for increasing on non-uniform query....3 CPU query time of NN Methos for increasing on uniform query A. KDTrees construction time for increasing n A. KDTrees construction time for increasing A.3 O(.5 ) construction time A. Degraation of KDTrees towars n at higher s A.5 CPU query time of KDTrees for increasing n on non-uniform query A.6 CPU query time of KDTrees for increasing n on uniform query A.7 CPU query time of KDTrees for increasing on non-uniform query..... A. CPU query time of KDTrees for increasing on uniform query B. Metric Trees construction time for increasing n with = B. Metric Trees construction time for increasing with n=k B.3 CPU query time of Metric Trees for increasing n on non-uniform query... B. CPU query time of Metric Trees for increasing n on uniform query B.5 CPU query time of Metric Trees for increasing on non-uniform query viii

25 B.6 CPU query time of Metric Trees for increasing on uniform query C. Preprocessing time of NN methos for increasing n with = C. Preprocessing time of NN methos for increasing with n=6k ix

26

27 List of Tables. Exact an approx. NN techniques Evaluate methos Measurements mae for each NN metho on each generate ataset Aitional measurements mae for tree base NN methos Distributions on which the NN methos were evaluate xi

28

29 Chapter Introuction: The Nearest Neighbour Problem Nearest neighbour (NN) search is an ol problem that is of practical importance in a number fiels. It involves fining, from a given set of points, one or more points that are nearest to another given point, calle the query point. The problem comes up in fiels as iverse as ata compression, computational biology, computer vision an information retrieval. It was originally propose in 969 by Minsky an Papert (Minsky & Papert, 969). Since then, it has been extensively stuie an a vast number of ata structures, algorithms an techniques have been propose for its solution. Though nearest neighbour is the most ominant term use, it is also known as the best-match, closest-match, closest point an the post office problem. The term similarity search is also often use in the information retrieval fiel an the atabase community.. Basic Definition The NN search problem is: Given a set of n points S in some -imensional space X an a istance (or issimilarity) measure M, our task is to preprocess the points in S in such a way that, given a query point q X, we can quickly fin the point in S which is nearest (or most similar) to q.. Extensions an Variations of the Problem A natural an straight forwar extension of this problem is k-nearest neighbour (knn) search, in which we are intereste in the k ( S ) nearest points to q, containe in S. The NN search then just becomes a special case of knn search with k=. A slight variation of NN search, avocate by some (Inyk & Motwani, 99; Datar et al., 00) in place of NN search, is ɛ-approximate NN (ɛ-nn) search, where given a

30 p q r p' Figure.: ɛ-approximate Nearest Neighbour. user efine error boun ɛ 0, the task is to fin a point p in S which is at most ɛ times farther than the exact NN p (also in S), i.e. p, p S M(p, q) ( + ɛ)m(p, q). For the knn case this simply extens to fining k neighbours p, p,..p k such that any of the i th ɛ-approximate NN p i is at most ɛ times farther than the exact ith NN p i, i.e. M(p i, q) ( + ɛ)m(p i, q). Fig.. illustrates this graphically..3 Applications of the NN Problem The knn search is of practical significance in a number of fiels. Some of those, along with examples of their use of knn search, inclue: Data compression: Here, it is use in a metho calle vector quantization for speech an image compression (Gersho & Gray, 99). It involves blocking speech or image waveform signals into vectors of fixe length. A set of coevectors is first compute base on a set of training vectors, then each new vector is encoe with the inex of its nearest neighbour among the coevectors. Pattern recognition, atamining an machine learning: Here, one of the most wiely use classifier/learner is the knn classifier (Cover & Hart, 967). It is base on straight forwar aoption of the knn search, an works by assigning a given test point the majority class of its k-nearest neighbours. Also, Locally weighte learning (Atkeson et al., 997) is another technique which utilizes knn search. It works by training its base classifier/learner on training points that are nearest neighbours of a given test point. Bioinformatics: Here, knn an its variant classifiers have been applie with success to biological an clinical ata. They have been use for cancer classification

31 (Niijima & Kuhara, 005), for etecting rrna sequences (Robinson-Cox et al., 995), an, using gene-expression ata for tumour classification (Duoit et al., 00) an tissue classification, (Li et al., 00). In cases involving gene selection (Niijima & Kuhara, 005; Duoit et al., 00; Li et al., 00), these classifiers have been observe to perform as well or even better than state-of-the-art SVM base classifiers. Multimeia atabases an libraries: Here, similarity search is often use to retrieve multimeia content similar to a user query. The systems usually allow content-base queries, i.e. queries in the form of object shapes, texture, ominant colours, an scene escriptions etc. for images an vieo (Flickner et al., 995; Pentlan et al., 99; Smith & Chang, 996; Bach et al., 996), an in the form of ominant frequency an pitch (which can also be given as an acoustic input from the user) etc. in case of an auio/music library (Ghias et al., 995; McNab et al., 997; Tseng, 999; Uitenboger & Zobel, 00; Zhu et al., 003). Computer vision: Here, NN search is an important tool use for the task of object classification, which involves fining similarities between images (Marshall, 006). It has also been applie recently in a prototype of a robust multi-target tracking system, which tracks players in a hockey rink (Cai et al., 006). Document/information retrieval: Here, NN search is often use metho to retrieve an rank ocuments given a user query (Lucarella, 9; Deerwester et al., 990; Faloutsos & Oar, 995).. Characteristics Common to NN Applications In almost all of the above, the general representation of objects of interest (ocuments, images, etc.) incluing the queries is as vectors or points in some real -imensional space X = R, sometimes also calle the feature vector representation; since each of the imensions of the vectors correspon to some feature or attribute of the objects they represent. The closeness or (is)similarity of the vectors (or points) is measure using a measure M which is usually a metric (a.k.a. istance function). However, both the object representation an the choice of similarity/nearness measure are usually epenent on the application omain. In string classification, for example, the objects are generally represente as string sequences rather than vectors (Aggarwal, 00; Mollinea et al., 003), even in cases when the employe istance measure (usually eit istance) is a 3

32 metric (Mollinea et al., 003). Similarly, the Dynamic Time Warping (DTW) istance measure use in speech recognition (Vial et al., 9), an the NEM r shape-istance measure employe in IBM s QBIC system (Flickner et al., 995), as note by Faragó et al. (Faragó et al., 993) an Fagin an Stockmeyer (Fagin & Stockmeyer, 99) respectively, are not exact metrics. Nevertheless, almost all of the stuies of (k)nn search that were reviewe as part of the research for this thesis, have concentrate on points/vectors in a real -imensional metric space. Because often it is easier to represent objects in terms of vectors, an for some omain specific measure to be mappe to a metric. Some of the techniques that have been propose for the problem have even been evise for some particular metric (e.g. the initial version of LSH (Inyk & Motwani, 99)). Many times even the NNsearch problem itself has been efine with insistence on points being in a metric space (Arya et al., 99; Maneewongvatana & Mount, 00; Inyk & Motwani, 99; Inyk, 99, 00). This, though, is pretty restrictive, given some of the earliest stuies of the problem use more general measures instea of metrics (Frieman et al., 977)..5 Objectives an Scope of the Thesis It was observe uring the review of the literature that even the most popular of the large number of techniques propose since the initial inception of the problem, have not been thoroughly evaluate against each other. Also observe was the fact that there exists some consierable variation in how many of these techniques can be implemente. This also has not been thoroughly investigate either (either theoretically or empirically). When this research project was starte it was intene to implement an empirically evaluate all the algorithms an techniques propose in the literature. However, given the sheer volume of solutions propose, compoune with the fact of ifferent ways of implementing many of them an their ifferent possible parameter settings, plus the fact that many solutions have been propose for some specific similarity measure, an not to mention the fact that a number of those eal with some slight variation of the problem (e.g. ɛ-knn) or the ifficulties of gaps that exist between abstract theoretical escriptions to concrete implementations of the algorithms, that the initially intene goal was eeme unattainable. This was further exacerbate by the fact that a large number of techniques that exist have been esigne for external memory (in orer to reuce the I/O overhea in atabases), an often cannot be fairly compare to those esigne to work in main memory. Inyk an his group have correcte this in their latest publications (Anoni & Inyk, 006; Shakhnarovich et al., 006a)

33 Hence, to make the project more tractable, the goal of the research was narrowe own to inclue only the most popular of the ol an the most promising of the novel techniques, that (a) work in main memory, (b) are applicable to vector/point representation, (c) use the Eucliean metric as istance measure, () have not been cross-evaluate, an (e) support exact NN search. This resulte in a etaile investigation of two popular ata structures, KDTrees an Metric Trees, an their ifferent propose implementation methos were compare against each other. Later, the best implementations of the two techniques were evaluate against each other, an against one inepenently evelope oler technique, the Metho, an one recently propose promising but not well evaluate ata structure, the Cover Trees..6 Thesis Overview The remainer of this thesis is structure as follows. The next chapter is evote to solving the NN problem. It iscusses the properties that woul be require of an ieal solution, an the apparent ifficulties of arriving at one. It also lists an escribes many of the methos propose so far, incluing the ones that were selecte for evaluation, along with the reasons of their selection. The chapter that follows next (Chapter 3) elves more eeply into the etails of each of the selecte methos. Chapter outlines the strategy use for evaluating these methos an presents the results. First, the various construction methos of the two popular algorithms, KDTrees an Metric Trees, are compare, an then the best of their construction methos are cross-evaluate against each other an against the rest of the selecte methos. The thesis conclues with some comments an remarks in Chapter 5. 5

34

35 Chapter Towars Solving the NN problem Given n -imensional ata points, a simple linear search (the brute force metho) takes O(n) time to fin an exact NN of a given query point, while taking O(n) space an only O() preprocessing time (since no preprocessing is performe). Hence, in orer to o better than that, a ata structure or an algorithm must try to fin a NN in time sublinear in either or n, or both an n. Moreover, it shoul try to achieve that in space an preprocessing time that is linear or near linear in an n. Since intuitively it s far simpler to eliminate points rather than trying to reuce the number imensions looke at uring a NN search, all known NN search techniques with the exception of two, PDS & TV-Tree, try to achieve a query time sublinear in n. In terms of n, ieal solutions exist with logarithmic query time, an near linear preprocessing an linear space requirements, for. For =, a simple binary search on a sorte array gives O() query time in the worst case, while requiring only O(n) space an O(n) preprocessing time (which equates to time require for sorting the array in this case). For =, O() query time in the worst case, with linear space an near linear preprocessing time, is possible using methos base on Voronoi iagrams (Lee, 9; Aurenhammer, 99). However, for >, no known solution exists that can guarantee a sublinear query time while still keeping the space complexity linear an the preprocessing time near linear. Still, for moerate values of ( ), algorithms exist that, though not in the worst case, but in practice (i.e. query time, with linear space an near linear preprocessing. expecte case) give sublinear However, for higher values of ( ) almost all of the known techniques are plague with what is calle the curse-of-imensionality. Roughly speaking what this means is that for points which are istribute somewhat uniformly, as their imensionality increases, they become more an more sparse/scattere, an ten to lie more towars the bounaries of the space, making them almost equally istant from each other. This breaks own the iea of locality of Time to fin (k)nns of given a query q, here an elsewhere. Originally coine by Bellman in 96 in Aaptive Control Processes: A Guie Tour, Princeton University Press, 96. 7

36 the points, since the local neighbourhoos become so large that they ten to comprise the entire point space. As a consequence most of the propose algorithms, when searching for the nearest neighbour of a given query, egenerate to simple linear search, since (almost) all the ata points have to be looke at, an often they take longer than the linear search itself ue to their involve overheas. The curse was the main motivation behin algorithms for ɛ-nn search, an also the newer algorithms that are base on the concept of intrinsic imensionality (efine below) of the ata. A more meticulous coverage of the curse can be foun in (Hastie et al., 00), (Katayama & Satoh, 00) an (Fayya et al., 996). Also, there is some ebate regaring the usefulness of the NN problem at such high imensions (Beyer et al., 999; Hinneburg et al., 000). In practical problems it often turns out that the ata points, even though being embee in a high imensional space, are not so wiely scattere after all. Due to the epenencies among the imensions (i.e co-relation among the attributes/features of the objects) they are usually clustere in some lower imensional subspace. Inee, many of the algorithms that shoul break own in such cases (e.g. KDTrees, Metric Trees etc.) sometimes o not an work better than expecte. Such low-imensional nature in high imensional embeings is calle the intrinsic imensionality of the points. Some recently propose techniques such as Cover Trees an Navigating Nets try to exploit this peculiar nature common in real worl ata to circumvent the curse. A number of imensionality reuction techniques are also common in practice to mitigate the effect of the curse, even though the inter-point istances are not exactly preserve after the reuction. In the ocument/information retrieval fiel some sort of imensionality reuction technique is essential, since it s often the case that n. Ranom Projections, Singular Value Decomposition (SVD), Principal Components Analysis (PCA), an Latent Semantic Inexing (LSI) (which is specific to IR), are some of the techniques that are generally use.. Techniques Propose as Solution Most of the techniques propose for (k)nn an ɛ-(k)nn search, that are applicable to vector/point representation an try to achieve query time sublinear in n, are base aroun two general traitions of solving the problem. They either partition/ecompose the point space to reuce the points looke at by eliminating regions unlikely to contain a NN, or they project the vectors to one or more scalar values an somehow try to reuce the points looke at using the scalar projections of ata an query vectors. Those of such

37 Exact NN Approx NN Base on Space Partitioning. KDTrees 6. X-Trees. Metrics Trees 7. M-Trees 3. vp-trees. SR-Trees. Voronoi Diagrams 9. TV-Trees 5. R-Trees. VA-Files Base on Scalar Projection. Orchar s Metho. LAESA. Metho 5. LSH 3. AESA Base on Intrinsic Dimensionality. Cover Trees Base on Space Partitioning. BBF-Trees (KDTrees with Priority Queue). BBD-Trees 3. Hybri Sp-Trees Base on Scalar Projection. LSH Base on Intrinsic Dimensionality. Navigating Nets Table.: Exact an approx. NN techniques techniques that have capture the interest of other researchers, an the novel ones base on the iea of trying to exploit the intrinsic imensionality of the ata, are summarize in Table.. The only known techniques that try to achieve query time sublinear in are Partial Distance Search (PDS) (also known as Partial Distance Calculation) an TV- Trees. TV-Trees, however, try to reuce not only but also n (hence are also mentione in Table.). These an the techniques in Table. are escribe in brief in the paragraphs below... Partial Distance Search (PDS) Originally propose by Bei an Gray (Bei & Gray, 95) it provies only moerate acceleration on its own. However, it is extremely simple an is general enough to be applie in conjunction with almost any other technique known for NN search. The technique works by stopping the calculation of a point s istance to the query, if at any point uring the calculation, the accumulate istance of the point becomes larger than the istance of the best (k th )NN, encountere among all the points so far looke at uring the search. So, for example, for any Minkowsky-p (L p ) metric (e.g. Manhattan ( i= (L ) metric an Eucliean (L ) metric) of the form: (x, y) p = x i y i p) /p, it will skip the sum calculation for all imensions > i, if at i the sum becomes larger than the overall sum of the best (k th )NN among all the points so far encountere. 9

38 The technique can be applie to any other issimilarity measure that accumulates or as values for ifferent imensions uring calculation of the istance... k-dimensional Trees (KDTrees), BBF-Trees an Variants Multiimensional binary search trees, calle in short by the author as KDTrees (where k is the imensionality of the space), were originally propose by Bentley in 975 (Bentley, 975) for associative retrieval of recors in a file. Their potential for NN search was observe by Bentley, an hence were quickly aopte for NN searching, with an optimize version by Frieman, Bentley an Finkel in 977 (Frieman et al., 977). Since then, these trees are by far the most popular search technique employe for NN search. The trees hierarchically partition the point space into mutually exclusive rectangular regions by recursively splitting it with axis-parallel hyperplanes. The splitting is binary, with each non-terminal internal noe splitting a region into two sub regions. The search for (k)nn is carrie out recursively, with the region containing the query point being searche first an then only those of the remaining ones which are likely to contain the (k th )NN. More specifically, after recursively narrowing own to the region of a leaf noe containing the query, the points insie the region are looke at, an then a ball (hypersphere to be exact), centre at the query an with raius equal to the query s istance to the best (k th )NN foun so far, is compute. Afterwars uring backtracking only those regions which intersect with this query ball are searche, an the ball is upate each time a better (k th ) NN is encountere in another region. The trees require ata in vector representation. They utilize this representation very efficiently an o not require any istance computation either uring construction or uring much of NN search (istance computations are performe only when looking at points insie a region of a leaf uring the NN search). During both processes they only look at the value of a point s imension that is orthogonal to the hyperplane use to split a region. The original version propose by Frieman, Bentley, an Finkel (Frieman et al., 977) requires O(n) construction (preprocessing) time, an O(n) storage. For a given query it takes O() time in the expecte case for moerate imensions. The query time, however, usually grows exponentially in, an the tree, suffering from the curse-of-imensionality, usually egenerates to simple linear search at higher imensions (with slightly higher query time). The original version is general enough to be applie even to non-metric istance measures, an requires measures to satisfy only a few constraints given by the authors. Still, however, it is only known to be evaluate an foun efficient

39 3 6 7 y [] 9.0<= 9.0> x [] x [3] 76.0<= 76.0> 76.0<= 76.0> y [] [5] y [] y [] 95.5<= 95.5> 6.5<= 6.5> 6.5<= 6.5> 7 x [6] [7] [] [3] [] x [5] 5.0<=.0> 6.0<= 6.0> [] [9] [6] [7] 9 q (a) (b) Figure.: (a) A typical imensional KDTree/BBFTree with (b) its graphical representation. The numbers in the rectangular regions in (b) correspon to the inices (given in brackets) of the leaf noes in (a). for Minkowsky-p metrics. Since the initial version a number of ifferent constructions methos, moifications, an enhancements have been propose for the trees by ifferent researchers. All, however, fall far short of removing the curse-of-imensionality an all, from what is known, have only been evise with at most Minkowsky-p metrics in view. Best-Bin-First (BBF) trees, evelope by (Arya & Mount, 993) an inepenently by (Beis & Lowe, 997), are a moification of KDTrees for ɛ-nn search. Using a priority queue, they visit those regions first uring back tracking that are nearer to the query point, an terminate the search early if the istance to the nearest region in the queue becomes greater than /( + ɛ) times the istance to the best encountere (k th )NN (i.e. > /( + ɛ) the raius of the query ball). As an example, consier Fig..(b). For a given query q, after searching region 9, a typical KDTree woul then search the leaf noe of region 7, which is the first one encountere uring backtracking as can be seen in Fig..(a). A BBF-Tree on the other han, woul first go on to search leaf 5 which is nearer to the query. If L is a region associate with a noe an r the raius of the query ball (equal to the istance of the best encountere (k th )NN), for a given istance measure M a KDTree woul carry on the search until for all remaining noes M(q, L ) > r, whereas a BBF-Tree woul terminate the search if for all remaining noes M(q, L ) > /( + ɛ) r. This way it can be guarantee that any (k th ) neighbour foun is at most by a factor of ɛ further than the true (k th ) nearest neighbour (see fig..).

40 (a) (b) Figure.: A (a) KDTree vs a (b) BBD-Tree on the same ata. A slight variation of BBF-Trees/KDTrees are the Balance Box Decomposition (BBD) trees (Arya et al., 99). The only ifference between these an BBF-Trees/KDTrees is that for some noes in BBD-Trees, their associate region is shrunk into two hyperrectangles (base on some shrinkage rule) instea of being split. The resulting two hyperrectangles after shrinking are set theoretic ifferences of each other an one of them completely encloses the other. These structures guarantee worst case ɛ-nn query time of O(c,ɛ ), where c,ɛ 6/ɛ. However, some new enhancements to BBF/KDTrees by the same authors have remove the performance isparity between BBF/KDTrees an BBD- Trees, an the former are not known to perform significantly worse than the latter. The ifference between the two trees is illustrate graphically in Fig....3 Metric (Ball) Trees, vp-trees an Hybri Sp-Trees Metric trees, which are also known as Ball Trees, were presente by Omohunro (Omohunro, 99) an Uhlmann (Uhlmann, 99a,b). These trees hierarchically partition the point space into (hyper)spherical regions. These regions, unlike the ones in KDTrees, are not mutually exclusive an are allowe to overlap. The points ivie among the regions, however, are not allowe to overlap an can only belong to one region or one of its sub regions. Like in KDTrees, the partitioning is binary, an also like in KDTrees, the search for a (k)nn is one recursively, with the region containing (or in some cases nearest to) the query being examine first, followe by those regions that intersect with the query ball (i.e. the ball centre at the query, with raius equal to the query s istance to the

41 Figure.3: A typical metric tree ecomposition. best (k th )NN encountere so far, just like in KDTrees). Unlike KDTrees, they o not require ata to be in vector form. They are general enough to be applie to any ata representation as long as it is in a metric space. The trees can be constructe in a number ifferent of ways, given by a number of ifferent people, with Omohunro (Omohunro, 99) alone giving 5 ifferent construction methos. Most of the propose construction methos work in O(n) time, while the space requirement regarless of the construction metho, is always O(n). No bouns for query time are known to be given for any of the construction methos, an also no evaluation of the trees seems to have ever been carrie out against establishe NN search methos such as KDTrees. In spite of that these trees have been the centre of attention of researchers recently (Moore, 000; Liu et al., 00; Liu et al., 005), an are even claime to be the state-of-the-art so far for exact NN search in moerately high imensions (Liu et al., 005); though without much theoretical or empirical unerpinning. Like KDTrees, these trees are also known to suffer from the curse-of-imensionality at higher imensions. Fig.3 illustrates graphically a typical metric tree ecomposition of -imensional ata. Yianilos inepenently evelope a slight variation of metric trees, which he calle Vantage Point Trees (vp-trees) (Yianilos, 993). These trees ivie the point space using spherical cuts into mutually exclusive (usually) semi-spherical regions. Each internal noe is associate with a ata point, calle the vantage point, which is the centre of the spherical cut use to ivie a (sub)space. The vantage point is usually chosen from one of the corner points in the (sub)space. Fig. graphically illustrates a vp-tree ecomposition. Liu et al. (Liu et al., 005) note that uring (k)nn search of a given query a metric tree can spen up to 95% of the time in backtracking, i.e. after fining a goo initial 3

42 Figure.: Illustration of a vp-tree ecomposition. caniate for the (k th )NN in a leaf noe, up to 95% of its total time can be taken in making sure it has got the correct answer. To avoi this behaviour, they suggeste a variation of metric trees that reports an approximate answer, that is an ɛ-(k)nn, but in a much shorter time. They calle this variant Spill Tree (Sp-Tree). The key iea behin Sp-Tree is to allow ata points that are at the split bounary to spill over between the two sub regions each time a region is ivie, so that both sub regions contain the points near the bounary. The amount of spill-over is controlle by a istance parameter τ, which allows ɛ guarantees of ɛ-(k)nn. As note by the authors, the Sp-Tree on its own is not pratically viable, as the istance parameter τ can introuce cases where a large number of points overlap an after ivision of a region both sub regions contain the same number of points as the parent region, thereby making the sub-ivision routine go into an infinite loop/recursion. Hence, for practical purposes they suggeste a Hybri Sp-Tree, a cross between Metric Trees an Spill Trees, that contains both overlapping an non-overlapping regions, an makes sure that at least a certain number of points of a region are ivie among the sub regions. It marks a region overlapping only if after ivision with parameter τ both of its sub regions contain < ρ of its total number of points (the authors set ρ to 70%), otherwise it is marke non-overlapping. The search for a (k)nn in a Hybri Sp- Tree is also carrie out in a hybri fashion, with backtracking being skippe only for overlapping regions an not for non-overlapping ones. These structures work very well for ɛ-(k)nn, even in high imensions. The authors in (Liu et al., 005) also presente an empirical evaluation of the structures against the initial version of LSH (iscusse below

43 in section..9), the only technique known to work well in high imensions, an have shown Hybri Sp-Trees to perform consistently better on a number of real-worl atasets... Voronoi Diagrams Voronoi iagrams are of practical importance in a number of fiels, which inclue, in aition to computer science, meteorology, physics, astronomy, archaeology an biology, e.g. see (Okabe et al., 000). The iagrams have a long history of use, with their earliest use being trace back to 6 when they were use by the well known French philosopher an mathematician René Descartes. Their general -imensional case was stuie an formalize by Russian mathematician Georgy Feoseevich Voronoy, whom they are name after. These are are also known as Voronoi tessellations or Dirichlet tessellations. Voronoi iagrams partition a point space into convex polygons. The polygons are constructe in such a way that each polygon is associate with a single ata point an any point that woul be closer to this ata point than any other woul lie in this polygon. The NN search then just becomes a search for the region that contains the query point, an, because of the properties of Voronoi iagrams, the ata point associate with that region is guarantee to be the NN of the query. For knn search with k >, an orer-k Voronoi iagram of the ata is compute. In orer-k Voronoi iagrams, each polygonal partition of the point space is associate with exactly k ata points, an any point falling in that region is closer to those k points than any other k points among the ata. Figure.5 graphically illustrates an orer- Voronoi space ecomposition. These structures are computationally only efficient for at most =, for which they require O(n) preprocessing an O(n) space. For values higher than, their complexity grows exponentially in (O(n / ) accoring to (Arya et al., 00)). The NN search on Voronoi iagrams takes only O() time in the worst case, while knn search takes only O( + k) time. The search is one using an approach for planar point location in straight line graphs. For a long time, a major rawback of this technique was that k for knn search neee to be known in avance before preprocessing, an ha to remain fixe for all queries. However, an approach evelope by Aggarwal et al. (Aggarwal et al., 990) for compacting orer-k Voronoi iagrams, allows all possible orer-k Voronoi iagrams (i.e. for k =...n ) to be store in O(n) space, while still guaranteeing O( + k) query time. An extensive survey of these structures is available in (Aurenhammer, 99) an (Okabe et al., 000). 5

44 Figure.5: An orer- Voronoi space ecomposition...5 Orchar s Metho Orchars s metho (Orchar, 99) was esigne specifically for Vector Quantization. It works by selecting a point p as a caniate NN, computing its istance r to the query q, an then eliminating all the points that are further than r from p, thereby exploiting the geometrical property that they can not be nearer to q than p (See Fig..6). After elimination, the metho looks for a neighbour better than p, in the points remaining, within the circle of raius r centre at p (hypersphere in higher imensions). If a neighbour better than p is foun, the elimination proceure is applie again, an again a better neighbour is sought, in the new circle centre at the new best neighbour (with raius equal to twice the new neighbour s istance to the query). The selection of point p for the first query is arbitrary, an for subsequent queries, it turns out in image compression, that the best caniate is the NN of the last query processe, since a goo initial caniate NN translates to faster performance. The metho requires for the elimination roun compute (an sorte) istances from the point selecte as a caniate NN to all the rest of the points. Since for the first an all the subsequent queries, any point can be selecte as a caniate NN (which can subsequently be upate), it effectively requires all inter-ata point istances to be compute at preprocessing, an thus requires O(n ) space an O(n ) pre-processing time; making it impractical for all but the smallest of NN problems. A slight variation of the metho computes istances of all the points to only m other points, an requires only O(mn) time an space. If uring search a nee arises to go past the m th item on the sorte istance lists, it simply resorts to simple linear search. The original version (the 6