k-nn Graph Construction: a Generic Online Approach

Transcription

1 k-nn Graph Construction: a Generic Online Approach Wan-Lei Zhao arxiv:80.00v [cs.ir] Sep 08 Abstract Nearest neighbor search an k-nearest neighbor graph construction are two funamental issues arise from many isciplines such as information retrieval, ata-mining, machine learning an computer vision. Despite continuous efforts have been taken in the last several ecaes, these two issues remain challenging. They become more an more imminent given the big ata emerges in various fiels an has been expane significantly over the years. In this paper, a simple but effective solution both for k-nearest neighbor search an k-nearest neighbor graph construction is presente. Namely, these two issues are aresse jointly. On one han, the k-nearest neighbor graph construction is treate as a nearest neighbor search task. Each ata sample along with its k-nearest neighbors are joine into the k-nearest neighbor graph by sequentially performing the nearest neighbor search on the graph uner construction. On the other han, the built k- nearest neighbor graph is use to support k-nearest neighbor search. Since the graph is built online, ynamic upating of the graph, which is not esirable from most of the existing solutions, is supporte. Moreover, this solution is feasible for various istance measures. Its effectiveness both as a k-nearest neighbor construction an k-nearest neighbor search approach is verifie across various atasets in ifferent scales, various imensions an uner ifferent metrics. Inex Terms k-nearest neighbor graph, nearest neighbor search, high-imensional, hill-climbing I. INTRODUCTION Given a ataset S, k-nn graph refers to the structure that keeps top-k nearest neighbors for each sample in the ataset. It is the key ata structure in the manifol learning [], [], computer vision, machine learning an information retrieval, etc []. Due to the funamental role it plays, it has been stuie for several ecaes. Basically, given a metric, the construction of k-nn graph is to fin the top-k nearest neighbors for each ata sample. When it is built in brute-force way, the time complexity is O( n ), where is the imension an n is the size of ataset. Due to the eminence of big ata issue in various contexts, both an n coul be very large. As a result, it is computationally prohibitive to buil an exact k-nn graph in an exhaustive manner. For this reason, works in the literature [] [6] only aim to search for an approximate but efficient solution. Despite numerous progress has been mae in recent years, the major issues latent in k-nn graph construction still remain challenging. First of all, many existing approaches only perform well on low-imensional ata. An the scale of ata they are assume to cope with is usually less than one million. Moreover, most of approaches are esigne uner specific metric i.e., l -norm. Only recent few works [], [7], [8] aim to Fujian Key Laboratory of Sensing an Computing for Smart City, Xiamen University, Fujian, China. wlzhao@xmu.eu.cn aress this issue in the generic metric spaces. Thanks to the introuction of in [], the construction complexity has been reuce from O(n.9 ) [7] to O(n. ) for ata with meium imensions (e.g., ) []. However, the performance of turns out to be unstable for ata with high intrinsic imension. Besies the major issues aforementione, many existing approaches still face another potential issue. In practice, it is possible that the ataset changes from time to time. This is especially the case for large-scale Internet applications. For instance, the photos an vieos in Flickr grows on a aily basis. In visual object tracking [9], the new object templates are joine into the caniate set, an the obsolete templates shoul be swappe out, as the tracking continues. In these scenarios, one woul expect the k-nn graph that works behin shoul be upate from time to time. Unfortunately, for most of the existing approaches, the ataset is assume to be fixe. Any upate on the ataset invokes a complete reconstruction on the k-nn graph. As a consequence, the aggregate cost is high even the ataset is in small-scale. It is more convenient if it is allowe to simply insert/remove the samples into/from the existing k-nn graph. Nevertheless, it is complicate to upate the k-nn graph with the support of conventional inexing structure such as locality sensitive hashing [0], R-Tree [] or k- tree []. Recent stuy [] shows that it is possible to buil k-nn graph incrementally by invoking k-nearest neighbor search irectly on an existing k-nn graph. Unfortunately, limite spee-up (0 to 0 times) is observe in []. In orer to support fast inexing, k-meois is calle in the approach to partition the samples that are in k-nn graph, which becomes very slow when both an n are large. Moreover, since it is built upon k-meois, it only works in certain metric spaces. Another problem that is closely relate to k-nn graph construction is k-nearest neighbor search (k-nn search), which, like k-nn graph construction, arises from a wie range of applications. The nearest neighbor search problem is efine as follows. Given a query vector (q R ), an n caniates in S that are uner the same imensionality. It is require to return sample(s) for the query that are closest to it accoring to a given metric. Traitionally, this issue has been aresse by various space partitioning strategies. However, these methos are harly scalable to high imensional (e.g., > 0), large-scale an ense vector space. In such case, most of the traitional approaches such as k- tree [], R-tree [] an locality sensitive hashing (LSH) [0] are unable to return ecent results. Recently, there are two major trens in the literature that

2 aim to aress this issue. In one irection, k-nn search is unertaken base upon vector quantization [] [6]. The primary goal of this way is to compress the reference set by vector quantization. Such that it is possible to loa the whole reference set (after compression) into the memory in the case that the reference set is extremely large. The istance between query an the reference set is measure in the compresse space, which is more efficient than it is unertaken in the original space. Due to the quantization loss, high accuracy is unesirable from this type of approaches. Alternatively, another more promising way is to conuct the k-nn search base on an approximate k-nn graph [], [7], [8] or the like [9], [0] with hill-climbing strategy []. In this paper, a generic k-nn graph construction approach is presente. The k-nn graph construction is treate as a k-nn search task. The k-nn graph is incrementally built by invoking each sample to query against the k-nn graph uner construction. After one roun of k-nn search, the query sample is joine into the graph with the resulting topk nearest neighbors. The k-nn lists of samples (alreay in the graph) that are visite uring the search are accoringly upate. The k-nn search basically follows the hill-climbing strategy []. In orer to achieve high performance in terms of both efficiency an quality, three major innovations are propose. The hill-climbing proceure is unertaken on both k- NN graph an its reverse k-nn graph. In orer to avoi the high cost of converting the intermeiate k-nn graph to its reverse k-nn graph each time, the ata structure orthogonal list is aopte, in which k-nn graph an reverse k-nn graph are maintaine as a whole. To further boost the performance, a lazy graph iversification (LGD) scheme is propose. It helps to avoi unnecessary istance computations uring the hill-climbing while involving no aitional computations. Moreover, recursive neighborhoo propagation (RNP) is propose to introuce the new sample to its most likely neighbors, which enhances the quality of k-nn graph consierably. The avantages of this approach are several fols. Firstly, the online construction avois repetitive istance computations that most of the current k-nn graph construction approaches suffer from. Seconly, online construction is particularly suitable for the scenario that ataset is ynamically changing. Moreover, our approach has no specification on the istance measure, it is therefore a generic solution, which is confirme in our experiments. Thanks to aforementione two innovations, the k-nn search turns out to be very cost-effective. When turning off the graph upate operations, it is also an effective k-nn search algorithm. Namely, the problems of k- NN graph construction an k-nn search have been jointly aresse in our solution. The remaining of this paper is organize as follows. In Section II, a brief review about the research works on k-nn graph construction an approximate k-nn search is presente. Section III presents an enhance hill-climbing algorithm upon which the k-nn graph construction approach is built. Section IV presents two online k-nn graph construction approaches, which are NN search approach as well when turning off the graph upate operations. The experimental stuies over the k-nn graph construction an k-nn search are presente in Section V. Section VI conclues the paper. A. k-nn Search II. RELATED WORKS The early stuy about the k-nn search issue coul be trace back to 970s, when the nee of NN search on the file system arises. In those ays, the ata to be processe are in very low imension, typically D. This problem is well-aresse by B-Tree [] an its variant B + -tree [], base on which the NN search complexity coul be as low as O(log(n)). B-tree is not naturally extensible to more than D case. More sophisticate inexing structures were esigne to hanle NN search in multi-imensional ata. Representative structures are k--tree [], R-tree [] an R*-tree []. For k- tree, pivot vector is selecte each time to split the ataset evenly into two. By applying this bisecting repeately, the hyper-space is partitione into embee hierarchical subspaces. The NN search is performe by traversing over one or several branches to probe the nearest neighbors. Unlike B-tree in D case, the partition scheme oes not exclue the possibility that nearest neighbor resies outsie of these caniate sub-spaces. Therefore, extensive probing over the large number of branches in the tree becomes inevitable. For this reason, NN search with k- tree an the like coul be very slow. Recent inexing structure FLANN [], [5] partitions the space with hierarchical k-means an multiple k- trees. Although efficient, sub-optimal results are achieve. For all the aforementione tree partitioning methos, another major isavantage lies in their heavy eman in memory. On one han, in orer to support fast comparison, all the caniate vectors are loae into the memory. On the other han, the tree noes that are use for inexing also take up consierable amount of extra memory. Overall, the memory consumption is usually several times bigger than the size of reference set. Aiming to reuce the memory consumption, quantization base approaches [], [6], [6] [8] compress the reference vectors by quantization [9]. For all the quantization base methos, they share two things in common. Firstly, the caniate vectors are all compresse via vector (or sub-vector) quantization. This makes it easier than previous methos to hol the whole ataset in the memory. Seconly, NN search is conucte between the query an the compresse caniate vectors. The istance between query an caniates is approximate by the istance between query an vocabulary wors that are use for quantization. Due to the heavy compression on the reference vectors, high search quality is harly esirable. Furthermore, this type of approaches are only suitable for metric spaces of l p -norm. Apart from above approaches, several attempts have been mae to apply LSH [0], [0] on NN search. In general, there are two steps involve in the search stage. Namely, step. collects the caniates that share the same or similar hash

3 keys as the query; step. performs exhaustive comparison between the query an all these selecte caniates to fin out the nearest neighbor. Similar as FLANN, computational cost remains high if one expects high search quality. Aitionally, the esign of hash functions that are feasible for various metrics is non-trivial. Recently, the hill-climbing (HC) strategy that performs NN search base on k-nn graph has been also explore [], [8], [7], [9], []. The hill-climbing starts from a group of ranom sees (ranom locations in the vector space). The search traverses iteratively over an approximate k-nn graph (built in avance) by best-first search. Guie by the k-nn graph, the search proceure ascents closer to the true nearest neighbor in each roun until no better caniates coul be foun. Approaches in [], [9] [] follow similar search proceure. The major ifference lies in the strategies that are use to buil the graph. Accoring to recent stuy [], these graph base approaches emonstrate superior performance over other types of approaches across variety types of ata. B. k-nn Graph Construction The approaches for k-nn graph construction can be roughly groupe into two categories. Approaches such as [6], [] basically follow the ivie-an-conquer strategy. On the first step, samples are partitione into a number of small sub-sets, exhaustive comparisons are carrie out within each sub-set. The closeness relations (viz. eges in the k-nn graph) between any two samples in one sub-set are establishe. In the secon step, these closeness relations are collecte to buil the k-nn graph. To enhance the performance, the first step is repeate for several times with ifferent partitions. The prouce closeness relations are use to upate the k-nn graph. Since it is har to esign partition scheme that is feasible for various generic spaces, they are generally only effective in l -space. Another category of k-nn Graph Construction, typically NNescent [] avois such isavantage. The graph construction in NN-escent starts from a ranom k-nn graph. Base on the principle neighbor s neighbor is likely to be the neighbor, the k-nn graph evolves by invoking comparisons between samples in each sample s neighborhoo. Better closeness relations that are prouce in the comparisons are use to upate the neighborhoo of one sample. This approach turns out to be generic an efficient. Essentially, it can be viewe as performing hill-climbing [] batchfully. Recently, the mixture scheme erive from the above approaches is also seen in the literature []. It is worth noting that approaches propose in [8], [9], [0], [] are not k-nn graph construction algorithms. The graphs are built primarily for k-nearest neighbor search. In these approaches, the samples which shoul be in the k-nn list of one sample are eliberately omitte for comparison efficiency. While the links to remote neighbors are maintaine [9], [0]. As a consequence, graphs constructe by these approaches are not k-nn graph in the real sense. Such kin of graphs are harly supportive to tasks beyon k-nn search. In most of the approaches aforementione, one potential issue is that the construction proceure has to keep recors (b) -NN graph G for (a) 5 5 (a) a graph with vertices (c) reverse -NN graph G for (a) Fig. : An illustration of k-nn graph G an its reverse k-nn graph G. In the illustration, k is set to. on the comparisons that have been mae between sample pairs to avoi repetitive comparisons. However, its space complexity coul be as high as O(n ). Otherwise the repetitive comparisons are inevitable even by aopting specific sampling schemes []. In this paper, the k-nn graph construction an k- NN search are aresse jointly. The k-nn graph construction is unertaken by invoking each sample as a query to query against the k-nn graph that is uner construction. Since the query is new to the graph uner construction each time, the issue of repetitive comparisons is overcome. More interestingly, we iscover that the k-nn graph construction an k-nn search are beneficial to each other. Namely, high quality k-nn search leas to the high quality of intermeiate k-nn graphs. In turn, the efficiency an quality of k-nn search are guarantee with the support of high quality of intermeiate k-nn graphs. III. BASELINE SEARCH ALGORITHM Since our k-nn graph construction approach is base on k-nn search. Before we introuce the construction approach, the baseline k-nn search algorithm that is base on a prebuilt k-nn graph is presente. It essentially follows hillclimbing strategy [], while consierable moifications have been mae. Algorithm : Enhance Hill-Climbing NN search (EHC) Input: q: query, G: k-nn Graph, G: reverse k-nn Graph, S n : reference set Output: Q: k-nn list of q : Q ; Flag[ n] 0; : while upate o : R[ p] p ranom sees in [0, n) : for each r R o 5: insertq(r, m(q, r), Q); 6: en for

4 en 7: for r top(q) o 8: if Flag[r] == 0 then 9: for each n i G[r] o 0: InsertQ(n i, m(q, n i ), Q); : en for : for each n j G[r] o : insertq(n j, m(q, n j ), Q); : en for 5: en if 6: Flag[r] = ; 7: en for 8: it = it + ; 9: en while Given k-nn graph G, G[i] returns k-nn list of sample i. Accoringly, G is the reverse k-nn graph of G, which is nothing more than a re-organization of graph G. G[i] keeps ID of samples that sample i appears in their k-nn lists. Notice that the size of G[i] is not necessarily k an there woul be overlappings between G[i] an G[i]. An illustration about graphs G an G are seen in Fig.. With the support of k-nn graph G an its reverse graph G, the baseline search algorithm is presente in Alg.. As shown in Alg., the hill-climbing starts from p ranom sees, the compare samples are kept in Q, which is sorte in ascening orer all the way. The neighbors an reverse neighbors of sample r, which are not visite an ranke closest to the query, are compare sequentially (Line 5 9). Notice that Q is upate as soon as a closer sample is foun. The istance function m( ) coul be any istance metric efine on the input ataset. This search algorithm is generic in the sense that there is no specification or assumption relate to the istance metric. This is also true for the k-nn construction algorithm that will be presente afterwars. The iteration (Line ) continues until no closer sample r is ientifie. Although this algorithm starts from ranom location, only minor performance fluctuation is observe across ifferent runs. In orer to avoi repetitive comparisons, variable Flag keeps the status whether a sample has been expane. The major ifference between Alg. an original hillclimbing [] lies in the use of both k-nn graph an its reverse k-nn graph. Thanks to the introuction of reverse k-nn graph, its performance is alreay very competitive in comparison to the state-of-the-art approaches [9], [0], []. IV. ONLINE k-nn GRAPH CONSTRUCTION The prerequisite for k-nn search algorithm in Alg. are the k-nn graph G an its reverse k-nn graph G. In this section, we are going to show how a k-nn Graph an its reverse Graph are built base on k-nn search algorithm itself. Aitionally, a more efficient scheme for k-nn search an in turn for k-nn graph construction is presente. Moreover, a strategy calle recursive neighborhoo propagation is propose to further enhance the quality of k-nn graph. Without the loss of generality, it is assume that the smaller of the istance the closer of two samples across the paper for escription convenience. A. k-nn Construction Graph by Search In Alg., when the hill-climbing is conucte, the comparison to the caniates starts from several ranom locations, moving along several trails, the search moves towars the closer neighborhoo of a query. In the ieal case, top-k nearest neighbors will be iscovere. On one han, the top-k nearest neighbors of this query are known after the search. On the other han, this query will be joine into the neighborhoos of samples being visite as a new neighbor uring the search. As a result, a k-nn graph is augmente to inclue this new sample. Motivate by this observation, the online construction algorithm is conceive. Generally, there are two major steps in the algorithm. Firstly, an initial graph is built exhaustively from a small subset of S. The size of S is fixe to 56 across the paper. In the secon step, each of the remaining samples is treate as query to query against the graph following the flow of Alg.. Each time, the k-nn list of one sample is joine into the graph being rightnow searche over. The k-nn lists of samples which have been compare uring the search are accoringly upate. The general proceure of the construction algorithm is summarize in Alg.. en Algorithm : Online k-nn Graph Constr. () : Input: S n : ataset; k: num. of NN; p: num. of sees : Output: G: k-nn Graph : Q ; Flag[ n] 0; : Extract a small subset I from S; 5: Initialize G an G with I; 6: G = G G; 7: for each q S I o 8: while upate o 9: R[ p] p ranom sees in [0, n) 0: for each r R o : InsertQ(r, m(q, r), Q); : en for : for r top(q) o : if Flag[r] == 0 then 5: for each n i G[r] o 6: InsertQ(n i, m(q, n i ), Q); 7: InsertG(n i, q, m(q, n i ), G); 8: en for 9: for each n j G[r] o 0: InsertQ(n j, m(q, n j ), Q); : InsertG(n i, q, m(q, n i ), G); : en for : en if : Flag[r] = ; 5: en for 6: en while 7: for each r Q o 8: InsertG(q, r, m(q, r), G); 9: en for 0: Q ; Flag[ n] 0; : en for

5 5 r a e b k-nn List Reverse k-nn List Fig. : An illustration of orthogonal list G for Fig. (a), which is the union of G an G. As shown in the figure, two links are kept for each vertex, they point to the k-nn list an reverse k-nn list of a vertex respectively. In our implementation, the samples in k-nn list are kept in ascening orer accoring to their istances to the vertex. While the samples in the reverse k-nn list are kept in the orer of being inserte. c Fig. : An illustration of occlusion in D l -space happens in the neighborhoo of sample r. Samples a, b, c, an e are in the k-nn list of r. m(r, b) is greater than m(r, a), while m(a, b) is smaller than m(r, b), we say that b is occlue by a in r s neighborhoo, while sample c an are not occlue by b. Actually all the samples locate in the moon shape shaow are occlue by a. Notice that the region that is occlue by a coul be beyon this moon shape region. In orer to support efficient search an the k-nn graph construction, an orthogonal list G (as shown in Fig. ) is aopte to keep the both G an G (Line 8) in the implementation. As shown in Fig., two linke lists are kept for one sample, one is k-nn list an another is reverse k-nn list. While for clarity, graph G an G are still referre in our escription. In Alg., the proceure of k-nn graph construction is basically a repetitive call of the search algorithm in Alg. (Line 0 6). The major ifference is that function insertg(a, b, m(a, b), G) is calle after the query is compare to a sample in the graph. Function insertg( ) is responsible for inserting an ege into k-nn list of a in graphs G an G. The major operations insie insertg( ) involve upating the k-nn list of a an the reverse k-nn list of b. Sample in the rear of k-nn list is elete if a closer sample is joine in. Distance m(a, b) is kept in k-nn list of a, which allows the list to be sorte all the time. In the en of each loop, insertg( ) is calle again to join the k-nn list of query q into G. Although the size of input ataset is fixe in Alg., it is apparently feasible for an open set, where new samples are allowe to join in from time to time. As will be reveale in the experiments (Section V), Alg. alreay performs pretty well. In the following, two novel schemes are presente to further boost its performance. B. Lazy Graph Diversification In Alg., when expaning sample r in the rank list Q, all the samples in the neighborhoo of r will be compare with the query. Accoring to recent stuies [8], [0], [], when samples in the neighborhoo of r are so close to each other that the istances between them are smaller than their istances to r, it is no nee to compare to all of them uring the expansion. The expansion on these close samples later will likely lea the climbing process to the same local region. The phenomenon that samples in the k-nn list are closer to each other than they are to r is calle as occlusion [8]. An illustration of occlusion is shown in Fig.. In the illustration, samples b an e are occlue by sample a. It is easy to see one sample can only be occlue by samples which are closer to r than that of it. Accoring to [8], [0], [], the hill-climbing will be more efficient when samples like b an e are not consiere when expaning r. In orer to know whether samples in a k-nn list are occlue one by another, the pair-wise comparisons between samples in the k-nn list are require, which is the practice in [8], [0], []. This is unfeasible for an online construction proceure (i.e., Alg. ). First of all, k-nn list is ynamically changing, pair-wise istances cannot be simply compute an kept once for all. Seconly, it is still too costly to upate the pair-wise occlusion relations as long as a new sample is joine, which involves a complete comparison between the new neighbor an the rest. Moreover, the occlue samples cannot be simply remove from a k-nn list since our primary goal is to buil the k-nn graph. In this paper, a novel scheme calle lazy graph iversification is propose to ientify the occlusions between samples uring the online graph construction. To achieve that, an occlusion factor λ is introuce as the attribute attache to each sample in a k-nn list. λs of all the samples in the list are initialize to 0, when the k-nn list of a new query is joine into the graph. Factor λ will be upate when another new sample is joine into the k-nn list in the later stage. Given a new sample q to be inserte into sample r s k-nn list, we shoul know the istances of all the neighbors to r an the istances between q an other neighbors in the list. The istances to r are known as they are kept for sorting. While the istances between this new sample an the rest are unknown. Instea of performing a costly thorough comparison between q an the rest neighbors, we make use of istances that are compute uring the hill-climbing. To o that, another variable D is introuce to keep the istances between sample q an the samples that have been compare uring the hillclimbing. It is possible that not all the samples in k-nn list of r are joine in the comparisons uring hill-climbing. However, accoring to the principle neighbor of a neighbor is likely to be a neighbor, the ol samples in the k-nn list of r are likely being encountere by q. With the support of D,

6 6 occlusion factor λ of all the samples in the k-nn list is upate with following three rules. Rule : λ is kept unchange for samples ranke before q; Rule : λ of sample q is incremente by if a sample ranke before q is closer to q than q is to r; Rule : λ of a sample ranke after q is incremente by if its istance to q is smaller than q is to r. The efault istance of each sample to q is set to. As a result, the λs of not-being-visite neighbors will not be upate accoring to Rule an Rule. This is reasonable because the not-being-visite neighbors shoul be sufficiently far away from q, otherwise they are alreay being visite accoring to the principle a neighbor of a neighbor is also likely to be a neighbor. Since we have all the possible istances (between q an samples in the graph) only after the the hill-climbing converges, the operations of inserting q into k-nn list of r an upating factor λ in the list are postpone to the en of hill-climbing search. Fig. illustrates a search trail that is forme by the hill-climbing. In the k-neighborhoo of r, the LGD operations are applie. Once the occlusion factor is available, the search algorithm (Alg. ) an the construction algorithm (Alg. ) are accoringly moifie. When the query is compare to the neighbors in one k-nn list, we only consier the samples whose λ is no greater than the average occlusion factor λ of this list. Notice that this is ifferent from the way propose in [8], [0], in which the occlue samples (λ > 0) are simply omitte. We fin such kin of rule is too restrictive in our case. In our case, only samples that are occlue by many other neighbors are ignore. The operation of skipping samples with high factor λ coul be interprete as performing iversification in the graph []. Different from [], the iversification is unertaken in a lazy way in the sense no exhaustive comparison is involve witin the k-nn list. This scheme is therefore calle as lazy graph iversification (LGD). The three rules use to calculate occlusion factor are calle as LGD rules. The k-nn graph construction algorithm with LGD is accoringly name as LGD, which is summarize in Alg.. Algorithm : Online k-nn Graph Constr. with LGD (LGD) : Input: S n : ataset; k: size of NN list; p: num. of sees : Output: G: k-nn Graph : Q ; D[ n] ; E[ n] 0; : Extract a small subset I from S; 5: Initialize G an G with I; 6: G = G G; 7: for each q S I o o 8: while isupate o 9: R[ p] p ranom sees in [0, n) 0: for each r R o : InsertQ(r, m(q, r), Q); : en for : for r top(q) o en : if E[r] == 0 then 5: for n i G[r] && λ(n i ) λ(r) o 6: insertq(n i, (q, n i ), Q); 7: D[n i ] = m(q, n i ); 8: en for 9: for n i G[r] && λ(n i ) λ(r) o 0: insertq(n j, m(q, n i ), Q); : D[n i ] = m(q, n i ); : en for : en if : E[r] = ; 5: en for 6: en while 7: for each visite n i o 8: upateg(n i, q, D, G); 9: en for 0: for each r Q o : insertg(q, r, (q, r), G); : en for : Q ; D[ n] ; E[ n] 0; : en for Compare to Alg., Alg. is ifferent in three major aspects. In Alg., query sample is compare to samples whose occlusion factor is no greater than average factor λ in both k-nn list an reverse k-nn list (see line 5 an Line 9). After q is compare to a sample n i in the k-nn list, k-nn list of n i is not upate immeiately. Instea, the istance from q to sample n i is collecte into D (Line 7, Line ) for later use. The upate of k-nn lists for all the samples so far encountere are postpone to the en of k-nn search (Line 7 0). Function upateg( ) is basically similar as insertg( ). The aitional operation insie upateg( ) is to upate λ of all the neighbors accoring to LGD rules. It is easy to see Alg. becomes a fast k-nn search algorithm when the upateg an insertg operations are turne off. In the experiment section, we are going to show Alg. turns out be more cost-effective than Alg. in both k-nn construction an k-nn search approach when turning off their upateg( ) an insertg( ) operations. C. Recursive Neighborhoo Propagation In the last steps of both Alg. an Alg., the query sample q is inserte to the neighborhoo of rs as long as q is in their k-nn range. Notice that only a few rs that are sufficiently close to q will be consiere. In the neighborhoo of such rs, it is possible that there are some samples are not compare to q uring the hill-climbing search (as is shown in the ashe circle of Fig. ). Base on the principle that neighbor s neighbor is likely to be the neighbor, these unvisite samples are likely to be close neighbors of sample q. It is therefore reasonable to insert q an these unvisite samples to the neighborhoos of each other, in terms of k-nn range an reverse k-nn range as well. After this insertion, it is possible that q is introuce to meet with more unvisite neighbors. As a result, such kin of insertion coul be unertaken recursively until no new

7 7 r q Query sample See Sample being visite Sample not visite Vertex r k-neighborhoo of r Fig. : A trail of hill-climbing proceure in D l -space. The hill-climbing starts from a single see an converges when it reaches to the neighborhoo of the query. Query sample q is to be inserte into k-nn list of r. The occlusion relations between q an the rest samples in r s neighborhoo have to be upate. The istances from samples in the list to r are known. The istances from q to visite samples in r s neighborhoo are also known. The istances from q to not-being-visite are. Base on the LGD rules, occlusion factor λs of samples in r s neighborhoo coul be upate. unvisite neighbors are encountere. This operation is calle as recursive neighborhoo propagation (RNP). We foun this operation further boosts the quality of k-nn graph from - 5%, while only inucing minor computational overhea. The LGD graph construction metho with the support of RNP is given as D. Sample Removal from the Graph In practice, we shoul allow samples to be roppe out from the k-nn graph. A goo case is to maintain a k-nn graph for prouct photos for an e-shopping website, where olfashione proucts shoul be withrew from sale. The removal of samples from the k-nn graph ynamically is supporte in our approach. If the graph is built by Alg., the removal operation is as easy as eleting the sample from k-nn lists of its reverse neighbors an releasing its own k-nn list. If the graph is built by Alg., before the sample is elete, the occlusion factors of the samples living in the same k-nn list have to be upate. Fortunately not all the samples in the list shoul be consiere. Accoring to LGD Rule, only samples ranke after current sample shoul be consiere. The upate operations involves k / times istance computations in average. Given k is a small constant, the time cost is much lower than fulfilling a query. Notice that such kin of ynamic upating operation is not necessarily supporte by other online algorithms [], [0], in which the eleting operation may lea to collapse of the inexing structure. Discussions Two k-nn graph construction approaches namely (Alg. ) an LGD (Alg. ) in general follow the same framework. In both processes, the construction starts from a small-scale k-nn graph of 00% quality. The search process appens a k-nn list of a new sample to the graph each time. At the same time, the k-nn lists of the alreay inserte samples will be possibly upate when the new sample happen to be in their neighborhoos. It is therefore a win-win situation for both graph construction an NN search. Effective search proceure returns high quality k-nn list. While k-nn graph with high quality gives a goo guiance for the hill-climbing process. Besies the size of NN list k, there is another parameter involve in an LGD. Namely, the number of sees p. Usually, the size of NN list k shoul be no less than the intrinsic ata imension, which is less than or equal to the ata imension. The number of sees is usually set to be no bigger than k. When is very high (i.e., several hunres to thousans) an is close to, the construction process coul be slow when k is set to be close to. In such situation, a goo trae-off is harly achieve between the quality of k-nn graph an the efficiency of the construction. V. EXPERIMENTS In this section, the performance of the propose algorithms is stuie both as a k-nn graph construction an an approximate nearest neighbor search approach. In the evaluation, the performance is reporte on both synthetic ranom ata an ata from real worl. It is believe that the intrinsic imension of synthetic ata is roughly equal to the ata imension [], [8]. While for the atasets from real-worl, the intrinsic imension is usually smaller than the ata imension [8]. The brief information about the atasets that are aopte in the evaluation are summarize in Table I. In the nearest neighbor search task, the performance of the propose search approaches is stuie in comparison to the representative approaches of ifferent categories. Namely they are graph base approaches such as [], [0]. The typical locality sensitive hash approaches [0], [], namely SRS [] is consiere. For quantization base approach, prouct quantizer (PQ) [] is incorporate in the comparison. FLANN [] an Annoy [5] are selecte as the representative tree partitioning approaches, both of which are popular NN search libraries in the literature. A. Evaluation Protocol For k-nn graph construction, five synthetic ranom atasets size of 00K are use in the evaluation. Their imension ranges from to 00. Data in each imension are inepenently rawn from the range [0, ) uner uniform istribution. It guarantees the intrinsic imension of the synthetic ata largely equals to the ata imension. The top- (recall@) an top-0 (recall@0) recalls on each ataset are stuie uner l an l metrics respectively. Given function R(i, k) returns the

8 8 TABLE I: Summary on Datasets use for Evaluation Name n # Qry m( ) Type Ran00K l synthetic Ran00K l synthetic SIFTM [] l SIFT [6] SIFT0M [] l SIFT GISTM [7] l GIST [7] GloVeM [8] Cosine Text NUSW [9], l BoVW [0] NUSW [9], κ BoVW YFCCM [] l Deep Feat. RanM l synthetic Average recall@ (%) 5 5 Gr-NN+HC Gr-NN+EHC NN-Desc.+HC NN-Desc.+EHC Search time per query (ms) (a) SIFTM Average recall@ (%) Gr-NN+HC Gr-NN+EHC NN-Desc.+HC NN-Desc.+EHC Search time per query (ms) (b) GISTM number of truth-positive neighbors at top-k NN list of sample i, the recall at top-k on the whole set is given as recall@k = n i= R(i, k). () n k Besies k-nn graph quality, the construction cost is also stuie by measuring the scanning rate [] of each approach. Given C is the total number of istance computations in the construction, the scanning rate is efine as c = C n (n )/. () In aition, another seven atasets are aopte to evaluate the performance of both nearest neighbor search an k-nn graph construction. Among them, six atasets are erive from real worl images or text ata. In particular, all four atasets (namely GISTM, GloveM, NUSW an RanM) that are marke as most challenging atasets in [] are aopte in the evaluation. For each of the ataset, another,000 or 0,000 queries of the same ata type are prepare. Different metrics such as l, Cosine an κ are aopte in accorance with the ata type of each set. The search quality is measure by the top- recall for the first nearest neighbor. In orer to make our stuy comparable uner ifferent harware settings, the search quality is reporte along with the spee-up one approach achieves over brute-force search. All the coes of ifferent approaches consiere in this stuy are compile by g In orer to make our stuy to be fair, we isable all the multithreas, SIMD an pre-fetching instructions in the coes. All the experiments are pulle out on a PC with.6ghz CPU an G memory setup. B. Performance of Baseline NN-Search In the first experiment, the focus is to verify the effectiveness of the baseline search algorithm upon which our k-nn graph construction is built. Four ifferent configurations are teste. Firstly, the NN search is supplie with prebuilt k-nn graph from [] an the true k-nn graph. In the experiment, k is fixe to 0. The experiment is pulle out on SIFTM an GISTM atasets. The quality of 0-NN graph from is above 0 in terms of its top- an top- 0 recall for both atasets. These k-nn graphs are supplie to hill-climbing (HC) [7] an enhance hill-climbing (EHC, Alg. ). EHC iffers from HC mainly in the use of reverse k- NN graph uring the search. The search performance in terms Fig. 5: The top- recall curve of four configurations on hillclimbing (HC) approach. The enhance hill-climbing algorithm (Alg. ) is given as EHC. The performance of ifferent configurations is compare to [0]. of recall@ is shown in Fig. 5. The performance is compare to the state-of-the-art approach [0]. As seen from the figure, there is a significant performance gap between EHC an HC. The NN list of each sample in EHC is usually longer than that of HC ue to the incorporation of reverse nearest neighbors. On one han, EHC has to visit more samples uring the expansion, an therefore shoul be slower. On the other han, the reverse nearest neighbors also provie short-cuts to the remote neighbors for the hill-climbing, which is similar as the mechanism offere by small-worl graph [9], [0]. As a result, EHC turns out to be more efficient. Another interesting iscovery is that, the NN search performance base on approximate k-nn graph is very close to that of being base on true k-nn graph. This inicates minor ifference in k-nn graph quality oes not lea to any big ifference in the search performance. Above observations apply to other atasets that are consiere in this paper. With the support of the effective search proceure, it becomes possible to buil the k-nn graph with the search results. In the following, we are going to show the quality of k-nn Graph that is built base on this search algorithm. C. k-nn Graph Construction In the secon experiment, the performance about k-nn graph construction is stuie when the enhance hill-climbing is employe as a graph construction approach. In the evaluation, the performance of (Alg. ), LGD (Alg. ) an (LGD with recursive neighborhoo propagation) is compare to [], which is still recognize as the most effective k-nn graph construction approach that works in the generic metric spaces. In orer to be in line with the experiments in [], synthetic ata in the same series of imensions are use. In the test, the share parameter k among ifferent approaches are tune to be close to the ata imension an no higher than 50. Meanwhile, we ensure the scanning rate of ifferent approaches to be on the same level. Usually, the higher is the scanning rate the better is the k-nn graph quality. The scanning rate of all four approaches are reporte in Table II. While the top- an top-0 recall of all

9 9 TABLE II: Scanning rate c of, LGD an in comparison to [] uner l -norm an l -norm m( ) = l NN-Desc LGD l NN-Desc LGD Recall@ SIFT0M SIFTM GISTM RANDM YFCCM NUS-k NUS-l GloVeM Recall@ SIFT0M SIFTM GISTM RANDM YFCCM NUS-k NUS-l GloVeM (a) Recall@ (b) Recall@0.05 LGD.05 LGD Fig. 7: Top- an Top-0 recall of k-nn graphs prouce by, an on eight atasets. 5 5 Recall@ Recall@ (a) Recall@ with l -norm LGD (c) Recall@ with l -norm Recall@0 Recall@ (b) Recall@0 with l -norm LGD () Recall@0 with l -norm Fig. 6: Top- an Top-0 recall for five 00K synthetic atasets uner l -norm an l -norm. the approaches uner l an l istance measures are shown in Fig. 6. As seen from the figure, in most of the cases, the quality of k-nn graph from, LGD an is consierably better than when the their scanning rates are similar to each other. This is particularly true uner l -norm. The scanning rates of all the approaches increase steaily as the imension of ata goes higher. Meanwhile, the recall that one approach coul reach rops. As shown in the figure, when reaches to 00, the scanning rates of all the approaches are above 0%, which is too high for them to be practically useful. In general,,, LGD an are effective in the similar range of ata imensions, namely [, 50]. Within this imensional range, achieves the best trae-off between scanning cost an the graph quality. Due to the superior performance over LGD, only the performance of is reporte in the following experiments for clarity. While is treate as the comparison baseline. D. Nearest Neighbor Search In our thir experiment, the performance of NN search with the support of k-nn graph built by an is evaluate. Six atasets erive from real worl ata are aopte. Among them, NUSW is teste uner both l an κ istance measures. In aition, another 00-imensional ranom ataset size of one million is aopte. The brief information about all the atasets are summarize in Table I. When we use (Alg. ) an (LGD with recursive neighborhoo propagation) to buil the approximate k-nn graph for search, the parameter k an p are fixe to 0 for all the atasets. Once k-nn graphs are built, algorithms an LGD (Alg. ) are use as NN search proceures as we turn off their upate an insertion operations. For convenience, the NN search approaches base on the graph constructe by an are given as an respectively. In the evaluation, the NN search performance is compare to three representative graph base approaches. Namely, they are [], [] an [0], all of which work in generic metric spaces. Aitionally, all the approaches use the similar hill-climbing search proceure. The major ifference lies in the graph upon which the search proceure is unertaken. graph is basically built upon. In, the k-nn graph built by is iversifie by an off-line post-process. Aitionally, the iversifie NN list of each sample has been appene with its reverse NN list. In the experiment, shares the same k-nn graph with for each ataset. Parameter k in is fixe to 0 for all the atasets. searches over graph which is a merge of k-nn graph an its reverse k-nn graph that are prouce by. NN search in is on a k- NN graph merge with its reverse k-nn graph that have been iversifie online by LGD rules. While for, the search is unertaken on a approximate small worl graph. The graph maintains links between close neighbors as well as long range links to the remote neighbors, which are kept in a hierarchy. The parameter M in is fixe to 0. The eges kept for each sample in the bottom-layer is 0. Its size of NN list is therefore on the same level as,, an. This is to be in line with the experiments in []

10 z 0 TABLE III: Scanning rate c of an on eight atasets in comparison to []. The share parameter k in all the approaches is fixe to 0 Dataset m( ) SIFTM l SIFT0M l GISTM l GloVeM Cosine NUSW-l l NUSW-κ κ YFCCM l RanM l TABLE IV: Time cost (s) in total (of all the queries) by exhaustive search on eight atasets Dataset SIFTM SIFT0M GISTM GloVeM # Qry Time Dataset NUSW-l NUSW-κ YFCCM RanM # Qry Time Since the search is conucte base on a k-nn graph for,, an, the recall@ an recall@0 of corresponing k-nn graph from an our approaches are shown in Fig. 7. Accoringly, their scanning rate c on each ataset is reporte in Table III. As shown in the table, the scanning rates of on five outof eight atasets from real worl are at least 0% lower than that of an. While as is shown in Fig. 7, the recall of k-nn graph from LGD is the best on all the atasets. The graph quality that achieve by ifferent approaches is generally similar, while one exception comes from. It shows consierably poor performance on GloVeM, whose intrinsic imension is believe to be high []. The search performance on eight atasets is shown in Fig. 9. In orer to make the search results comparable to the results that are pulle out uner ifferent harware setups, the performance is reporte as the recall curve against the spee-up achieve over exhaustive search. The time costs of exhaustive search on each ataset are reporte separately on Table IV. It is therefore also convenient for the reaers who want to see the efficiency that our approach achieves with current setup. As shown in the figure, the performance from an are unstable across ifferent atasets. performs marginally better than on most of the atasets. While shows close performance as on two atasets (SIFTM an YFCCM) an outperforms it on the other six atasets, most of which are marke as most challenging in []. As the relative better performance is observe from, an, it is clear to see the performance boost mainly owes to the use of reserve k-nn an the introuction of graph iversification. The best performance is observe from in most of the cases. On one han, this inicates the search proceure is closely relate to the quality Query x Fig. 8: The illustration of hill-climbing in a D sub-space. As illustrate in the figure, the samples in blue form a D plane embee in the D space. The hill-climbing is more efficient than it is thought to be since it climbs along the D slope instea of exploring in the D space. of k-nn graph. On the other han, it also shows the search gets more cost-effective when skipping occlue samples. Comparing the result presente in Fig. 9(a) to the one presente in Fig. 9(b), the high scalability is achieve by the propose approaches. As seen in the figure, the size of reference set has been increase by 0 times, while the time cost only increases from 0.0ms (per query) to 0.07ms (per query), when the search quality is maintaine on level. Similar high scalability is observe on eep features i.e., YFCCM (Fig. 9()). This is goo news given the eep features have been wiely aopte in ifferent applications nowaays. In contrast, such kin of high spee-up is not achievable on NUSW, GloVeM an RanM, although the imensionality of GloVeM an RanM is lower than that of SIFTM an YFCCM. We foun that the spee-up that graph-base approaches coul achieve is partly relate to the intrinsic ata imension [], [8]. When the intrinsic ata imension is low, with the guiance of a k-nn graph, the hill-climbing search is actually unertaken on the sub-spaces where most of the ata samples lying in. Due to the low imensionality of these sub-spaces, the search complexity is lower than it seemingly is. Fig. 8 illustrates this phenomenon. This is one of the major reasons that the graph base approaches exhibit superior performance over other type of approaches. 6 0 See E. Comparison to state-of-the-art k-nn Search Fig. 0 further compares our approach with the most representative approaches of ifferent categories in the literature. Besies aforementione, an, approaches consiere in the comparison inclue tree partitioning approaches Annoy [5] an FLANN [], locality sensitive hashing approach SRS [], an vector quantization approach prouct quantizer (PQ) []. In the figures, the spee-ups that each approach achieves are reporte when recall@ is set to an levels. For PQ, it is impossible to achieve top- recall above 0.5 ue to its heavy quantization loss. As an exception, its recall is measure at top-6 for SIFTM an NUSW, an measure at top-8 for GISTM an RanM respectively. y 5 6

11 000 Annoy FLANN SRS PQ 000 Annoy FLANN SRS PQ Recall@ (a) SIFTM Recall@ (b) SIFT0M SIFTM GISTM NUSW-l RanM Dataset (a) Recall@= SIFTM GISTM NUSW-l RanM Dataset (b) Recall@= Fig. 0: The NN search performance on four atasets ranging from easy to har (best viewe in color). Eight representative approaches in the literature are consiere in the comparison. Figures (a) an (b) report respectively the speeup that one approach coul achieve when the top- recall is on an levels Recall@ (c) GISTM Recall@ (e) NUSW-l Recall@ (g) GloVeM Recall@ () YFCCM Recall@ (f) NUSW-κ Recall@ (h) RanM spee-up is observe for SRS on any of the atasets. This basically inicates SRS is not suitable for the tasks which require high NN search quality. Another interesting observation is that the performance gap between graph base approaches an the rest is wier on the easy ataset than that of har. Compare to the approaches of other categories, the NN search base on the graph makes goo use of the sub-space structures latent in a ataset. Since the intrinsic imension of easy ataset is low [8], the hill-climbing is actually unertaken on these low-imensional sub-spaces. The lower is the intrinsic imension the higher is the spee-up that graphbase approaches achieve. In contrast, there is no specific strategy in other type of approaches exploits on such latent structures in the ata. On one han, the high search spee-up is observe from on ata types such SIFT, GIST an eep features. With such efficiency, it is possible to realize an image search system with instant response on billion level ataset by a single PC. On the other han, it is still too early to say NN search on high-imensional ata has been solve. As shown on RanM an NUSW atasets, where both the ata imension an intrinsic ata imension are high, the efficiency achieve by our approach is still limite. Highly efficient NN search on these types of ata (i.e., intrinsic imension above 50) is still an open issue. Fig. 9: The NN search performance on eight atasets. Five graph base approaches are consiere in this stuy. an are propose in this paper. As shown in the figure, the best results come from graph base approaches. performs the best in most of the cases. This observation is consistent across ifferent atasets. The spee-up of all the approaches rops as the recall@ rises from to. The spee-up egraation is more significant for approaches such as PQ an FLANN. No consierable VI. CONCLUSION We have presente our solution for both k-nn graph construction an approximate nearest neighbor search. These two issues have been aresse uner a unifie framework. Namely, the NN search an NN graph construction are esigne as an interepenent proceure that one is built upon another. The avantages of this esign are several fols. First of all, the k-nn graph construction is an online proceure. It therefore allows the samples to be inserte in or roppe out from the graph ynamically, which is unesirable from most of the existing solutions. Moreover, no sophisticate