Approximate Nearest Neighbor Problem: Improving Query Time CS468, 10/9/2006

Transcription

1 Approximate Nearest Neighbor Problem: Improving Query Time CS468, 10/9/2006

2 Outline Reducing the constant from O ( ɛ d) to O ( ɛ (d 1)/2) in uery time Need to know ɛ ahead of time Preprocessing time and storage feature O(ɛ d ), O(ɛ (d 1)/2 ) etc.

3 Outline Reducing the constant from O ( ɛ d) to O ( ɛ (d 1)/2) in uery time Need to know ɛ ahead of time Preprocessing time and storage feature O(ɛ d ), O(ɛ (d 1)/2 ) etc. Timothy M. Chan. Approximate Nearest Neighbor Queries Revisited. Discrete and Computational Geometry Decomposition of space into cones BBD-tree for range searching in R d k + point location in R k Kenneth Clarkson. An Algorithm for Approximate Closest-point Queries. SoCG Additional log(ρ/ɛ) in space complexity Polytope approximation in R d+1

4 Chen s Algorithm: Motivation (1 + ɛ)-ann among (sorted) points in a narrow cone O(log n) by binary search Need a data structure that returns a sorted points given and a cone direction

5 Chen s Algorithm: Motivation (1 + ɛ)-ann among (sorted) points in a narrow cone O(log n) by binary search Need a data structure that returns a sorted points given and a cone direction Uses the BBD-tree data structure Given a uery point R d and a radius r one can find O(log n) cells of the BBD-tree which contain B(, r) and are contained in B(, 2r). This takes O(log n) time Use for approximate range searching in R d 1

6 Conic ANN (with a Hint) Input: Query point and a 2-approximation r to the NN distance Output: A points s such that s (1 + ɛ) p where p is the NN inside a cone with apex and angle δ = ɛ/16 r δ p s Note: s need not be in the cone! Note: The cone is fixed (not a part of input, mod. translation to )

7 Main (1 + ɛ)-ann Algorithm Uses the conic-ann with a hint as a subrotine Query (given only ) Obtain r by [Arya and Mount 1998] Get one point per data structure, return the one closest to

8 Main (1 + ɛ)-ann Algorithm Uses the conic-ann with a hint as a subrotine Query (given only ) Obtain r by [Arya and Mount 1998] Get one point per data structure, return the one closest to Preprocessing floating Tile R d with O(ɛ (d 1)/2 ) cones of angle δ = Θ( ɛ) Build a conic-ann data structure for each cone

9 Main (1 + ɛ)-ann Algorithm Uses the conic-ann with a hint as a subrotine Query (given only ) Obtain r by [Arya and Mount 1998] Get one point per data structure, return the one closest to Preprocessing floating Tile R d with O(ɛ (d 1)/2 ) cones of angle δ = Θ( ɛ) Build a conic-ann data structure for each cone Correctness s p true NN (1 + ɛ)-ann (returned from that cone s data structure)

10 Main (1 + ɛ)-ann Algorithm Uses the conic-ann with a hint as a subrotine Query (given only ) Obtain r by [Arya and Mount 1998] Get one point per data structure, return the one closest to Preprocessing floating Tile R d with O(ɛ (d 1)/2 ) cones of angle δ = Θ( ɛ) Build a conic-ann data structure for each cone Correctness s p true NN (1 + ɛ)-ann (returned from that cone s data structure) Query time O(ɛ (d 1)/2 log n) [# of cones] [conic uery]

11 Conic-ANN Data Structure For preprocessing given only direction of the cone (wlog: d-axis) and angle δ d-axis r δ

12 Conic-ANN Data Structure For preprocessing given only direction of the cone (wlog: d-axis) and angle δ Query Algorithm (given and r) Approximate range uery on the set of projections {p = [p 1 p 2 p d 1 ] T, p P } with B(, δr) returns O(log n) BBD-nodes (cells) in O(log n) time O(log n) binary searches Return the point s such that s d d is min d-axis r δ s δr 2δr

13 Conic-ANN Data Structure For preprocessing given only direction of the cone (wlog: d-axis) and angle δ Query Algorithm (given and r) Approximate range uery on the set of projections {p = [p 1 p 2 p d 1 ] T, p P } with B(, δr) returns O(log n) BBD-nodes (cells) in O(log n) time O(log n) binary searches Return the point s such that s d d is min d-axis Correctness (proof for s (1 + ɛ) p ) s d d p d d p s 2δr 4δ p s δ 2 p = (1 + ɛ) p r δ p s δr 2δr

14 Conic-ANN Data Structure For preprocessing given only direction of the cone (wlog: d-axis) and angle δ Query Algorithm (given and r) Approximate range uery on the set of projections {p = [p 1 p 2 p d 1 ] T, p P } with B(, δr) returns O(log n) BBD-nodes (cells) in O(log n) time O(log n) binary searches Return the point s such that s d d is min Correctness (proof for s (1 + ɛ) p ) d-axis s d d p d d p s 2δr 4δ p s δ 2 p = (1 + ɛ) p Data structure BBD-tree on the projection set For every tree node v the associated list of points is sorted in the d coordinate r δ δr p s 2δr

15 Conic-ANN Analysis Construction (preprocessing) BBD-tree O(n log n) +sorting O(n log n) = O(n log n) Query Approximate range uery O(log n) + bin. searches O(log 2 n) = O(log 2 n) Improving uery time by exploiting correlation [Lueker and Willard] v O(log n) O(1) O(1) O(log n) nodes O(1) O(1) left(v) right(v) O(1) O(1) O(1) O(1) O(1) O(1) O(1) O(1)

16 Summary and Remarks Variant with projecting to d 2 dimensions BBD tree + planar point location Rough ( d 3/2 ) approximation algorithms Polynomial dependence on d

17 Clarkson s Algorithm: Iterative Improvement Exact nearest neighbor problem Data structure For each site s, a (small) list L s of other sites such that for any uery point if s is not the nearest neighbor of, then L s contains a site closer to s

18 Clarkson s Algorithm: Iterative Improvement Exact nearest neighbor problem Data structure For each site s, a (small) list L s of other sites such that for any uery point if s is not the nearest neighbor of, then L s contains a site closer to Algorithm s arbitrary site while t L s : t < s do s t return s s

19 Clarkson s Algorithm: Iterative Improvement Exact nearest neighbor problem Data structure For each site s, a (small) list L s of other sites such that for any uery point if s is not the nearest neighbor of, then L s contains a site closer to Algorithm s arbitrary site while t L s : t < s do s t return s s Note The same L s valid for all!

20 Not Useful for Exact NN Reason 1: space complexity Ω(n 2 ) For all s, L s has to include all Delaunay neighbors of s For d > 2, Delaunay triangulation may have Ω(n 2 ) edges

21 Not Useful for Exact NN Reason 1: space complexity Ω(n 2 ) For all s, L s has to include all Delaunay neighbors of s t For d > 2, Delaunay triangulation may have Ω(n 2 ) edges s c Proof: t Delaunay neighbor of s, but t / L s t is the only site closer to than s

22 Not Useful for Exact NN Reason 1: space complexity Ω(n 2 ) For all s, L s has to include all Delaunay neighbors of s t For d > 2, Delaunay triangulation may have Ω(n 2 ) edges s c Proof: t Delaunay neighbor of s, but t / L s t is the only site closer to than s Reason 2: uery time Ω(n) No sufficient progress guarantee, may have to visit all sites s 2 s 1 s s 4 s 5 3

23 Not Useful for Exact NN Reason 1: space complexity Ω(n 2 ) For all s, L s has to include all Delaunay neighbors of s t For d > 2, Delaunay triangulation may have Ω(n 2 ) edges s c Proof: t Delaunay neighbor of s, but t / L s t is the only site closer to than s Conclusion No improvement over the trivial algorithm! Reason 2: uery time Ω(n) No sufficient progress guarantee, may have to visit all sites s 2 s 1 s s 4 s 5 3

24 Modification for ANN Data structure For each site s, a (small) list L s of other sites such that for any uery point if s is not a (1 + ɛ)-ann of, then L s contains a site (1 + ɛ/2)-closer to s t b s s 1+ɛ/2 s 1+ɛ

25 Modification for ANN Data structure For each site s, a (small) list L s of other sites such that for any uery point if s is not a (1 + ɛ)-ann of, then L s contains a site (1 + ɛ/2)-closer to s t b s s 1+ɛ/2 s 1+ɛ Algorithm (simple version) s arbitrary site while t L s : t s 1+ɛ/2 return s do s t

26 Query Algorithm Skip list approach [Arya and Mount 1993] R 0 = S

27 Query Algorithm Skip list approach [Arya and Mount 1993] R 1 R 0 = S

28 Query Algorithm Skip list approach [Arya and Mount 1993] R 2 R 1 R 0 = S

29 Query Algorithm Skip list approach [Arya and Mount 1993] R 3 R 2 R 1 R 0 = S

30 Query Algorithm Skip list approach [Arya and Mount 1993] R K R 3 R 2 R 1 R 0 = S

31 Query Algorithm Skip list approach [Arya and Mount 1993] R K R 3 R 2 R 1 R 0 = S Algorithm start with any t K 1 R K 1 for j = K 2, K 3,..., 0 [using naive algorithm] return t 0 find t j =(1 + ɛ)-ann of in R j starting from t j+1

32 Compare with a regular path Query Time Analysis Suppose that any node s list size is at most c Observation: Query time = c number of visited nodes Visit nodes in the order of proximity to, then go to the lower level

33 Compare with a regular path Query Time Analysis Suppose that any node s list size is at most c Observation: Query time = c number of visited nodes Visit nodes in the order of proximity to, then go to the lower level Claim: Our path visits at most 2K nodes more R j+1 t t j+1 t R j t t j+1 (1 + ɛ/2) ɛ t t

34 Compare with a regular path Query Time Analysis Suppose that any node s list size is at most c Observation: Query time = c number of visited nodes Visit nodes in the order of proximity to, then go to the lower level Claim: Our path visits at most 2K nodes more R j+1 t t j+1 t R j t t j+1 (1 + ɛ/2) ɛ t t Pr[regular path length C log n] O(n C ) [distribution of points across levels] [starting search point]

35 What about any? Query Time Analysis Skip list n possible search targets Probability of failure n O(n C ) = O(n (C 1) )

36 What about any? Query Time Analysis Skip list n possible search targets Probability of failure n O(n C ) = O(n (C 1) ) Only n O(d) combinatorially distinct regular paths If 1 and 2 incude the same distance ordering on the input sites, their regular paths are the same Arrangement of ( n 2) bisecting hyperplanes has (( n ) 2) (n 2 ) d = n 2d d d-dimensional cells

37 What about any? Query Time Analysis Skip list n possible search targets Probability of failure n O(n C ) = O(n (C 1) ) Only n O(d) combinatorially distinct regular paths If 1 and 2 incude the same distance ordering on the input sites, their regular paths are the same Arrangement of ( n 2) bisecting hyperplanes has (( n ) 2) (n 2 ) d = n 2d d d-dimensional cells Setting C = 2d + C Pr[regular path length O(d) log n] = O(n C )

38 Weighted Voronoi Diagrams Goal For each site s, compute L s such that R d b S : b s t L 1+ɛ s : t s 1+ɛ/2 [s is an (1 + ɛ)-ann of ] [no improvement in L s ]

39 Weighted Voronoi Diagrams Goal For each site s, compute L s such that R d b S : b s t L 1+ɛ s : t s 1+ɛ/2 [s is an (1 + ɛ)-ann of ] [no improvement in L s ] b [s, b, ɛ fixed] s t [s, t, ɛ fixed] s Q(b, ɛ) Q(t, ɛ/2) 1 s b ɛ(2+ɛ) 1 s t ɛ(2+ɛ/2) 2(1+ɛ) s b ɛ(2+ɛ) 2(1+ɛ/2) s t (ɛ/2)(2+ɛ/2)

40 Weighted Voronoi Diagrams Goal For each site s, compute L s such that R d b S : b s t L 1+ɛ s : t s 1+ɛ/2 [s is an (1 + ɛ)-ann of ] [no improvement in L s ] b S : Q(b, ɛ) t L s : Q(t, ɛ/2) b [s, b, ɛ fixed] s t [s, t, ɛ fixed] s Q(b, ɛ) Q(t, ɛ/2) 1 s b ɛ(2+ɛ) 1 s t ɛ(2+ɛ/2) 2(1+ɛ) s b ɛ(2+ɛ) 2(1+ɛ/2) s t (ɛ/2)(2+ɛ/2)

41 Weighted Voronoi Diagrams Goal For each site s, compute L s such that R d b S : b s t L 1+ɛ s : t s 1+ɛ/2 [s is an (1 + ɛ)-ann of ] [no improvement in L s ] b S : Q(b, ɛ) t L s : Q(t, ɛ/2) T T Q(b, ɛ) Q(t, ɛ/2) t L s b S b [s, b, ɛ fixed] s t [s, t, ɛ fixed] s Q(b, ɛ) Q(t, ɛ/2) 1 s b ɛ(2+ɛ) 1 s t ɛ(2+ɛ/2) 2(1+ɛ) s b ɛ(2+ɛ) 2(1+ɛ/2) s t (ɛ/2)(2+ɛ/2)

42 Linearization ( Lifting ) A point inside/outside a sphere in R d? A point above/below a hyperplane in R d+1? Example for d=1 y = 2 D D

43 Linearization ( Lifting ) A point inside/outside a sphere in R d? A point above/below a hyperplane in R d+1? Example for d=1 y = 2 D b s Q(b, ɛ) D Q(b, ɛ) = { R d : s (1 + ɛ) b }

44 Linearization ( Lifting ) A point inside/outside a sphere in R d? A point above/below a hyperplane in R d+1? Example for d=1 y = 2 D b s Q(b, ɛ) D Q(b, ɛ) = { R d : s (1 + ɛ) b } α 2ɛ P (b, ɛ) = {(, y) : αy 2, b b 2 } {(, y) : y = 2 } H(b, ɛ), halfspace in R d+1 (note: contains the origin) Ψ, standard paraboloid in R d+1 (note: independent of b, ɛ)

45 Final Formulation Paraboloid Ψ = {(, y) : y = 2 } y 4 α 2 b b 2 s b b 2 /α

46 Final Formulation Paraboloid Ψ = {(, y) : y = 2 } Halfspaces H(b, ɛ) = {(, y) : αy 2 b, b 2 } for all b S [can compute using S and ɛ] y H(b, ɛ) s uery points for which s is a (1 + ɛ)-ann

47 Final Formulation Paraboloid Ψ = {(, y) : y = 2 } Halfspaces H(b, ɛ) = {(, y) : αy 2 b, b 2 } for all b S [can compute using S and ɛ] Halfspaces G(t, ɛ = ɛ/2) = {(, y) : α y 2 t, t 2 } for all t L s [unknown] y Goal It suffices to make sure that s

48 Preprocessing initialize the weight of all sites to 1 repeat pick a (weighted) random sample R S of size C 1 cd log c T if G(t, ɛ/2) Ψ T H(b, ɛ) t R return R b S else v = a violating vertex of T t R G(t, ɛ/2) Ψ double the weight of V = {t S \ R : v / G(t, ɛ/2)} The sample size depends on c, the optimal size of L s Next we bound c using polytope approximation

49 Size of L s Exhibit a list of size O ( ɛ (d 1)/2 log ρ ɛ ), where ρ = max s,t S s t min s,t S s t Lemma For any convex and compact set P R d contained in the unit sphere and any ɛ (0, 1), there is a polytope P P with at most O(ɛ (d 1)/2 ) facets which is in the ɛ-neighborhood of P. Note Always outer approximation ɛ P P ɛ ɛ 1 ɛ ɛ

50 Size of L s Exhibit a list of size O ( ɛ (d 1)/2 log ρ ɛ ), where ρ = max s,t S s t min s,t S s t Lemma For any convex and compact set P R d contained in the unit sphere and any ɛ (0, 1), there is a polytope P P with at most O(ɛ (d 1)/2 ) facets which is in the ɛ-neighborhood of P. ɛ P P ɛ ɛ 1 Note Always outer approximation Recall We need an inner approximation of this y ɛ ɛ H(b, ɛ), b S

51 Want an inner approximation of this Size of L s y

52 Size of L s Want an inner approximation of this using only these hyperplanes as potential facets y y stretching 2 times

53 Size of L s Want an inner approximation of this using only these hyperplanes as potential facets y y stretching 2 times Goal: Subsample (as much as possible) the hyperplanes on the right so that

54 Size of L s Dudley approximation y Straightforward application of Dudley s Theorem does not work! The value of ɛ dictated by the smallest scale ɛ (in Dudley s Theorem)

55 Size of L s Solution: height-dependent slicing, per-slice Dudley approximations y vertical slice Slices have geometrically increasing height constant gap

56 Size of L s d m > 4 α 2 max b S b 2 y d i = 3 2 d i 1 d 0 = 1 4 min b S b 2 Number of slices m = O(log(ρ/α)) Recall: ρ spread Complexity (number of facets) of approximation O(ɛ (d 1)/2 ) per slice Key fact Red and blue projections into the -hyperplane within one slice are at least a factor of 1 + ɛ apart, so the same ɛ can be used in all approximations

57 Clarkson s Algorithm: Summary Improved uery time at the expense of specifying ɛ in advance O(ɛ (d 1)/2 ) instead of O(ɛ d ) Express the condition on L s in the form of P (S, ɛ) Q(L s, ɛ/2) Preprocessing by iterative random sampling from S and checking the containment condition Query procedure using top-down search on a skip list iterative improvement algorithm within one level