Computational Geometry

Transcription

1 Computational Geometry Search in High dimension and kd-trees Ioannis Emiris Dept Informatics & Telecoms, National Kapodistrian U. Athens ATHENA Research & Innovation Center, Greece Spring 2018 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

2 Contents 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

3 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

4 Range query interpreted geometrically salary G. Ometer born: Aug 19, 1954 salary: $ date of birth I.Emiris (Athens, Greece) Computational Geometry Spring / 57

5 Range query in 3 dimensions salary chlidren date of birth I.Emiris (Athens, Greece) Computational Geometry Spring / 57

6 The geometric approach We are interested in answering queries on d fields of the records in our database. Transform the records to points in d-dimensional space. The transformed range query asks for all points inside a d-dimensional axis-parallel box (may be unbounded). Such a query is called rectangular or orthogonal range query. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

7 1-Dimensional Range Search Problem Preprocess a set of points P = {p 1, p 2,..., p n } R so as to answer queries efficiently: Which points lie inside a query interval [x : x ]? Arrays O(n) space, O(n log n) preprocess, O(k + log n) query But, do not generalize in higher dim, do not allow efficient updates: O(n). Balanced Binary Search Trees (BBST) The leaves of T store the points of P, internal nodes store splitting values that guide the search. E.g. red-black trees, AVL trees. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

8 A search with the interval [18 : 77] µ µ I.Emiris (Athens, Greece) Computational Geometry Spring / 57

9 A search with the interval [x, x ] Search for x and x in T. The search ends to leaves µ and µ. Report all points stored at leaves between µ and µ plus, possibly, the points stored at µ and µ. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

10 A search with the interval [x, x ] Search for x and x in T. The search ends to leaves µ and µ. Report all points stored at leaves between µ and µ plus, possibly, the points stored at µ and µ. Remark The leaves to be reported are the ones of subtrees that are rooted at nodes whose parents are on the search paths to µ and µ. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

11 The selected subtrees root(t ) v split µ µ I.Emiris (Athens, Greece) Computational Geometry Spring / 57

12 Correctness and Performance Any reported point lies in the query range. Any point in the range is reported. O(n) storage. O(n log n) preprocessing. O(log n) update. Θ(n) worst case case query cost. O(k + log n) output sensitive query cost: O(k) to report the points plus O(log n) to follow the paths to x, x. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

13 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

14 kd-trees in the plane Problem Preprocess points P = {p 1, p 2,..., p n } R 2, to answer queries efficiently: Which points lie inside a query rectangle [x : x ] [y : y ]? p = (p x, p y ) lies in the rectangle iff p x [x, x ] & p y [y, y ]. kd-trees Generalize BBST: they split current pointset at median value, but use different coordinate at each level. Left subtree contains half (or one less) points with smaller coordinate and point with median value. Points shall correspond to leaves (and plane regions). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

15 The way the plane is subdivided p 2 l 5 p 1 l 1 l 7 p 8 l 8 p 3 p 6 l 9 p 7 p 5 p 9 l 2 l 3 p 4 p 10 l 4 l 6 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

16 The way the plane is subdivided p 2 p 1 l 1 p 8 p 3 p 6 p 5 p 7 p 9 p 4 p 10 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

17 The way the plane is subdivided p 2 p 1 l 1 p 8 p 3 p 6 p 5 p 7 l 2 p 9 p 4 p 10 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

18 The way the plane is subdivided p 2 p 1 l 1 p 8 p 3 p 6 p 7 p 5 p 9 l 2 l 3 p 4 p 10 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

19 The way the plane is subdivided p 2 p 1 l 1 p 8 p 3 p 6 p 7 p 5 p 9 l 2 l 3 p 4 l 4 p 10 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

20 The way the plane is subdivided l 5 p 1 l 1 p 2 p 8 p 3 p 6 p 7 p 5 p 9 l 2 l 3 p 4 l 4 p 10 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

21 The way the plane is subdivided p 2 l 5 p 1 l 1 p 8 p 3 p 6 p 7 p 5 p 9 l 2 l 3 p 4 p 10 l 4 l 6 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

22 The way the plane is subdivided p 2 l 5 p 1 l 1 l 7 p 8 p 3 p 6 p 7 p 5 p 9 l 2 l 3 p 4 p 10 l 4 l 6 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

23 The way the plane is subdivided p 2 l 5 p 1 l 1 l 7 p 8 l 8 p 3 p 6 p 7 p 5 p 9 l 2 l 3 p 4 p 10 l 4 l 6 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

24 The way the plane is subdivided p 2 l 5 p 1 l 1 l 7 p 8 p 3 p 6 l 8 l 9 p 7 p 5 p 9 l 2 l 3 p 4 p 10 l 4 l 6 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

25 The corresponding binary tree l 1 l 2 l 3 l 4 l 5 l 6 l 57 p 5 p 4 p 2 p 10 p 9 p 7 l 8 p 2 l 9 p 3 p 1 p 63 p 18 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

26 BuildKdTree(P, depth) if P contains only one point then return a leaf storing this point else if depth is even then split P with vertical l through median x-coord. of points in P P 1 points left of l or on l P 2 points right of l else {depth is odd} split P with horizontal l through median y-coord. of points in P P 1 points below l or on l P 2 points above l end if v left BuidKdTree(P 1, depth + 1); v right BuidKdTree(P 2, depth + 1) create a node v storing l lc(v) v left ; rc(v) v right return v end if I.Emiris (Athens, Greece) Computational Geometry Spring / 57

27 Building time and storage Remarks Split at the n 2-th smallest (median) coordinate: O(n) time, or preprocess by sorting both on x- and y-coordinates. The building time satisfies the recurrence: { O(1) if n = 1 T (n) = O(n) + 2T ( n 2 ) if n > 1 T (n) = O(n log n) which subsumes sorting. O(n) storage: points stored at leaves, leaf contains 1 points (alternatively stored at internal/splitting nodes). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

28 Nodes in a kd-tree and regions in the plane l 1 l 1 l 2 l 2 l 3 v l 3 region(v) I.Emiris (Athens, Greece) Computational Geometry Spring / 57

29 Regions and the query algorithm Internal nodes of a kd-tree correspond to rectangular regions of the plane: can be unbounded on one or more sides. Regions of all nodes at a specific level partition the plane. region(root(t )) is the whole plane. Point stored at (leaf of) subtree rooted at v iff it lies in region(v) Search the subtree of v only if the query rectangle intersects region(v). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

30 A query on a kd-tree p 2 l 5 p 1 l 1 l 7 p 8 l 1 p 3 p 6 l 8 l 9 l 2 l 3 p 7 p 5 l 2 l 3 p 4 p 9 l 4 l 5 p 5 p 4 p 2 p 10 l 6 l 57 p 9 p 7 l 8 p 2 l 9 p 10 p 3 p 1 p 63 p 18 l 4 l 6 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

31 Algorithm SearchKdTree(v, R) if v is a leaf then report point stored at v if in R else if region(lc(v)) is fully contained in R then ReportSubtree(lc(v)) else if region(lc(v)) intersects R then SearchKdTree(lc(v), R) end if end if if region(rc(v)) is fully contained in R then ReportSubtree(rc(v)) else if region(rc(v)) intersects R then SearchKdTree(rc(v), R) end if Input: root v, range R. Works for any query R, e.g. disk, triangle. O(k) to report k points. How many other nodes v are visited? i.e. for how many v, query range intersects region(v)? I.Emiris (Athens, Greece) Computational Geometry Spring / 57

32 Query time analysis Any vertical line intersects region(lc(root(t ))) or region(rc(root(t ))) but not both. If a vertical line intersects region(lc(root(t ))) it always intersects the regions corresponding to both children of lc(root(t )). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

33 Query time analysis The number of intersected regions (by vertical line) in a kd-tree storing n points, satisfies the recurrence: { O(1) if n = 1 Q(n) = 2 + 2Q( n 4 ) if n > 1 Q(n) = O( n) time = O( n + k) for rectangular query The analysis is rather pessimistic: In many practical situations the query range is small and will intersect much fewer regions. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

34 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

35 Introduction Given a distance function/metric: Preprocess: set of points/objects P = {p 1,..., p n } in d dimensions. Query: Given a d-dimensional query point/object q, report the closest p P to q. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

36 Motivation Points model general objects (e.g. handwritten digits) Distance between points inverse to similarity measure I.Emiris (Athens, Greece) Computational Geometry Spring / 57

37 Several applications Machine Learning: clustering/classification. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

38 Several applications Pattern Recognition and Classification I.Emiris (Athens, Greece) Computational Geometry Spring / 57

39 Several applications Searching multimedia databases. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

40 NN in R Sort/store the n points, use binary search for queries, then: Prepreprocessing in O(n log n) time Data structure requiring O(n) space Answer the query in O(log n) time I.Emiris (Athens, Greece) Computational Geometry Spring / 57

41 NN in R 2 Preprocessing: Voronoi Diagram in O(n log n). Storage = O(n). Given query q, find the cell it belongs (point location) in O(log n). NN = site of cell containing q. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

42 Exact NN in R d Is it faster than linear-time? Curse of Dimensionality: Complexity of Voronoi diagram grows rapidly = O(n d/2 ). Planar point location methods do not extend to higher dimensions. The volume of the space increases so fast that data becomes sparse State of the art: kd-trees: Sp = O(n), Query = O(d n 1 1/d ). Most practical for d log n: O(log n) expected for random points Randomized [Clarkson 88]: Sp = O(n d/2 +δ ), Q log n exp(d). n hyperplanes: point location O(d 5 log n), Sp = O(n d+δ ) [Meiser 93] I.Emiris (Athens, Greece) Computational Geometry Spring / 57

43 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

44 Nearest Neighbor in high dimension Exact NN Given set P in d dimensions, and query point q, its NN is point p 0 P: Approximate NN dist(p 0, q) dist(p, q), p P. Given set P in d dimensions, approximation factor 1 > ɛ > 0, and query point q, an ɛ-nn, or ANN, is any point p 0 P: dist(p 0, q) (1 + ɛ) dist(p, q), p P. x1 q (1 + ɛ)r r NN x2 I.Emiris (Athens, Greece) Computational Geometry Spring / 57

45 Approximate NN in R d BBD tree [Arya,Mount et al.98]: optimal query for d = O(1) In practice like kd-trees: cgal offers lazy kd-trees ann [Mount] for d 60 flann [Lowe-Muja], kd-geraf [E-Samaras]: randomized Locality sensitive hashing (LSH) for ɛ-nn Sp = O(dn 1+ρ ), Q = O(dn ρ ), ρ = 1/(1 + ɛ) 2. [Indyk,Motwani 98] [Panigrahy 06] [Andoni,Indyk 06] Dimensionality reduction [Anagnostopoulos,E,Psarros 15 17] Sp = O (dn), Q = O (dn ρ ), ρ = 1 + ɛ 2 / log ɛ < 1 BBD I.Emiris (Athens, Greece) Computational Geometry Spring / 57

46 NN formulations Standard computational geometry: the space is Euclidean R d, for constant d. Complex data: treat d as an asymptotic quantity and seek solutions having no exponential dependence on d. Wish to treat arbitrary metric (nonvector) spaces. Structure, especially for metric spaces: may assume a growth-limiting property, e.g. constant doubling dimension: twice the ball is included in constant number of balls (true for Euclidean). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

47 Grid for Uniform points n uniformly distributed points in [0, 1] d Cell structure (array) using parameter c = O(1) Expected #points per box = c (high c increases box search). Expected n/c boxes (high c reduces array size). Each box of volume = c/n, edge length (c/n) 1/d < 1. Query lands in a box in O(1), checks points in box. Given current best distance, check 3 d 1 adjacent boxes. In expectation, O(1) boxes visited, in time O(c). Expected query time = O(c + 3 d ). [Bentley-W.-Yao 80,Bentley 90]. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

48 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

49 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

50 kd-trees Assuming d > 2 but small. Iterate through splitting coordinates; various strategies to pick them Leaves contain 1 points; bound #levels. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

51 NN search Procedure NN(node), given query q if node is leaf then Search all points in node, update best-dist else {internal node} if split-coor(q) node s split-value then NN(left-child) // standard branch if split-coor(q) + best-dist > node s split-value then NN(right-child) // recurse to check end if else {split-coor(q) > node s split-value} NN(right-child) if split-coor(q) best-dist node s split-value then NN(left-child) end if end if left/right end if internal node Overall topdown algorithm: NN(root). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

52 Complexity Sp = O(d n). construction of balanced tree: O(d n log n) by sorting per dimension, O(n log n) by linear-time median computation. (Few) Insert/delete operations in balanced kd-tree = O(log n) I.Emiris (Athens, Greece) Computational Geometry Spring / 57

53 Complexity Sp = O(d n). construction of balanced tree: O(d n log n) by sorting per dimension, O(n log n) by linear-time median computation. (Few) Insert/delete operations in balanced kd-tree = O(log n) Exact Range query = O(d n 1 1/d + k). In practice, ANN O(log n) when d = O(1), since O(1) expected neighbors for random (e.g. uniform) distribution. See also BBD-trees: O((d/ɛ) d log n). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

54 Extension k Nearest Neighbors Store k current best points. Current ball encloses k current best points. Eliminate sibling if none of its points can be closer than any of k current best points, i.e. if sibling region outside current ball. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

55 Splitting at max spread median of set closest to box centre I.Emiris (Athens, Greece) Computational Geometry Spring / 57

56 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

57 Randomization Construct: Search: Create r kd-trees s.t. searches are largely independent. Find O(1) coord s maximizing variance: Pick one randomly May sample the data; split it about the mean of the sample. Use bounded #levels; bucket contain several points. Principal Component Analysis finds moment axes: rotate to align them with the coordinte axes. Or, random rotation. Upper bound on total #nodes to be searched. Priority queue stores candidates across r trees. Similar effect to r lower-dim projection [Silpa-Anan,Hartley 08] I.Emiris (Athens, Greece) Computational Geometry Spring / 57

58 FLANN: Fast Library for ANN Typically r 6 independent trees. Target d = 128, n > 10 4 (SIFT encoding of images). Given data: Automatic choice of configuration, and algorithm (Randomized kd-trees, Hierarchical k-means trees) [Lowe:IJCV04], software [Lowe,Muja] I.Emiris (Athens, Greece) Computational Geometry Spring / 57

59 kd-geraf Implement k-ann Simultaneous search, no backtracking. Quickselect algorithm to find median in O(n) Accelerated distance computations (dot product, see below) Public domain C++: (WebApp: :8080) [Avrithis,E,Samaras 15] I.Emiris (Athens, Greece) Computational Geometry Spring / 57

60 Randomization Rotation Every tree uses a randomly rotated pointset, thus using a different set of dimensions/coordinates. Split dimension Pick t dimensions of highest variance. Choose one randomly at every node while building the tree. Split Value Shuffling The pointset s median in split dimension plus uniformly distributed δ [ 3 d, 3 d ], = diameter of pointset. The split value may be witnessed in several points, instead of picking always the same point, shuffle them to break ties. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

61 Performance Parameters (auto or manual) Practical complexity r Number of trees in forest (points are stored once) t Number of hi-variance dimensions used for splits Maximum number of points-per-leaf c Maximum number of leaves to be checked during search ɛ Determine search accuracy Automatic parameter configuration yields fastest preprocessing, successful trade-off between accuracy and speed. Most competing methods suffer from slow parameter configuration, running out of memory, unstable search behaviour. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

62 Implementation Search Descend every tree to leaf, store unvisited branch nodes in min-priority queue Q. Examine nodes in Q, until c/1 + ɛ leaves are checked. On descending a tree: at leaf: update currently best distance. at node: if query in the left half-space: insert right child to Q, descend to left child; or vice versa. Distance computation x q 2 = x 2 + q 2 2q x, where the first two can be stored. Offers up to 10% speedup. Project idea: x q 2 y q 2 reduces to 2q (y x). I.Emiris (Athens, Greece) Computational Geometry Spring / 57

63 Experiments Faster than ann/bbd, flann for d 1, 000 (up to 10,000), n (i) SIFT images: n = 10 6, d = 128, BBD out of memory. GIST: d = 960 (ii) n = 10 5 (iii) n = 10 6, query < 1s, 90% exact. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

64 Experiments on Oxford set, CroW features (neural nets) n = 5062 images, d = 512. Brute force: 5.22 sec. Build takes 2 sec for kd-geraf. points per leaf trees no t max leaf check miss(%) time(ms) Search with Noisy queries. points per leaf trees no t max leaf check miss(%) time(s) Search with Oxford queries. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

65 Outline 1 Orthogonal Range Search 2D orthogonal range search 2 Nearest Neighbors Approximate nearest neighbor 3 Trees kd-trees Randomized kd-trees Balanced Box-Decomposition trees I.Emiris (Athens, Greece) Computational Geometry Spring / 57

66 BBD-trees Box: set theoretic difference of two boxes, one enclosed in the other (inner is optional) Empirical runtimes for most distributions show little/no practical advantage over kd-trees [Arya,Mount,et al 94,98]. Complexity: O(1) points per leaf, space = O(dn). Height = O(log n): every 4 levels reduce #points by > 2/3. Construct = O(dn log n). k ɛnns in time O((d/ɛ) d + k) log n). Dynamic: point insertion/deletion = O(log n). overview I.Emiris (Athens, Greece) Computational Geometry Spring / 57

67 Construction Tree constructed by applying 2 operations, when cell contains > 1 points: (Fair) Split: by hyperplane parallel to a coordinate plane, through midpoint. If inner box exists, do not intersect it. Exponential decrease of region size (quadtree). Shrink: partitions box into inner and outer boxes. If inner box exists, it lies inside new inner box. Exponential decrease in number of points per cell (kd-tree). Two strategies: Splits and Shrinks alternate. Split until both children with < 2/3 of parent s points, then Shrink. I.Emiris (Athens, Greece) Computational Geometry Spring / 57

68 ANN search Algorithm 1 Find leaf that contains query q; min-dist δ from q to points in leaf 2 Order leaves in increasing distance from q (priority search). 3 Find closest leaf to q, compute min-distance δ from q to points in cell 4 While distance of next closest leaf < δ(1 + ɛ), compute min-distance between q and points in cell: if < δ, this distance becomes δ. Time Complexity Point location = O(log n). #cells explored = c < (1 + 6d/ɛ) d. ANN query = O(cd log n). I.Emiris (Athens, Greece) Computational Geometry Spring / 57