Introduction to Indexing R-trees Dimitris Papadias Hong Kong University of Science and Technology 1
Introduction to Indexing 1. Assume that you work in a government office, and you maintain the records of 10 million people HK residents 2. The record of each resident contains the HK ID, address, telephone etc. 3. People come to your office, and ask you to retrieve the records of persons given their HK ID (i.e., "show me the record of person with HK ID: 5634569") 4. Lets forget computers for now. You just want to print the records in a catalog, so you answer these queries by manually looking up the catalog Assuming that you can put 10 records per printed page, the catalog will be 1 million pages 2
Introduction to Indexing (2) How would you arrange the records in the catalog? Your goal is to minimize the effort of finding records We measure this cost as the number of pages you have to "open" before finding the record Solution 1 - random order If the catalog contains records in random order of HK ID, then in the worst case you have to search the entire catalog (cost = 10 6 ) before finding a record, or to determine that the HK ID does not exist in the catalog Solution 2 - records sorted on HK ID You can apply binary search with cost log 2 10 6 = 20 Same discussion applies when we use computers; instead of the printed pages, we have disk pages (e.g., size 8 KB) Every time we read something from the disk (even a single number) we need to bring an entire page in main memory. The major cost is how many pages we read because disk operations are much more expensive than CPU operations Can you make it even faster? 3
Introduction to Indexing (3) Lets keep the sorted file and build an additional index (e.g., at the beginning of the catalog). Each index entry is a small record, that contains a HK ID and the page where you can find this ID (e.g., <5634569, 259> means that HK ID 5634569 is on page 259 of the catalog. HK ID is called the search key of the index Since each index entry is much smaller than the record, lets assume that we can fit 100 entries per page. The index entries are also sorted on HK ID Do we need an index entry for each of the 10,000,000 records? No: we only need an entry for the first record of each page Example: If I have two consecutive entries <5634569, 259>, <5700000, 260> in the index, then I know that every HK ID between 5634569 and 5700000 must be on page 259. Therefore, I only need only 1,000,000 index entries (one for each page of the main catalog). 4
Introduction to Indexing (4) Given that I can fit 100 entries per page, and I have 1,000,000 entries, my index is 10,000 pages. How I can use the index to speed up search for a record? Use binary search on the index to find the largest HK ID that is smaller or equal to input HK ID. The cost is log 2 10 4 = 14. Then, follow the pointer from that entry to the actual catalog (cost 1) Total cost: 14+1 = 15 Can I drop the cost even further? Yes: Build an index on the index The second level index contains 10,000 entries (one for each page of the first index) in 100 pages. Use binary search on the second level index to find the largest HK ID that is smaller or equal to input HK ID. The cost is log 2 10 2 = 7. Then, follow the pointer from that entry to first level index and finally to the actual catalog (cost 2) Total cost: 7+2 = 9 Finally build a third level index containing 100 entries, one for each page of the second level index. These entries fit in one page. Read this page - find the largest HK ID that is smaller or equal to input HK ID and follow the pointers to second level index, first level index and file. Total cost: 4 5
B+trees The B+ tree, the most common index for relational databases, applies the same ideas. Instead of the printed pages, we have index pages, called nodes There are several variations, but the main idea is the same - given a search key always follow a path from the root to the actual record. The cost of retrieving a record is always the number of levels in the tree plus 1 6
Additional Issues on Indexes In general, the index we choose depends on the type of queries that we want to process efficiently B+-trees are good for record retrieval based on the search key, and range queries: e.g., find all records with HK ID between 5600000 and 5700000 Complication: when we update the file, we also need to update the index In order to update the index efficiently, and at the same time guarantee performance, we need some clever algorithms Every index node must be at least half-full (i.e., it must contain a certain number of entries) If most nodes are almost empty then the index may be very large If we find a node that underflows we should do something about it (e.g., take entries from a sibling node - or merge with a sibling node) If a node overflows we must split it 7
Primary vs. Secondary Indexes Primary index (also called clustering index): in a sequentially ordered file, the index whose search key specifies the sequential order of the file. The search key of a primary index is usually but not necessarily the primary key. Example: File sorted on HKID and B+-tree on HKID Secondary index (also called non-clustering index): an index whose search key specifies an order different from the sequential order of the file. Example: File sorted on HKID and B+-tree on Name What is the maximum number of primary indexes that you can build on a single file?
Example of non-clustering (secondary) B+-tree 41 11 21 30 45 51 1 3 11 13 15 21 23 30 33 41 43 45 47 51 53 FILE WITH RECORDS record with search key 11 record with search key 3 record with search key 1 Should be always dense
Inserting a Data Entry into a B+ Tree Find correct leaf L. Put data entry onto L. If L has enough space, done! Else, must split L (into L and a new node L2) Redistribute entries evenly, copy up middle key. Insert index entry pointing to L2 into parent of L. This can happen recursively To split index node, redistribute entries evenly, but push up middle key. (Contrast with leaf splits.) Splits grow tree; root split increases height. Tree growth: gets wider or one level taller at top.
Deleting a Data Entry from a B+ Tree Start at root, find leaf L where entry belongs. Remove the entry. If L is at least half-full, done! If L less that half-full, Try to re-distribute, borrowing from sibling (adjacent node to the right). If re-distribution fails, merge L and sibling. If merge occurred, must delete entry (pointing to L or sibling) from parent of L. Merge could propagate to root, decreasing height.
B+-tree Update Examples Consider the B+-tree below where each node except for the root must contain at least two search key values and 3 pointers. Show the tree that would result after successively applying each of the following operations. 41 11 21 30 45 51 1 3 11 13 15 21 23 30 33 41 43 45 47 51 53 Remove 1 41 Remove 41 13 21 30 45 51 3 11 13 15 21 23 30 33 41 43 45 47 51 53
After removing 41 B+-tree Update Examples (cont) 30 13 21 43 51 3 11 13 15 21 23 30 33 43 45 47 51 53 Remove 3 Insert 41 21 30 43 51 11 13 15 21 23 30 33 43 45 47 51 53
After inserting 41 B+-tree Update Examples (cont) 21 30 43 51 3 11 13 15 21 23 30 33 41 43 45 47 51 53 Insert 1 30 13 21 43 51 1 3 11 13 15 21 23 30 33 41 43 45 47 51 53
Bulk Loading of a B+ Tree If we have a large collection of records, and we want to create a B+ tree on some field, doing so by repeatedly inserting records is very slow. Bulk Loading can be done much more efficiently. Initialization: Sort all data entries (using external sorting), insert pointer to first (leaf) page in a new (root) page. Root Sorted pages of data entries; not yet in B+ tree 3* 4* 6* 9* 10* 11* 12* 13* 20* 22* 23* 31* 35* 36* 38* 41* 44*
Index entries for leaf pages always entered into right-most index page just above leaf level. When this fills up, it splits. (Split may go up right-most path to the root.) Much faster than repeated inserts! Bulk Loading (Cont.) 6 Root 10 20 12 Data entry pages not yet in B+ tree 3* 4* 6* 9* 10* 11* 12* 13* 20*22* 23* 31* 35*36* 38*41* 44* 23 35 Root 20 10 35 Data entry pages not yet in B+ tree 6 12 23 38 3* 4* 6* 9* 10* 11* 12* 13* 20*22* 23* 31* 35*36* 38*41* 44*
What is Special About Spatial The search key is now the location of an object and we want to process spatial ranges (e.g., find all buildings that are within 1km of HKUST) or nearest neighbors (e.g., find the 2 closest MTR stations from HKUST) Relational indexes (e.g. B + -trees) and query processing methods are not readily applicable to spatial databases because there is no total ordering of objects in the multidimensional space that preserves spatial proximity 1D ordering using x-coordinate and space filling curves Multidimensional access methods index spatial data and facilitate efficient processing of simple spatial query types (i.e. range queries)
MBRs and 2-step Spatial Query Processing A spatial object is approximated by its minimum bounding rectangle (MBR) - i.e., the search key is the MBR A spatial query is then processed in two steps: 1. Filter step: The MBR is tested against the query predicate 2. Refinement step: The exact geometry of objects that pass the filter step is tested for qualification Example: find the objects that intersect the green object filtered pair non-qualifying pair that passes the filter step (false hit) qualifying pair Other approximations possible, but MBRs have in general the best performance
R-trees A height balanced tree similar to B + -tree that indexes spatial objects and processes the filter step. Objects are grouped in nodes (disk pages) according to spatial proximity (i.e., a problem similar to clustering) Minimum node utilization m is variable, usually 40% of maximum page capacity b For typical page sizes (e.g., 4Kbytes or more), m and b are in the order of hundreds Each entry is a pair (MBR, ptr), that contains: a pointer ptr to an indexed object or a lower level node the MBR of the pointed object or node R-trees can be used to of most spatial queries
10 8 6 4 2 y axis e b c f d g E 3 R-Tree Example h a i m j k l Minimum Bounding Rectangle (MBR) x axis 0 2 4 6 8 10 Root E 1 E 2 For simplicity node capacity = 3 In practice, in the order of 100 E 1 E 3 E 4 E 5 E 6 E 7 E2 a b c d e f g h i j k l m E E 3 4 E 5 E E 6 7 20
R-Tree, Leaf Nodes 10 8 6 4 2 y axis g h e f E 4 E 5 d b E 3 a c m E 7 l E 6 k i j x axis 0 2 4 6 8 10 Root E 1 E 2 E 1 E 3 E 4 E 5 E 6 E 7 E2 a b c d e f g h i j k l m E E 3 4 E 5 E E 6 7 21
10 8 6 4 2 R-Tree Intermediate Nodes y axis m g h l k e f E 2 i j d E 1 b a c x axis 0 2 4 6 8 10 Root E 1 E 2 E 1 E 3 E 4 E 5 E 6 E 7 E2 a b c d e f g h i j k l m E E 3 4 E 5 E E 6 7 22
10 8 6 4 2 R-tree, Range Query y axis m g h l e f k E 2 i j d E 1 b a c x axis 0 2 4 6 8 10 Root E 1 E 2 E 1 E 3 E 4 E 5 E 6 E 7 E2 a b c d e f g h i j k l m E E 3 4 E 5 E E 6 7 23
Desirable R-tree Properties 1. The area covered by a node MBR should be minimized (equivalently, minimize the dead space inside MBRs) 2. The overlap between node MBRs should be minimized. Similar to 1, this decreases the number of paths to be traversed during query processing 3. The margin (perimeter) of a node MBR should be minimized. Assuming fixed area, the object with the smallest margin is the square. Since quadratic objects can be packed easier, the MBRs of a level lead to smaller MBRs at the level above R-tree construction algorithms should aim at preserving the above properties
R-tree insertion 1. Invoke ChooseSubtree to find an appropriate node N, which to place the new object 2. If N has < b entries, add object in N (no overflow) 3. If N is already full, Split N and propagate split upwards 4. Adjust all covering rectangles in the insertion path such that they are the MBRs enclosing their children rectangles Algorithm ChooseSubtree 1. Set N to be the root 2. If N is a leaf, return N 3. else choose the entry of N that incurs the minimum MBR increase. Resolve ties by choosing the entry with the with the smallest MBR 4. Set N to be the childnode pointed to by the pointer of the chosen entry and goto 2 Deletion: Find the object to be deleted using a top down search of the R-tree and delete it. If there is an underflow (i.e., after deletion the node contains < m entries), delete and re-insert all the objects in the node.
Example of ChooseSubtree
Quadratic R-tree split algorithm 1. Apply PickSeeds to choose two entries to be the first elements of the groups. Assign each to a group 2. If all entries have been assigned, stop. If one group has so few entries that all the rest must be assigned to it, assign them and stop 3. Invoke Algorithm PickNext to choose the next entry to assign. Add it to the group whose MBR will have to be enlarged least to accommodate it. Resolve ties by adding the entry to the group with smaller MBR, then to the one with fewer entries, then to either 4. Go to 2 PickSeeds 1. For each pair of entries E1 and E2, compose a rectangle J including E1 and E2. Calculate d = area(j) - area(e1) - area(e2) 2. Choose the pair with the largest d (d is the dead space that contains nothing) PickNext 1. For each entry E not yet in a group, calculate d1=the area increase required in the covering rectangle of Group 1 to include E. Calculate d2 similarly for Group 2 2. Choose any entry with the maximum difference between d1 and d2
R*-tree R*-tree a more efficient version of the original R-tree that has more complex construction algorithms. R*-tree Insertion 1. Invoke ChooseSubtree to find an appropriate node N, which to place the new entry E 2. If N has < b entries, accommodate E in N (no overflow) 3. If N is already full invoke OverflowTreatment 4. If OverflowTreatment was called and a split was performed, propagate OverflowTreatment upwards 5. Adjust all covering rectangles in the insertion path such that they are the MBRs enclosing their children rectangles
R*-tree ChooseSubtree 1. Set N to be the root 2. If N is a leaf, return N 3. else if N points to intermediate nodes, choose the entry that incurs the minimum MBR increase. Resolve ties by choosing the entry with the with the smallest MBR (same as conventional R-tree) If N points to leaves, choose the entry in N whose MBR needs least overlap enlargement to include the new object. Resolve ties by choosing the entry that incurs the minimum MBR increase, then the entry with the smallest MBR 4. Set N to be the childnode pointed to by the pointer of the chosen entry and goto 2
R*-tree OverflowTreatment and Reinsert Algorithm OverflowTreatment 1. If the level is not the root level and this is the first call of OverflowTreatment in the given level, then invoke Reinsert 2. else invoke Split Algorithm Reinsert 1. Sort all b+1 entries of (full) node N in decreasing order of distance between their centers and the center of N 2. Remove the first p (30%b) entries and adjust the MBR of N 3. Reinsert the p entries in the tree Note that unlike conventional R-trees, re-insertion also happens at insertions. Reinsertion improves the structure of the tree because it alleviates the effect of insertion order.
R*-tree split algorithm Algorithm Split 1. Invoke ChooseSplitAxis to determine the axis, perpendicular to which the split is performed 2. Invoke ChooseSplitIndex to determine the best distribution into two groups along that axis 3. Distribute the entries into two groups Algorithm ChooseSplitAxis 1. For each axis Sort the entries by the lower (then by the upper value of their rectangles) and determine all distributions. Compute the sum S of all margin-values of the different distributions 2. Choose the axis with the minimum S as split axis Algorithm ChooseSplitlndex 1. Along the chosen split axis, choose the distribution with the minimum overlapvalue. Resolve ties by choosing the distribution with the minimum area
Example of R* split - ChooseSplitlndex Assume that node capacity is 7 and a node has 8 entries (overflows). The minimum node utilization is m=3 ( 40% 7). We first determine the split axis by considering all possible distributions on each axis. Let the split axis be x. Then we consider again all possible distributions (i.e., sorted orders on x) subject to m (3-5, 4-4, 5-3). The final grouping is the one with minimum overlap.
Bulk-loaded R-trees For static objects: Sort objects according to geometric criterion (e.g., x-coordinates, Hilbert value) and insert objects in nodes according to the sorted order (i.e., first b objects go to the first leaf node and so on). Much faster than individual insertions. Trees are packed (i.e., nodes are full) (a) California roads (b) Node MBRs of R* -tree (c) Sorting on x- coordinate (d) Sorting on Hilbert value
Selectivity Estimation- Multi-dimensional histograms Goal: Guess how may objects are there in a query window without actually processing the query. Important for Query Optimization and Cost Models Main idea: Divide the space in buckets, so that data in each bucket are almost uniform. Keep in memory the bucket extents and the number of objects per bucket Use this information to estimate the number of objects in query window MINSKEW example y 5 y 4 y 3 y 2 1 2 2 2 1 1 4 5 3 3 3 4 9 11 10 9 5 5 5 6 y 5 y 4 y 3 y 2 window query b 1 9 b 3 13 b 4 b 6 y 1 5 6 1 1 1 y 1 b 2 b 5 x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 x 5
Selectivity Estimation - Other Methods Random Sampling: Keep a sample of n objects out of N (data cardinality) objects in memory Let f be the number of samples that fall inside the query window Then, the expected cardinality of the query output is fn/n Power Laws : Given a dataset with intrinsic dimension d, the average selectivity of biased window queries with extent r (per axis) equals N r d. Assumptions: unit space and queries that follow the data distribution Note: the estimation is independent of the query location a a 2D uniform (d=2) Line dataset (d=1) Sierpinski (d=1.58)
Other Uses of R-trees Temporal Databases (1D R-trees) Keep the value of a search key over time Spatio-temporal Databases (3D R-trees) Keep extents of spatial objects over time y N2 d e time b a N 1 b a c d e c d' e' N 2 ' x d and e change location time t 1
Predictive Indexing of Spatial Objects The database stores the motion functions of moving objects. For each object o, its motion function gives its location o(t) at a future time t. A predictive window query specifies a query region q R and a future time interval q T retrieves the set of all objects that will fall in q R during q T. Examples Find all airplanes that will be over California in the next 10 minutes. Report all vessels that will enter the United States in the next hour.
Motion function We consider linear motion. 10 8 6 4 2 y axis -2-2 b 1-2 1 a 1 1-1 -2 1-1 d 1-2 at time 0 0 2 4 6 8 10 c 2 x axis 10 8 6 4 2 y axis For each object, the database stores Its MBR at the reference time 0 b a d c at time 1 0 2 4 6 8 10 Its current velocity bounding rectangle (VBR) Examples: MBR(a)={2,4,3,4}, VBR(a)={1,1,1,1}; MBR(c)={8,9,8,9}, VBR(c)={-2,0,-2,0}; x axis An update is necessary only when an object s VBR changes.
The Time Parameterized R-Tree Extends the R-tree by introducing the velocity bounding rectangle (VBR) in non-leaf entries. Queries are compared with conservative MBRs of non-leaf entries. 10 8 6 4 2 y axis 1 2-1 -1 N 1 d 2 N 1-2 -2-2 b 1-2 1-2 -2 2 1 1-2 a c 1 1-2 -1 0 2 4 6 8 10 at time 0 x axis 10 8 6 4 2 y axis N 1 b a d c N 2 0 2 4 6 8 10 at time 1 q R x axis