What we have covered?

Transcription

1 What we have covered? Indexing and Hashing Data warehouse and OLAP Data Mining Information Retrieval and Web Mining XML and XQuery Spatial Databases Transaction Management 1

2 Lecture 6: Spatial Data Management 2

3 Types of Spatial Data Point Data Points in a multidimensional space E.g., Raster data such as satellite imagery, where each pixel stores a measured value E.g., Feature vectors extracted from text Region Data Objects have spatial extent with location and boundary DB typically uses geometric approximations constructed using line segments, polygons, etc., called vector data. 3

4 Applications of Spatial Data Geographic Information Systems (GIS) E.g., ESRI s ArcInfo; OpenGIS Consortium Geospatial information All classes of spatial queries and data are common Computer-Aided Design/Manufacturing Store spatial objects such as surface of airplane fuselage Range queries and spatial join queries are common Multimedia Databases Images, video, text, etc. stored and retrieved by content First converted to feature vector form; high dimensionality Nearest-neighbor queries are the most common 4

5 Types of Spatial Queries Spatial Range Queries Find all cities within 50 miles of Madison Query has associated region (location, boundary) Answer includes overlapping or contained data regions Nearest-Neighbor Queries Find the 10 cities nearest to Madison Results must be ordered by proximity Spatial Join Queries Find all cities near a lake Expensive, join condition involves regions and proximity 5

6 Spatial Indexing Point Access Methods (PAMs) vs Spatial Access Methods (SAMs) PAM: index only point data Hierarchical (tree-based) structures Multidimensional Hashing Space filling curve SAM: index both points and regions Transformations Overlapping regions Clipping methods (non-overlapping) Data partitioning vs Space partitioning 6

7 Single-Dimensional Indexes B+ trees are fundamentally single-dimensional indexes. When we create a composite search key B+ tree, e.g., an index on <age, sal>, we effectively linearize the 2-dimensional space since we sort entries first by age and then by sal. Consider entries: <11, 80>, <12, 10> <12, 20>, <13, 75> SAL B+ tree order AGE

8 Multidimensional Indexes A multidimensional index clusters entries so as to exploit nearness in multidimensional space. Keeping track of entries and maintaining a balanced index structure presents a challenge! Consider entries: <11, 80>, <12, 10> <12, 20>, <13, 75> Spatial clusters B+ tree order

9 Motivation for Multidimensional Indexes Spatial queries (GIS, CAD). Find all hotels within a radius of 5 miles from the conference venue. Find the city with population 500,000 or more that is nearest to Kalamazoo, MI. Find all cities that lie on the Nile in Egypt. Find all parts that touch the fuselage (in a plane design). Similarity queries (content-based retrieval). Given a face, find the five most similar faces. Multidimensional range queries. 50 < age < 55 AND 80K < sal < 90K

10 What s the difficulty? An index based on spatial location needed. One-dimensional indexes don t support multidimensional searching efficiently. (Why?) Hash indexes only support point queries; want to support range queries as well. Must support inserts and deletes gracefully. Ideally, want to support non-point data as well (e.g., lines, shapes).

11 PAMs Point Access Methods Hierarchical methods: kd-tree based Space Filling Curves: Z-ordering Multidimensional Hashing: Grid File Exponential growth of the directory 11

12 The problem Given a point set and a rectangular query, find the points enclosed in the query We allow insertions/deletions on line Query 12

13 Tree-based PAMs Most of tb-pams are based on kd-tree kd-tree is a main memory binary tree for indexing k-dimensional points Needs to be adapted for the disk model Levels rotate among the dimensions, partitioning the space based on a value for that dimension kd-tree is not necessarily balanced 13

14 kd-tree At each level we use a different dimension B C x<5 x=5 x>=5 y=3 y=6 A E D x=6 14

15 Kd-tree properties Height of the tree O(log 2 n) Search time for exact match: O(log 2 n) Search time for range query: O(n 1/2 + k) 15

16 kd-tree example X=3 X=7 X=5 y=5 y=6 Y=6 x=3 x=8 x=7 Y=2 y=2 X=5 X=8 16

17 External memory kd-trees Similar to B-tree, tree nodes split many ways instead of two ways insertion becomes quite complex and expensive. No storage utilization guarantee since when a higher level node splits, the split has to be propagated all the way to leaf level resulting in many empty blocks. Pack many interior nodes (forming a subtree) into a block. it may not be feasible to group nodes at lower level into a block productively. Many interesting papers on how to optimally pack nodes into blocks recently published. 17

18 PAMs Point Access Methods Hierarchical methods: kd-tree based Space Filling Curves: Z-ordering Multidimensional Hashing: Grid File Exponential growth of the directory 18

19 Single-Dimensional Indexes B+ trees are fundamentally single-dimensional indexes. When we create a composite search key B+ tree, e.g., an index on <age, sal>, we effectively linearize the 2-dimensional space since we sort entries first by age and then by sal. Consider entries: <11, 80>, <12, 10> <12, 20>, <13, 75> SAL B+ tree order AGE

20 Z-Curve What is a Z-curve? A space filling curve Generated from interleaving bits x, y coordinate See Fig. 4.6 Alternative generation method see Fig. 4.5 Connecting points by z-order see Fig. 4.4 looks like Ns or Zs Fig 4.6 Implementing file operations Fig

21 Example of Z-values Figure 4.7 Left part shows a map with spatial object A, B, C Right part and Left bottom part Z-values within A, B and C Note C gets z-values of 2 and 8, which are not close Exercise: Compute z-values for B. Fig

22 Hilbert Curve A space filling curve Example: Fig. 4.5 More complex to generate due to rotations Illustration on next slide! Implementing file operations Fig

23 Calculating Hilbert Values (Optional Topic) Fig

24 PAMs Point Access Methods Hierarchical methods: kd-tree based Space Filling Curves: Z-ordering Multidimensional Hashing: Grid File Exponential growth of the directory 24

25 Grid File Hashing methods for multidimensional points (extension of Extensible hashing) Idea: Use a grid to partition the space each cell is associated with one page Two disk access principle (exact match) 25

26 Grid File Start with one bucket for the whole space. Select dividers along each dimension. Partition space into cells Dividers cut all the way. Each cell corresponds to 1 disk page. Many cells can point to the same page. Cell directory potentially exponential in the number of dimensions 26

27 Grid File Implementation Dynamic structure using a grid directory Grid array: a 2 dimensional array with pointers to buckets (this array can be large, disk resident) G(0,, nx-1, 0,, ny-1) Linear scales: Two 1 dimensional arrays that used to access the grid array (main memory) X(0,, nx-1), Y (0,, ny-1) 27

28 Grid Directory Example Buckets/Disk Blocks Linear scale Y Linear scale X 28

29 Grid File Search Exact Match Search: at most 2 I/Os assuming linear scales fit in memory. First use liner scales to determine the index into the cell directory access the cell directory to retrieve the bucket address (may cause 1 I/O if cell directory does not fit in memory) access the appropriate bucket (1 I/O) Range Queries: use linear scales to determine the index into the cell directory. Access the cell directory to retrieve the bucket addresses of buckets to visit. Access the buckets. 29

30 Grid File Insertions Determine the bucket into which insertion must occur. If space in bucket, insert. Else, split bucket how to choose a good dimension to split? If bucket split causes a cell directory to split do so and adjust linear scales. insertion of these new entries potentially requires a complete reorganization of the cell directory--- expensive!!! 30

31 Grid File Deletions Deletions may decrease the space utilization. Merge buckets We need to decide which cells to merge and a merging threshold Buddy system and neighbor system A bucket can merge with only one buddy in each dimension Merge adjacent regions if the result is a rectangle 31

32 Grid File Example (N=6) A 1 6 A 2 5 A

33 Grid File Example (N=6) A B A B A B

34 Grid File Example (N=6) A 14 B A B C B C A B C

35 Grid File Example (N=6) A D 14 B 1 6 A DB B 13 7 C BC B C A B C D

36 Grid File Example (N=6) y 4 y 3 A H I D F B A A H I D D F F B B y 2 E A I G F B y 1 C G E E G F B C C C C B x 1 x 2 x 3 x 4 36

37 The R-Tree The R-tree is a tree-structured index that remains balanced on inserts and deletes. Each key stored in a leaf entry is intuitively a box, or collection of intervals, with one interval per dimension. Example in 2-D: Root of R Tree Y X Leaf level

38 R-Tree Properties Leaf entry = < n-dimensional box, rid > key value being a box. Box is the tightest bounding box for a data object. Non-leaf entry = < n-dim box, ptr to child node > Box covers all boxes in child node (in fact, subtree). All leaves at same distance from root. Nodes can be kept 50% full (except root). Can choose a parameter m that is <= 50%, and ensure that every node is at least m% full.

39 Example of an R-Tree Leaf entry R1 R3 R8 R4 R9 R10 R11 R5 R14 R13 Index entry Spatial object approximated by bounding box R8 R12 R7 R18 R6 R16 R17 R19 R15 R2

40 Example R-Tree (Contd.) R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19

41 Search for Objects Overlapping Box Q Start at root. 1. If current node is non-leaf, for each entry <E, ptr>, if box E overlaps Q, search subtree identified by ptr. 2. If current node is leaf, for each entry <E, rid>, if E overlaps Q, rid identifies an object that might overlap Q. Note: May have to search several subtrees at each node! (In contrast, a B-tree equality search goes to just one leaf.)

42 Improving Search Using Constraints It is convenient to store boxes in the R-tree as approximations of arbitrary regions, because boxes can be represented compactly. But why not use convex polygons to approximate query regions more accurately? Will reduce overlap with nodes in tree, and reduce the number of nodes fetched by avoiding some branches altogether. Cost of overlap test is higher than bounding box intersection, but it is a main-memory cost, and can actually be done quite efficiently. Generally a win. 42

43 Insert Entry <B, ptr> Start at root and go down to best-fit leaf L. Go to child whose box needs least enlargement to cover B; resolve ties by going to smallest area child. If best-fit leaf L has space, insert entry and stop. Otherwise, split L into L1 and L2. Adjust entry for L in its parent so that the box now covers (only) L1. Add an entry (in the parent node of L) for L2. (This could cause the parent node to recursively split.)

44 Splitting a Node During Insertion The entries in node L plus the newly inserted entry must be distributed between L1 and L2. Goal is to reduce likelihood of both L1 and L2 being searched on subsequent queries. Idea: Redistribute so as to minimize area of L1 plus area of L2. GOOD SPLIT! BAD!

45 Spatial Data Warehousing Spatial data warehouse: Integrated, subject-oriented, time-variant, and nonvolatile spatial data repository for data analysis and decision making Spatial data integration: a big issue Structure-specific formats (raster- vs. vector-based, OO vs. relational models, different storage and indexing, etc.) Vendor-specific formats (ESRI, MapInfo, Integraph, etc.) Spatial data cube: multidimensional spatial database Both dimensions and measures may contain spatial components 45

46 Dimensions and Measures in Spatial Data Warehouse Dimension modeling nonspatial e.g. temperature: degrees generalizes to hot spatial-to-nonspatial e.g. region B.C. generalizes to description western provinces spatial-to-spatial e.g. region Burnaby generalizes to region Lower Mainland Measures numerical distributive (e.g. count, sum) algebraic (e.g. average) holistic (e.g. median, rank) spatial collection of spatial pointers (e.g. pointers to all regions with degrees in July) 46

47 Example: BC weather pattern analysis Input A map with about 3,000 weather probes scattered in B.C. Daily data for temperature, precipitation, wind velocity, etc. Concept hierarchies for all attributes Output A map that reveals patterns: merged (similar) regions Goals Interactive analysis (drill-down, slice, dice, pivot, roll-up) Fast response time Minimizing storage space used Challenge A merged region may contain hundreds of primitive regions (polygons) 47

48 Spatial data warehouse Dimensions region_name time temperature precipitation Measurements region_map area count Star Schema of the BC Weather Warehouse Dimension table Fact table 48

49 Spatial Merge Precomputing all: too much storage space On-line merge: very expensive 49

50 Methods for Computation of Spatial Data Cube On-line aggregation: collect and store pointers to spatial objects in a spatial data cube expensive and slow, need efficient aggregation techniques Precompute and store all the possible combinations huge space overhead Precompute and store rough approximations in a spatial data cube accuracy trade-off Selective computation: only materialize those which will be accessed frequently a reasonable choice 50

51 Spatial Association Analysis Spatial association rule: A B [s%, c%] A and B are sets of spatial or nonspatial predicates Topological relations: intersects, overlaps, disjoint, etc. Spatial orientations: left_of, west_of, under, etc. Distance information: close_to, within_distance, etc. s% is the support and c% is the confidence of the rule Examples is_a(x, large_town) ^ intersect(x, highway) adjacent_to(x, water) [7%, 85%] is_a(x, large_town) ^adjacent_to(x, georgia_strait) close_to(x, u.s.a.) [1%, 78%] 51

52 Progressive Refinement Mining of Spatial Association Rules Hierarchy of spatial relationship: g_close_to: near_by, touch, intersect, contain, etc. First search for rough relationship and then refine it Two-step mining of spatial association: Step 1: Rough spatial computation (as a filter) Using MBR or R-tree for rough estimation Step2: Detailed spatial algorithm (as refinement) Apply only to those objects which have passed the rough spatial association test (no less than min_support) 52

53 Spatial Classification and Spatial Trend Analysis Spatial classification Analyze spatial objects to derive classification schemes, such as decision trees in relevance to certain spatial properties (district, highway, river, etc.) Example: Classify regions in a province into rich vs. poor according to the average family income Spatial trend analysis Detect changes and trends along a spatial dimension Study the trend of nonspatial or spatial data changing with space Example: Observe the trend of changes of the climate or vegetation with the increasing distance from an ocean 53

54 LSD-tree Local Split Decision tree Use kd-tree to partition the space. Each partition contains up to B points. The kdtree is stored in main-memory. If the kd-tree (directory) is large, we store a sub-tree on disk Goal: the structure must remain balanced: external balancing property 54

55 Example: LSD-tree N2 N6 N7 x:x 1 y 3 y 1 y:y 1 y:y 2 x:x 2 x:x 3 (internal) (external) directory y 4 y 2 N5 N8 y:y 3 y:y 4 N1 N2 N3 N4 N5 N6 N7 N8 buckets N1 N3 N4 x 1 x 2 x 3 55

56 LSD-tree: main points Split strategies: Data dependent Distribution dependent Paging algorithm Two types of splits: bucket splits and internal node splits 56

57 Handling Regions with Z-curve Fig