Roadmap DB Sys. Design & Impl. Reference. Detailed roadmap. Spatial Access Methods - problem. z-ordering - Detailed outline.

Size: px

Start display at page:

Download "Roadmap DB Sys. Design & Impl. Reference. Detailed roadmap. Spatial Access Methods - problem. z-ordering - Detailed outline."

Ashlie Waters
6 years ago
Views:

1 D Sys. Design & Impl. Z-ordering Christos Faloutsos Roadmap 1) Roots: System R and Ingres 2) Implementation: buffering, indexing, q-opt 3) Transactions: locking, recovery 4) Distributed DMSs 5) Parallel DMSs: Gamma, lphasort 6) OO/OR DMS 7) Data nalysis - data mining 8) enchmarks 9) vision statements extras (streams/sensors, graphs, multimedia, web, fractals) C. Faloutsos 2 Detailed roadmap 1) Roots: System R and Ingres 2) Implementation: buffering, indexing, q-opt OS support for DMS R-trees and GiST Z-ordering uffering... 3) Transactions: locking, recovery Reference Jack. Orenstein: Spatial Query Processing in an Object-Oriented Database System. SIGMOD Conference 1986, pp C. Faloutsos C. Faloutsos 4 - Detailed outline What is the problem / S..M. main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations Spatial ccess Methods - problem Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer spatial queries (like??) C. Faloutsos C. Faloutsos 6 1

2 Spatial ccess Methods - problem Given a collection of geometric objects (points, lines, polygons,...) organize them on disk, to answer point queries range queries k-nn queries spatial joins ( all pairs queries) Q: applications? SMs - motivation C. Faloutsos C. Faloutsos 8 SMs - motivation Q: applications? : GIS multimedia indexing traditional db... SMs: solutions R-trees (grid files) Q: how would you organize, e.g., n-dim points, on disk? (C points per disk page) C. Faloutsos C. Faloutsos 10 - Detailed outline What is the problem / S..M. main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations Q: how would you organize, e.g., n-dim points, on disk? (C points per disk page) Hint: reduce the problem to 1-d points(!!) Q1: why? : Q2: how? C. Faloutsos C. Faloutsos 12 2

3 Q: how would you organize, e.g., n-dim points, on disk? (C points per disk page) Hint: reduce the problem to 1-d points (!!) Q1: why? : -trees! Q2: how? Q2: how? : assume finite granularity; = bitshuffling = N-trees = Morton keys = geocoding = C. Faloutsos C. Faloutsos 14 Q2: how? : assume finite granularity (e.g., 2 32 x2 32 ; 4x4 here) Q2.1: how to map n-d cells to 1-d cells? Q2.1: how to map n-d cells to 1-d cells? C. Faloutsos C. Faloutsos 16 Q2.1: how to map n-d cells to 1-d cells? : row-wise Q: is it good? Q: is it good? : great for x axis; bad for y axis C. Faloutsos C. Faloutsos 18 3

4 Q: How about the snake curve? Q: How about the snake curve? : still problems: 2^32 2^ C. Faloutsos C. Faloutsos 20 Q: Why are those curves bad? : no distance preservation (~ clustering) Q: solution? Q: solution? (w/ good clustering, and easy to compute, for 2-d and n-d?) 2^32 2^ C. Faloutsos C. Faloutsos 22 Q: solution? (w/ good clustering, and easy to compute, for 2-d and n-d?) : /bit-shuffling/linear-quadtrees looks better: few long jumps; scoops out the whole quadrant before leaving it a.k.a. space filling curves /bit-shuffling/linear-quadtrees Q: How to generate this curve (z = f(x,y) )? : 3 (equivalent) answers! C. Faloutsos C. Faloutsos 24 4

5 /bit-shuffling/linear-quadtrees Q: How to generate this curve (z = f(x,y))? 1: z (or N ) shapes, RECURSIVELY Notice: self similar ( fractal ) method is hard to use: z =? f(x,y) order-1 order-2... order (n+1) order-1 order-2... order (n+1) C. Faloutsos C. Faloutsos 26 /bit-shuffling/linear-quadtrees Q: How to generate this curve (z = f(x,y) )? : 3 (equivalent) answers! Method #2? bit-shuffling y x 0 0 y 1 1 z =( ) 2 = x C. Faloutsos C. Faloutsos 28 bit-shuffling y x 0 0 y 1 1 bit-shuffling y x 0 0 y x z =( ) 2 = 5 How about the reverse: (x,y) = g(z)? x z =( ) 2 = 5 How about n-d spaces? C. Faloutsos C. Faloutsos 30 5

6 /bit-shuffling/linear-quadtrees Q: How to generate this curve (z = f(x,y) )? : 3 (equivalent) answers! Method #3? linear-quadtrees : assign N->1, S->0 e.t.c. W E 1 N S C. Faloutsos C. Faloutsos and repeat recursively. Eg.: z blue-cell = WN;WN = (0101) 2 = 5 W E N S Drill: z-value of magenta cell, with the three methods? W E 1 N 0 S C. Faloutsos C. Faloutsos 34 Drill: z-value of magenta cell, with the three methods? W E method#1: 14 1 method#2: shuffle(11;10)= N (1110) 2 = S Drill: z-value of magenta cell, with the three methods? W E method#1: 14 1 method#2: shuffle(11;10)= N (1110) 2 = 14 method#3: EN;ES =... = 14 0 S C. Faloutsos C. Faloutsos 36 6

7 - Detailed outline main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations - usage & algo s Q1: How to store on disk? : Q2: How to answer range queries etc C. Faloutsos C. Faloutsos 38 - usage & algo s Q1: How to store on disk? : treat z-value as primary key; feed to -tree usage & algo s MJOR DVNTGES w/ -tree: already inside commercial systems (no coding/debugging!) concurrency & recovery is ready C. Faloutsos C. Faloutsos 40 - usage & algo s Q2: queries? (eg.: find city at (0,3) )? usage & algo s Q2: queries? (eg.: find city at (0,3) )? : find z-value; search -tree C. Faloutsos C. Faloutsos 42 7

8 - usage & algo s - usage & algo s Q2: range queries? 5 12 Q2: range queries? : compute ranges of z-values; use -tree 9, C. Faloutsos C. Faloutsos 44 - usage & algo s Q2 : range queries - how to reduce # of qualifying of ranges? 9, usage & algo s Q2 : range queries - how to reduce # of qualifying of ranges? : ugment the query! 9, > C. Faloutsos C. Faloutsos 46 - usage & algo s Q2 : range queries - how to break a query into ranges? - usage & algo s Q2 : range queries - how to break a query into ranges? : recursively, quadtree-style; decompose only non-full quadrants 9, , C. Faloutsos C. Faloutsos 48 8

9 - usage & algo s Q2 : range queries - how to break a query into ranges? : recursively, quadtree-style; decompose only non-full quadrants , 11 9, Detailed outline main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations C. Faloutsos C. Faloutsos 50 - usage & algo s Q3: k-nn queries? (say, 1-nn)? usage & algo s Q3: k-nn queries? (say, 1-nn)? : traverse -tree; find nn wrt z-values and C. Faloutsos C. Faloutsos 52 - usage & algo s... ask a range query. - usage & algo s... ask a range query. nn wrt z-value nn wrt z-value C. Faloutsos C. Faloutsos 54 9

10 - usage & algo s Q4: all-pairs queries? ( all pairs of cities within 10 miles from each other? ) (we ll see spatial joins later: find all P counties that intersect a lake) - Detailed outline main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations C. Faloutsos C. Faloutsos 56 - regions Q: z-value for a region? z =?? z C =?? C - regions Q: z-value for a region? : 1 or more z-values; by quadtree decomposition C z =?? z C =?? C. Faloutsos C. Faloutsos 58 - regions don t care - regions don t care Q: z-value for a region? z = 11** Q: z-value for a region? z = 11** 1 W E N z C =?? W E N z C = {0010; 1000} S S C 1 0 C C. Faloutsos C. Faloutsos 60 10

11 - regions Q: How to store in -tree? Q: How to search (range etc queries) - regions Q: How to store in -tree? : sort (*<0<1) Q: How to search (range etc queries) C C z obj-id etc 0010 C C 11** C. Faloutsos C. Faloutsos 62 - regions Q: How to search (range etc queries) - eg red range query C z obj-id etc 0010 C C 11** - regions Q: How to search (range etc queries) - eg red range query : break query in z-values; check -tree C z obj-id etc 0010 C C 11** C. Faloutsos C. Faloutsos 64 - regions lmost identical to range queries for point data, except for the don t cares - i.e., C 1100?? 11** z obj-id etc 0010 C C 11** - regions lmost identical to range queries for point data, except for the don t cares - i.e., z1= 1100?? 11** = z2 Specifically: does z1 contain/avoid/intersect z2? Q: what is the criterion to decide? C. Faloutsos C. Faloutsos 66 11

12 - regions z1= 1100?? 11** = z2 Specifically: does z1 contain/avoid/intersect z2? Q: what is the criterion to decide? : Prefix property: let r1, r2 be the corresponding regions, and let r1 be the smallest (=> z1 has fewest * s). Then: C. Faloutsos 67 - regions r2 will either contain completely, or avoid completely r1. it will contain r1, if z2 is the prefix of z1 C 1100?? 11** region of z1: completely contained in region of z C. Faloutsos 68 - regions Drill (True/False). Given: z1= ** z2= 01****** z3= 0100**** T/F r2 contains r1 T/F r3 contains r1 T/F r3 contains r2 - regions Drill (True/False). Given: z1= ** z2= 01****** z3= 0100**** T/F r2 contains r1 - TRUE (prefix property) T/F r3 contains r1 - FLSE (disjoint) T/F r3 contains r2 - FLSE (r2 contains r3) C. Faloutsos C. Faloutsos 70 - regions - regions Drill (True/False). Given: z1= ** z2= 01****** z3= 0100**** z2 Drill (True/False). Given: z1= ** z2= 01****** z3= 0100**** z3 z2 T/F r2 contains r1 - TRUE (prefix property) T/F r3 contains r1 - FLSE (disjoint) T/F r3 contains r2 - FLSE (r2 contains r3) C. Faloutsos C. Faloutsos 72 12

13 - regions Spatial joins: find (quickly) all counties intersecting lakes - regions Spatial joins: find (quickly) all counties intersecting lakes C. Faloutsos C. Faloutsos 74 - regions Spatial joins: find (quickly) all counties intersecting lakes - regions Spatial joins: find (quickly) all counties intersecting lakes Naive algorithm: O( N * M) Something faster? z obj-id etc 0010 LG 1000 WS 11** LG z obj-id etc 0011 Erie 0101 Erie 10** Ont C. Faloutsos C. Faloutsos 76 - regions Spatial joins: find (quickly) all counties intersecting lakes Solution: merge the lists of (sorted) z-values, looking for the prefix property - Detailed outline main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations footnote#1: * needs careful treatment footnote#2: need dup. elimination C. Faloutsos C. Faloutsos 78 13

14 - variations Q: is the best we can do? - variations Q: is the best we can do? : probably not - occasional long jumps Q: then? C. Faloutsos C. Faloutsos 80 - variations Q: is the best we can do? : probably not - occasional long jumps Q: then? 1: Gray codes - variations 2: Hilbert curve! (a.k.a. Hilbert-Peano curve) C. Faloutsos C. Faloutsos 82 - variations Looks better (never long jumps). How to derive it? - variations Looks better (never long jumps). How to derive it? order-1 order-2... order (n+1) C. Faloutsos C. Faloutsos 84 14

15 - variations Q: function for the Hilbert curve ( h = f(x,y) )? : bit-shuffling, followed by post-processing, to account for rotations. Linear on # bits. See textbook, for pointers to code/algorithms (eg., [Jagadish, 90]) - variations Q: how about Hilbert curve in 3-d? n-d? : Exists (and is not unique!). Eg., 3-d, order- 1 Hilbert curves (Hamiltonian paths on cube) #1 # C. Faloutsos C. Faloutsos 86 - Detailed outline main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations - analysis Q: How many pieces ( quad-tree blocks ) per region? : C. Faloutsos C. Faloutsos 88 - analysis Q: How many pieces ( quad-tree blocks ) per region? : proportional to perimeter (surface etc) - analysis (How long is the coastline, say, of England? Paradox: The answer changes with the yardstick -> fractals...) C. Faloutsos C. Faloutsos 90 15

16 - analysis Q: Should we decompose a region to full detail (and store in -tree)? - analysis Q: Should we decompose a region to full detail (and store in -tree)? : NO! approximation with 1-3 pieces/zvalues is best [Orenstein90] C. Faloutsos C. Faloutsos 92 - analysis Q: how to measure the goodness of a curve? - analysis Q: how to measure the goodness of a curve? : e.g., avg. # of runs, for range queries C. Faloutsos 93 4 runs 3 runs (#runs ~ #disk accesses on -tree) C. Faloutsos 94 - analysis Q: So, is Hilbert really better? : 27% fewer runs, for 2-d (similar for 3-d) Q: are there formulas for #runs, #of quadtree blocks etc? : Yes ([Jagadish; Moon+ etc]) - fun observations Hilbert and curves: space filling curves : eventually, they visit every point in n-d space - therefore: order-1 order-2... order (n+1) C. Faloutsos C. Faloutsos 96 16

17 - fun observations... they show that the plane has as many points as a line (-> headaches for 1900 s mathematics/topology). (fractals, again!) - fun observations Observation #2: Hilbert (like) curve for video encoding [Y. Matias+, CRYPTO 87]: Given a frame, visit its pixels in randomized hilbert order; compress; and transmit order-1 order-2... order (n+1) C. Faloutsos C. Faloutsos 98 - fun observations In general, Hilbert curve is great for preserving distances, clustering, vector quantization etc - Detailed outline main idea - 3 methods use w/ -trees; algorithms (range, knn queries...) non-point (eg., region) data analysis; variations Conclusions C. Faloutsos C. Faloutsos 100 Conclusions is a great idea (n-d points -> 1-d points; feed to -trees) used by TIGER system and (most probably) by other GIS products works great with low-dim points C. Faloutsos

Roadmap DB Sys. Design & Impl. Review. Detailed roadmap. Interface. Interface (cont d) Buffer Management - DBMIN

Roadmap DB Sys. Design & Impl. Review. Detailed roadmap. Interface. Interface (cont d) Buffer Management - DBMIN 15-721 DB Sys. Design & Impl. Buffer Management - DBMIN Christos Faloutsos www.cs.cmu.edu/~christos Roadmap 1) Roots: System R and Ingres 2) Implementation: buffering, indexing, q-opt 3) Transactions: