Roadmap DB Sys. Design & Impl. Review. Detailed roadmap. Interface. Interface (cont d) Buffer Management - DBMIN

Transcription

1 DB Sys. Design & Impl. Buffer Management - DBMIN Christos Faloutsos Roadmap 1) Roots: System R and Ingres 2) Implementation: buffering, indexing, q-opt 3) Transactions: locking, recovery 4) Distributed DBMSs 5) Parallel DBMSs: Gamma, Alphasort 6) OO/OR DBMS 7) Data Analysis - data mining 8) Benchmarks 9) vision statements extras (streams/sensors, graphs, multimedia, web, fractals) C. Faloutsos 2 Detailed roadmap 1) Roots: System R and Ingres 2) Implementation: buffering, indexing, q-opt OS support for DBMS R-trees and GiST Z-ordering Buffering Joins... 3) Transactions: locking, recovery Review DB accesses are page-oriented Need to cache DBMS disk pages Buffer pool: a set of page frames, each of which can hold a disk page. frames C. Faloutsos C. Faloutsos 4 Interface Interface (cont d) Hash table maps pageid to BP index entries Getpage (pageno) returns memory address - Check buffer pool for page - If not found, get from disk - Fix page in buffer pool - Note the reference - Return address of page C. Faloutsos 5 Unfixpage(pageNo) decrements fix count Flushpage(pageNo) force page to disk C. Faloutsos 6 1

2 Chou and DeWitt - Outline Review of Algorithms Domain separation (Reiter) Extensions to domain separatilgorithm Ingres proposal Hot-set approach DBMIN - ideas and algorithms Experiments Domain Separation (Reuter) Classify pages as types Each type has a domain of buffers LRU within domain Example: B + -tree index One domain per index level One domain per leaf/data pages C. Faloutsos C. Faloutsos 8 Domain Separation (cont.) Problems? Domain Separation (cont.) Problems with this approach Static domains (relative importance depends on query) Doesn t prevent interference among users Doesn t prevent thrashing C. Faloutsos C. Faloutsos 10 Domain Separation (cont.) Extensions Priority ranking for domains to find free pages Dynamically vary domain sizes in WS-like way INGRES proposal (Kaplan) New algorithm: each relation needs a working set Subdivide buffer pool / allocate per-relation Link resident sets in priority order / global free list on top Search for free page via priority chain Use MRU for resident sets (but keep >=1 active buffer) C. Faloutsos C. Faloutsos 12 2

3 INGRES proposal (cont d) Problems? INGRES proposal (cont d) Problems (except that it didn t add much!) How to determine priority? MRU not always good Costly search under high loads Not multi-user (hard to determine priority) C. Faloutsos C. Faloutsos 14 Hot Set (Sacco & Schkolnick) hot set: set of pages over which there is looping behavior hot set in memory Ÿ efficient query processing #page faults vs. size of buffers in partitions Discontinuities: hot points Hot Set (cont d) Key ideas Give query hot set pages Allow 1 deficient query to execute Hot set size computed by query optimizer Use LRU replacement within each partition C. Faloutsos C. Faloutsos 16 Problems? Hot Set (cont d) Hot Set (cont d) Problems LRU not always fast => allocate more memory! Over-allocates pages for some phases of query C. Faloutsos C. Faloutsos 18 3

4 Chou and DeWitt - Outline Review of Algorithms Domain separation (Reiter)... DBMIN - ideas and algorithms Experiments DBMIN (Chou & DeWitt) Based on Query Locality Set Model DBMSs support a limited set of operations Reference patterns exhibited are predictable Decompose complex patterns into simple ones Identify locality sets C. Faloutsos C. Faloutsos 20 Which patterns? Which patterns? Sequential (+ variations) Random Hierarchical C. Faloutsos C. Faloutsos 22 Sequential Patterns Straight sequential (SS) File scan #pages? Straight sequential (SS) File scan Need one page C. Faloutsos C. Faloutsos 24 4

5 Straight sequential (SS) File scan Need one page Replaced with next one Clustered sequential (CS) Like inner S for mergejoin (sequential w/ backup) Join condition: R.a=S.a # of pages? a=4 a=4 a=4 a=7 a=7 a=8 a=4 a=4 a=4 a=4 a=7 a= C. Faloutsos C. Faloutsos 26 Clustered sequential (CS) Like inner S for mergejoin (sequential w/ backup) Need # of pages in largest cluster a=4 a=4 a=4 a=7 a=7 a=8 a=4 a=4 a=4 a=4 a=7 a=7 Clustered sequential (CS) Like inner S for mergejoin (sequential w/ backup) Need # of pages in largest cluster FIFO or LRU are good a=4 a=4 a=4 a=7 a=7 a=8 a=4 a=4 a=4 a=4 a=7 a= C. Faloutsos C. Faloutsos 28 Looping sequential (LS) Like inner S for nestedloop-join # of pages? Looping sequential (LS) Like inner S for nestedloop-join As many pages as possible C. Faloutsos C. Faloutsos 30 5

6 Random Patterns Looping sequential (LS) Like inner S for nestedloop-join As many pages as possible MRU Independent Random (IR) Non-clustered index scan # of pages? non-clustered index C. Faloutsos C. Faloutsos 32 Random Patterns (cont.) Random Patterns (cont.) Independent Random (IR) Non-clustered index scan One page (assuming low prob. of reaccess) non-clustered index Independent Random (IR) Non-clustered index scan One page (assuming low prob. of reaccess) Any replacement algorithm! non-clustered index C. Faloutsos C. Faloutsos 34 Random Patterns (cont.) Random Patterns (cont.) Clustered Random (CR) Inner, non-clustered index on join column Clustered Random (CR) Inner, non-clustered index on join column # of pages? # of records in largest cluster C. Faloutsos C. Faloutsos 36 6

7 Random Patterns (cont.) Hierarchical Patterns Clustered Random (CR) Inner, non-clustered index on join column # of records in largest cluster As in CS Straight Hierarchical (SH) Access index pages ONCE (retrieve a single tuple) # of pages? index C. Faloutsos C. Faloutsos 38 Hierarchical Patterns (cont.) Hierarchical Patterns (cont.) Straight Hierarchical (SH) Access index pages ONCE (retrieve a single tuple) Like SS index C. Faloutsos 39 Hierarchical w/ straight/clustered sequential (H/SS or H/CS) Hierarchical w/ SS or CS leaf scan Like SS/CS 2 a2= Lo a2= Hi C. Faloutsos 40 e.g., clustered index Hierarchical Patterns (cont.) Looping Hierarchical (LS) When inner index in join is repeatedly accessed LIFO need to keep root Chou and DeWitt - Outline Review of Algorithms Domain separation (Reiter)... DBMIN - ideas and algorithms Experiments C. Faloutsos C. Faloutsos 42 7

8 DBMIN policy Buffers allocated per-file instance basis Active instances of same file have different BPs Those are independently managed May share a same buffered page through global table DBMIN policy (cont d) Each file instance has its locality set of pages Each page in buffer belongs to at most 1 lset Global, shared table of buffers too C. Faloutsos C. Faloutsos 44 Parameters N total number of buffers I ij max number of buffers for file instance j of query i (desired size) r ij number of buffers allocated for file instance j of query i (actual size) DBMIN Algorithm When a query requests a page, search global table: Found in global table and locality set Update usage stats In memory, not in locality set If already owned by someone else, return it Else, return to locality set and increment r ij If r ij > I ij, release a page to global free list C. Faloutsos C. Faloutsos 46 DBMIN Algorithm (cont.) Not in memory Get a free buffer Schedule a read, then do in memory from above On file open/close, do load control: (Open): if Σ i Σ j I ij < N, query can proceed, w/o blocks (Close): release buffers to free list, unblock one or more other queries Chou and DeWitt - Outline Review of Algorithms Domain separation (Reiter)... DBMIN - ideas and algorithms Experiments C. Faloutsos C. Faloutsos 48 8

9 Performance Results Workload? Compared to Rand FIFO Clock WS Hot Set DBMIN C. Faloutsos C. Faloutsos 50 Workload? 6 queries, 3 mixes - Queries: q1: selection (clustered index) q2: selection (non-clustered index) q3: sel (cl-ind) + join (index-join) q4: seq scan + index join (non-cl-index) q5: sel (cl-ind) + join (n.l.) q6: sel (cl-ind) + hash join C. Faloutsos 51 Workload? 6 queries, 3 mixes - Mixes: Mix1: all 6 queries equally Mix2: more of q1 and q2 (selections) Mix3: much more of q1, q C. Faloutsos 52 Throughput Mix 1 no data sharing who is who? Throughput HOT DBMIN Mix 1 no data sharing who is who? CLOCK RAND FIFO WS NCQ NCQ C. Faloutsos C. Faloutsos 54 9

10 Throughput HOT DBMIN Mix 1 w/ data sharing what changes? Throughput HOT DBMIN Mix 1 w/ data sharing what changes? A: all move up 0.20 CLOCK RAND FIFO WS 0.20 CLOCK RAND FIFO WS NCQ NCQ C. Faloutsos C. Faloutsos 56 What about the lighter, M2 and M3 mixes? What about the lighter, M2 and M3 mixes? A: similar performance (higher throughput) C. Faloutsos C. Faloutsos 58 Performance Results (cont.) DBMIN did best Hot set was next WS was next (trouble with join loops) Then: clock, FIFO, rand (thrashing as MPL increases) Load control helps C. Faloutsos 59 10