EECS 647: Introduction to Database Systems

Transcription

1 EECS 647: Introduction to Database Systems Instructor: Luke Huan Spring 2009

2 External Sorting Today s Topic Implementing the join operation 4/8/2009 Luke Huan Univ. of Kansas 2

3 Review DBMS Architecture User/Web Forms/Applications/DBA query transaction Query Parser Transaction Manager Query Rewriter Today s lecture Query Optimizer Query Executor Lock Manager Logging & Recovery Files & Access Methods Buffer Manager Storage Manager Buffers Lock Tables Main Memory Storage 4/8/2009 Luke Huan Univ. of Kansas 3

4 A query s trip through the DBMS <Query> <SFW> <select-list> <where-cond> <from-list> <table> <table> Enroll Course PROJECT (title, SID) Sort MERGE-JOIN (CID) SORT (CID) SORT (CID) SCAN (Enroll) SCAN (Course) SQL query Parser Parse tree Rewritor Logical plan Optimizer Physical plan Executor Result SELECT title, SID FROM Enroll, Course WHERE Enroll.CID = Course.CID; π title, SID σ Enroll.CID = Course.CID Enroll Course 4/8/2009 Luke Huan Univ. of Kansas 4

5 Index and Data File Organization Heap file vs. B + -tree Index file Header Page Data Page Data Page Dat a Pag e Data Page Data Page Data Page Full Pages Pages with Free Space 4/8/2009 Luke Huan Univ. of Kansas 5

6 Cost of Operations B: The number of data pages R: Number of records per page D: (Average) time to read or write disk page Heap File Sorted File Tree-indexed File Scan all records BD BD 1.5 BD Equality Search 0.5 BD (log 2 B) * D (log F 1.5B) * D Range Search BD search + #match pg*d search+ #match pg*d Insert 2D search + BD search+ D Delete search + D search + BD search + D

7 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing gpa order Sorting is first step in bulk loading B+ tree index. Sorting useful for eliminating duplicate copies in a collection of records (Why?) Sorting is useful for summarizing related groups of tuples Sort-merge join algorithm involves sorting. Problem: sort 100Gb of data with 1Gb of RAM. why not virtual memory?

8 2-Way Sort: Requires 3 Buffers Pass 0: Read a page, sort it, write it. only one buffer page is used (as in previous slide) Pass 1, 2, 3,, etc.: requires 3 buffer pages merge pairs of runs into runs twice as long three buffer pages used. INPUT 1 INPUT 2 OUTPUT Disk Main memory buffers Disk

9 Two-Way External Merge Sort Each pass we read + write each page in file. N pages in the file => the number of passes = log + 2 N 1 So total cost is: ( log N + ) 2N 1 2 Idea: Divide and conquer: sort subfiles and merge 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 2,3 4,4 6,7 8,9 4,7 8,9 1,2 2,3 3,4 4,5 6,6 7,8 9 1,3 5,6 2 1,2 3,5 6 Input file PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 8-page runs

10 Using B+ Trees for Sorting Scenario: Table to be sorted has B+ tree index on sorting column(s). Idea: Can retrieve records in order by traversing leaf pages. Is this a good idea? Cases to consider: B+ tree is clustered B+ tree is not clustered Good idea! Could be a very bad idea!

11 Clustered B+ Tree Used for Sorting Cost: root to the left-most leaf, then retrieve all leaf pages (primary index) Index (Directs search) If clustering index is used? Additional cost of retrieving data records: each page fetched just once. Data Records Data Entries ("Sequence set") Always better than external sorting!

12 Unclustered B+ Tree Used for Sorting Unclustered index for data entries; each data entry contains rid of a data record. In general, one I/O per data record! Index (Directs search) Data Entries ("Sequence set") Data Records

13 SQL and Relational algebra Review Student (Sid: string, name: string, GPA: float) Register (Sid: string, Cid: string) SELECT name, cid FROM Student, Register WHERE Student.Sid = Register.Sid π name, cid R S.sid = R.sid S 4/8/2009 Luke Huan Univ. of Kansas 13

14 Relational algebra operators Selection: σ p R Projection:π L R Cross product: R S Union: R S Difference: R - S Other ones: Join R p S Renaming ρ S(A1, A 2, ) R Review 4/8/2009 Luke Huan Univ. of Kansas 14

15 Review and Preview How to implement the following SQL query: Student (Sid: string, name: string, GPA: float) SELECT * FROM Student WHERE GPA > 3.0 σ S.GPA>3.0 S Heap File Sorted File Clustered File with index Range Search BD search + #match pg*d search+ #match pg*d 4/8/2009 Luke Huan Univ. of Kansas 15

16 P.buyer, P.item SELECT FROM Purchase P, Person Q WHERE P.buyer=Q.name AND Q.city= lawrence AND Q.age < 20 Query Plan: logical plan (declarative) physical plan (procedural) Review Query Tree σ City= Lawrence age < 20 Purchase buyer,item buyer=name Person procedural implementation of each logical operator scheduling of operations π (Table scan) (Simple nested loops) (Index scan) 4/8/2009 Luke Huan Univ. of Kansas 16

17 Logical v.s. Physical Operators Logical operators: what they do e.g., union, selection, project, join, grouping Physical operators: how they do it e.g., nested loop join, sort-merge join, hash join, index join How should we design the physical operators? Query execution has to do with how to implement physical operators 4/8/2009 Luke Huan Univ. of Kansas 17

18 Today s Focus How to implement the query processing unit in a DBMS? Logical and physical operators Selection, Projection Join Set operation Union, Difference, Intersection 4/8/2009 Luke Huan Univ. of Kansas 18

19 Notation Relations: R, S Tuples: r, s Number of tuples: R, S Number of disk blocks: B(R), B(S) Number of memory blocks available: M Cost metric Number of I/O s Memory requirement 4/8/2009 Luke Huan Univ. of Kansas 19

20 Linear scan Scan table R and process the query Selection over R Projection of R without duplicate elimination I/O s: B(R) Trick for selection: stop early if it is a lookup by key Memory requirement: 2 Not counting the cost of writing the result out Same for any algorithm! Maybe not needed results may be pipelined into another operator 4/8/2009 Luke Huan Univ. of Kansas 20

21 Join Operation Nested-loop join (brute force) Sort-merge join Indexed join 4/8/2009 Luke Huan Univ. of Kansas 21

22 R p S Nested-loop join For each block of R, and for each r in the block: For each block of S, and for each s in the block: Output rs if p evaluates to true over r and s R is called the outer table; S is called the inner table I/O s: B(R) + R B(S) Memory requirement: 3 Improvement: block-based nested-loop join For each block of R, and for each block of S: For each r in the R block, and for each s in the S block: I/O s: B(R) + B(R) B(S) Memory requirement: same as before 4/8/2009 Luke Huan Univ. of Kansas 22

23 More improvements of nested-loop join Stop early if the key of the inner table is being matched Make use of available memory Stuff memory with as much of R as possible, stream S by, and join every S tuple with all R tuples in memory I/O s: B(R) + B(R) / (M 2 ) B(S) Or, roughly: B(R) B(S) /M Memory requirement: M (as much as possible) 4/8/2009 Luke Huan Univ. of Kansas 23

24 When do we need nested-loop join? May be best if many tuples join Example: non-equality joins that are not very selective Necessary for black-box predicates Example: WHERE user_defined_pred(r.a, S.B) 4/8/2009 Luke Huan Univ. of Kansas 24

25 R R.A = S.B S Sort-merge join Sort R and S by their join attributes, and then merge r, s = the first tuples in sorted R and S Repeat until one of R and S is exhausted: If r.a > s.b then s = next tuple in S else if r.a < s.b then r = next tuple in R else output all matching tuples, and r, s = next in R and S I/O s: sorting + 2 B(R) + 2 B(S) 4/8/2009 Luke Huan Univ. of Kansas 25

26 Example R: S: R R.A = S.B S: r 1.A = 1 s 1.B = 1 r 2.A = 3 s 2.B = 2 r 3.A = 3 s 3.B = 3 r 4.A = 5 s 4.B = 3 r 5.A = 7 s 5.B = 8 r 6.A = 7 r 7.A = 8 r 1 s 1 r 2 s 3 r 2 s 4 r 3 s 3 r 3 s 4 r 7 s 5 4/8/2009 Luke Huan Univ. of Kansas 26

27 Optimization of SMJ Idea: combine join with the merge phase of merge sort Sort: produce sorted runs of size M for R and S Merge and join: merge the runs of R, merge the runs of S, and merge-join the result streams as they are generated! Disk Memory Merge Sorted runs R S Merge Join 4/8/2009 Luke Huan Univ. of Kansas 27

28 Performance of two-pass SMJ I/O s: 3 (B(R) + B(S)) Memory requirement To be able to merge in one pass, we should have enough memory to accommodate one block from each run: M > B(R) /M+ B(S) /M M > sqrt(b(r) + B(S)) 4/8/2009 Luke Huan Univ. of Kansas 28

29 Other Operators that are supported by sort-based algorithms Union (set), difference, intersection More or less like SMJ Duplication elimination External merge sort Eliminate duplicates in sort and merge GROUP BY and aggregation External merge sort Produce partial aggregate values in each run Combine partial aggregate values during merge Partial aggregate values don t always work though Examples: SUM(DISTINCT ), MEDIAN( ) 4/8/2009 Luke Huan Univ. of Kansas 29

30 Equality predicate: σ A = v (R) Selection using index Use an ISAM, B + -tree, or hash index on R(A) Range predicate: σ A > v (R) Use an ordered index (e.g., ISAM or B + -tree) on R(A) Hash index is not applicable Indexes other than those on R(A) may be useful Example: B + -tree index on R(A, B) How about B + -tree index on R(B, A)? 4/8/2009 Luke Huan Univ. of Kansas 30

31 Index versus table scan Situations where index clearly wins: Index-only queries which do not require retrieving actual tuples Example: π A (σ A > v (R)) Primary index clustered according to search key One lookup leads to all result tuples in their entirety 4/8/2009 Luke Huan Univ. of Kansas 31

32 R R.A = S.B S Index nested-loop join Idea: use the value of R.A to probe the index on S(B) For each block of R, and for each r in the block: Use the index on S(B) to retrieve s with s.b = r.a Output rs I/O s: B(R) + R (index lookup) Typically, the cost of an index lookup is 2-4 I/O s Beats other join methods if R is not too big Better pick R to be the smaller relation Memory requirement: 2 4/8/2009 Luke Huan Univ. of Kansas 32

33 Zig-zag join using ordered indexes R R.A = S.B S Idea: use the ordering provided by the indexes on R(A) and S(B) to eliminate the sorting step of sort-merge join Trick: use the larger key to probe the other index Possibly skipping many keys that don t match B + -tree on R(A) B + -tree on S(B) 4/8/2009 Luke Huan Univ. of Kansas 33

34 Scan Summary of tricks Selection, duplicate-preserving projection, nested-loop join Sort External merge sort, sort-merge join, union (set), difference, intersection, duplicate elimination, GROUP BY and aggregation Index Selection, index nested-loop join, zig-zag join 4/8/2009 Luke Huan Univ. of Kansas 34