Query and Join Op/miza/on 11/5

Transcription

1 Query and Join Op/miza/on 11/5

2 Overview Recap of Merge Join Op/miza/on Logical Op/miza/on Histograms (How Es/mates Work. Big problem!) Physical Op/mizer (if we have /me)

3 Recap on Merge

4 Key (Simple) Idea To find an element that is no larger than all elements in two lists, one only needs to compare minimum elements from each list. A 1 <= A 2 <= <= A N B 1 <= B 2 <= <= B M Then Min {A 1, B 1 } <= A i for i=1.n and Min {A 1, B 1 } <= B j for j=1.m

5 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory Main Memory Two Sorted Files (disk) 1, 5 7,11 20,31 2, 22 23,24 25,30

6 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory Main Memory Two Sorted Files (disk) 1, 5 2, 22 7,11 20,31 23,24 25,30

7 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 1,5 2,22 Main Memory Two Sorted Files (disk) 7,11 20,31 23,24 25,30

8 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 5 22 Main Memory 1,2 Two Sorted Files (disk) 7,11 20,31 23,24 25,30

9 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 5 22 Main Memory 1,2 Two Sorted Files (disk) 7,11 20,31 23,24 25,30

10 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 22 Main Memory 5 1,2 What next? Two Sorted Files (disk) 7,11 20,31 23,24 25,30

11 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 22 Main Memory 5 1,2 Two Sorted Files (disk) 7,11 23,24 20,31 25,30

12 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 7,11 22 Main Memory 5 1,2 Two Sorted Files (disk) 23,24 20,31 25,30

13 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory Main Memory 5,7 1,2 Two Sorted Files (disk) 23,24 20,31 25,30

14 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory Main Memory 1,2 5,7 Two Sorted Files (disk) 23,24 20,31 25,30

15 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 22 Main Memory 11 1,2 5,7 Two Sorted Files (disk) 23,24 20,31 25,30

16 Merge BIG sorted files to produce BIGGER Sorted Files With SMALL memory 20,31 22 Main Memory 11 1,2 5,7 Two Sorted Files (disk) 23,24 25,30

17 We can merge lists of arbitrary length with only 3 buffer pages. If Lists of size N and M, then Cost: 2(N+M) if lists of size N,M. What if we merge B lists with B+1 buffer pages?

18 Query Op/miza/on

19 Op/miza/on Order the opera/ons within a query to reduce the cost. Major component of the database Most mysterious & important Heart of Query Processing (QP) QP is not rocket science. When you flunk out of QP, we make you go build rockets. anonymous

20 Join Op/miza/on

21 RA Reminder Find Names of sailors who ve reserved a red boat Find the names of sailors who reserved both a red boat and a green boat Warning: Keys Sailors(sid,sname,ra/ng,age) Reserves(sid,bid,date) Boats(bid,bname,color)

22 Schema for Examples Sailors (sid: integer, sname: string, rating: integer, age: real) Reserves (sid: integer, bid: integer, day: dates, rname: string) Reserves: Each tuple is 40 bytes long, 100 tuples per page, 1000 pages. Sailors: Each tuple is 50 bytes long, 80 tuples per page, 500 pages.

23 What is this doing? You too can type EXPLAIN! (you may also want to know ANALYZE) When it s slow, you d like to know!

24 Joins One of the most important for performance Many, many algorithms: All fun. SELECT * FROM Reserves R1, Sailors S1 WHERE R1.sid = S1.sid What is this in RA?

25 Some dry nota/on Given Rela/on R. Define the following two func/ons. T(R) = # of tuples in R B(R) = # of pages/blocks in R NB: I omit ceiling in calcula=ons. A good exercise is to put them in the appropriate places! NB2: We don t write the output wri=ng to disk cost!

26 Nested loop join 26

27 Nested Loop Joins Tuple- based nested loop R S for each tuple r in R do for each tuple s in S do if r and s join then output (r,s) Cost: B(R) + T(R) B(S). Why? B(R) = 500 T(R) = 50,000 B(S) = 1000 T(S) =200,000 then, 5e7 IOs. ~ 140 hours! What is the cost if we switch the R and S? 27

28 Block Nested Loop Joins for each (M- 1) blocks br of R do for each block bs of S do for each tuple s in bs do for each tuple r in br do if r and s join then output(r,s) Let M be the number of blocks in memory (M=11) B(R) = 500 T(R) = 50,000 B(S) = 1000 T(S) =200,000 NLJ =140 hrs BNLJ=.14 hrs NLJ = B(R) + T(R)B(S) BNLJ = B(R) + B(R)B(S)/(M- 1) 28

29 Nested Loop Joins Block- based Nested Loop Join S/ll a smart cross product. Nevertheless, useful! NB: it is faster to iterate over the smaller rela/on first R S: R=outer rela/on, S=inner rela/on 29

30 Smarter than Cross Products

31 Index Nested Loop Joins Index - based nested loop R S on A for each tuple r in R do for each tuple s find all s.t. r.a = s.a Does not evaluate the full cross product! Clustered B+ tree on S.A. All dis/nct values fit on a page. How much does this join cost? ~ B(R) + T(R)*3 (rule of thumb) 31

32 Sort Merge

33 Join: Sort- Merge (R S) Sort R and S on the join column, then scan them to do a ``merge (on join col.), and output result tuples. R is scanned once; each S group is scanned once per matching R tuple. Mul/ple scans of an S group are likely to find needed pages in buffer. If R, S are already sorted on the join key, SMJ is awesome!

34 Example of Sort- Merge Join sid sname rating age 22 dustin yuppy lubber guppy rusty Cost: 6 M + 6N + (M+N) sid bid day rname /4/96 guppy /3/96 yuppy /10/96 dustin /12/96 lubber /11/96 lubber /12/96 dustin The cost of scanning, M+N, could be M*N (very unlikely! When does this happen?) Here M (resp. N) is the size in Pages of R (resp. N)

35 Sort Merge v. Nested Loops steel cage match If we have 100 buffer pages, reserves is 1000 pages and Sailors 500 pages then Sort both in two passes: 2 * 2 * * 2 * 500 Merge phase so 7500 IOs What is BNLJ? *500/99 = 5550 But, if we have 35 buffer pages? Sort Merge has same behavior (s/ll 2 pass) BNLJ? ~ 15k IOs! NB: SMJ both rela/ons sorted in two passes

36 A simple op/miza/on: Merged! Observe. The last phase of the external sort is a merge, and we can merge the merge phases. Create sorted runs 2 * ( ) Each run is of length (B- 1) (approximately) There are 1000/(B- 1) + 500/(B- 1) such runs If ( )/(2(B- 1)) < B- 1 then all runs fit in memory, roughly if (M+N) < 2B 2 or max { M, N } < 2B 2 One can create runs of length 2(B- 1) using what s called a tournament sort used in PostgreSQL So we ll say max { M, N } < B 2 implies cost 3(M+N)

37 Hash Join

38 Hash- Join Original Rela9on OUTPUT 1 Par99ons (1) Par//on both rela/ons using hash fn h: R tuples in par==on i will only match S tuples in par==on i.... Disk INPUT hash func9on h B main memory buffers 2 B- 1 Disk 1 2 B- 1 (2) Read a par//on of R, hash it using h2. Scan matching par//on of S for matches. Par99ons of R & S hash fn h2 Hash table for par99on Ri (k < B- 1 pages) h2 Join Result Input buffer for Si Output buffer Should h = h2? Disk B main memory buffers Disk

39 How much memory does Hash join need to perform well? Good case=perform the join in 2 passes 1 st Point: How large are the par//ons? R is of size M S is of size N (wlog M < N) Par//on R into B- 1 buffer pages (why B- 1?) How many par//ons result? How big are they? Roughly, each par//on of R is f M/(B- 1) where f is some fudge factor.

40 How much memory does Hash join need to perform well? Good case=perform the join in 2 passes 2 nd Ques/on: During the probe phase, how much memory do we need? Key :Only smaller par//on needs to fit! Buffer needs to fit 1 par99on of R, 1 page of S, & output: B > f M / (B- 1) i.e., B 2 > fm The li le dog!

41 Sort- Merge v. Hash Join In par//oning phase, read+write both R,S; 2(M+N). In matching phase, read both R,S; M+N I/Os. Given a minimum amount of memory (what is this, for each?) both have a cost of 3(M+N) I/Os. Minimum memory: HJ : B 2 > min {M,N} pages i.e., the smaller rela/on SMJ: B 2 > max {M,N} pages i.e., the larger rela/on. Hash Join superior if rela/on sizes differ greatly. Why?

42 Further Comparisons of Hash and Sort Joins Hash Joins are highly parallelizable. Sort- Merge less sensi/ve to data skew and result is sorted

43 Observa/ons about Hash- Join In- memory hash table speeds up matching tuples, so li le more memory is needed (fudge factor). If the hash func/on does not par//on uniformly, one or more R par//ons may not fit in memory. What then? Can apply hash- join technique recursively to do the join of this R- par//on with corresponding S- par//on. SKEW!

44 Recall: Logical Op/miza/on

45 Single block SQL to RA Highly rated sailors SELECT DISTINCT S.sid FROM Sailors S, Reserves R WHERE s.sid = r.sid and s.ra/ng > 8 Highly rated sailors who reserve many different boats and how many boats they reserve SELECT S.sid, COUNT(DISTINCT Bid) FROM Sailors S, Reserves R WHERE s.sid = r.sid and s.ra/ng > 8 GROUP BY S.sid HAVING COUNT(DISTINCT Bid) > 5 How would you op/mize these?

46 Logical Op/miza/on Summary Use query equivalence to compute same output via different plans Key reason to use an algebra Oen logical rewri/ngs applied heuris/cally: Always convert selec/on + cross product to Join Asympto/c reduc/on Push down selec/ons and projec/ons Oen, but not always a good idea!

47 Physical Op/miza/on

48 One concept: Pipelining Intermediate results: could write them to disk or pipeline them to next operator RA Tree: sname bid=100 rating > 5 We can apply selec/on & projec/on on the fly. Why? sid=sid Reserves Sailors

49 Overview of Query Op/miza/on A Plan is Tree of R.A. ops with choice of algorithm for each op. Each operator typically implemented using a `pull interface: Two main issues: For a given query, what plans are considered? Algorithm to search plan space for cheapest (es/mated) plan. How is the cost of a plan es/mated? Ideally: Want to find best plan. Prac/cally: Avoid worst plans! We will study the System R approach.

50 Highlights of System R Op/mizer Impact: Most widely used; works well for < 10 joins. Cost es9ma9on: Approximate art at best. Sta/s/cs, maintained in system catalogs, used to es/mate cost of opera/ons and result sizes. Considers combina/on of CPU and I/O costs. Enumerates an en9re plan space: Too many plans so only leq- deep plans considered. Le- deep plans allow output of each operator to be pipelined into the next operator Cartesian products avoided. There are other styles now rule based.

51 An Example

52 How does it get those costs?

53 Cost Es/ma/on

54 Cost Es/ma/on For each plan considered, must es/mate cost: Must es/mate cost of each opera/on in plan tree. You know or can guess this: We ve already discussed how to es/mate the cost of opera/ons (sequen/al scan, index scan, joins, etc.) All es/mates depend on input cardinality Must also es/mate size of result for each opera/on in tree! For selec/ons and joins, assume independence of predicates. Let s see how to es/mate

55 Es/ma/ng Results Sizes Es/mate the reduc=on factor Column = value (e.g. Salary = 100k) If there is an index I then 1/#keys(I) If no index? 1/10 Column1 = Column2 (e.g. Salary = Age ) Index I1, I2, then 1/max(#keys(I1), #keys(i2)) If no Index? 1/10 Column > value If Index I then (High(I) value) / (High(I) Low(I)) No Index? 1/2 Later: Do be er with histograms

56 Histograms

57 Histograms A histogram idea is to make buckets count how many are in each bucket How to choose the buckets? Equiwidth & Equidepth Turns out high- frequency values are very important

58 Frequency 10 Abstract Example Values How do we compute how many values between 8 and 10? (Yes, it s obvious) Problem: Counts take too much space!

59 The Uniform is Red 10 How much space does this take to store?

60 Fundamental Tradeoffs Want high resolu/on (like the full counts) Want low space (like uniform) Histograms are a compromise!

61 The Uniform is Red 10 How do you es/mate # of tuples? What about point queries? A query

62 Equi width All buckets roughly the same width

63 Equidepth All buckets contain roughly the same number of items

64 Range Query: x in [5,8] All buckets roughly the same width

65 Histograms Simple, intui/ve and popular Parameters # of buckets and type Can extend to many a ributes (mul/dimensional)

66 Maintaining Histograms Histograms require that we update them! Typically, you must run/schedule a command to update sta/s/cs on the database Out of date histograms can be terrible! There is research work on self- tuning histograms and the use of query feedback Oracle 11g

67 Nasty example we insert many tuples with value > we do not update the histogram 3. we ask for values > 20?

68 When es/mates behave badly If we underes/mate the number of tuples, what kinds of plans suffer? Think about using unclustered indexes. If we overes/mate the number of tuples, what kinds of plans suffer? We could have used that index! Or we could have used a hash join in one pass instead of sor/ng in two!

69 Compressed Histograms One popular approach: 1. Store the most frequent values and their counts explicitly 2. Keep an equiwidth or equidepth one for the rest of the values People con/nue to try all manner of fanciness here that people try wavelets, graphical models, entropy models,