Overview of Query Processing and Optimization

Transcription

1 Overview of Query Processing and Optimization Source: Database System Concepts Korth and Silberschatz Lisa Ball, 2010 (spelling error corrections Dec 07, 2011)

2 Purpose of DBMS Optimization Each relational algebra query can generate multiple execution plans ordering of operations (σ, π, ) use of access structures (index files) Different strategies vary greatly in # of disk accesses Goal: Free user from worrying about writing efficient SQL

3 Basic Query Processor Operations Input: SQL Query (declarative) Parse Query checks syntax, table existence Input: relational algebra (procedural) Find equivalent, but more efficient expressions Input: optimized rel algebra expressions Select execution plan-- order of operations & index files to use Goal: minimize the # of disk accesses Execute Query

4 Example Database cust_name, street, city CUSTOMER branch_name, acct_no, cust_name, balance DEPOSIT branch_name, assets, branch_city BRANCH

5 Order of Operations Heuristic # 1: PERFORM SELECTIONS EARLY Q1: List the name and assets of all banks with depositors living in Port Chester sql query: select B.branch_name, B.assets from BRANCH B, DEPOSIT D, CUSTOMER C where B.branch_name=D.branch_name and D.cust_name=C.cust_name and C.city= Port Chester

6 Heuristic # 1: Options for relational algebra translation Option 1: joins 1 st π branch_name,assets (σ city= Port Chester (customer * deposit * branch)) Option 2: select 1 st π branch_name,assets ((σ city= Port Chester customer) * deposit * branch) Both options are equivalent, in that they return the same data, but option 1 first joins all 3 tables, generating huge intermediate results on disk. We only need a fraction of the rows! KEY HERE: the select only applies to 1 relation (CUSTOMER)

7 Order of Operations Heuristic # 1: PERFORM SELECTIONS EARLY Q2: List the name and assets of all banks with depositors living in Port Chester with a balance > $1000. sql query: select B.branch_name, B.assets from BRANCH B, DEPOSIT D, CUSTOMER C where B.branch_name=D.branch_name and D.cust_name=C.cust_name and C.city= Port Chester and D.balance > 1000

8 Heuristic #2: Apply selects to minimum required tables Option 1: join all 1 st π branch_name,assets (σ city= Port Chester and balance>1000 (customer * deposit * branch)) Option 2: apply select to join of needed tables π branch_name,assets ((σ city= Port Chester and balance>1000 customer * deposit) * branch) Option 3: apply select to join of needed tables using rule at end of slide π branch_name,assets (σ city= Port Chester (σ balance>1000 (customer * deposit )) branch)) Option 4: apply select to minimal tables needed π branch_name,assets ((σ city= Port Chester customer) * (σ balance>1000 deposit) * branch) All options are equivalent, in that they return the same data, but option 4 reduces the size of the intermediate tables by performing the selects 1 st KEY HERE: σ P1 AND P2 (e) σ P1 (σ P2 (e))

9 Heuristic # 3: Perform Projections Early Option 1: project 1 st, select 2 nd, join last π branch_name,assets ((π branch_name, ((σ city= Port Chester (customer)) *deposit)) * branch)) Looking back at Q1, we get a relation with many unnecessary columns KEY HERE: reduce size of intermediate tables by only using columns needed (ones that appear in result OR are needed in subsequent operations, e.g. joins)

10 Heuristic # 4: Order join operations to reduce the size of temporary tables Query 1: List the name and assets of all banks with depositors living in Port Chester Query 1 relational algebra (join of 3 tables) π branch_name,assets ((σ city= Port Chester customer) * deposit * branch) Natural join is associative, so we can choose the order in which tables are joined Option 1 (NOTbest choice): join deposit and branch 1 st Likely to be large 1 row for each deposit account (it matches 1 row in in branch) So if deposit has 10,000 rows, the result of this intermediate join will be 10,000 Option 2 (WORST choice): do 1 st (σ city= Port Chester customer) * branch No common join attribute Cartesian product So if there are 200 Port Chester customers and 3000 branches, the result of this intermediate join is 2000 X 3000 = 600,000 Option 3 (BEST choice): do 1 st (σ city= Port Chester customer) * deposit If the bank has many branches in different cities, this will be much smaller So if there are 200 Port Chester customers, each matching 2 deposit records on average, the result of this intermediate join is 200 X 2 = 400

11 Other Rewrite Rules to Aid Efficiency σ p (r 1 r 2 ) = σ p (r 1 ) (r 2 ) select before take union σ p (r 1 - r 2 ) = σ p (r 1 ) - r 2 = σ p (r 1 ) - σ p (r 2 ) example: list CS students (r1) with GPA>3.5 that are not also EE students (r2) (r 1 r 2 ) r 3 = r 1 (r 2 r 3 )

12 Strategies for Complex Queries May have a LARGE # of possible expressions that APPEAR to be efficient Ways to cope: Use heuristics to choose an expression Generate all promising expressions, estimate processing cost for every one (we ll see next) In choosing strategies, look at size of each relation distribution of values within columns (selectivity), e.g. Name is very selective (few duplicates) Gender is not very selective (about ½ rows for each)

13 Estimating Query Processing Cost To estimate query processing costs, the DB needs to store some stats, e.g., n r = the # of records in relation r s r = the size of a record in bytes in r V(A,r) = the # of distinct values that appear in relation r for attribute A (e.g., gender=2)

14 Estimating Query Processing Cost So, an upper limit on the size of a binary join is the Cartesian product r s, which has n r * n s rows each record is s r + s s bytes long V(A,r) lets us estimate the # of rows that satisfy a selection predicate of the form <attribute-name> = <value> How? next slide

15 Estimating Query Processing Cost How we estimate # rows of an natural join assume a UNIFORM distribution of values then σ A=a (r) has approximately Example, if V(Gender,2), a table with 1000 rows will have = 500 rows Note: this may not be very accurate large bank branches may have many more depositors than small branches

16 Estimating Natural Join Size r 1 * r 2 : join on the key of r 1 will have r 2 rows Example (Company DB ) Employee * Dependent (join attribute is SSN) will have the same # of rows as Dependent (n r2 ), because each Dependent joins with exactly 1 row of Employee

17 Estimating Natural Join Size r 1 * r 2 : join attribute is not a key of r 1 or r 2 Example (Company DB ) Employee * Dept_Location (join attribute is DNO) each row in r 1 will join with rows in r 2 Equation 1 so, the join will have a total estimate of: Equation 2

18 Estimating Natural Join Size Example of the Employee*Dept_Location: Let Employee (r 1 ) have 50 rows Dept_Location (r 2 ) have 15 rows V(DNO, Dept_Location) = 5 Then we have An estimate of (15/5) =3 rows from Dept_Location for each join with Employee (using eq. 1) Giving us (eq. 2): 3 * 50 = 150 rows in this join

19 Estimating Natural Join Size The previous result generally is the same as using Equation 3 Using same # of rows as before, Employee (r 1 ) has 50 rows Dept_Location (r 2 ) has 15 rows V(DNO, Dept_Employee) = 5 Then we have An estimate of (50/5) =10 rows from Employee for each join with Dept_Location (using variant of eq. 1) Giving us (eq. 3): 10* 15 = still an estimate of 150 rows in this join Note: the 2 estimates could be different if there are dangling tuples (nulls that won t partcipate in the join)

20 To keep in mind All this estimation requires that the DBMS keeps the statistics needed for these calculations Often these stats are only updated during periods of lighter system load But these stats are giving us estimates of the # of rows we really care about the # of disk accesses, which are influenced by: physical organization (we ll be using worst case in our discussions) access structures (indices) available

21 Access Cost Estimation Using Index Files and Hashing An access plan for a query has: relational operations to be performed the order in which the operations are to be performed the access structures to be used the order in which the rows are to be accessed 1 st we ll look at queries involving 1 tables

22 Cost Estimation Example Query 3 (Q3): SELECT acct_no FROM deposit WHERE branch-name= Hurst and cust-name= Fagin and balance > 1000 We have the following statistics bfr = 20 for deposit (20 records/block) V(branch-name,deposit) = 50 V (cust-name, deposit) = 200 V(balance, deposit) = 5000 Deposit has 10,000 rows (10,000/20 = 500 blocks) Index structures (next page )

23 Cost Estimation Example We have the following index files on deposit: A clustering 2-level index on branch-name A non-clustering (2 ), 2-level index on cust-name 1 st level of index 2 nd (base) level list of ptrs (extra level of indirection)- we ll ignore actual file

24 Cost Estimation: Plan #1 Use branch-name index (clustering) Analysis: Since V(branch-name,deposit) = 50 10,000 rows / 50 possible values = 200 rows for Hurst We need to read each of these blocks to check for Fagin and balance > 1000 The 200 rows are clustered into about 200/20 = 10 blocks for the deposit file Using this index (2 levels) requires 10 blocks + 2 index block reads = 12 block accesses

25 Cost Estimation: Plan #2 Use cust-name 2 index (non-clustering, 2 level) Analysis: Since V(cust-name,deposit) = ,000 rows / 200 possible values = 50 rows for Fagin We need to read each of these blocks to check for Hurst and balance > 1000 The 50 rows are not clustered, so in the worst case each record will be in a separate block, so we must read 50 blocks Using this index (2 levels) requires 50 blocks + 2 index block reads = 52 block accesses (vs. 12) If branch-name was non-clustering: we would have = 202 block accesses (worst case)

26 Cost Estimation: Plan #3 This time, both attributes (cust-name and deposit) use a non-clustering (2 ) index Since we can use index files for non-key attributes, we have index pointers to the blocks of record pointers (the extra level of indirection that we aren t counting here) Idea: take the intersection of the indirect record pointers of each index.

27 Cost Estimation: Plan #3 Let P 1 = pointers to records with cust-name Fagin (200) P 2 = pointers with branch-name Hurst (50) This means P 1 P 2 = set of pointers with cust-name= Fagin AND branch-name= Hurst To estimate block accesses we assume uniform distribution and attribute independence (Hurst doesn t imply we must have Fagin records or vice versa) P 1 P 2 = min(200, 50) 50 records at most have both attributes

28 Cost Estimation: Plan #3 Since 50 records, at most, have both attributes, out of 10,000 records, we get Probability(Fagin and Hurst) = 50/10,000 = 1/2000 1/2000 * 10,000 = 5 records So, we get = 5 block accesses (vs. 12, 52, 202 in our previous examples)

29 Cost Estimation: Plan #4 What about using balance in our plan? No index on balance attribute a non equality comparison (balance > 1000) is usually less selective that =

30 Single Table Optimization: Final Comments For any relational algebra expression, the optimizer may be able to formulate many different access plans During access plan selection, the query optimizer chooses the best strategy for a given expression Different plans can vary significantly in terms of real cost (# of block accesses) What appears to be a more efficient relational algebra expression may in fact have only mediocre plans, depending on file organization and available access structures

31 Cost Estimation: Joins Best strategy factors physical order of rows index (1, 2 ) possible creation of temporary index files memory buffer size worst case: 1 block per tables best case: 1 or both files can fit in memory Join example: deposit * customer n deposit = 10,000 n customer = 200

32 Cost Estimation: Joins # of comparisons to be done How many comparisons do we have to do? (this does not consider blocking factors) 10,000 * 200 = 2,000,000 comparisons!

33 Cost Estimation: Joins Algorithom 1: Simple Iteration deposit=10,000 and customer = 200 rows * this is just for illustrative purposes, or when bfr=1 for each row d in deposit for each row c in customer test(d,c) // cust-name match // worst case (1 read for each row) // for each outer loop, inner loop is runs 200 times // outer loop runs 10,000 times // total = (200 * 10,000) + 10,000 = 2,010,000 reads // best case (inner table or both fit in memory) // for each outer loop, inner loop is runs 200 times, but // this all fits in memory, so is read 1 time // outer loop runs 10,000 times // total = ,000 = 10,200 reads

34 Cost Estimation: Joins Algorithom 2: Block Oriented Iteration deposit=10,000 and customer = 200 rows bfr deposit = 20 requires 10,000/20 ==500 blocks bfr customer=20 requires 200/20 = 10 blocks for each block Bd in deposit for each block Bc in customer for each row d in Bd // in memory for each row c in Bc //in memory test (d,c) worst case (1 disk read for each block) 500 reads for deposit, 10 * 500 reads for customer Total: 5500 block reads best case (inner table or both fit in memory) 10 for customer for deposit = Total: 510 block reads

35 Algorithm 3: Merge-Join When to use: Neither table fits in memory Both tables are sorted by the join attribute Method: Associate a pointer with each table A group of rows with the join attribute value will match a consecutive group of matching rows in the other table Reads each block one time (same as when inner or both tables fit into memory) Example: next slide

36 Merge Join P 1 (STARTS AT 1ST BLOCK) Customer DB EJ OO RP once we ve read the last matching block in P 2, we know there will be no more matches P 2 (STARTS AT 1ST BLOCK) Deposit DB DB DB EJ OO OO OO OO RP RP

37 Algorithm Comparison Summary for Join Algorithms Number of disk reads Algorithm1: Simple iteration (single row reads) 2,010,000 Algorithm 1: Simple iteration (inner table in memory) 10, 200 Algorithm 2: Block oriented iteration (single block reads) 5,500 Algorithm 2: Block oriented iteration (inner table in memory) 510 Algorithm 3: Merge Join (single block reads)* 510 Algorithm 4: Index join ** (note: save 1,970,000 vs. 1 st row) 40,000 Remember: selection of these algorithms depend on file size, memory buffer size, file sort order, and index availability Both table are sorted by join attribute; neither file fits in memory Neither table is sorted by join attribute