Discuss physical db design and workload What choises we have for tuning a database How to tune queries and views

Transcription

1 TUNING AND DB DESIGN 1

2 GOALS Discuss physical db design and workload What choises we have for tuning a database How to tune queries and views 2

3 STEPS IN DATABASE DESIGN Requirements Analysis user needs; what must the database capture? Conceptual Design high level description (often done w/er model) Logical Design translate ER into DBMS data model Typically: relational model as implemented by SQL

4 STEPS IN DATABASE DESIGN Schema Re nement consistency, normalization Physical Design indexes, disk layout Security Design who accesses what, and how 3. 1

5 WHERE CAN WE TUNE 1. Tuning indexes 2. Tuning conceptual schema 3. Tuning queries 3. 23

6 INTRODUCTION Physical design linked tightly to query optimization We must begin by understanding the workload: The most important queries and how often they arise. The most important updates and how often they arise. The desired performance for these queries and updates.

7 UNDERSTANDING THE WORKLOAD Which relations does it access? Which attributes are retrieved? For each query in the workload: Which attributes are involved in selection/join conditions? How selective are these conditions likely to be? 4. 1

8 UNDERSTANDING THE WORKLOAD For each update in the workload: Which attributes are involved in selection/join conditions? How selective are these conditions likely to be? The type of update (INSERT/DELETE/UPDATE), and the attributes that are affected. 4. 2

9 CREATING AN ISUD CHART Employee Table Transaction Frequency % table Name Salary Address Payroll Run Monthly 100 S S S Add Emps daily 0.1 I I I Delete Emps daily 0.1 D D D Give Raises monthly 10 S SU 4. 3

10 DECISIONS TO MAKE What indexes should we create? Which relations should have indexes? What eld(s) should be the search key? Should we build several indexes? For each index, what kind of an index should it be? Clustered? Hash/B+tree? 4. 4

11 DECISIONS TO MAKE Should we make changes to the conceptual schema? Normal Forms Vertical partitioning/denormalization Horizontal partitioning Replication Views 4. 5

12 INDEX SELECTION

13 GUIDELINES FOR INDEX SELECTION 1. Whether to index 2. Choise of search key 3. Multi-attribute search keys 4. Whether to cluster 5. Type if index (Hash vs Tree) 6. Balancing the cost of index maintenance

14 INDEX SELECTION SELECT E.ename, D.mgr FROM Emp E, Dept D WHERE E.dno=D.dno AND D.dname= 'Toy' Attributes mentioned in a WHERE clause are candidates for index search keys

15 INDEX SELECTION One approach: Consider most important queries in turn. Consider best plan using the current indexes, and see if better plan is possible with an additional index. If so, create it. Before creating an index, must also consider the impact on updates in the workload! Trade-off: indexes can make queries go faster, updates slower. Require disk space, too.

16 ISSUES IN INDEX SELECTION Attributes mentioned in a WHERE clause are candidates for index search keys. Try to choose indexes that bene t as many queries as possible. SELECT E.ename, D.mgr FROM Emp E, Dept D WHERE E.dno=D.dno AND D.dname= 'Toy' 5. 45

17 ISSUES IN INDEX SELECTION When considering a join condition: B+tree on inner is very good for Index Nested Loops. Should be clustered if join column is not a key for inner, and inner tuples need to be retrieved. Clustered B+ tree on join column(s) good for Sort-Merge.

18 ISSUES IN INDEX SELECTION 5. 6

19 EXAMPLE 1 (A) SELECT E.ename, D.mgr FROM Emp E, Dept D WHERE E.dno=D.dno AND D.dname= 'Toy' B+ tree index on D.dname supports 'Toy' selection. Given this, index on D.dno is not needed. B+ tree index on E.dno allows us to get matching (inner) Emp tuples for each selected (outer) Dept tuple. 5. 7

20 EXAMPLE 1 (B) SELECT E.ename, D.mgr FROM Emp E, Dept D WHERE E.dno=D.dno AND D.dname= 'Toy' AND E.age=25 What if WHERE included: AND E.age=25? Could retrieve Emp tuples using index on E.age, then join with Dept tuples satisfying dname selection. How do you compare it with the strategy that used E.dno index? 5. 8

21 EXAMPLE 2 SELECT E.ename, D.mgr FROM Emp E, Dept D WHERE E.sal BETWEEN AND AND E.hobby='Stamps' AND E.dno=D.dno What index should we build on Emp? B+ tree on E.sal could be used, OR an index on E.hobby could be used. Only one of these is needed, and which is better depends upon the selectivity of the conditions. As a rule of thumb, equality selections more selective than range selections. 5. 9

22 EXAMPLE 2 Should Emp be outer or inner relation in Index NL join? Outer, because all selections are on Emp Suggests that we build a B+ tree index on D.dno for INL Join As both examples indicate, our choice of indexes is guided by the plan(s) that we expect an optimizer to consider for a query. Have to understand optimizers! 5. 10

23 CLUSTERED AND INDEXING Only one index can be clustered per relation! Choose based on important queries bene tting most from clustering Range conditions are sensitive to clustering 5. 11

24 EXAMPLES OF CLUSTERING SELECT E.dno FROM Emp E WHERE E.age>40 B+ tree index on E.age can be used to get qualifying tuples. How selective is the condition? Is the index clustered? 6. 1

25 EXAMPLES OF CLUSTERING SELECT E.dno, COUNT (*) FROM Emp E WHERE E.age>10 GROUP BY E.dno Consider the GROUP BY query. If many tuples have E.age > 10, using E.age index and sorting the retrieved tuples may be costly. Clustered E.dno index may be better! 6. 2

26 EXAMPLES OF CLUSTERING SELECT E.dno FROM Emp E WHERE E.hobby=Stamps Equality queries and duplicates: Clustering on E.hobby helps! 6. 34

27 INDEX-ONLY PLANS 7. 1

28 ISSUES IN INDEX SELECTION Multi-attribute search keys should be considered when a WHERE clause contains several conditions. If range selections are involved, order of attributes should be carefully chosen to match the range ordering. Such indexes can sometimes enable index-only strategies for important queries. For index-only strategies, clustering is not important!

29 INDEX-ONLY PLANS Queries can be answered without retrieving any tuples from the relations involved if a suitable index is available. SELECT D.mgr FROM Dept D, Emp E WHERE D.dno=E.dno <E.dno> 7. 23

30 INDEX-ONLY PLANS <E.dno,E.eid> SELECT D.mgr, E.eid FROM Dept D, Emp E WHERE D.dno=E.dno 7. 4

31 INDEX-ONLY PLANS <E.dno> SELECT E.dno, COUNT(*) FROM Emp E GROUP BY E.dno 7. 5

32 INDEX-ONLY PLANS <E.dno,E.sal> SELECT E.dno, MIN(E.sal) FROM Emp E GROUP BY E.dno 7. 6

33 SELECT AVG(E.sal) FROM Emp E WHERE E.age=25 AND E.sal BETWEEN 3000 AND 5000 INDEX-ONLY PLANS <E.age,E.sal> OR <E.sal, E.age> 7. 7

34 INDEX TUNING "WIZARDS" Both IBM s DB2, Oracle and MS SQL Server have automated index advisors 8. 1

35 They take a workload of queries INDEX TUNING WIZARDS Basic idea: Possibly based on logging what s been going on They use the optimizer cost metrics to estimate the cost of the workload over different choices of sets of indexes Enormous # of different choices of sets of indexes: Heuristics to help this go faster

36 TUNING THE CONCEPTUAL SCHEMA

37 WEAKER NORMAL FORM We may settle for 3NF instead of BCNF Often more ways to come to 3NF or BCNF. Consider if queries would bene t from one compared to the other. Choise should be guided by workload 9. 2

38 DECOMPOSITION Further decompose a relation that is in BCNF Does some queries indicate a performance optimization if vertical partitioning is used Split large relation into multiple based on columns frequently and less frequently used Can avoid hotspots as less locking contention on each of the smaller tables

39 DENORMALIZATION Would denormalization bene t performance? Less joins at the cost of some redundency Choose to add redundant elds to some relations to speed up queries 9. 34

40 HORIZONTAL DECOMPOSITIONS Contracts (Cid, Sid, Jid, Did, Pid, Qty, Val) Suppose that contracts with value > are subject to different rules. So queries on Contracts will often say WHERE val> One approach: clustered B+ tree index on the val eld. 9. 5

41 HORIZONTAL DECOMPOSITIONS Second approach: replace contracts by two new relations, LargeContracts and SmallContracts, with the same attributes (CSJDPQV). Performs like index on such queries, but no index overhead. Can build clustered indexes on other attributes, in addition! 9. 6

42 HORIZONTAL DECOMPOSITIONS Replace relation by a collection of relations that are selections. Each new relation has same schema as original, but subset of rows. Collectively, new relations contain all rows of the original. Typically, the new relations are disjoint. 9. 7

43 MASKING CONCEPTUAL SCHEMA CHANGES CREATE VIEW Contracts(cid, sid, jid, did, pid, qty, val) AS SELECT * FROM LargeContracts UNION SELECT * FROM SmallContracts Horizonal Decomposition masked by a view. queries with condition val > must be asked wrt LargeContracts for ef ciency. which means non-transparency Arguably OK a solution is needed to improve performance

44 TUNING QUERIES AND VIEWS If a query runs slower than expected, check if an index needs to be reclustered, or if statistics are too old. Sometimes, the DBMS may not be executing the plan you had in mind

45 TUNING QUERIES AND VIEWS Common areas of weakness: Selections involving null values (bad selectivity estimates) Selections involving arithmetic or string expressions (ditto) Selections involving OR conditions (ditto) Complex subqueries Lack of evaluation features like index-only strategies or certain join methods or poor size estimation.

46 TUNING QUERIES AND VIEWS Check the plan that is being used! Then adjust the choice of indexes or rewrite the query/view. E.g. check via Explain command Some systems rewrite for you under the covers (e.g. DB2) Can be confusing and/or helpful! 9. 10

47 MORE GUIDELINES FOR QUERY TUNING Minimize the use of DISTINCT : don t need it if duplicates are acceptable, or if answer contains a key. SELECT MIN (E.age) FROM Employee E GROUP BY E.dno HAVING E.dno=102 Minimize the use of GROUP BY and HAVING : 9. 11

48 MORE GUIDELINES FOR QUERY TUNING SELECT MIN (E.age) FROM Employee E WHERE E.dno=102 Consider DBMS use of index when writing arithmetic expressions: E.age=2*D.age will bene t from index on E.age, but might not bene t from index on D.age!

49 SELECT * INTO Temp FROM Emp E, Dept D WHERE E.dno=D.dno AND D.mgrname= Joe SELECT T.dno, AVG(T.sal) FROM Temp T GROUP BY T.dno SELECT E.dno, AVG(E.sal) FROM Emp E, Dept D WHERE E.dno=D.dno AND D.mgrname= Joe GROUP BY E.dno GUIDELINES FOR QUERY TUNING Avoid using intermediate relations: vs

50 GUIDELINES FOR QUERY TUNING Later query does not materialize the intermediate reln Temp. If there is a dense B+ tree index on <dno, sal>, an index-only plan can be used to avoid retrieving Emp tuples in the second query!

51 IMPACT OF CONCURRENCY 10. 1

52 LOCKING IMPACT Blocking delays can be minimized by Reducing the time that a transaction hold locks Reducing hot spots (pages that are shared, fx root page in index tree) 10. 2

53 REDUCING LOCK DURATIONS Delay Lock Requests Make transactions faster Replace long running transactions by short ones Build a warehouse Consider a lower isolation level

54 DELAY LOCK REQUESTS Tune transactions by writing to local program variables and defer changes to the database until end of transaction. This will reduce the time locks are held (especially write locks)

55 MAKE TRANSACTIONS FASTER Have discussed how we can tune indexes etc. Consider having multiple disks and index on one and data on another 10. 5

56 REPLACE LONG RUNNING TRANSACTIONS BY SHORT ONES If too much work is done in one transaction. Consider updating your application to have less work in each

57 BUILD A WAREHOUSE Many complex queries are for statistical purposes. Consider exporting data to a datawarehouse or NoSQL storage, where these queries can be performed

58 CONSIDER A LOWER ISOLATION LEVEL For statistics, these complex queries could maybe use EPEATABLE READ` or READ COMMITTED Lower isolation level lowers locking overheads - design trade-off 10. 8

59 REDUCING HOT SPOTS Delay Operations on Hot Spots Optimize Access Patterns Choice of index 10. 9

60 DELAY OPERATIONS ON HOT SPOTS Already discussed, but even more relevant for frequently accessed objects

61 OPTIMIZE ACCESS PATTERNS If inserts follow key of B+tree index, rightmost path is frequently visited Hot spots This is not a problem for HASH indexes, hash process randomizes bucket into which element is inserted

62 CHOICE OF INDEX If relation is updated frequently, root of B+tree is bottleneck. Check if a Hash index is usable, that does not have these problems unless data distribution is skewed

63 PRACTICAL EXAMPLES IN POSTGRES 11. 1

64 DATABASE boatdb=# \d List of relations Schema Name Type Owner public boat table boats public hibernate_sequence sequence boats public reserves table boats public sailor table boats (4 rows) 11. 2

65 BOAT boatdb=# \d boat Table "public.boat" Column Type Modifiers id bigint not null version bigint not null name character varying(255) not null year_created integer not null color character varying(255) not null Indexes: "boat_pkey" PRIMARY KEY, btree (id) Referenced by: TABLE "reserves" CONSTRAINT "fkbn3186y4a41xy2wu72wmybh4x" FOREIGN KEY (boat_id) REFEREN

66 BOAT boatdb=# EXPLAIN SELECT * FROM boat ; QUERY PLAN Seq Scan on boat (cost= rows=100 width=30) (1 row)

67 SAILOR boatdb=# \d sailor Table "public.sailor" Column Type Modifiers id bigint not null version bigint not null rating integer not null age numeric(19,2) not null name character varying(255) not null Indexes: "sailor_pkey" PRIMARY KEY, btree (id) Referenced by: TABLE "reserves" CONSTRAINT "fkqcig9cdpc3al540k2qwjkynif" FOREIGN KEY (sailor_id) REFER

68 SAILOR boatdb=# EXPLAIN SELECT * FROM sailor ; QUERY PLAN Seq Scan on sailor (cost= rows=40000 width=31) (1 row)

69 RESERVES boatdb=# \d reserves Table "public.reserves" Column Type Modifiers id bigint not null version bigint not null rname character varying(255) not null sailor_id bigint not null day timestamp without time zone not null boat_id bigint not null Indexes: "reserves_pkey" PRIMARY KEY, btree (id) Foreign-key constraints: "fkbn3186y4a41xy2wu72wmybh4x" FOREIGN KEY (boat_id) REFERENCES boat(id) "fkqcig9cdpc3al540k2qwjkynif" FOREIGN KEY (sailor_id) REFERENCES sailor(id)

70 RESERVES boatdb=# EXPLAIN SELECT * FROM reserves ; QUERY PLAN Seq Scan on reserves (cost= rows= width=45) (1 row)

71 QUERY Select S.id, COUNT(*) AS number FROM Sailor S, Reserves R, Boat B WHERE S.id = R.sailor_id AND R.boat_id = B.id AND B.color = 'red' GROUP BY S.id; 11. 9

72 EXPLAIN ANALYZE EXPLAIN ANALYZE Select S.id, COUNT(*) AS number FROM Sailor S, Reserves R, Boat B WHERE S.id = R.sailor_id AND R.boat_id = B.id AND B.color = 'red' AND S.name = 'Pi' GROUP BY S.id;

73 EXPLAIN QUERY PLAN GroupAggregate (cost= rows=1 width=16) (actual time= rows=4 l Group Key: s.id -> Sort (cost= rows=1 width=8) (actual time= rows=9 loops= Sort Key: s.id Sort Method: quicksort Memory: 25kB -> Nested Loop (cost= rows=1 width=8) (actual time= ro Join Filter: (r.boat_id = b.id) Rows Removed by Join Filter: 511 -> Seq Scan on boat b (cost= rows=10 width=8) (actual time=0.019 Filter: ((color)::text = 'red'::text) Rows Removed by Filter: 90 -> Materialize (cost= rows=5 width=16) (actual time= > Nested Loop (cost= rows=5 width=16) (actual time=0.73 -> Seq Scan on sailor s (cost= rows=2 width=8) (ac Filter: ((name)::text = 'Pi'::text) Rows Removed by Filter: > Bitmap Heap Scan on reserves r (cost= rows=3 wid Recheck Cond: (sailor_id = s.id) Heap Blocks: exact=52 -> Bitmap Index Scan on sailor_idx (cost= rows Index Cond: (sailor_id = s.id) Planning time: ms Execution time: ms (23 rows)

74 AUCH - WHAT TO DO? EXPLAIN ANALYZE Select S.id, COUNT(*) AS number FROM Sailor S, Reserves R, Boat B WHERE S.id = R.sailor_id AND R.boat_id = B.id AND B.color = 'red' AND S.name = 'Pi' GROUP BY S.id;

75 INDEXES? CREATE INDEX sailor_name ON sailor(name);

76 EXPLAIN QUERY PLAN GroupAggregate (cost= rows=1 width=16) (actual time= rows=4 loops Group Key: s.id -> Sort (cost= rows=1 width=8) (actual time= rows=9 loops=1) Sort Key: s.id Sort Method: quicksort Memory: 25kB -> Nested Loop (cost= rows=1 width=8) (actual time= rows= -> Nested Loop (cost= rows=5 width=16) (actual time= > Bitmap Heap Scan on sailor s (cost= rows=2 width=8) (a Recheck Cond: ((name)::text = 'Pi'::text) Heap Blocks: exact=21 -> Bitmap Index Scan on sailor_name (cost= rows=2 wi Index Cond: ((name)::text = 'Pi'::text) -> Bitmap Heap Scan on reserves r (cost= rows=3 width=16) Recheck Cond: (sailor_id = s.id) Heap Blocks: exact=52 -> Bitmap Index Scan on sailor_idx (cost= rows=3 wid Index Cond: (sailor_id = s.id) -> Index Scan using boat_pkey on boat b (cost= rows=1 width=8) ( Index Cond: (id = r.boat_id) Filter: ((color)::text = 'red'::text) Rows Removed by Filter: 1 Planning time: ms Execution time: ms (23 rows)

77 POINTS TO REMEMBER Indexes must be chosen to speed up important queries Index maintenance overhead on updates to key elds. Choose indexes that can help many queries, if possible. Build indexes to support index-only strategies. Clustering is an important decision; only one index on a given relation can be clustered! Order of elds in composite index key can be important

78 POINTS TO REMEMBER Static indexes may have to be periodically re-built. Statistics have to be periodically updated. Over time, indexes have to be ne-tuned (dropped, created, re-clustered, ) for performance. Should determine the plan used by the system, and adjust the choice of indexes appropriately

79 POINTS TO REMEMBER System may still not nd a good plan, so may have to rewrite the query/view: Avoid nested queries, temporary relations, complex conditions, and operations like DISTINCT and GROUP BY

80 QUESTIONS? 13