Robust Optimization of Database Queries

Transcription

1 Robust Optimization of Database Queries Jayant Haritsa Database Systems Lab Indian Institute of Science July 2011 Robust Query Optimization (IASc Mid-year Meeting) 1

2 Database Management Systems (DBMS) Efficient i and convenient mechanisms for storing, maintenance and querying of enterprise data banking, inventory, insurance, travel, Cornerstone of computer industry Uses more than 80 percent of computers worldwide Employs more than 70 percent of IT professionals Largest monetary sector of computer business July 2011 Robust Query Optimization (IASc Mid-year Meeting) 2

3 Current Database Systems Commercial IBM DB2 / Oracle / Microsoft SQL Server / Sybase ASE / CA Ingres Public-Domain PostgreSQL / MySQL July 2011 Robust Query Optimization (IASc Mid-year Meeting) 3

4 Relational Database Systems [RDBMS] Based on first-order logic Edgar Codd of IBM Research, Turing Award (1981) We believe in Codd, not God Data is stored in a set of relations (i.e. tables) with attributes, relationships, constraints STUDENT COURSE Roll No N Name Add Address Course No Titl Title C Credits REGISTER Roll No Course No Grade July 2011 Robust Query Optimization (IASc Mid-year Meeting) 4

5 QUERY INTERFACE Structured Query Language (SQL) Invented by IBM, 1970s Example: List names of students t and their course titles select STUDENT.Name, Name COURSE.Title from STUDENT, COURSE, REGISTER where STUDENT.RollNo = REGISTER.RollNo and REGISTER.CourseNo = COURSE.CourseNo STUDENT Roll No Name Address COURSE Course No Title Credits REGISTER Roll No Course No Grade July 2011 Robust Query Optimization (IASc Mid-year Meeting) 5

6 Query Execution Plans SQL is a declarative language Specifies only what is wanted, but not how the query should be evaluated (i.e. ends, not means) select STUDENT.Name, COURSE.Title from STUDENT, COURSE, REGISTER where STUDENT.RollNo = REGISTER.RollNo and REGISTER.CourseNo = COURSE.CourseNo Unspecified: join order [ ((S R) C) or ((R C) S)? ] join techniques [ Nested-Loops or Sort-Merge or Hash? ] DBMS query optimizer identifies the optimal evaluation strategy: query execution plan July 2011 Robust Query Optimization (IASc Mid-year Meeting) 6

7 Sample Execution Plan RETURN Cost: Total Execution Cost (estimated) HASH JOIN Cost: MERGE JOIN Cost: TABLE SCAN Cost: 6834 INDEX SCAN Cost: 6745 SORT Cost: TABLE SCAN Cost: STUDENT COURSE REGISTER July 2011 Robust Query Optimization (IASc Mid-year Meeting) 7

8 CONCEPT: PLAN DIAGRAMS July 2011 Robust Query Optimization (IASc Mid-year Meeting) 8

9 Relational Selectivities Cost-based Query Optimizer s choice of execution plan = f (query, database, system, ) For a given database and system setup, execution plan chosen for a query = f (selectivities of query s base relations) selectivity is the estimated percentage of rows of a relation used in producing the query result April 2011 Plan Diagrams Tutorial (ICDE 2011) 9

10 SQL Query Template [Q7 of TPC-H HBenchmark] Determines the values of goods shipped between nations in a time period select supp_nation, cust_nation, l_year, sum(volume) as revenue from (select n1.n_name as supp_nation, n2.n_name as cust_nation, extract(year from l_ shipdate) as l_y year, l_extendedprice * (1 - l_discount) as volume from supplier, lineitem, orders, customer, nation n1, nation n2 where s_suppkey Value determines = l_suppkey and o_orderkey = l_orderkey and c_custkey selectivity it= of o_custkey and s_nationkey = n1.n_nationkey selectivity of and ORDERS c_nationkey relation = n2.n_nationkey and ((n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY') or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')) and l_shipdate between date ' ' and date ' ' and o_ totalprice C1 and c_ acctbal C2 ) as shipping group by supp_nation, cust_nation, l_year order by supp_nation, cust_nation, l_year Value determines CUSTOMER relation July 2011 Robust Query Optimization (IASc Mid-year Meeting) 10

11 Relational Selectivity Space ctivity Selec Selectivity July 2011 Robust Query Optimization (IASc Mid-year Meeting) 11

12 Plan and Cost Diagrams A plan diagram is a pictorial enumeration of the plan choices of the query optimizer over the relational selectivity space A cost diagram is a visualization of the (estimated) plan execution costs over the same relational selectivity space July 2011 Robust Query Optimization (IASc Mid-year Meeting) 12

13 PICASSO Picasso is a Java tool developed in our lab that automatically generates plan and cost diagrams ~50000 lines of code ( ) with ~100 classes Operational on DB2/Oracle/SQLServer/Sybase/PostgreSQL Copyrighted by IISc in May 2006 Released as free software in Nov 2006 by Prof. N Balakrishnan, Associate Director of IISc Release of v1.0 in May 2007, v2.0 in Feb 2009, v2.1 in Feb 2011 In use at academic and industrial labs worldwide CMU, Purdue, Duke, TU Munich, NU Singapore, IIT-B, IBM, Microsoft, Oracle, Sybase, HP, Received Best Software award in Very Large Data Base (VLDB) conference, 2010 July 2011 Robust Query Optimization (IASc Mid-year Meeting) 13

14 The Picasso Connection Plan diagrams are often similar to cubist paintings! Woman with a guitar Georges Braque, 1913 [ Pablo Picasso founder of cubist genre ] July 2011 Robust Query Optimization (IASc Mid-year Meeting) 14

15 Complex Plan Diagram [QT8,OptA*] Highly irregular plan boundaries # of plans: 76 The Picasso Connection Intricate Complex Patterns Extremely finegrained coverage (P76 ~ 0.01%) July 2011 Robust Query Optimization (IASc Mid-year Meeting) 15

16 Cost Diagram [QT8, Opt A*] All costs are within 20 percent of the maximum MinCost: 8.26E5 MaxCost: 1.05E6 July 2011 Robust Query Optimization (IASc Mid-year Meeting) 16

17 PHENOMENA: PLAN DIAGRAM REDUCTION July 2011 Robust Query Optimization (IASc Mid-year Meeting) 17

18 Problem Statement Can the plan diagram be recolored with a smaller set of colors (i.e. some plans are swallowed by others), such that Guarantee: No query point in the original diagram has its estimated cost increased, post-swallowing, by more than λ percent (user-defined) d) Analogy: (with due apologies to Sri Lankans in the audience) Sri Lanka agrees to be annexed by India if it is assured that the cost of living of each Lankan citizen is not increased by more than λ percent July 2011 Robust Query Optimization (IASc Mid-year Meeting) 18

19 Reduced Complex Plan Diagram [λ=10%] [QT8, OptA*, Res=100] QTD: QT8_OptA*_100 Reduced to 5 plans from 76! Comparatively smoother contours July 2011 Robust Query Optimization (IASc Mid-year Meeting) 19

20 Anorexic Reduction Extensive empirical evaluation with a spectrum of multi-dimensional query templates indicates that With a cost-increase-threshold of just 20%, virtually all complex plan diagrams [irrespective of query templates, data distribution, query distribution, system configurations, etc.] reduce to anorexic levels (~10 or less plans)! February 2011 Picasso (IIITM Kerala) 20

21 APPLICATION: ROBUST PLANS July 2011 Robust Query Optimization (IASc Mid-year Meeting) 21

22 Selectivity Estimation Errors q e (x e, y e ) : estimated location by optimizer q a (x a, y a ): actual location during execution The difference could be substantial due to Outdated Statistics (expensive to maintain) Coarse Summaries (histograms) Attribute Value Independence (AVI) assumptions Chronic problem in database design July 2011 Robust Query Optimization (IASc Mid-year Meeting) 22

23 Impact of Error Example L I N E I T E M Estimated Query Location (q e ): (1, 40) Cost(P oe ) = 9 x 10 4 (optimal) Actual Query Location (q a ): (80,80) Cost(P oa ) = 50 x 10 4 (optimal) Cost(P oe ) = 110 x 10 4 (highly sub-optimal) Selectivity Space S Space S ORDERS July 2011 Robust Query Optimization (IASc Mid-year Meeting) 23

24 Error Locations wrt Plan Replacement Regions??? Inherently robust 100 Inherently robust q a S Y q a Endo-optimal re Selec ctivity q e `` q a Swallow re Exo-optimalre 0 Selectivity X July 2011 Robust Query Optimization (IASc Mid-year Meeting) 24

25 Positive Impact of Reduction In most cases, replacement plan provides robustness to selectivity errors even in exo-optimal region QT5 q e = (0.36, 0.05) q e (, ) Original Plan (P oe ) Replacement Plan (P re ) P 37 P 1 P 3 P 6 P 4 P 4 Local Optimal Plan (P oa ) July 2011 Robust Query Optimization (IASc Mid-year Meeting) 25

26 Negative Impact of Reduction But, occasionally, the replacement is much worse than the original plan! QT5 q e = (0.03, 0.14) Replacement Plan (P re ) Original Plan (P oe ) P 37 P 1 P 3 P 6 P 4 P 4 Local Optimal Plan (P oa ) July 2011 Robust Query Optimization (IASc Mid-year Meeting) 26

27 Research Challenge How do we ensure that plan replacements can only help, but never materially hurt the expected performance? July 2011 Robust Query Optimization (IASc Mid-year Meeting) 27

28 Our Contribution Mathematical analysis to show that only the perimeter of the selectivity space suffices to determine global safety Border Safety Interior Safety! April 2011 Plan Diagrams Tutorial (ICDE 2011) 28

29 Take Away We can efficiently i produce plan diagrams that simultaneously possess the desirable properties of being online, anorexic, safe and robust. This result could play a meaningful role in designing g the next generation of database query optimizers. July 2011 Robust Query Optimization (IASc Mid-year Meeting) 29

30 More Details i / /PICASSO Publications, Software, Sample Diagrams July 2011 Robust Query Optimization (IASc Mid-year Meeting) 30

31 Dedication Name: Dr B K Narayana Rao (Founding Fellow) Born: 1881 Elected: 1934 Section: Medicine Council Service: ; Vice President Name: Prof T S Subbaraya Born: 1905 Elected: 1935 Section: Physics Name: Dr B N Balakrishna Rao Born: 1910 Elected: 1945 Section: Medicine July 2011 Robust Query Optimization (IASc Mid-year Meeting) 31

32 QUESTIONS? July 2011 Robust Query Optimization (IASc Mid-year Meeting) 32

33 END PRESENTATION July 2011 Robust Query Optimization (IASc Mid-year Meeting) 33