SQL Query Writing Tips To Improve Performance in Db2 and Db2 Warehouse on Cloud

Transcription

1 SQL Query Writing Tips To Improve Performance in Db2 and Db2 Warehouse on Cloud Calisto Zuzarte IBM St. Louis Db2 User s Group Tue, March 06, 2018 Db2 Warehouse Db2 Warehouse on Cloud Integrated Analytics System 1

2 Agenda Rewriting SQL to improve performance Queries with Outer Joins Queries with Aggregation Queries with High Cardinality Aggregation DISTINCT Queries with Correlated Subqueries Queries with OR predicates Queries With OR EXISTS / AND NOT EXISTS Predicates Queries with UNION ALL Queries With a Large Number of Joins

3 Queries With Outer Joins (1/4) - Reordering Inner Joins STLDIG Outer Joins do not filter rows Put them at the end of the statement or after expanding inner joins. SELECT.C10,.C20, T3.C30, T4.C40, T5.C50 FROM LEFT OUTER JOIN ON.C2 =.C2 LEFT OUTER JOIN T3 ON.C3 = T3.C3 INNER JOIN T4 ON.C4 = T4.C4 INNER JOIN T5 ON.C5 = T5.C5 WHERE T4.C0 = 2 SELECT.C10,.C20, T3.C30, T4.C40, T5.C50 FROM INNER JOIN T4 ON.C4 = T4.C4 INNER JOIN T5 ON.C5 = T5.C5 LEFT OUTER JOIN ON.C2 =.C2 LEFT OUTER JOIN T3 ON.C3 = T3.C3 WHERE T4.C0 = 2 Cost based automatic reordering may not be done without complete statistics information 3

4 Queries With Outer Joins (2/4) - Reordering Inner Joins STLDIG K 5K 5K T5 5K LOJ LOJ T3 LOJ T4 5K LOJ T3 5K T5 T4 4

5 Queries With Outer Joins (3/4) - Large NULL producing side Very large NULL producing side and small (or highly filtered) row producing side. In an MPP system, the NULL producing side may need to be broadcast or redistributed during the query SELECT.C1,.CX,.C2,.C3,.C20 FROM LEFT OUTER JOIN ON.C2 =.C2 WHERE.C0 = 2 SELECT.C1,.CX.C2,.C3,.C20 FROM LEFT OUTER JOIN (SELECT.* FROM, (SELECT DISTINCT C2 FROM WHERE.C0 = 2) WHERE.C2 =.C2 ) ON.C2 =.C2 WHERE.C0 = 2 5

6 Queries With Outer Joins (4/4) - Large NULL producing side LOJ 100M DTQ 100M ROJ DTQ 10K 10K 10K 10K DTQ 100M DTQ 5K 5K 10K Semi-Join 6

7 Queries with Aggregation (1/4) Reducing Aggregation Filtering joins are sometimes specified after aggregation STLDIG SELECT.C2,.C10,.C20,.C30, TX.S FROM INNER JOIN (SELECT.C2, SUM(.SALES) AS S FROM INNER JOIN T3 ON.C3 = T3.C3 GROUP BY.C2) TX ON.C2 =.C2 WHERE.C0 = 5 SELECT.C2,.C10,.C20,.C30, TX.S FROM INNER JOIN (SELECT.C2, SUM(.SALES) AS S FROM INNER JOIN T3 ON.C3 = T3.C3 INNER JOIN (SELECT DISTINCT.C2 FROM WHERE.C0 = 5 ) ON.C2 =.C2 GROUP BY.C2) TX ON.C2 =.C2 WHERE.C0 = 5 7

8 Queries with Aggregation (2/4) Reducing Aggregation STLDIG M 100M 100M GB 100K T3 100K 50M 100K 50M 100K T3 100M GB 50K 1M 1M 100K Semi-Join 8

9 Queries with Aggregation (3/4) - Eager Aggregation STLDIG A GROUP BY at the end of the query may have too much baggage SELECT.C20, T3.C30, T4.C40, T4.C41, T4.C42, T5.C50, SUM(.SALES) FROM INNER JOIN ON.C2 =.C2 INNER JOIN T3 ON.C3 = T3.C3 INNER JOIN T4 ON.C4 = T4.C4 INNER JOIN T5 ON.C5 = T5.C5 WHERE.C0 = 2 AND T3.C0 = MONTH( ) GROUP BY.C20, T3.C30, T4.C40, T4.C41, T4.C42, T5.C50 Filtering Joins Look up Joins with wide columns Or Expanding Joins SELECT TX.C20, TX.C30, T4.C40, T4.C41, T4.C42, T5.C50, SUM(TX.S) FROM (SELECT.C20, T3.C30,.C4,.C5, SUM(.SALES) AS S FROM INNER JOIN ON.C2 =.C2 INNER JOIN T3 ON.C3 = T3.C3 WHERE.C0 = 2 AND T3.C0 = MONTH( ) GROUP BY.C20, T3.C30,.C4,.C5 ) TX INNER JOIN T4 ON TX.C4 = T4.C4 INNER JOIN T5 ON TX.C5 = T5.C5 GROUP BY TX.C20, TX.C30, T4.C40, T4.C41, T4.C42, T5.C50 9

10 Queries with Aggregation (4/4) - Eager Aggregation 100M 50M T3 10K GB 200M T4 T5 Eager GROUP BY - Includes T4 and T5 Join columns - Excludes wide payload columns - Effective before expanding joins 100M 50M 500K 500K GB T3 10K GB T4 T5 10

11 Queries with DISTINCT (1/2) - Eliminating Duplicates Early Duplicate removal is sometimes deferred with DISTINCT subqueries SELECT.C10,.C20, T3.C30, T4.C40, T5.C50 FROM INNER JOIN ON.C2 =.C2 INNER JOIN T3 ON.C3 = T3.C3 INNER JOIN (SELECT DISTINCT C4, CX FROM T4 ) T4 ON.C4 = T4.C4 INNER JOIN (SELECT DISTINCT C5, CY FROM T5 ) T5 ON.C5 = T5.C5 WHERE SELECT.C10,.C20, T3.C30, T4.C40, T5.C50 FROM INNER JOIN ON.C2 =.C2 INNER JOIN T3 ON.C3 = T3.C3 INNER JOIN (SELECT C4, CX FROM T4 GROUP BY C4, CX) T4 ON.C4 = T4.C4 INNER JOIN (SELECT C5, CY FROM T5 GROUP BY C5, CY) T5 ON.C5 = T5.C5 WHERE 11

12 Queries with DISTINCT (2/2) - Eliminating Duplicates Early SORT SORT 10X 500M 5X 50M T5 GB T4 GB T5 T3 T3 T4 20M 20M GROUP BY forces duplicate removal 12

13 Queries With High Cardinality Aggregation DISTINCT (1/4) COUNT(DISTINCT C1) can be a memory hog 100s of millions of distinct values need to be kept in memory Cannot do a partial-final GROUP BY SELECT COUNT(DISTINCT C1) FROM GROUP BY C0 High number of distinct values Low number of distinct values or no GROUP BY SELECT COUNT(*) FROM (SELECT C0, C1 FROM GROUP BY C0, C1) AS T GROUP BY C0 Db2 may not do this automatically if costs are off Partial-Final GROUP BY is possible 13

14 Queries With High Cardinality Aggregation DISTINCT (2/4) 10 CD High count distinct values 10 C* 100M FGB 500M TQ 100M GB 250M TQ 500M 500M 250M PGB 14

15 Queries With High Cardinality Aggregation DISTINCT (3/4) Rewriting with multiple COUNT DISTINCTs may be a little more involved SELECT C0, COUNT(DISTINCT C1) CDC1, COUNT(DISTINCT C2) CDC2 FROM GROUP BY C0 SELECT C0, CDC1, CDC2 FROM ( SELECT COUNT(*) AS CDC1 FROM (SELECT C0, C1 FROM GROUP BY C0, C1) AS T GROUP BY C0) TX, ( SELECT COUNT(*) AS CDC2 FROM (SELECT C0, C2 FROM GROUP BY C0, C2) AS T GROUP BY C0) TY WHERE TX.C0 = TY.C0 15

16 Queries With High Cardinality Aggregation DISTINCT (4/4) 10 CD High count distinct values CDC1 10 CDC2 Each COUNT DISTINCT is done in a separate leg 500M 100M GB GB 20M 500M 500M Multiple scans required but better than a memory error 16

17 Queries With Correlated Subqueries (1/4) - Predicate Db2 query rewrite generally decorrelates the query SELECT * FROM VERSIONED_TBL WHERE.SALARY >= AND.DATE_COL >= (SELECT MIN(DATE_COL) FROM VERSIONED_TBL WHERE.EMP_ID =.EMP_ID) If not decorrelated with row tables an index helps. With column tables rewriting is generally better STLDIG SELECT * FROM VERSIONED_TBL, (SELECT.EMP_ID, MIN(DATE_COL) AS M FROM VERSIONED_TBL GROUP BY.EMP_ID) TX WHERE.SALARY >= AND.DATE_COL >= TX.M AND.EMP_ID = TX.EMP_ID) 17

18 Queries With Correlated Subqueries (2/4) - Predicate STLDIG scans required if no indexes GB SCAN GB SCAN 10K One scan needed 18

19 Queries With Correlated Subqueries (3/4) Select List STLDIG Aggregation subquery in the SELECT list (simple subqueries handled) SELECT C1, (SELECT MAX(C2) FROM, T3 WHERE.C1 =.C1 AND.C3 = T3.C3 ) as C2, C3 FROM WHERE.C0 > 10 SELECT.C1, TX.MAXC2 AS C2,.C3 FROM LEFT OUTER JOIN (SELECT C1, MAX(C2) AS MAXC2 FROM, T3 WHERE.C3 = T3.C3 GROUP BY C1 ) TX ON (.C1 = TX.C1) WHERE.C0 > 10 Use a WITH clause for TX if there are multiple similar SELECT list aggregation subqueries 19

20 Queries With Correlated Subqueries (4/4) Select List STLDIG scans required if no indexes GB SCAN LOJ GB SCAN 10K One scan needed 20

21 Queries With OR Predicates (1/6) - OR Join Predicates STLDIG OR Join predicates cannot be processed by a Hash Join Nested Loop Joins work well with row tables and indexes Column oriented tables perform better with a UNION DISTINCT must be present to transform SELECT DISTINCT.C1,.C2,.C3,.CX FROM, WHERE ((.C6 =.C2 and.c3 IN ('1', 2') ) OR (.C7 =.C2 and.c3 NOT IN IN ('1', 2') ) ) SELECT.C1,.C2,.C3,.CX FROM (SELECT.C1,.C2,.C3,.CX FROM, WHERE.C6 =.C2 and.c3 IN ('1', 2') UNION SELECT.C1,.C2,.C3,.CX FROM, WHERE.C7 =.C2 and.c3 NOT IN ('1', 2') 21

22 Queries With OR Predicates (2/6) - OR Join Predicates STLDIG OR predicate applied to cartesian join (nested loop join) 100M 10K UNION Each branch of the UNION has a simple equality predicate join and can use a hash join 22

23 Queries With OR Predicates (3/6) OR EXISTS / AND NOT EXISTS If no complexity, Db2 tries to use Left Outer Joins for better performance The following example is with NOT EXISTS AND NOT EXISTS IS NOT NULL is used for the EXISTS OR EXISTS transformation SELECT.C1,.C2,.C3 FROM WHERE NOT EXISTS (SELECT 1 FROM WHERE.C2 =.C2) AND NOT EXISTS (SELECT 1 FROM T3 WHERE.C3 = T3.C3) SELECT.C1,.C2,.C3 FROM LEFT OUTER JOIN ON.C2 =.C2 LEFT OUTER JOIN T3 ON.C3 = T3.C3 WHERE.C2 IS NULL AND T3.C3 IS NULL 23

24 Queries With OR Predicates (2/3) - OR EXISTS Subqueries OR EXISTS predicate may be applied as a FILTER predicate FILTER OR predicate uses IS NULL to test whether the values EXIST FILTER LOJ 100M SCAN SCAN 10K 5K LOJ T3 T3 24

25 Queries With OR Predicates (5/6) - Alternative OR To UNION Db2 handles this particular scenario. Consider rewriting if still correlated and slow. Scans and joins are repeated Needs DISTINCT to be equivalent SELECT DISTINCT.C1,.C2,.C3 FROM WHERE.C0 = 0 OR EXISTS (SELECT 1 FROM WHERE.C2 =.C2) SELECT.C1,.C2,.C3 FROM (SELECT.C1,.C2,.C3 FROM WHERE.C0 = 0 UNION SELECT.C1,.C2,.C3 FROM WHERE EXISTS (SELECT 1 FROM WHERE.C1 =.C2) ) 25

26 Queries With OR Predicates (5/6) - Alternative OR To UNION OR EXISTS predicate may be applied as a FILTER predicate FILTER UNION This branch has the EXISTS predicate converted into a join 100M SCAN 10K 26

27 Queries With UNION ALL (1/2) - UNION ALL over tiny tables Db2 tries to push Joins through a UNION ALL Great with UNION ALL over row oriented Fact Tables UNION ALL over small dimension tables may not perform on column oriented tables SELECT FROM F1 INNER JOIN D2 ON F1.C2 = D2.C2 INNER JOIN D3 ON F1.C3 = D3.C3 INNER JOIN D4_UNION_ALL D4 ON F1.C4 = D4.C4 SELECT FROM F1 INNER JOIN D2 ON F1.C2 = D2.C2 INNER JOIN D3 ON F1.C3 = D3.C3 INNER JOIN (SELECT * FROM (SELECT, RAND( ) R FROM D4_UNION_ALL ) UA WHERE R >= 0 ) D4 ON F1.C4 = D4.C4 27

28 Queries With UNION ALL (2/2) - UNION ALL over tiny tables UNION N Join Pushdown through UNION results in multiple scans of the fact table D4 UNION N D5 D3 D4 D5 D3 D3 F1 D2 F1 D2 F1 D2 RAND( ) prevents the pushdown 28

29 Queries With a Large Number of Joins Excessive Compile Time? Consider GROUP BY instead of DISTINCT to break up a query block. SELECT.C10,.C20, T3.C30,, T5.CX 5.C150 FROM INNER JOIN ON.C2 =.C2 INNER JOIN T3 ON.C3 = T3.C3 INNER JOIN (SELECT DISTINCT T7.C7, T9.CX FROM T7, T8, T9 WHERE ) TX ON.C7 = TX.C7 INNER JOIN 5 ON.C15 = 5.C15 WHERE SELECT.C10,.C20, T3.C30,, T5.CX 5.C150 FROM INNER JOIN ON.C2 =.C2 INNER JOIN T3 ON.C3 = T3.C3 INNER JOIN (SELECT T7.C7, T9.CX FROM T7, T8, T9 WHERE GROUP BY T7.C7, T9.CX) TX ON.C7 = TX.C7 INNER JOIN 5 ON.C15 = 5.C15 WHERE Other option : breaking up the query using DGTTs, 29

30 Queries With UNION ALL (2/2) - UNION ALL over tiny tables Excessive compile time? GB T4 T9 T4 T3 T7 T8 T3 T9 T7 T8 Tables below the GROUP BY not compiled with the rest of the query 30

31 Calisto Zuzarte IBM SQL Query Writing Tips To Improve Performance in Db2 and Db2 Warehouse on Cloud 31