Query Optimizer, Who Influences & How it works ++ optimization techniques

Transcription

1 Query Optimizer, Who Influences & How it works ++ optimization techniques AIOUG : ODevC Yatra 2018, India Chandan Tanwani Senior Application Engineer Oracle Financial Services Software Ltd. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 1

2 Journey so far... Working on various Oracle Database version from Oracle 7 to Oracle 12c Working on Oracle Application Server 10g, Weblogicserver 8 to 12c Worked on forms/reports 3, 6i, 9i Passionate for Oracle Technologies Frequent Blogger & Speaker in various events Frequent Contributor in OTN database forum /tanwanichandan cb.t@oracle.com Copyright 2018 Oracle and/or its affiliates. All rights reserved. 2

3 Agenda Introduction to Query Optimization How Query Optimizer Works? Optimizer Components and Operations Who influences optimizer? Adaptive Query Optimization Optimizer Limitation vs Improvements New Optimization Techniques Q&A Copyright 2018 Oracle and/or its affiliates. All rights reserved. 3

4 Safe Harbor Statement The Technical Observations& Views here are my own and not necessarily those of Oracle or its affiliates. These are purely based on my understanding, learning and resolution of various issues. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 4

5 Introduction to Query Optimizer Copyright 2018 Oracle and/or its affiliates. All rights reserved. 5

6 Introduction to Query Optimizer I am asked to Tune Query. I do generate explain plan, Checking if there is full table scan or not, If yes, looking why index not used, Checking if cost is optimal, Looking for better Plan, Go here and there and tried few stuff, Hope something works, but no success. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 6

7 Introduction to Query Optimizer Is there any guidelines which I follow and my query willbetuneitself? Copyright 2018 Oracle and/or its affiliates. All rights reserved. 7

8 Introduction to Query Optimizer Are you looking something like this? Copyright 2018 Oracle and/or its affiliates. All rights reserved. 8

9 Introduction to Query Optimizer Why don t we give this task to our Optimizer. Optimizer is the Brain of Oracle Database. Feed Optimizer and Get Best Performance Copyright 2018 Oracle and/or its affiliates. All rights reserved. 9

10 How Optimizer Works? Copyright 2018 Oracle and/or its affiliates. All rights reserved. 10

11 How Optimizer Works? Continue Query Transformer OR Expansion You wrote SELECT * FROM sales WHERE region_id=2 OR prod_id=11; Optimizer Transformation SELECT * FROM sales WHERE prod_id=11 UNION ALL SELECT * FROM sales WHERE region_id=2 AND LNNVL(prod_id=11); Copyright 2018 Oracle and/or its affiliates. All rights reserved. 11

12 How Optimizer Works? Continue Query Transformer Subquery Un-nesting You wrote SELECT * FROM sales WHERE cust_id IN ( SELECT cust_id FROM customers ); Optimizer Transformation SELECT sales.* FROM sales, customers WHERE sales.cust_id = customers.cust_id; Copyright 2018 Oracle and/or its affiliates. All rights reserved. 12

13 How Optimizer Works? Continue Query Transformer Predicate Pushing You wrote CREATE VIEW empvw AS ( SELECT emp_id, last_name, job_id, commission_pct, dept_id FROM employees ) UNION ( SELECT emp_id, last_name, job_id, commission_pct, dept_id FROM contract_workers); You then query the view as follows: SELECT last_name FROM empvw WHERE dept_id = 50; Optimizer Transformation SELECT last_name FROM ( SELECT emp_id, last_name, job_id, commission_pct, dept_id FROM employees WHERE dept_id=50 UNION SELECT emp_id, last_name, job_id, commission_pct, dept_id FROM contract_workers WHERE dept_id=50); Copyright 2018 Oracle and/or its affiliates. All rights reserved. 13

14 How Optimizer Works? Continue Estimator Selectivity Estimator Cardinality Cost Copyright 2018 Oracle and/or its affiliates. All rights reserved. 14

15 How Optimizer Works? Continue Plan Generator Copyright 2018 Oracle and/or its affiliates. All rights reserved. 15

16 Optimizer Operations Evaluation of expressions and conditions Statement transformation The optimizer first evaluates expressions and conditions containing constants as fully as possible. For complex statements involving, for example, correlated subqueries or views, the optimizer might transform the original statement into an equivalent join statement. Choice of optimizer goals The optimizer determines the goal of optimization. Optimizer mode plays role to choose goal. Choice of access paths Choice of join orders For each table accessed by the statement, the optimizer choosesoneormoreoftheavailableaccesspathstoobtain table data. For a join statement that joins more than two tables, the optimizer chooses which pair of tables is joined first, and thenwhichtableisjoinedtotheresult,andsoon. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 16

17 DoYouKnow? Selectivity is an internal calculation that is not visible in execution plans. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 17

18 DoYouKnow? If no statistics are available then the optimizer either uses dynamic sampling or an internal default value Copyright 2018 Oracle and/or its affiliates. All rights reserved. 18

19 DoYouKnow? The cost calculation is done based on disk I/O, CPU usage, and memory usage as units of work. The cost of a query plan is the number of work units that are expected to be acquired when the query is executed and produce the result. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 19

20 DoYouKnow? CostisthemeasurementfortheOptimizerto compare plans for the same query. You cannot tune or change cost. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 20

21 Optimizer is smart enough Right??? Then, Why SQLs performing Bad? Copyright 2018 Oracle and/or its affiliates. All rights reserved. 21

22 Who Influences Optimizer? Copyright 2018 Oracle and/or its affiliates. All rights reserved. 22

23 Who Influences Optimizer? Copyright 2018 Oracle and/or its affiliates. All rights reserved. 23

24 Controlling Behavior of Optimizer init Parameters are the one who Control Optimizer behavior. OptimizerMode CursorSharing OptimizerDynamicSampling OptimizerFeaturesEnable OptimizerIndexCaching OptimizerIndexCostAdj DbFileMultiblockReadCount StarTransformationEnabled Optimizerusependingstatistics Optimizerusesqlplanbaseline Copyright 2018 Oracle and/or its affiliates. All rights reserved. 24

25 Controlling Behavior of Optimizer OptimizerAdaptiveFeatures OptimizerIn-memoryAware OptimizerAdaptiveFeatures(Obsoleted) OptimizerAdaptivePlans OptimizerAdaptiveStatistics OptimizerIn-memoryAware Copyright 2018 Oracle and/or its affiliates. All rights reserved. 25

26 Adaptive Query Optimization Copyright 2018 Oracle and/or its affiliates. All rights reserved. 26

27 Adaptive Query Optimization Copyright 2018 Oracle and/or its affiliates. All rights reserved. 27

28 Adaptive Query Optimization How it works Adaptive query optimization enable optimizer to adjust execution plan at run time when better information is available. AdaptivePlans: Different Join Methods(change NL to HASH) or Parallel Distribution Adaptive Plans does not pick the final plan until execution time based on statistics collection. Information learned at execution time is used in future executions. TherearethreetypesofAdaptiveStatistics: Dynamic Statistics(previously dynamic sampling in 10g/11g) or runtime statistics Automatic Reoptimization or statistics generated after the initial execution SQL Plan Directives direct optimizer to dynamic statistics & gets accurate cardinality In 12cR2, Oracle introduced Continuous Adaptive Query Plans (CAQP) where certain queries, based on input data, can benefit from continuous adaptive join methods. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 28

29 Adaptive Query Optimization How it works AdaptivePlans changedfromdefaulton1stexecution AutomaticRe-optimization 2ndexecution 1. Determine default plan and include alternative sub-plans which are pre-computed 2. Sub-plans stored in the cursor 3. Stats collector inserted before join 4. Rows buffered until final decision is made 5. Number of rows seen in statistics collector if exceeds threshold, Plan switches to other join method 6. Statistics collector disabled 7. Plan resolved on first execution& remains the same for subsequent executions Copyright 2018 Oracle and/or its affiliates. All rights reserved. 29

30 DoYouKnow? Howlongsameplanisusedbyoptimizer? Till the plan ages out from the cache, or a different optimizer feature (for example, adaptive cursor sharing or statistics feedback) invalidates the plan. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 30

31 DoYouKnow? When optimizer choose nested loop and when chooses hash join? The statistics collector buffers enough rows to determine which join method to use. If the row count is below the threshold identified by the optimizer, then optimizer chooses the nested loops join; otherwise, optimizer chooses the hash join. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 31

32 Optimizer Limitations vs Improvements Copyright 2018 Oracle and/or its affiliates. All rights reserved. 32

33 Optimizer Limitation vs Improvements Adaptive Query Optimization 11g Limitations There will be variations of statistics by optimizer for many reasons i.e. stale or incomplete statistics, complex expressions used etc. No way to adopt run time statistics. 12c Improvements Run-time adjustments to query plans. Discovery of additional information at execution time leads to better statistics. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 33

34 Optimizer Limitation vs Improvements continue Histograms 11g Limitations Histograms can underestimation of cardinality for nearly popular values. Number of histogram buckets is limited to c Improvements Top-frequency histograms providing more accurate information on the popular values Actual frequencies of bucket endpoint values are recorded in histograms. More histogram buckets. Hybrid histogram with more endpoint and buckets. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 34

35 Optimizer Limitation vs Improvements continue SQL Plan Directives 11g Limitations Better plan and SQL information stored in cursor only. We cannot share this information across statements. Information is lost when the cursor ages out from shared pool. 12c Improvements SQL plan directives allow to persist information learnt during compilation and execution. Collected on query expressions not at a statement level. Automatically created and maintained for you. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 35

36 Optimizer Limitation vs Improvements continue Online Statistics Gathering for Direct Path Load 11g Limitations After inserting rows in newly created table statistics are required to ensure accurate cardinality estimate. Gathering statistics for the first time requires a full table scan. Potentially expensive in terms of time and resource. 12c Improvements Statistics will be gathered as part of the direct path load operations. Statistics available directly after data load. No additional table scan required to gather statistics. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 36

37 Optimizer Limitation vs Improvements continue Statistic gathering options for GTTs 11g Limitations All Sessions can have same statistics, which is gathered on a GTT This needs to understand that not all session in GTT will have the same data distribution 12c Improvements Each session can gather their own set of statistics for GTT. Provides more accurate cardinality estimates within each session. Controlled by new preference GLOBAL_TEMP_TABLE_STATS. Default value is SESSION (non shared). Copyright 2018 Oracle and/or its affiliates. All rights reserved. 37

38 Optimizer Limitation vs Improvements continue Adaptive SQL Plan Management 11g Limitations DBA needs to monitor SQL plan baselines to find new unaccepted plans. SPM evolve process needed to manually invoked. 12c Improvements New evolve auto task running in the maintenance window. Ranks all non-accepted plans and runs evolve process for them. Automatically accepts any new plan that performs better than exist plan. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 38

39 DoYouKnow? Dynamic stats are stored in memory and reused by other SQLs too. (In result cache). Copyright 2018 Oracle and/or its affiliates. All rights reserved. 39

40 New Query Optimization Techniques in 12c onwards Copyright 2018 Oracle and/or its affiliates. All rights reserved. 40

41 New Optimizer Techniques Approximate Query Processing Partial Join Evaluation (PJE) Multi-Table Left Outer Join Optimizer Statistics Advisor Copyright 2018 Oracle and/or its affiliates. All rights reserved. 41

42 Approximate Query Processing Approximate query processing is a set of optimization techniques that speed analytic queries by calculating results within an acceptable range of error. In 12c R1 APPROX_COUNT_DISTINCT In 12c R2 (newly added) APPROX_COUNT_DISTINCT_AGG APPROX_COUNT_DISTINCT_DETAIL APPROX_MEDIAN APPROX_PERCENTILE APPROX_PERCENTILE_AGG APPROX_PERCENTILE_DETAIL TO_APPROX_COUNT_DISTINCT TO_APPROX_PERCENTILE In 18c (extended in this release) APPROX_COUNT APPROX_SUM APPROX_RANK Copyright 2018 Oracle and/or its affiliates. All rights reserved. 42

43 Approximate Query Processing To control behavior approximate query processing, following parameters are introduce in 12c R2, which can be set at system and session level. Initialization Parameter Default Description APPROX_FOR_AGGREGATION FALSE Approximate query processing is used for aggregation queries and analytic queries. APPROX_FOR_COUNT_DISTINCT FALSE If TRUE converts COUNT(DISTINCT...) calls to APPROX_COUNT_DISTINCT calls. APPROX_FOR_PERCENTILE NONE This can be set to NONE PERCENTILE_CONT, PERCENTILE_CONT DETERMINISTIC, PERCENTILE_DISC, PERCENTILE_DISC DETERMINISTIC, ALL, ALL DETERMINISTIC. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 43

44 Approximate Query Processing Exact query Lots of temp Approx query no temp Copyright 2018 Oracle and/or its affiliates. All rights reserved. 44

45 Join Elimination SELECT e.ename, d.deptno FROM employees e, departments d WHERE e.deptno = d.deptno; SELECT e.ename FROM employees e WHERE exists ( SELECT 1 FROM departments d WHERE e.deptno = d.deptno); SELECT e.ename, e.project_id FROM employees e, projects p WHERE e.project_id = p.project_id (+); SELECT e.ename, e.deptno FROM employees e WHERE e.deptno is not null; SELECT e.ename FROM employees e WHERE e.deptno is not null; SELECT e.ename, e.project_id FROM employees e; Copyright 2018 Oracle and/or its affiliates. All rights reserved. 45

46 Multi-table left outer Join Copyright 2018 Oracle and/or its affiliates. All rights reserved. 46

47 Optimizer Statistics Advisor Oracle Database 12.2 introduce new feature called Optimizer Statistics Advisor The goal of the statistics advisor is to analyze how statistics are gathered, validate statistics and check the status of auto stats gathering. Consists of 4 components Rules Define the best way to manage statistics Findings Violations of the rules in the system Recommendations Solutions to the violations Actions Implementation of the recommendations Rules Findings Recommendations Actions Copyright 2018 Oracle and/or its affiliates. All rights reserved. 47

48 Optimizer Statistics Advisor Auto Happens daily in the maintenance window Partial cycle: Rules -> Findings -> Recommendations, Actions not implemented Report can be viewed any time and Actions implemented Manual Can be run anytime Full cycle: Rules -> Findings -> Recommendations -> Actions The Actions can generate a script instead of implementation Copyright 2018 Oracle and/or its affiliates. All rights reserved. 48

49 Wrapping it all up Query optimizer is the brain of Oracle Database. More you feed more powerful it becomes. Statistics calculation, User, SQL profiles, Hints, Plan Management, Init Parameters are the components who Influences to Optimizer. Many Optimizer Limitation in 11g which is improved in 12c and many more in 18c version. Adaptive Optimizer, one of the smartest way to perform SQL better and faster New Query Optimization Techniques in 12c onwards which increases query performance. Copyright 2018 Oracle and/or its affiliates. All rights reserved. 49

50 Copyright 2018 Oracle and/or its affiliates. All rights reserved. 50