Die Wundertüte DBMS_STATS: Überraschungen in der Praxis

Size: px

Start display at page:

Download "Die Wundertüte DBMS_STATS: Überraschungen in der Praxis"

Anis Newman
5 years ago
Views:

1 Die Wundertüte DBMS_STATS: Überraschungen in der Praxis, 14. Mai 2018 Dani Schnider, Trivadis danischnider.wordpress.com BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

of the books Data Warehousing mit Oracle Data Warehouse Blueprints Certified

2 Dani Schnider Working for Trivadis in Glattbrugg/Zurich Senior Principal Consultant Data Warehouse Lead Architect Trainer of several Courses Co-Author of the books Data Warehousing mit Oracle Data Warehouse Blueprints Certified Data Vault Data Modeler Oracle danischnider.wordpress.com

3 DBMS_STATS

A Short History of DBMS_STATS 4 Oracle 8i Gather / Delete Object Statistics Set / Get Statistic Values Export / Import Statistics Oracle 9i Gather / Delete System Statistics Oracle 10g Automatic

4 A Short History of DBMS_STATS 4 Oracle 8i Gather / Delete Object Statistics Set / Get Statistic Values Export / Import Statistics Oracle 9i Gather / Delete System Statistics Oracle 10g Automatic Statistics Gathering Job Dictionary / Fixed Objects Statistics Locking Statistics Statistics History Set / Get Parameters Oracle 11g Statistics Preferences Pending Statistics Extended Statistics Incremental Statistics Oracle 12c Additional Preferences New Synopsis Calculation (12.2) Optimizer Statistics Advisor (12.2)

5 Histograms

Histograms: Statistics about Data Distribution Useful for columns with skewed data distribution Histogram contains multiple buckets Until Oracle 11g: 254 Since Oracle 12c: 2048 Different histogram

6 Histograms: Statistics about Data Distribution Useful for columns with skewed data distribution Histogram contains multiple buckets Until Oracle 11g: 254 Since Oracle 12c: 2048 Different histogram types Frequency histograms Rows AD BG CH DK GB HT JP MA NC PE RW SY UY Country Height-balanced histograms Top-frequency histograms Hybrid histograms

7 Histograms: Parameter METHOD_OPT Parameter to define histograms SIZE: number of buckets resp REPEAT AUTO SKEWONLY Example: dbms_stats.gather_table_stats ( ownname => USER, tabname => 'ADDRESSES', method_opt => 'FOR ALL COLUMNS SIZE SKEWONLY');

8 Histograms: Real-Life Example What is wrong here? DBMS_STATS.gather_schema_stats ( ownname => 'APPL_USER', estimate_percent => 5, method_opt => 'FOR ALL INDEXED COLUMNS SIZE 254', cascade => TRUE, degree => DBMS_STATS.default_degree );

9 Histograms: Real-Life Example What is wrong here? DBMS_STATS.gather_schema_stats ( ownname => 'APPL_USER', estimate_percent => 5 DBMS_STATS.auto_sample_size, method_opt => 'FOR ALL INDEXED COLUMNS SIZE 254' SKEWONLY', cascade => TRUE, degree => DBMS_STATS.default_degree );

10 Histograms: Lessons Learned Avoid FOR ALL INDEXED COLUMNS (or column statistics will be missing) Number of buckets can be less then SIZE number for frequency histograms SIZE AUTO (Default) works only on used columns User SIZE SKEWONLY if possible Oracle 12c: Use AUTO_SAMPLE_SIZE for new histogram types SIZE AUTO / SKEWONLY still create maximum of 254 buckets

11 Gathering Statistics in ETL Jobs

Statistics Gathering in ETL Job T1 T2 2000 ETL Job T3 1500 DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T1'); DBMS_STATS.

12 Statistics Gathering in ETL Job T1 T ETL Job T DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T1'); DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T2'); INSERT STATEMENT INSERT T HASH JOIN TABLE ACCESS FULL T TABLE ACCESS FULL T

13 Statistics Gathering in ETL Job The Next Day... T1 T2 2 ETL Job T3 DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T1'); DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T2'); INSERT STATEMENT 2 2 INSERT T3 2 3 NESTED LOOP 2 4 TABLE ACCESS FULL T1 2 5 TABLE ACCESS FULL T

14 Statistics Gathering in ETL Job Rerun T1 T ETL Job T3 DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T1'); DBMS_STATS.gather_table_stats (ownname => 'DWH',tabname => 'T2'); INSERT STATEMENT 2 2 INSERT T3 2 3 NESTED LOOP 2 4 TABLE ACCESS FULL T1 2 5 TABLE ACCESS FULL T

15 Why is the Old Execution Plan Used? no_invalidate Does not invalidate the dependent cursors if set to TRUE. The procedure invalidates the dependent cursors immediately if set to FALSE. Use DBMS_STATS.AUTO_INVALIDATE to have Oracle decide when to invalidate dependent cursors. This is the default. Oracle Database PL/SQL Packages and Types Reference 11g Release 2 (11.2) DBMS_STATS.gather_table_stats (ownname => 'STG',tabname => 'T1',no_invalidate => FALSE); DBMS_STATS.gather_table_stats (ownname => 'STG',tabname => 'T2',no_invalidate => FALSE);

16 Gathering Statistics in ETL Jobs: Lessons Learned Object statistics should be gathered as part of every ETL job Loaded table can be used as input for next ETL jobs Disable automatic gathering job when ETL runs in the night Default behavior of NO_INVALIDATE not suitable for ETL jobs Recommendation: no_invalidate => FALSE Oracle 12c: Online Statistics Gathering used for CREATE TABLE AS SELECT Direct-Path INSERT into empty table (after TRUNCATE)

17 Incremental Statistics

18 Global Statistics Global statistics are essential for good execution plans num_distinct, low_value, high_value, density, histograms Gathering global statistics is time-consuming All partitions must be scanned Jan 18 Feb 18 Mar 18 Apr 18 Mai 18 Jun 18 Jul 18 Aug 18 Sep 18 Oct 18 gather statistics for current partition gather global statistics Data Dictionary

19 Incremental Global Statistics Synopsis-based gathering of statistics (since Oracle 11g) For each partition a synopsis is stored in SYSAUX tablespace Statistics metadata for partition and columns of partition Global statistics by aggregating the synopses from each partition Jan 18 Feb 18 Mar 18 Apr 18 Mai 18 Jun 18 Jul 18 Aug 18 Sep 18 Oct 18 gather statistics for current partition gather incremental global statistics synopsis

Synopsis Tables Two tables in SYSAUX tablespace: WRI$_OPTSTAT_SYNOPSIS_HEAD$ One row per table/partition/column WRI$_OPTSTAT_SYNOPSIS$ One row per distinct value

20 Synopsis Tables Two tables in SYSAUX tablespace: WRI$_OPTSTAT_SYNOPSIS_HEAD$ One row per table/partition/column WRI$_OPTSTAT_SYNOPSIS$ One row per distinct value Since Oracle 12.1, this table is LIST- HASH partitioned per table/partition In Oracle 12.2, this table is only used for backward compatibility (see later)

Some Key Figures of a Customer Data Warehouse Oracle Database 12.1.0.

21 Some Key Figures of a Customer Data Warehouse Oracle Database Enterprise Edition on Exadata Data Warehouse with around 24 TB of data 891 partitioned tables, most of them with INTERVAL daily partitions Total of partitions (state end of November 2017) 21 Synopsis Table Rows Size WRI$_OPTSTAT_SYNOPSIS_HEAD$ MB WRI$_OPTSTAT_SYNOPSIS$ GB

22 Gathering Incremental Statistics Gathering statistics with DBMS_STATS is very slow This query was always on the Top Activity list of SQL statments SELECT DISTINCT BO#, GROUP# FROM SYS.WRI$_OPTSTAT_SYNOPSIS_HEAD$ H WHERE GROUP# <> 0 AND GROUP# NOT IN (SELECT T.OBJ# * 2 FROM SYS.TABPART$ T UNION ALL SELECT T.OBJ# * 2 FROM SYS.TABCOMPART$ T )

Bugs, Bugs, Bugs Examples: 23100700 (13-OCT-2017) Patch 23100700: PERFORMANCE ISSUE

ISSUE WHEN RUNNING GATHER TABLE STATS WITH INCREMENTAL STATS 20602794 (10-OCT-2017)

23 Bugs, Bugs, Bugs Examples: (13-OCT-2017) Patch : PERFORMANCE ISSUE WITH RECLAIM_SYNOPSIS_SPACE (01-NOV-2017) Patch : KN:LNX:PERFORMANCE ISSUE WHEN RUNNING GATHER TABLE STATS WITH INCREMENTAL STATS (10-OCT-2017) Patch : INCREMENTAL STATS GATHER REPEATEDLY GATHERING ON SAME PARTITIONS

24 Our Solution on Oracle 12.1 Disable Incremental Statistics Weekly job every Sunday to gather global statistics Waiting for Oracle 12.2 or Oracle 18c

25 Synopsis Calculation ( Oracle 12.1) Synopsis calculation with Adaptive Sampling Number of distinct values (NDV) stored in WRI$_OPTSTAT_SYNOPSIS$ Size of synopsis tables: WRI$_OPTSTAT_SYNOPSIS_HEAD$ rows = number of partitions number of columns WRI$_OPTSTAT_SYNOPSIS$ rows = ;<=>?@ AB CAD<=;E CADF G NDV(col)

26 Synopsis Calculation ( Oracle 12.2) Synopsis calculation with HyperLogLog (HLL) algorithm Approximate NDV stored in WRI$_OPTSTAT_SYNOPSIS_HEAD$ Size of synopsis tables: WRI$_OPTSTAT_SYNOPSIS_HEAD$ rows = number of partitions number of columns WRI$_OPTSTAT_SYNOPSIS$ rows =

27 Configuration of Synopsis Calculation in Oracle 12.2 Set Optimizer Statistics Preferences in DBMS_STATS DBMS_STATS.set_database_prefs('APPROXIMATE_NDV_ALGORITHM', '<value>') DBMS_STATS.set_global_prefs('APPROXIMATE_NDV_ALGORITHM', '<value>') DBMS_STATS.set_schema_prefs('<schema>', 'APPROXIMATE_NDV_ALGORITHM', '<value>') DBMS_STATS.set_table_prefs('<schema>','<table>','APPROXIMATE_NDV_ALGORITHM','<value>') Values for APPROXIMATE_NDV_ALGORITHM: 'REPEAT OR HYPERLOGLOG' 'ADAPTIVE SAMPLING' 'HYPERLOGLOG' Preserve format for existing tables, create new tables with HyperLogLog (Default) Adaptive sampling is used for all synopses HyperLogLog is used for all new and stale synopses

Incremental Statistics: Lessons Learned Incremental Statistics can be problematic for tables with very high number of partitions Oracle 12.

28 Incremental Statistics: Lessons Learned Incremental Statistics can be problematic for tables with very high number of partitions Oracle 12.2: Reduced space requirements in tablespace SYSAUX WRI$_OPTSTAT_SYNOPSIS$ not needed anymore WRI$_OPTSTAT_SYNOPSIS_HEAD$ grows (blob column SPARE2) Better performance for gathering incremental statistics Problems of Oracle 12.1 seem to be solved HyperLogLog algorithm is very efficient

29 Bonus Demo

30 Bonus Demo What do you expect after this DBMS_STATS call? BEGIN DBMS_STATS.gather_table_stats ( ownname => USER, tabname => 'ADDRESSES', estimate_percent => 0.01, method_opt => 'FOR ALL COLUMNS SIZE 2', cascade => FALSE ); END;

31 DBMS_STATS Preferences The behavior of DBMS_STATS and many parameters can be set with preferences: For all existing tables in the database DBMS_STATS.set_database_prefs('<preference>', '<value>') For all existing tables of a schema DBMS_STATS.set_schema_prefs('<schema>', '<preference>', '<value>') For a particular table DBMS_STATS.set_table_prefs('<schema>','<table>','<preference>','<value>') For all new objects in the database DBMS_STATS.set_global_prefs('<preference>', '<value>')

32 DBMS_STATS Preferences APPROXIMATE_NDV_ALGORITHM REPEAT OR HYPERLOGLOG Oracle 12.2 AUTO_STAT_EXTENSIONS OFF Oracle 12.2 CASCADE TRUE DBMS_STATS.AUTO_CASCADE Oracle 10.1 CONCURRENT OFF Oracle 12.1 DEGREE NULL Oracle 10.1 ESTIMATE_PERCENT DBMS_STATS.AUTO_SAMPLE_SIZE Oracle 10.1 GLOBAL_TEMP_TABLE_STATS SESSION Oracle 12.1 GRANULARITY AUTO Oracle 10.2 INCREMENTAL FALSE Oracle 11.1 INCREMENTAL_STALENESS ALLOW_MIXED_FORMAT Oracle 12.1 INCREMENTAL_LEVEL PARTITION Oracle 12.1 METHOD_OPT FOR ALL COLUMNS SIZE SKEWONLY AUTO Oracle 10.1 NO_INVALIDATE FALSE DBMS_STATS.AUTO_INVALIDATE Oracle 10.1 OPTIONS GATHER Oracle 12.1 PREFERENCE_OVERRIDES_PARAMETER TRUE FALSE Oracle 12.2 PUBLISH TRUE Oracle 11.1 STALE_PERCENT 10 Oracle 11.1 STAT_CATEGORY OBJECT_STATS Oracle 12.2 TABLE_CACHED_BLOCKS 1 Oracle 12.1 WAIT_TIME_TO_UPDATE_STATS 15 Oracle

Further Information Incremental Statistics A Real World Scenario https://danischnider.wordpress.

33 Further Information Incremental Statistics A Real World Scenario ntal-statistics-a-real-world-scenario/ Online Statistics Gathering in Oracle 12c Avoid dbms_stats.auto_invalidate in ETL jobs danischnider.wordpress.com

34 Thank you. Dani Schnider, Trivadis danischnider.wordpress.com BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Pitfalls & Surprises with DBMS_STATS: How to Solve Them

Pitfalls & Surprises with DBMS_STATS: How to Solve Them Dani Schnider, Trivadis AG @dani_schnider danischnider.wordpress.com BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN