Reducing Recollections: Summary Statistics, Extrapolation, & Threshold Carrie Ballinger Teradata 14.0 Certified Master

Transcription

1 Reducing Recollections: Summary Statistics, Extrapolation, & Threshold Carrie Ballinger Teradata 14.0 Certified Master Sr. Technical Consultant Teradata

2 The Optimizer Relies on Statistics for Query Plans 2 PARSING ENGINE Optimizer Dictionary Cache Table Header Random AMP Samples Data Dictionary DBC.StatsTbl Statistics Cache Collected Statistics Histograms Subtable 0 1.Looks first for statistics it needs in the Statistics Cache 2.If not found, looks for collected statistics in DBC.StatsTbl on disk 3.Random AMP samples (RAS) are kept in the Dictionary Cache > Accessed whenever the table header (subtable 0) is accessed from disk NOTE: This presentation reflects capabilities in Teradata Database 14.0 and 14.10

3 Evolution of Statistics Histograms (for your reference) Version Description 1 Does not including sampling fields, up to 100 detail intervals 2 Includes sampling fields in Interval 0, up to 100 detail intervals Default for Teradata Database 12.0, up to 200 detail intervals, new fields provided for the interval 0 Introduced in , adds new fields for interval 0 to record single-amp and all-amp random AMP sample estimates for enhanced table growth detection Introduced in 14.0, new layout, larger number of bytes for values, modifiable number of intervals, supports SUMMARY statistics and history records 6 Introduced in 14.10, UPDATE/INSERT/DELETE counts and other new detail 3

4 The Histogram Keeps History (Version 5 and above) 4 General information about the statistic A listing of up to 400 to 800 skewed values along with the number of rows carrying that value The remaining non-skewed values are spread equally across a default of 250 equal heights intervals, with each interval carrying summarized demographics SummaryInfo data from previous collections is saved at the end of the histogram SummaryInfo MinValue MaxValue NumOfDistinctValues NumOfRows (and other columns) Biased Values and Frequencies BiasedValAndFreq[1)] BiasedValAndFreq[2] BiasedValAndFreq[3]. Intervals Interval[1],MaxVal,ModeVal,ModeFreq,LowFreq... Interval[2],MaxVal,ModeVal,ModeFreq,LowFreq... Interval[3],MaxVal,ModeVal,ModeFreq,LowFreq.... SummaryRecord[1] MinValue MaxValue NumOfDistinctValues NumOfRows (and other columns)

5 New Table-Level SUMMARY Statistics Table-level statistics only, no histogram is built Provides row count and other information to the optimizer for better extrapolations Highly-optimized, runs fast, neglible impact on the system Summary Stats 1 second EXAMPLE Column Stats 540 seconds Execution time in seconds Very helpful after a big load job when you don t have time for full recollections of the table s statistics

6 6 SHOW SUMMARY STATISTICS VALUES ON table; /** TableLevelSummary **/ /* Version */ 6, /* NumOfRecords */ 1,. /* SummaryRecord[1] */ /* Temperature */ 0, /* TimeStamp */ TIMESTAMP, /* NumOfAMPs */ 72, /* OneAMPSampleEst */ 72936, /* AllAMPSampleEst */ 73415, /* RowCount */ 73414, /* DelRowCount */ 0, /* PhyRowCount */ 73415, /* AvgRowsPerBlock */ , /* AvgBlockSize (bytes) */ , /* BLCPctCompressed */ 0.00, /* BLCBlkUcpuCost */ , /* BLCBlkURatio */ , /* AvgRowSize */ , /* StatsSkipCount */ 0, /* SysInsertCnt */ 0, /* SysDeleteCnt */ 0, /* SysUpdateCnt */ 0, /* SysInsDelLastResetTS */ TIMESTAMP Single and all-amp RAS Actual number of rows Table update counts Summary statistics include data about: > Temperature > Compression > Columnar deletes > Update counts > Average blocksize > Random AMP samples Updated each time any statistic for the table is collected

7 History is Maintained for Both Individual Statistics and Table-Level SUMMARY Statistics Statistics histogram history Current SummaryInfo Current statistics intervals Past SummaryInfo (1) Past SummaryInfo (2) Past SummaryInfo (3) SHOW STATISTICS VALUES COLUMN (column-name) ON table-name; Table summary statistics history Current table summary statistics Past summary statistics (1) Past summary statistics (2) Past summary statistics (3) SHOW SUMMARY STATISTICS VALUES ON table-name; 7

8 Statistics Extrapolation Intended to reduce frequency of re-collection Most useful with large tables that are growing If statistics are determined to be stale Demographics are adjusted to account for table updates 8 When stale statistics are detected, the optimizer attempts to extrapolate the following: > Table row count > Number of distinct values, number of NULLs > High-mode frequency > Maximum value of the histogram

9 Viewing the Impact of SUMMARY Statistics Load of 1 million rows and collect full statistics on these columns HELP STATISTICS ON MyOrders; Date Time Unique Values Column Names 13/03/11 9:24:25 1,000,000 * 13/03/11 9:23:47 999,842 custdate 13/03/11 9:24:16 1,000,000 O_ORDERKEY 13/03/11 9:24:25 2,406 O_ORDERDATE 9 Add an additional 1 million rows and re- collect SUMMARY statistics COLLECT SUMMARY STATS ON MyOrders; HELP STATISTICS ON MyOrders; Date Time Unique Values Column Names 13/03/11 9:35:59 2,000,000 * 13/03/11 9:23:47 999,842 custdate 13/03/11 9:24:16 1,000,000 O_ORDERKEY 13/03/11 9:24:25 2,406 O_ORDERDATE Unchanged

10 Viewing & Validating Extrapolated Statistics Extrapolation changes the number of unique values for each statistic HELP CURRENT STATISTICS ON MyOrders; Date Time Unique Values Column Names 13/03/11 9:40:31 2,000,000 * 13/03/11 9:40:31 1,999,684 custdate 13/03/11 9:40:31 2,000,000 O_ORDERKEY 13/03/11 9:40:31 4,812 O_ORDERDATE Re-collect full statistics to validate extrapolation Changed 10 HELP STATISTICS ON MyOrders; Date Time Unique Values Column Names 13/03/11 9:44:31 2,000,000 * 13/03/11 9:44:07 1,999,376 custdate 13/03/11 9:44:16 2,000,000 O_ORDERKEY 13/03/11 9:44:31 2,406 O_ORDERDATE

11 Extrapolation Accuracy Improves With History Insert 1m rows Collect full stats Insert 1m rows Collect summary stats Collect full stats Insert 1m rows Collect summary stats Collect full stats Insert 1m rows Collect summary stats Collect full stats Insert 1m rows Collect summary stats # History records Orderdate # actual (distinct) values Orderdate # (distinct) values extrapolated Building up the number of histogram history records by re-collecting full statistics 11

12 UseCount Logging Tracks Statistics Usage UseCount is a new DBQL logging option in Teradata Database > Turn on by database Will create one row per statistic in the DBC.ObjectUsage table Number of times the optimizer used each statistic is captured Tracks INSERTs/UPDATEs/ DELETEs made to tables in the database being logged # rows Inserted, updated, deleted Base Table DBC.ObjectUsage Data Dictionary Optimizer Looked at statistics on Column A, Column B, Column (D,F) 12

13 13 Growth Detection with UseCount & Summary Statistics in Step 1. Check if SUMMARY statistics reflect recent table updates SUMMARY Row count Assess the difference Adjust SUMMARY row counts Go to Part 2 UseCount Update counts Step 2. Use adjusted SUMMARY row count to detect stale statistic histograms Adjusted SUMMARY row counts Row count from the histogram Compare and decide if statistic is stale If stale, extrapolate the statistic Extrapolation uses UseCount and histogram history records, if available

14 UseCount Logging and Threshold Functionality In Teradata Database the Optimizer will consider thresholds to determine whether or not to recollect statistics Thresholds can be expressed as > Percent of change (the preferred metric) > Time (some number of days) Default System Threshold only recognizes percent of change > With default system threshold the Optimizer selects a reasonable change percent threshold for each statistic individually 14 For information on the Threshold functionality, see:

15 Threshold Can Be Expressed at Three Levels 15 The System Default SysChange Threshold Option Enabled by default in DBS Control Uses Percent of Change only DBA-Defined System Defaults Overrides the system default DefaultUser ChangeThreshold Disabled by default in DBS Control Uses Percent of Change DefaultTime Threshold Disabled by default in DBS Control Uses Number of Days Individualized USING clauses Overrides all system-level threshold for that statement only Statement Threshold Uses percent of change and/or number of days Uses Percent of Change or Number of Days

16 Several Factors Must Align for the Optimizer to Skip Statistic Recollections Default System Threshold enabled? Use Count Logging? Adequate history records? Predictable patterns in growth? Below systemdetermined threshold? Skip USING Threshold specified? Percent Change? Number of Days? Below change threshold? Below time threshold? Skip Skip 16

17 Viewing Skipped Statistic Recollections EXPLAIN COLLECT STATISTICS tells you if the stat was skipped We SKIP collecting STATISTICS for ('o_orderdate'), because the estimated data change of 5% does not exceed the user-specified change threshold of 20%. Can view SkipCount in DBC.StatsTbl (use the view TableStatsV) SELECT DatabaseName, TableName,ColumnName,CAST(LastCollectTimeStamp AS Date) As CollectionDate,CAST(LastAlterTimeStamp AS Date) As LastSubmitDate,StatsSkipCount FROM DBC.TableStatsV WHERE ColumnName IS NOT NULL; 17

18 18 Give Skipping a Closer Look Extrapolation accuracy has improved, easy to validate Percent of change threshold uses UseCount numbers, if available Update patterns can be identified with the help of history records You can observe whether or not skipping has taken place Recommendation: Try it out > Turn on UseCount logging for a non-critical database only > Turn off the system default and use statement-based percent of change thresholds just for the test database table collections > Run updates, execute queries, issue collect statistics statements > View extrapolation, evaluate skip frequency, check query plans

19 In-Depth Descriptions of Statistics Enhancements 19

20 20 More Information on Developer Exchange