Real-World Performance Training Star Query Edge Conditions and Extreme Performance

Transcription

1

2 Real-World Performance Training Star Query Edge Conditions and Extreme Performance Real-World Performance Team

3 Dimensional Queries The Dimensional Model and Star Queries Star Query Execution Star Query Prescription Edge Conditions and Extreme Performance

4 Edge Conditions Snowflake Schema product customer region product cust_mv date sales sales month supplier cal_mv supplier year Collapse your dimension tables into materialized views

5 Edge Conditions Relationships between Dimensions product customer product customer sales sales calendar supplier calendar supplier Examine data model, collapse your dimension tables into materialized views

6 Edge Conditions Common Join Columns product customer product customer part sales sales calendar supplier calendar supplier Examine data model, look to see if there are missing columns

7 Edge Conditions Not Completing Joins calendar sales supplier customer region SELECT GROUP BY ORDER BY d.year, s.nation, c.nation, SUM(l.extendedprice) FROM lineorder l JOIN calendar d ON l.lo_orderdate = d_datekey JOIN supplier s ON l.lo_suppkey = s.suppkey JOIN customer c ON l.lo_suppkey = c.custkey JOIN region r ON s.region_id = r.region_id WHERE d.year IN (1993, 1994, 1995) AND r.region_code = ASIA AND c.region_id = s.region_id d.year, s.nation, c.nation, d.year, s.nation, c.nation; Snowflake schema with queries providing filter predicates once for both dimension joins and not completing joins

8 Edge Conditions Completing Joins calendar sales supplier region SELECT GROUP BY ORDER BY d.year, s.nation, c.nation, SUM(l.extendedprice) FROM lineorder l JOIN calendar d ON l.lo_orderdate = d_datekey JOIN supplier s ON l.lo_suppkey = s.suppkey JOIN customer c ON l.lo_suppkey = c.custkey JOIN region r1 ON s.region_id = r1.region_id JOIN region r2 ON c.region_id = r2.region_id WHERE d.year IN (1993, 1994, 1995) AND r1.region_code = ASIA AND r2.region_code = ASIA d.year, s.nation, c.nation, d.year, s.nation, c.nation; customer region Join each dimension to outer table in snowflake schema

9 Finesse Moves Materialized Views product calendar sales customer supplier Operation Object Name Predicate information SELECT STATEMENT SORT GROUP BY HASH JOIN TABLE ACCESS STORAGE FULL HASH JOIN SUPPLIER LO_SUPPKEY = S_SUPPKEY JOIN FILTER CREATE :BF0001 PART JOIN FILTER CREATE :BF0000 LO_ORDERDATE = D_DATEKEY TABLE ACCESS STORAGE FULL CALENDAR D_YEAR IN ( 1993, 1994, 1995 ) HASH JOIN JOIN FILTER CREATE :BF0002 LO_PARTKEY = P_PARTKEY TABLE ACCESS STORAGE FULL PRODUCT P_CONTAINER = 'JUMBO PACK' JOIN FILTER USE JOIN FILTER USE :BF0001 :BF0002 PARTITION RANGE JOIN-FILTER TABLE ACCESS STORAGE FULL SALES :BF0000

10 Finesse Moves Materialized Views product customer sales SALES_MV calendar supplier CREATE MATERIALIZED VIEW SALES_MV [storage options] [rewrite options] AS SELECT [columns from dimension tables and fact tables] FROM sales JOIN customer ON c_custkey = sales_custkey JOIN supplier ON s_suppkey = sales_suppkey JOIN product ON p_partkey = sales_partkey JOIN calendar ON s_suppkey = sales_suppkey [GROUP BY clause, if possible/necessary]

11 Finesse Moves Materialized Views sales_mv Operation Object Name Predicate information SELECT STATEMENT SORT GROUP BY MAT_VIEW REWRITE ACCESS STORAGE FULL SALES_MV P_CONTAINER = 'JUMBO PACK D_YEAR IN ( 1993, 1994, 1995 )

12 Finesse Moves Clustering Mechanisms of co-locating data with common values within a table If you co-locate data you can retrieve them with the minimal amount of IO Sometimes data is naturally clustered due to the method of loading (eg transaction data loaded by day will naturally be grouped by that day) Other methods can be used to guarantee the co-location of the data

13 Finesse Moves Techniques Used for Clustering Sorting the data before/during loading means that the data will be naturally sorted around the sort key, reducing the number of physical I/Os required to retrieve rows with the predicate based on the sort key if retrieving data via indexes or storage indexes Clusters Groups of one or more tables that are clustered around a common key which can be accessed via index or hash If we access rows on the key then we can retrieve all the related data with less I/Os

14 Finesse Moves Partitioning to Cluster Data Divide data based on the partitioning column Only need to access required partitions when data is queried by the partitioning key For Example List Partitioning based on country, all the orders for a country would be in a single partition might lead to asymmetric sizes though UK USA GERMANY KOREA ITALY SPAIN CHINA

15 Finesse Moves Attribute Clustering New in Sorts the data as it is loaded or reorganised on the clustering key The rows with a given value of the clustering column are stored using a small number of blocks Used in conjunction with Zone Maps to reduce the amount of IO needed to scan a table while searching for clustering column values Will improve the efficiency of DBIM queries on the clustering column

16 Category Attribute Clustering Category Boys Boys Boys Boys Girls Country AR JP SA US AR AR Women Interleaved-Ordered Table JP Women Country SA Women US Women Linear-Ordered Table Girls Girls Girls JP SA US AR Men JP Men SA Men US Men Men Men Men Men AR JP SA US AR Girls JP Girls SA Girls US Girls Women Women Women AR JP SA AR Boys JP Boys SA Boys US Boys Women US

17 Star Query Multi-User Demo Part II

18 Multi User Demo Recall Star Schema Benchmark Data Model Four racers, B*Tree Indexes, Bitmap Indexes, Exadata, and In-Memory Each racer executes as many randomized queries as it can during the same interval Begin with four users and increase

19 Multi-User Race Demo Prior Configuration, 32 users Queries running in parallel with DoP=2 32 concurrent users B*Tree Indexes with Nested Loop Joins Quick config Bitmap Indexes with Star Transformation Exadata with Smart Scan Oracle Database In-Memory

20 Multi-User Race Demo Previous results with 32 Users (DoP=2)

21 Multi-User Race Demo Previous results with 32 Users (DoP=2) Exadata and Indexes access methods constrained on I/O In-Memory was nearly all CPU

22 Multi-User Race Demo Previous results with 32 Users (DoP=2) Exadata has reached its limits

23 Multi-User Demo Exadata vs. Database In-Memory Most of the queries in the demo were I/O intensive By using Hybrid Columnar Compression and a sound partitioning strategy, you can reduce the I/O load on Exadata and get better throughput

24 Multi-User Race Demo Configuration, 32 users Queries running in parallel with DoP=2 32 concurrent users B*Tree Indexes with Nested Loop Joins Quick config Bitmap Indexes with Star Transformation Exadata Partitioned with HCC Oracle DBIM and Partitioning

25 Multi-User Race Demo 32 Users, Partitioning (Exa & DBIM) and HCC (Exa) Without compression or partitioning Exadata with HCC query high and partitioning on both Exa & DBIM Exadata with HCC Query High and Partitioning provided almost 5x throughput compared to uncompressed and no partitioning DBIM was also better (2275 vs. 1932)

26 Multi-User Race Demo 32 Users, Partitioning and HCC (DoP=2) Exadata without compression or partitioning Exadata with HCC and partitioning

27 Multi-User Demo Materialized Views and Clustering Let s expand on the previous test by testing with In-Memory Materialized Views We ll have attribute clustered In- Memory MVs and partitioned In- Memory MVs with attributing clustering

28 Multi-User Race Demo Configuration, 32 users Queries running in parallel with DoP=2 32 concurrent users Exadata Partitioned with HCC Quick config Oracle DBIM and Partitioning MVs In-Memory Attributed Clustered Partitioned In-Memory MVs with Attribute Clustering

29 Multi-User Race Demo 32 Users, Partitioning, HCC, In-Memory MVs, Attribute Clustering Exadata Partitioned HCC In-Memory Partitioned In-Memory MVs with Attribute Clustering Partitioned In-Memory MVs with Attribute Clustering

30 Multi-User Demo Lessons Learned Partitioning and compression on Exadata is important Materialized Views can make a difference Clustering can make a difference Already have good performance? You can probably do better

31 Reference Section

32 Star Query Exceptions

33 Demo: Dimensional Query Exceptions Snowflake schemas Dimension table joins Snowflake without completing predicates Common dimension joins to fact table Meta-model

34 Exceptions Low cardinality snowflake query

35 Exceptions High cardinality snowflake query

36 Exceptions Low cardinality snowflake 7 seconds elapsed Filtering on partition JOIN filter from DAY_DIM and join filter from CAT_DIM/PART_DI M join Filters 300m rows down to 25m after applying Bloom Filters Spends time doing large hash joins

37 Exceptions High cardinality snowflake 12 seconds elapsed Filters on date dimensions and no others Filters 300m rows down to 46m after applying Bloom Filters Spends time doing large hash joins

38 Exceptions Low cardinality snowflake with dimensions collapsed 0.1 seconds elapsed Filters on DAY_DIM_MV and PART_DIM_MV, yielding only 42k rows from fact table after filter Because we filtering early, downstream cost of joins is reduced

39 Exceptions High cardinality snowflake with dimensions collapsed 1 second elapsed Filters on DAY_DIM_MV and CUSTOMER table Able to filter much more data from fact table

40 Exceptions Dimension Table Joins

41 Exceptions Dimension Table Joins 15 seconds elapsed Filtering on SUPPLIER Bloom Filter Significant time completing hash join to CUSTOMER

42 Exceptions No Dimension Table Joins Before Join each dimension to fact Apply predicates directly to dimension

43 Exceptions Dimension Table Joins 3 seconds elapsed Able to use vector transformation

44 Exceptions Snowflake without completing predicates supp day_dim lo region cust Only apply predicate to REGION on CUSTOMER join

45 Exceptions Snowflake without completing predicates 34 seconds Only filtering on DAY_DIM

46 Exceptions Snowflake with completing predicates supp region day_dim lo cust region Apply predicates on both REGION dimensions

47 Exceptions Snowflake with completing predicates 9 seconds Filtering on both DAY_DIM and SUPP 21m rows after filtering compared to 131m without completing predicates

48 Exceptions Snowflake with complete predicates and MVs supp region day_dim lo cust region

49 Exceptions Snowflake with complete predicates and MVs 4 seconds More filtering

50 Exceptions Common Dimension Joins to Fact PART_EXT is an extension of PART Apply some filter predicates against PART and some against PART_EXT

51 Exceptions Common Dimension Joins to Fact Although inherently correlated, PART and PART_EXT filters are applied independently to fact table Fact table is filtered using predicates & Bloom Filters from DATE_DIM and PART_EXT, but not PART This leads to under-estimate of join cardinality downstream Since the correlation is across tables, we can t apply extended stats techniques

52 Exceptions No Common Dimension Joins to Fact Here, PART has all the columns we need All PART filters applied against same table, allowing is to be able to understand correlation

53 Exceptions No Common Dimension Joins to Fact Now we can filter optimally and achieve the right cardinality estimates Extract 700k rows from fact table after filters applied

54 Exceptions Meta-model lineorder (fact) part_meta part_lookups

55 Exceptions Meta-model lineorder (fact) part_meta part_lookups

56 Exceptions Meta-model 42 seconds elapsed Bloom filter on PART% joins, not on DATE_DIM or SUPPLIER Low cardinality estimates on PART_LOOKUPS/PART_ META joins cause NL joins to DATE_DIM and SUPPLIER, after LINEORDER is scanned with poor filtering

57 Exceptions Meta-model using Materialized View to Flatten lineorder (fact) part_meta part_lookups

58 Exceptions Meta-model with MV to Flatten 3 seconds elapsed Right-deep bloom filters from PART_META_MV, SUPPLIER, and DATE_DIM. Filters ~299m rows (early)

59 Star Query Finesse

60 Finesse Moves Demo Exadata & DBIM Partitioning Zonemaps & Attribute Clustering Materialized Views In-Memory Aggregation Fixed-With Bitmap Indexes

61 Finesse Moves Query Display 4th quarter 1995 sales to customers in Asia that post no sales in 1st quarter 1996

62 Finesse Moves 1) Exadata Baseline 30S seconds elapsed 60 GB of IO

63 Finesse Moves 2) Exadata with HCC 30S seconds elapsed 16 GB of IO

64 Finesse Moves 3) Exadata with Partitioning and HCC Query Low 3 seconds elapsed 915 MB of IO

65 Finesse Moves 4) Exadata with Zonemaps (No compression or partitioning) 26 seconds elapsed 44 GB of IO Zonemap pruning

66 Finesse Moves 5) Exadata with Zonemaps and HCC (no partitioning) 27 seconds elapsed 16 GB of IO Zonemap pruning

67 Finesse Moves 6) Exadata with Zonemaps, Partitioning, and HCC 2 seconds elapsed 915 MB of IO Zonemap & partition pruning

68 Finesse Moves 7) Exadata with MV 9 seconds elapsed

69 Finesse Moves 8) Exadata with MV and HCC 3 seconds elapsed

70 Finesse Moves 9) Exadata with MV with Attribute Clustering (no HCC) 1 second elapsed

71 Finesse Moves 10) Exadata with MV with Attribute Clustering and HCC 2 seconds elapsed

72 Finesse Moves 11) Exadata with MV with Partitioning, Attribute Clustering and HCC 1 second elapsed

73 Finesse Moves 12) DBIM Baseline 28 seconds elapsed Same as Exadata

74 Finesse Moves 13) DBIM with Partitioning 3 seconds elapsed

75 Finesse Moves 14) DBIM with Zonemaps (unpartitioned) 4 seconds elapsed Zonemap pruning and Bloom filtering

76 Finesse Moves 15) DBIM with Partitioning and Zonemaps 3 seconds elapsed Zonemap pruning, Bloom pruning, and Bloom filtering

77 Finesse Moves 16) DBIM with MV 2 seconds elapsed

78 Finesse Moves 17) DBIM with MV and Attribute Clustering 1 second elapsed

79 Finesse Moves 18) DBIM with Partitioned MV and Attribute Clustering 0.58 seconds

80 Finesse Moves 19) DBIM without In-Memory Aggregation 5 seconds elapsed 67% of time completing joins and aggregation

81 Finesse Moves 20) DBIM with In-Memory Aggregation 2 seconds elapsed All time on scan, filtering, and VGBY transformation

82 Finesse Moves 21) Bitmap indexes with Star Transformation 16 GB of IO 1.7 seconds database time and 49k buffer gets

83 Finesse Moves 22) Bitmap Fixed Width (Hakan) with Star Transformation 1.1 seconds database time and 47k buffer gets

84