SEIZE THE DATA SEIZE THE DATA. 2015

Size: px

Start display at page:

Download "SEIZE THE DATA SEIZE THE DATA. 2015"

Toby Greene
6 years ago
Views:

2 Advanced Projection Design Herb Collins, Vertica Training August 10, 2015

Projections Associated with Tables ANCHOR TABLE Logical Customer table A B C D PROJECTIONS Physical C A B D B D customer_p1 (superprojection) C customer_p2

4 Projections Associated with Tables ANCHOR TABLE Logical Customer table A B C D PROJECTIONS Physical C A B D B D customer_p1 (superprojection) C customer_p2 (query-specific) C A customer_p3 (query-specific 4 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

5 Create Projection DDL CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) Define Columns and Encoding Types 5 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

6 Create Projection DDL CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) AS SELECT product_desc, sku_number, category_desc, product_cost FROM public.product_fact Define Columns and Encoding Types Base Query (More complex queries allowed) 6 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

7 Create Projection DDL CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) AS SELECT product_desc, sku_number, category_desc, product_cost FROM public.product_fact ORDER BY sku_number, category_desc Define Columns and Encoding Types Base Query (More complex queries allowed) Sort Order 7 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

8 Create Projection DDL CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) AS SELECT product_desc, sku_number, category_desc, product_cost FROM public.product_fact ORDER BY sku_number, category_desc SEGMENTED BY HASH (category_desc, department_desc); Define Columns and Encoding Types Base Query (More complex queries allowed) Sort Order Segmentation 8 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

10 Determine Projection Sort Order In the ORDER BY statement: 1. Start with Query Predicates Order predicates from lowest to highest cardinality Run Length Encode columns in sort order 2. Order the columns based on Group By Order from lowest to highest cardinality 3. Order remaining columns until average run length is less than ~10 4. Factor in special conditions Join optimization Identically Segmented projections Group By optimization 10 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

11 ORDER BY For Query Predicate CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) AS SELECT product_desc, sku_number, category_desc, product_cost FROM public.product_fact ORDER BY sku_number, category_desc SEGMENTED BY HASH (category_desc, department_desc); Define Columns and Encoding Types SELECT * FROM public.product_fact WHERE category_desc = housewares AND sku_number = KM745 ; Base Query (More complex queries allowed) Sort Order Segmentation 11 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

12 ORDER BY For GROUP BY CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) SELECT category_desc, sum(product_cost) FROM public.product_fact AS GROUP BY category_desc; SELECT product_desc, sku_number, category_desc, product_cost FROM public.product_fact ORDER BY category_desc Define Columns and Encoding Types Base Query (More complex queries allowed) Sort Order SEGMENTED BY HASH (category_desc, department_desc); Segmentation 12 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

13 ORDER BY For JOIN CREATE PROJECTION product_fact_data ( product_desc ENCODING AUTO, sku_number ENCODING RLE, category_desc ENCODING RLE, product_cost ENCODING AUTO ) AS SELECT product_desc, sku_number, F.sku_number = D.sku_number; category_desc, product_cost FROM public.product_fact ORDER BY sku_number Define Columns and Encoding Types SELECT * FROM product_fact F, product_dim D WHERE Base Query (More complex queries allowed) Sort Order SEGMENTED BY HASH (category_desc, department_desc); Segmentation 13 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

14 Optimize Joins Design projections so that the Optimizer will select the optimal join operator Create projections for Merge Join or Hash Join Minimize network operations Segment projections on the join keys so that join record matching is local Eliminate query-time join entirely Use a Pre-Join projection Denormalize externally 14 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

15 HASH JOIN In Explain Plan => EXPLAIN SELECT CD.annual_income,OSI.sale_date_key -> FROM online_sales.online_sales_fact OSI -> FULL OUTER JOIN customer_dimension CD ON CD.customer_key = OSI.customer_key; Access Path: +-JOIN HASH [FullOuter] [Cost: 18K, Rows: 5M] (PATH ID: 1) Outer (RESEGMENT) Inner (FILTER) Join Cond: (CD.customer_key = OSI.customer_key) Execute on: All Nodes +-- Outer -> STORAGE ACCESS for OSI [Cost: 3K, Rows: 5M] (PATH ID: 2) Projection: online_sales.online_sales_fact_dbd_12_seg_vmartdb_design_vmartdb_design Materialize: OSI.sale_date_key, OSI.customer_key Execute on: All Nodes +-- Inner -> STORAGE ACCESS for CD [Cost: 264, Rows: 50K] (PATH ID: 3) Projection: public.customer_dimension_dbd_1_rep_vmartdb_design_vmartdb_design_node0001 Materialize: CD.annual_income, CD.customer_key Execute on: All Nodes 15 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

16 HASH JOIN Operator The most common join operator requires no special optimization Hash table is built on dimension table (inner) Fact table (outer) is scanned, rows that match dimension rows are output as result set C B D E A E B A Join Column (Not 1 st in ORDER BY) No match Found a match Output tuple Hash Table (In Memory) Value 16 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. D B C A Build Hash Table D B C A Join Column (Not 1 st in ORDER BY)

17 MERGEJOIN In Explain Plan => EXPLAIN SELECT * FROM fact JOIN dim ON x=y JOIN ext on y=z; Access Path: +-JOIN MERGEJOIN(inputs presorted) [Cost: 815, Rows: 10K (NO STATISTICS)] (PATH ID: 1) Join Cond: (dim.y = ext.z) Materialize at Output: fact.x Execute on: All Nodes +-- Outer -> JOIN MERGEJOIN(inputs presorted) [Cost: 408, Rows: 10K (NO STATISTICS)] (PATH ID: 2) Join Cond: (fact.x = dim.y) Execute on: All Nodes +-- Outer -> STORAGE ACCESS for fact [Cost: 202, Rows: 10K (NO STATISTICS)] (PATH ID: 3) Projection: public.fact_super Materialize: fact.x Execute on: All Nodes 17 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

18 MERGEJOIN Operator If dimension table does not fit into memory, use merge join Faster for large joins because both tables are streamed through memory No need to go to disk Join Column (1 st in ORDER BY) A B B C D D D E X D M M N T X N Output Tuple C D B E A = B C A D 18 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. A B C D No more values. Join completed. Join Column (1 st in ORDER BY)

19 Merge Join Requirements Data on both sides of the join must be sorted by the join key column(s) By ordering the join key in projections By using a subquery with an ORDER BY clause Equality predicates will be executed first to reduce the size of the merge join Optimizer will bias towards projections for predicates Predicate column(s) can be first in projection ORDER BY only if the query has single-value equality predicates for the column(s) Recommended for large Dimensions to avoid disk spill 19 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

20 Optimize for a Merge Join Order by the join columns for both Fact and Dimension projections Sample query: SELECT * FROM Fact F, Dim D WHERE F.id = D.id; Sample projections: CREATE PROJECTION Fact_p ( a ENCODING RLE, b ENCODING RLE, c, id ENCODING RLE ) AS SELECT a, b, c, id FROM Fact ORDER BY id, b SEGMENTED BY HASH(c) ALL NODES; CREATE PROJECTION Dim_p ( x ENCODING RLE, y ENCODING RLE, z, id ENCODING RLE ) AS SELECT x, y, z,id FROM Dim ORDER BY id, x, y UNSEGMENTED ALL NODES; 20 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

21 Merge Join with Predicate Pushdown Order the fact projection by filter column(s) in predicate Generally best to optimize for filter first to eliminate any data that does not need to be joined. Then, order by the join columns for both Fact and Dimension projections Sample query: SELECT * FROM Fact F, Dim D WHERE F.id = D.id AND f.a = 10; Sample projections: CREATE PROJECTION Fact_p ( a ENCODING RLE, b ENCODING RLE, c, id ENCODING RLE ) AS SELECT a, b, c, id FROM Fact ORDER BY a,id, b SEGMENTED BY HASH(c) ALL NODES; CREATE PROJECTION Dim_p ( x ENCODING RLE, y ENCODING RLE, z, id ENCODING RLE ) AS SELECT x, y, z,id FROM Dim ORDER BY id, x, y UNSEGMENTED ALL NODES; 21 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

23 Optimize Joins Network Operations Requirement for local joins Matching rows from fact and dimension table must be on the same node Segmented fact to replicated dimension join Fact: SEGMENTED BY HASH (id) ALL NODES; Dimension: UNSEGMENTED ALL NODES; Identically Segmented Projections Both projections segmented on join key Final results aggregated on initiator node 23 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

24 Design for Local Joins Replicate small dimension projections and segment large fact projections Dimension Data Carol Jim Kim Fact Data trial results ABC's storm rt Javier This week Go Broncos! Pats-onside Segmented Projection trial results ABC's storm rt Javier This week Go Broncos! Pats-onside Replicated/Unsegmente d Projection Carol Jimmy Kim Carol Jimmy Kim Carol Jimmy Kim Node1 Node2 Node3 24 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

25 Segmented and Replicated Projections Segment fact, replicate dimension Sample query: SELECT * FROM fact F JOIN dim D ON F.id = D.id; Sample projections: CREATE PROJECTION fact_p ( a ENCODING RLE, b ENCODING RLE, c ENCODING RLE, id ) AS SELECT a, b, c, id FROM fact ORDER BY id, a, b, c SEGMENTED BY HASH(id) ALL NODES; CREATE PROJECTION dim_p ( x ENCODING RLE, y ENCODING RLE, z, id ) AS SELECT x, y, z,id FROM dim ORDER BY id, x, y UNSEGMENTED ALL NODES; 25 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

26 Identically Segmented Projections Segment by the join columns for both the fact and dimension Sample query: SELECT * FROM fact F JOIN dim D ON F.id = D.id; Sample projections; CREATE PROJECTION fact_p ( a ENCODING RLE, b ENCODING RLE, c ENCODING RLE, id ) AS SELECT a, b, c, id FROM fact ORDER BY id, a, b, c SEGMENTED BY HASH(id) ALL NODES; CREATE PROJECTION dim_p ( x ENCODING RLE, y ENCODING RLE, z, id ) AS SELECT x, y, z,id FROM dim ORDER BY id, x, y SEGMENTED BY HASH(id) ALL NODES; 26 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

27 Network Operations If data to be joined is not locally available, data is redistributed at runtime, requiring network operations Look for these network operators in the Explain Plan BROADCAST (sends temporary full copy of data to each node) Access Path: +-JOIN HASH [LeftOuter] [Cost: 40K, Rows: 10K (NO STATISTICS)] (PATH ID: 1) Inner (BROADCAST) Join Filter: (T1.a > T2.y) Materialize at Output: T1.b Execute on: All Nodes RESEGMENT (sends temporary ISP segments to each node) Access Path: +-JOIN HASH [Cost: 639, Rows: 10K (NO STATISTICS)] (PATH ID: 1) Inner (RESEGMENT) Join Cond: (T1.a = T2.y) Materialize at Output: T1.b Execute on: All Nodes 27 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

29 Optimize Group By Design for optimized Group By operator Create projections for Group By Pipe or Group By Hash Design for local Group By Segment such that each group will be only one node 29 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

30 Group By Hash Operator Query: SELECT count(*) FROM cust GROUP BY cust.state; Hash table must be completely built before results can be output to the user A A B B B C D D CA MA AL CA DE AL MA DE GROUP BY Column (Not 1 st in ORDER BY) Hash Map (Stored In Memory) Value Count CA 12 MA 12 AL 12 DE 12 Return to User 30 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

31 Group By Pipe Operator SELECT count(*) FROM cust GROUP BY cust.state; Less memory is used and runs faster than Group By Hash GROUP BY Column (1 st in ORDER BY) AL AL CA CA DE DE MA MA C D L M A M E L Count( CB ) 12 = Return to User 31 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

32 Group By Pipe Group By Pipe is critical to aggregate large amounts of data OR a large number of groups Can stream an infinite number of tuples Total number of groups does not matter If there is a selective predicate, optimize for that initially then for group by Equality predicates are executed before a Group by Pipe Output of Group by Pipe is sorted Look for Group by operator in query Explain Plan GROUP BY HASH or GROUP BY PIPELINED 32 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

33 Optimize for a Group By Pipe All grouping columns must be first in projection sort order Sample query: SELECT count(*) FROM cust GROUP BY a, b, c; Sample projection: CREATE PROJECTION cust_p ( a ENCODING RLE, b ENCODING RLE, c ENCODING RLE, d, e ) AS SELECT a, b, c, d, e FROM cust ORDER BY a, b, c SEGMENTED BY HASH(d) ALL NODES; 33 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

34 Distributed Group By If data is randomly distributed among nodes, Vertica must resegment data to accomplish Group by Resegment on Group By column(s) What if each group exists only on one node? Achieved by segmenting on the Group By columns Query explain plan will show resegmentation Distributed Group By will show RESEGMENT GROUP 34 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

35 Optimize for a Local Group By Segment by columns in the Group By Sample query: SELECT count(*) FROM cust GROUP BY a, b, c; Sample projection: CREATE PROJECTION cust_p ( a ENCODING RLE, b ENCODING RLE, c ENCODING RLE, d, e ) AS SELECT a, b, c, d, e FROM cust ORDER BY a, b, c SEGMENTED BY HASH(a, b, c) ALL NODES; 35 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

36 Resegment for Group By If projection SEGMENTED BY HASH(a,b) ALL NODES and query uses GROUP BY a Resegmentation required due to additional segmentation value not present in Group By GROUP BY a,b No resegmentation at runtime GROUP BY a,b,c No resegmentation at runtime GROUP BY a+1,b Resegmentation required due to expression on column a 36 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

37 Partially Sorted Group By Feature enabled by default Allows the Optimizer to improve the performance of aggregate queries on large data sets by leveraging the sort order of columns in the GROUP BY clause 37 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Explain Plan Example The query plan shows that the optimizer uses the partially sorted GROUP BY column, story_key is one of the sort columns in the projection, store.

38 Explain Plan Example The query plan shows that the optimizer uses the partially sorted GROUP BY column, story_key is one of the sort columns in the projection, store.store_sales_fact, and contains more than one call to aggregate functions 38 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

40 Live Aggregated Projections An anchor table must exist Anchor Table At least one regular projection must exist to serve as the base for the aggregated projection Base Projection Live aggregated projection is based off of one regular projection Live Aggregated Projection A B C B A C B A C D = A G G E = A G G Queries containing aggregates are made directly to the Live Aggregate Projection select * from projection_name_agg; 40 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

41 Projections with Expressions An anchor table must exist Anchor Table At least one regular projection must exist to serve as the base for the projection with expressions Base Projection Projection with Expressions is based off of one regular projection Projection with Expressions A B C B A C B A C D = A * B E = C / A Queries containing columns with expressions are made directly to the Projection with Expressions. select * from projection_name; 41 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

High Cardinality Column Lookups SELECT Address FROM cdr_table WHERE Number='978-263-4563'; Problem: Number column is high cardinality, so RLE

43 High Cardinality Column Lookups SELECT Address FROM cdr_table WHERE Number=' '; Problem: Number column is high cardinality, so RLE is ineffective 43 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

High Cardinality Column Split (1 of 3) Create and order by a new column called Area_Code If SELECT COUNT (distinct Number) = 13 million, then we would create a new column

44 High Cardinality Column Split (1 of 3) Create and order by a new column called Area_Code If SELECT COUNT (distinct Number) = 13 million, then we would create a new column such that SELECT COUNT (distinct Area_Code) 3.6k 44 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

45 High Cardinality Column Split (2 of 3) Add the new column to the projection sort order ORDER BY Area_Code, Number... Add the additional predicate to the SQL query Original Query SELECT Address FROM cdr_table WHERE Number=' '; New Query SELECT Address FROM cdr_table WHERE Number=' ' and Area_Code='978'; 45 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

46 High Cardinality Column Split (3 of 3) Pro Low cardinality column can be mixed in with other low-cardinality predicate columns early in the sort order Con Tables, projections, and queries must be modified 46 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

47 QUESTIONS? Please attend our Q&A with HP Big Data experts today Marina Ballroom, Lobby level 10:15 am 10:30 am 12:00 pm 1:00 pm 2:30 pm 3:00 pm 4:30 pm 5:00 pm

SEIZE THE DATA SEIZE THE DATA. 2015

SEIZE THE DATA SEIZE THE DATA. 2015 1 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Vertica Resource Management Ramesh Narayanan, Vertica Professional Services