Overview. DW Performance Optimization. Aggregates. Aggregate Use Example

Size: px

Start display at page:

Download "Overview. DW Performance Optimization. Aggregates. Aggregate Use Example"

Colleen Kennedy
5 years ago
Views:

1 Overview DW Performance Optimization Choosing aggregates Maintaining views Bitmapped indices Other optimization issues Original slides were written by Torben Bach Pedersen Aalborg University 07 - DWML course 2 Aggregates Observations DW queries are simple, follow same schema Aggregate (GROUP-BY) queries Idea Compute and store query results in advance (pre-aggregation) Example: store total sales per month and product Yields large performance improvements (e.g., factor 000) No need to store everything reuse whenever possible Example: quarterly total can be quickly computed from monthly total Prerequisites Tree-structured dimensions with fixed height Many-to-one relationships from fact to dimensions Facts mapped to bottom level in all dimensions Otherwise, re-use is not possible Aggregate Use Example Imagine billion sales rows, 000 products, 00 locations CREATE VIEW TotalSales (pid,locid,total) AS SELECT s.pid,s.locid,sum(s.sales) FROM Sales s GROUP BY s.pid,s.locid The materialized view has 00,000 rows Query rewritten to use view SELECT p.category,sum(s.sales) FROM Products p, Sales s WHERE p.pid=s.pid GROUP BY p.category Rewritten to SELECT p.category,sum(t.total) FROM Products p, TotalSales t WHERE p.pid=t.pid GROUP BY p.category Query becomes 0,000 times faster! Aalborg University 07 - DWML course 3 Aalborg University 07 - DWML course 4

2 Pre-Aggregation Choices Full pre-aggregation: (all combinations of levels) Fast query response Takes a lot of space/update time (0-500 times raw data) No pre-aggregation Slow query response (for terabytes ) Practical pre-aggregation: chosen combinations A good compromise between response time and space use Most (R)OLAP tools now support practical pre-aggregation IBM DB2 UDB Oracle 0g MS Analysis Services Using Aggregates Application: aggregates used via aggregate navigator Given a query, the best aggregate is found, and the query is rewritten to use it Done by the system, not by the user Traditionally done in middleware, e.g., ODBC Performed directly by DBMS Four design goals for aggregate usage Aggregates stored separately from detail data Shrunk dimensions mapped to aggregate facts Connection between aggregates and detail data known by system All queries (SQL) refers only detail data SUM, MIN, MAX, COUNT, AVG can all be handled The last one requires little trick Why? Aalborg University 07 - DWML course 5 Aalborg University 07 - DWML course 6 Choosing Aggregates Practical pre-aggregation, decide what aggregates to store Non-trivial (NP-complete) optimization problem Space use Update speed Response time demands Actual queries Index and/or aggregates Choose an aggregate if it is considerably smaller than available, usable aggregates (factor 3-5-0) Supported (semi)-automatically by DBMS Oracle, DB2, MS SQL Server Implementing Data Cubes Efficiently SIGMOD 96 paper Greedy approach: simple but effective Above DBMS now use similar, but more advanced techniques Data Cube The data cube stores multidimensional GROUP BY relations of tables in data warehouses Red White Blue By make Group By (with total) By color Sum Red White Blue Aggregate Sum Database capable By make Data warehouse Cross Tab capable Chevy Ford By color Sum Ford Chevy By color and year By make and year Sum By make and color Aalborg University 07 - DWML course 7 Aalborg University 07 - DWML course 8

A Data Cube example Lattice Framework (I) 8 possible views for 3 dimensions. Each view gives the total sales as per that grouping. (II) 8 views organized into a Lattice.

3 A Data Cube example Lattice Framework (I) 8 possible views for 3 dimensions. Each view gives the total sales as per that grouping. (II) 8 views organized into a Lattice. part, supplier, customer (6M rows) 2. part, customer (6M) 3. part, supplier (0.8M) 4. supplier, customer (6M) 5. part (0.2M) 6. supplier (0.0M) 7. customer (0.M) 8. none () psc 6M pc 6M ps 0.8M sc 6M p 0.2M s 0.0M c 0.M none 9 M rows total Scenario: A query asks for the sales of a part a) If view pc is available, need to process about 6M rows; OR b) If view p is available, need to process about 0.2M rows Questions: a) How many views to materialize to get reasonable performance? - and b) Given that we have space S, what views to materialize to minimize average query costs? (III) Picking the right views to materialize improve performance (IV) View pc and sc not needed Effective rows reduced from 9M to 7M Each lattice node represents a view / query Dependence relationship: Q Q2 Q is the descendant of Q2 Q can be answered using only the results of Q2 In other words, Q is dependent on Q2 The operator imposes a partial ordering on the queries Top view = base data (i.e., most detail) Essentially, the lattice captures dependency relationship among queries / views and can be represented by a lattice graph Aalborg University 07 - DWML course 9 Aalborg University 07 - DWML course 0 Cost Model Greedy Algorithm Cost of answering a query based on a view and it has major assumptions Model simple and realistic, Enable the design and analysis of powerful algorithms Greedy algorithm: each time choose the view with the maximum benefit Explanation: 2 3 Time to answer a query is equal to the space occupied by the query from which the query is answered All queries are identical to some queries in the given lattice The clustering of the materialized query and indexes have not been considered An illustration: To answer query Q, choose an ancestor of Q, say, Qa, that has been materialized We thus need to process the table of Qa Cost of answering Q: number of rows in the table of Qa The Greedy algorithm S = {top view} for i= to k do begin Select that view v not in S such that B(v,S) is maximized; S = S union {v}; Given a data cube lattice with space costs associated with each view Always include the top view because it cannot be generated from other views Suppose we may only select k number of views in addition to the top view The benefit of view v (relative to S), is based on how v can improve the costs of evaluating views, including itself Experimental validation of the cost model: almost linear relationship between size and running time Source Size S Time T Ratio m From cell itself From view s 0, From view ps 0.8M end; return S; The total benefit of v is the sum over all views w of the benefit of using v to evaluate w, providing that benefit is positive From view psc 6M Aalborg University 07 - DWML course Aalborg University 07 - DWML course 2

4 Greedy Algorithm an example Greedy Algorithm vs optimal choice 50 b 00 a 30 C 75 Round 2: views b, d, e, g, h start with cost 50 and the other views start with cost 00 Which queries can be improved by view f? Benefit of view f? Ans: (00-40)+(50-40)=70 Situations where the algorithm does poorly Round : Picks c whose benefit is 44 Round 2: Picks b or d with benefit of 200 each e d f 40 g h 0 Lattice with space costs Initial: each query answered by a, with cost 00 Which queries can be answered by view b? What is the benefit of view b? Ans: 5*(00-50)=250 Can we find a better view than view b? End of round : view b selected Benefits of possible choices at each round b c d e f g h Round Round 2 Round 3 50 x 5 = x 5 = x 2 = x 2 =40 60 x 2 = 99 x = x = x 2 = x 2 = 60 x 2 = = x = x = x = x 2 = 60 2 x + 0 =50 49 x = x = 30 After 3 rounds, total cost is 4 (initial cost 800) In summary, each round pick the view with the maximum benefit Nodes total b Nodes total a c 99 Nodes total 000 d 00 Nodes total 000 A Lattice where the greedy does poorly Greedy results in = 624 But, the optimal choice is to pick b and d b and d improve by 00 for itself and all 80 nodes below resulting in total benefits of 80 Ratio of greedy/optimal=624/80=76% but the benefit is at least 63% of the benefit of the optimal algorithm as reasoned by the authors Aalborg University 07 - DWML course 3 Aalborg University 07 - DWML course 4 Greedy algorithm space-time trade-off Optimal cases and anomalies Experiment with a lattice: Important to materialize only the first few (5) views, performance increases at first Greedy order of view selection example Selection Benefit Time Space cp ns nt c p cs np ct t n s none infinite 24M rows 2M 5.9M 5.8M M M 0.0M small small small small 72M rows 48M 36M 30.M 24.3M 6M rows 6M 6M 6.M 6.3M.3M 6.3M Time / Space Total Time Total Space Views Two situations where the algorithm is optimal If a is much larger than the other a s, the greedy is close to optimal If all the a s are equal then greedy is optimal but there are also two situations where the algorithm is not realistic Views in a lattice are unlikely to have the same probability of being requested in a query Instead of asking for some fixed number of views to materialize, should instead allocate a fixed amount of space to views Aalborg University 07 - DWML course 5 Aalborg University 07 - DWML course 6

Hypercube lattices some observations Space-optimal and time-optimal solutions The size of views grows exponentially, until it reaches the size of the raw data at rank [log r m], i.e. the cliff Assumptions and basis of reasoning Each domain size is r Top element has m cells appearing in raw data Is there sense to minimize time by materializing all views?

5 Hypercube lattices some observations Space-optimal and time-optimal solutions The size of views grows exponentially, until it reaches the size of the raw data at rank [log r m], i.e. the cliff Assumptions and basis of reasoning Each domain size is r Top element has m cells appearing in raw data Is there sense to minimize time by materializing all views? Size of views Log r m m If group on i attributes, cube has r i cells If r i m, then each cell will have at most one data point and m of the cells with be nonempty. Use m as the size of any view. What is the average time for a query when the space is optimal? No gain past the cliff No point to do so n Number of group-by attributes Effect of the number of group-by attributes on the size of views * (p,s) does not have 6M rows because the benchmark made it so If r i <m, then almost all r i cells will have at least one data point. Space cost is r i as several data points can be collapsed into one aggregate Which explains why grouping of 2 attributes (p,c), (s,c) have the same size as (p,s,c) at 6 M rows* in Figure Space is minimized when only the top view is materialized Every query would take time m Total time cost for all 2 n queries is m2 n Nature of time-optimal solution depends on. Rank k = [log r m] at which the cliff occurs; and 2. Rank j, such that r j = ( n j) is maximized Aalborg University 07 - DWML course 7 Aalborg University 07 - DWML course 8 Hierarchies & the Lattice Framework Composite lattices Hierarchies are important as they underlie two commonly used query operations, Drilldown and Roll-up Dependencies caused by different dimensions and attribute hierarchies can be combined into a Direct Product lattice A common hierarchy and its dependency relations cp 6M Week Day none Month Year Year Month Day Week Day; but Month week; week month but, hierarchies introduce query dependencies that must be accounted for when determining which queries to materialize; and this can be complex customer 0.M nation 25 none + product 6M size type none Combining two hierarchical dimensions = s 50 p 0.2M none t 50 ns 250 np 5M n 25 cs 5M nt 3750 c 0.M A Direct Product lattice ct 5.99M assuming views can be created by independently grouping any or no member of the hierarchy for each of n dimensions Aalborg University 07 - DWML course 9 Aalborg University 07 - DWML course

6 Applicability of the Lattice Framework Summary on choosing aggregates Advantages of the lattice framework Clean framework A clean framework to work with dimensional hierarchies, since hierarchies are themselves lattices Easy to model dependencies Dependency relationship among queries captured by the lattice Order of materialization A simple descending-order topological sort on lattice nodes gives the required order of materialization Why?. Lattice framework 2. Linear cost model 3. Greedy algorithm (How it works?) 4. Space-time trade-off Aalborg University 07 - DWML course 2 Aalborg University 07 - DWML course 22 Overview Choosing aggregates Maintaining views Bitmapped indices Other optimization issues View Maintenance How and when should we refresh materialized views? Total re-computation Too expensive Incremental view maintenance Apply only changes since last refresh to view Insert new rows R i and delete existing rows R d Update = delete + insert Additional info must be stored in views To make the views self-maintainable Store number of derivations c v along with each row v in V Aalborg University 07 - DWML course 23 Aalborg University 07 - DWML course 24

7 View Maintenance Projection views (with DISTINCT in SQL) V=Π(R) (r,c): r appears c times in R If (r,c i ) Π(R i ) and (r,c r ) V then c r =c r +c i, otherwise insert (r,c i ) into V If (r,c d ) Π(R d ) and (r,c r ) V then c r =c r -c d, delete from V if c r =0 Join views V=R S Compute R i S and add to V, update counts Compute R d S and subtract from V, update counts Aggregation View Maintenance COUNT Maintain <g,count> Update count based on inserts/deletes Insert <g,> rows for new values, delete rows from V with count=0 SUM Maintain <g,sum,count> Update count and sum based on inserts/deletes Insert <g,a,> rows for new values, delete rows from V with count=0 AVG computed as sum/count MIN (similar for MAX) Maintain <g,minimum,count> Insert (g,m): update minimum,count based on m =,<,> minimum Delete (g,m): update minimum,count based on m =,<,> minimum If count=0, base table must be searched (expensive!) Aalborg University 07 - DWML course 25 Aalborg University 07 - DWML course 26 Practical View Maintenance When to synchronize? Immediate in same transaction as base change Lazy when V is used for the first time after base updates Periodic, e.g., once a day, often together with base load Forced after a certain number of changes Updating aggregates Computation outside DBMS in flat files (still relevant?) Built by loader Computation in DBMS using SQL Can be expensive: DBMS must be tuned for this Supported by tool/dbms Oracle, SQL Server, DB2 Indexing Index used in combination with aggregates Index on dimension tables and on materialized views Fact table Build primary B-tree index on dimension keys (primary key)? Build indices on each dimension key separately (index intersection) Indices on combinations of dimension keys? (many!) Sort order is important (index-organized tables) Compressing data can be possible (values not repeated) Can save aggregates due to fast sequential scan Best sort order (almost) always Time Dimension tables Build indices on many/all individual columns (think of type) Build indices on common combinations Hash indices Efficient for un-sorted data Aalborg University 07 - DWML course 27 Aalborg University 07 - DWML course 28

8 Bitmap Indices A B-tree index stores a list of RowIDs for each value A RowID takes ~8 bytes Large space use for columns with low cardinality (gender, color) Example: Index for bio. rows with gender takes 8 GB Not efficient to do index intersection for these columns Idea: make a position bitmap for each value (only two) Female: Male: Takes only (no. of values)*(no. of rows)* bit Example: bitmap index on gender (as before) takes only 256 MB Very efficient to do index intersection (AND/OR) on bitmaps Can be improved for for higher cardinality using compression Supported by some RDBMS (DB2, Oracle) Using Bitmap Indices Query example Find female customers in Aalborg with black hair and blue eyes Female: Aalborg: Black: 0000 Blue: 000 Result use AND, only one such customer Range queries can also be handled and Salary BETWEEN 0,000 AND 300, ,000: ,000: OR together: 000 Use as regular bitmap Aalborg University 07 - DWML course 29 Aalborg University 07 - DWML course 30 Compressed Bitmaps Problem: space use With m possible values and n records: n*m bits required However, probability of a is /m => very few s Solution: compressed bitmaps Run-length encoding A run is i 0 s followed by a Concatenating binary numbers won t work no unique decoding Instead, determine j number of bits in binary representation of i Encode as <j- s> <i in binary> Encode next run similarly, trailing 0 s not encoded Example: encoded as 0 j> => first bit of i is this bit can be saved encoded as 0 Decoding: scan bits to find j, scan next j- bits to find i, find next Example: j = 3, i = 7 () => bitmap = (trailing) Managing Bitmaps Compression factor Assume m=n (unique values) Each value has just one run of length i < n Each run takes at most 2 log 2 n bits (j <= log 2 n) Total space consumption: 2n log 2 n bits (compared to n 2 ) Operations on compressed bitmaps Decompress one run at a time and produce relevant s in output Finding and storing bitvectors Index with B-trees + store in blocks/block chains Finding records Use secondary or primary index (hashing or B-tree) Handling modifications Deletion: retire record number + update bitmaps with s Insertion: add new record to file + update bitmaps with s (trail 0 s) Updates: update bitmaps with old and new s (expensive) Aalborg University 07 - DWML course 3 Aalborg University 07 - DWML course 32

Physical Storage Partitioning Data stored in large lumps (partitions) Example: one partition per quarter Queries need only read the relevant partitions Can yield large performance improvements

9 Physical Storage Partitioning Data stored in large lumps (partitions) Example: one partition per quarter Queries need only read the relevant partitions Can yield large performance improvements Operations on partitions are independent Creation, deletion, update, indexing Aggregation level can be different among partitions Column storage Data stored in columns, not in rows A different kind of partitioning Works well for typical DW queries (only few columns accessed) Supports good compression of data Physical Configuration RAID Gives (depending on level) error tolerance and improved read speed DW optimized for reads, not for writes DW well suited for, e.g., RAID5 (% redundancy) Disk type Small drives (with many controllers) are more expensive, but faster Large drives are cheaper, can store more aggregates for the same price Block size Large sequential reads faster with large blocks (32K) Scattered index reads faster with small blocks (4K) Memory RAM is cheap: buy a lot RAM caching must be per user session Monitoring user activity Can give feedback to, e.g., choice of aggregates Aalborg University 07 - DWML course 33 Aalborg University 07 - DWML course 34 DBMS Possibilities Aggregate navigation/use, Aggregate choice, Aggregate maintenance Oracle 0g, DB2 UDB, MS Analysis Services Using ordinary indices Oracle 0g, DB2 UDB, MS SQL Server can do star joins Bitmap indices Oracle 0g, DB2 UDB not yet in MS SQL Server Partitioning Oracle 0g, DB2 UDB, MS SQL Server+Analysis Services Column storage Redbrick (Informix->IBM->?) MOLAP/ROLAP/HOLAP Oracle 0g, DB2 UDB, MS SQL Server Mini Project New subtask DW performance optimization Build aggregates SQL Server Management Studio Connect to Analysis Services Find your cube and Aalborg University 07 - DWML course 35 Aalborg University 07 - DWML course 36

Advanced Data Management Technologies

ADMT 2017/18 Unit 13 J. Gamper 1/42 Advanced Data Management Technologies Unit 13 DW Pre-aggregation and View Maintenance J. Gamper Free University of Bozen-Bolzano Faculty of Computer Science IDSE Acknowledgements: