Improving the Performance of OLAP Queries Using Families of Statistics Trees

Transcription

1 Improving the Performance of OLAP Queries Using Families of Statistics Trees Joachim Hammer Dept. of Computer and Information Science University of Florida Lixin Fu Dept. of Mathematical Sciences University of North Carolina DaWak 2001 München,, Germany September 7, 2001

2 Outline of Talk Background and Related Work Cubing with Families of Statistics Trees (CubiST ++) Statistics Tree Data Structure Cube Query Language (CQL) Query Evaluation Experimental Results Conclusion DaWak

3 Overview Subset of OLAP queries that arise naturally in DSS Highly aggregated along multiple dimensions Return unary results rather than tuples Cube queries Efficient encoding for data cube plus set of algorithms for evaluating cube queries Multi-way tree data structure (Statistics( Tree) Cube query language (CQL) Set of query processing algorithms (CubiST( ++ Superior to current OLAP technologies DaWak )

4 Data size Challenges Lots of record Large number of dimensions Large domain size Variety of aggregate functions Distributive (SUM, COUNT, MIN ) Algebraic (AVG, ) Holistic (MEDIAN, RANK, ) Queries can involve arbitrary combination of dimensions, any subset of domain values DaWak

5 Data cube systems State-of of-the-art ROLAP: Store the data in relational tables using star or snowflake schema design» COTS: Redbrick Dynamic Server, MOLAP: Use multi-dimensional arrays Indexing» COTS: Oracle Express, (Encoded) bitmap, projection indexes, join indexes, View Materialization View selection and maintenance View computation: pipelining and overlapping of computations DaWak

6 Multidimensional Data Model Location city, region, state Car Sales Data model, type, class Manufacturer Time day, week, year Operations: Slicing/dicing, roll-up and drill-down down E.g., data cube operator DaWak

7 Operations on Data Cube Queries select desired regions of the cube Slicing and dicing Choice of partition for each dimension dices cube (i.e., group by) Focusing on particular partitions along one or more dimension (i.e., where) Drill-down/roll down/roll-up: selecting finer/coarser partitions E.g., data cube operator (Gray et al.) Computes aggregates along one or more dimensions DaWak

8 Statistics Tree (ST) New, efficient encoding for data cube Stores aggregates for one or more functions Level 1 Root * ALL Pointer Level 2 Level 3 Level 4 * Interior Nodes A * A A A A A * * Leaf Nodes DaWak

9 Highlights Multi-way tree storing aggregate information based on one or more aggregate functions Each level in tree corresponds to dimension in data set Interior nodes Pointer for each dimension value + additional (star) pointer for ALL value Leaf nodes Form linked list Store aggregate information for each dimension Memory required to store a k-dim ST is bounded by: c (d i +1), c = 1~2 Compares favorably to B-Tree B or multi-dimensional array DaWak

10 Important Metadata Information For efficiency, map domain values into integers before creating ST E.g., 1 byte to represent 256 domain values Car sales example Jan/99 1, Feb/99 2, Honda 1, Mercedes 2, Toyota 3, For each dimension, represent the inclusion relationships for hierarchies (if present) {Jan/99, Feb/99, Mar/99} Q1/99, {Q1/99, Q2/99, Q3/99, Q4/99} 1999, DaWak

11 Initializing a Statistics Tree Using count() function D1 1 2 * Insert(1,2,4) D * * * D * * * * DaWak

12 Statistics Tree after Insert(1,2,4) DaWak

13 Families of STs Reduce the size of the ST if dimension values are at higher-level granularity E.g., represent time at year level Improves query processing CubiST ++ maintains a set (family( family) ) of ST s 1 base tree representing dimensions at finest granularity» Calculated from detailed data set 1 or more derived trees,, each of which represents a different combination of hierarchy levels for underlying dimensions» Can be materialized from base tree DaWak

14 Generating the Family Step 1: Select from set of all candidate trees, best set of STs to derive Greedy: At each step, include unselected candidate tree that adds least amount of space/overhead to current family Step 2: Materialize selected STs using only existing trees Starting with set of currently materialized trees, roll-up dimension(s) to necessary level of abstraction to produce new derived tree DaWak

15 Generating a Family of STs Base tree T 0, d 1 =10,000, d 2 =25, d 3 =100 Two-level hierarchy for each dimension d 1 : factor 100, d 2 : factor 5, d 3 : factor 10 7 candidate trees T 1 T 7 Hierarchy vectors: : (0,0,0), (0,0,1),, (1,1,1) e.g., select T 1 (1,0,0) and T 6 (1,0,1) Generate T 1 from T 0 by rolling-up D 1, T 6 from T 1 by rolling up D 3 Recursive procedure Rollup() DaWak

16 Cube Query Language CQL Cube query q is an aggregate operation on cells of a k-dimensional data cube q=agg measure ((D-index,H index,h-level):s; ) agg is the aggregate function s either a singleton, range or partial selection No constraint for dimension, assume ALL values selected Drop H-level H if no hierarchy Example q=sum sales ((Time,Month):[Mar,Oct];(Location,City):Orlando) DaWak

17 Query Processing Step 1: Given an input query, chose smallest ST which can answer query Hierarchies in tree must be at same level of granularity or (finer) than the ones specified in query Step 2: Rewrite query (if necessary) Use inclusion relationships to re-write constraints on hierarchies to match representation in ST DaWak

18 Step 3: Use ST to Answer Query 1 2 * q = COUNT sales (2, [1,3], {1,3}) = * * * * DaWak

19 Data Generator CubiST ++ Prototype System Files on Disk Base Tree Initialization Input Query (CQL) CQL Compiler Family Generation Query Matching & Evaluation ST Repository Query Result DaWak

20 Testbed and Experiments SUN ULTRA 10, 90MB RAM, JDK 1.2 SUN E450 with 1GB of RAM for experiments with commercial ROLAP installation Synthetic data sets varying in #records, #dimensions, domain size Experiment: Setup and response times for CubiST ++ on different data sets Cube queries of varying degree of complexity Comparison with: Single base tree Bitmap-based query answering algorithm (bitmap) Commercial ROLAP system DaWak

21 vs. Single ST 1,000,000 records, 3 dimensions, 2-level hierarchy 2 nd level hierarchy contains equal number 1 st level values q1 = COUNT Sales (0:[0,29]) q2 = COUNT Sales (0:[0, 9]) q3 = COUNT Sales (0:[0,7]) base ST derived ST derived ST Factors 1, 1, 1 3, 2, 2 4, 4, 3 Dimension Sizes 60, 60, 60 20, 30, 30 15, 15, 20 Number of Leaves 226,981 20,181 5,376 I/O Time (ms) 16,818 1, Total Time (ms) 25,387 1,

22 vs. Bitmap-based Algorithm Setup Time Query Response Time 60, ,000 Setup time (ms) 40,000 Response time (ms) 10,000 1, , Bitmap CubiST ++ 10K 100K 200K Number of Records 10K 100K 200K 1,045 10,039 21,660 16,961 32,776 50,786 Scanning Bitmap 1 10K 100K 200K Number of Records 10K 100K 200K 979 9,196 18, CubiST q = COUNT Sales ( (0:[2,8]); (1:[2,8]); (2:[5,10]))

23 vs. Commercial ROLAP System 1,000,000 records, 5 dimensions (10, 10, 10, 10,15) Five randomly generated CQL queries (translated into SQL) Time (sec) Time (milsec) 4,000 3,000 2,000 1,000 0 q1 q2 q3 q4 q5 0 Writing Loading View ST Setup Times W/o View 2,52 2,64 2,30 1,48 3,58 With View ST Query Response Times

24 ++ Summary of CubiST ++ Viable alternative to current ROLAP and MOLAP Fast, space-efficient, efficient, scalable, incremental Single scan of detailed data set Memory required to store a k-dim ST is bounded by: c (d i +1), c = 1~2 Time Complexity» Updating ST for each record: O(2 k )» Answering a singleton query: O(k)» Answering an arbitrary query: O(size of ST) ) (Worst Case) Step towards supporting ad-hoc cube queries Many applications: data marts, big picture analysis DaWak

25 Future Additional experiments incl. MOLAP TPC-H H benchmark data Integration with data warehouse system Parallel version of CubiST ++ How to distribute ST across nodes? More long-term: OLAM DaWak