Data Warehousing & OLAP

Transcription

1 CMPUT 391 Database Management Systems Data Warehousing & OLAP Textbook: (first edition: ) Based on slides by Lewis, Bernstein and Kifer and other sources University of Alberta 1

2 Why Data Warehouses Businesses have a lot of data, operational data and facts, stored in heterogeneous and distributed databases. in different databases in different physical locations in different formats Decision makers need fast access to this information in a summarized form, with a focus often on historical data University of Alberta 2

3 What Is Data Warehouse? consolidates the information from different data sources, enabling OLAP (online analytical processing), to help decision support. is maintained separately from an operational database (which is used for OLTP online transaction processing). Option 1: Consolidate Data Marts Corporate Data Warehouse Data Mart Data Mart Data Mart Data Mart Option 2: Build from scratch Corporate data University of Alberta 3

4 OLTP Compared With OLAP On Line Transaction Processing -- OLTP Maintain a database that is an accurate model of some real-world enterprise Short simple transactions Relatively frequent updates Transactions access only a small fraction of the database On Line Analytic Processing -- OLAP Use information in database to guide strategic decisions Complex aggregation queries Infrequent updates Transactions access a large fraction of the database University of Alberta 4

5 Why Do We Separate DWs From Operational DBs? Performance reasons: OLAP necessitates special data organization that supports multidimensional views. OLAP queries would degrade operational DB. OLAP is read only. No concurrency control and recovery. Decision support requires historical data. Decision support requires consolidated data. University of Alberta 5

6 Fact Tables Many OLAP applications are based on a fact table For example, a supermarket application might be based on a table Sales (Market_Id, Product_Id, Time_Id, Sales_Amt) The table can be viewed as a multidimensional data cube The first three columns are the dimensions representing specific supermarkets products time intervals The fourth column, the Sales_Amt, is a function of the other three, called a measure University of Alberta 6

7 Dimension Tables The dimensions of the fact table can be further described with dimension tables Fact table Sales (Market_id, Product_Id, Time_Id, Sales_Amt) Dimension Tables Market (Market_Id, City, Province, Region) Product (Product_Id, Name, Category, Price) Time (Time_Id, Week, Month, Quarter) University of Alberta 7

8 Star Schema The fact and dimension relations can be displayed in an E-R diagram, which suggests a star and is called a star schema University of Alberta 8

9 Table View of a Star Schema Time TimeId Day Month Year Store StoreID City Province Country Region Sales Fact Table Time Product Store Customer unit_sales dollar_sales Product ProductNo ProdName ProdDesc Category Cust CustId CustName CustCity CustCountry Two different measures (Source: JH) University of Alberta 9

10 Aggregation Many OLAP queries involve aggregation of the data in the fact table For example, to find the total sales (over time) of each product in each market, we might use SELECT S.Market_Id, S.Product_Id, SUM (S.Sales_Amt) FROM Sales S GROUP BY S.Market_Id, S.Product_Id The aggregation is over the entire time dimension and thus produces a two-dimensional view of the data University of Alberta 10

11 Aggregation over Time The output of the previous query Product_Id Market_Id M1 M2 M3 M4 SUM(Sales_Amt) P P P P P5 University of Alberta 11

12 Concept-Hierarchies Many dimensions form an aggregation hierarchy (total or partial orders) Examples: Markets(Market_Id City Province Country Region) Time(year quarter week month day) University of Alberta 12

13 Drilling Down and Rolling Up Executing a series of queries that moves down a hierarchy (e.g., from aggregation over regions to that over provinces) is called drilling down Requires the use of the fact table or information more specific than the requested aggregation (e.g., cities) Executing a series of queries that moves up the hierarchy (e.g., from provinces to regions) is called rolling up Note: In a rollup, coarser aggregations can be computed using prior queries for finer aggregations University of Alberta 13

14 Drilling Down Drilling down on market: from Region to Province Sales (Market_Id, Product_Id, Time_Id, Sales_Amt) Market (Market_Id, City, Province, Region) 1. SELECT S.Product_Id, M.Region, SUM (S.Sales_Amt) FROM Sales S, Market M WHERE M.Market_Id = S.Market_Id GROUP BY S.Product_Id, M.Region 2. SELECT S.Product_Id, M.Province, SUM (S.Sales_Amt) FROM Sales S, Market M WHERE M.Market_Id = S.Market_Id GROUP BY S.Product_Id, M.Province, University of Alberta 14

15 Rolling Up Rolling up on market, from Province to Region If we have already created a table, Province_Sales, using 1. SELECT S.Product_Id, M.Province, SUM (S.Sales_Amt) INTO FROM WHERE Province_Sales Sales S, Market M M.Market_Id = S.Market_Id GROUP BY S.Product_Id, M.Province then we can roll up from there to: 2. SELECT T.Product_Id, M.Region, SUM (T.Sales_Amt) FROM Province_Sales T, Market M WHERE M.Province = T.Province GROUP BY T.Product_Id, M.Region University of Alberta 15

16 Pivoting When we view the data as a multi-dimensional cube and group on a subset of the axes, we are said to be performing a pivot on those axes Pivoting on dimensions D 1,,D k in a data cube D 1,,D k,d k+1,,d n means that we use GROUP BY A 1,,A k and aggregate over A k+1, A n, where A i is an attribute of the dimension D i Example: Pivoting on Product and Time corresponds to grouping on Product_id and Quarter and aggregating Sales_Amt over Market_id: SELECT S.Product_Id, T.Quarter, SUM (S.Sales_Amt) FROM Sales S, Time T WHERE T.Time_Id = S.Time_Id GROUP BY S.Product_Id, T.Quarter Pivot University of Alberta 16

17 Dicing When we use GROUP BY to specify part of a hierarchy, we are performing a range selection called a dice Dicing Sales in the time dimension: total sales for each product in each quarter. SELECT FROM WHERE GROUP BY S.Product_Id, T.Quarter, SUM (Sales_Amt) Sales S, Time T T.Time_Id = S.Time_Id T.Quarter, S.Product_Id Dice University of Alberta 17

18 Slicing When we use WHERE to specify a particular value for an axis (or several axes), we are performing a slice Slicing the data cube in the Time dimension (choosing sales only in week 12) then pivoting to Product_id (aggregating over Market_id) SELECT FROM WHERE S.Product_Id, SUM (Sales_Amt) Sales S, Time T T.Time_Id = S.Time_Id AND T.Week = Wk-12 GROUP BY S. Product_Id Slice University of Alberta 18

19 Slicing-and-Dicing Typically slicing and dicing involves several queries to find the right slice. For instance, change the slice and the axes: Slicing on Time and Market dimensions then pivoting to Product_id and Week (in the time dimension) SELECT S.Product_Id, T.Week, SUM (Sales_Amt) FROM Sales S, Time T WHERE T.Time_Id = S.Time_Id AND T.Quarter = 4 AND S.Market_id = M1 GROUP BY S.Product_Id, T.Week Pivot Slice University of Alberta 19

20 The extended Multi-dimensional Data Cube/Fact Table City Edmonton Calgary Lethbridge Sum Year Sum All Years Drama, Edmonton Drama Comedy... Category Sum Contains all possible aggregates in addition to the facts in the fact table University of Alberta 20

21 Product_Id The CUBE Operator To construct the following table, would take 4 queries (next slide) Market_Id M1 M2 M3 Total SUM(Sales_Amt) P P P P Total University of Alberta 21

22 The Four Queries For the table entries, without the totals (aggregation on time) SELECT S.Market_Id, S.Product_Id, SUM (S.Sales_Amt) FROM Sales S GROUP BY S.Market_Id, S.Product_Id For the row totals (aggregation on time and supermarkets) SELECT S.Product_Id, SUM (S.Sales_Amt) FROM Sales S GROUP BY S.Product_Id For the column totals (aggregation on time and products) SELECT S.Market_Id, SUM (S.Sales) FROM Sales S GROUP BY S.Market_Id For global total: SELECT SUM (S.Sales) FROM Sales S University of Alberta 22

23 Definition of the CUBE Operator Doing these four queries is wasteful The first does much of the work of the other three: if we could save that result and aggregate over Market_Id and Product_Id, we could compute the other queries more efficiently The CUBE clause is part of SQL:1999 GROUP BY CUBE(v1, v2,, vn) Equivalent to a collection of GROUP BYs, one for each of the 2 n subsets of v1, v2,, vn University of Alberta 23

24 Example of CUBE Operator The following query returns all the information needed to obtain the previous products/markets table: SELECT S.Market_Id, S.Product_Id, SUM (S.Sales_Amt) FROM Sales S GROUP BY CUBE (S.Market_Id, S.Product_Id) University of Alberta 24

25 ROLLUP ROLLUPis similar to CUBE except that instead of aggregating over all subsets of the arguments, it creates subsets moving from right to left GROUP BY ROLLUP (A 1,A 2,,A n ) is a series of these aggregations: GROUP BY A 1,, A n-1,a n GROUP BY A 1,, A n-1 GROUP BY A 1, A 2 GROUP BY A 1 No GROUP BY ROLLUPis also in SQL:1999 University of Alberta 25

26 Example of ROLLUP Operator SELECT S.Market_Id, S.Product_Id, SUM (S.Sales_Amt) FROM Sales S GROUP BY ROLLUP (S.Market_Id, S. Product_Id) first aggregates with the finest granularity: GROUP BY S.Market_Id, S.Product_Id then with the next level of granularity: GROUP BY S.Market_Id then the grand total is computed with no GROUP BY clause University of Alberta 26

27 ROLLUP vs. CUBE The same query with CUBE: - first aggregates with the finest granularity: GROUP BY S.Market_Id, S.Product_Id - then with the next level of granularity (both subsets): GROUP BY GROUP BY S.Market_Id S.Product_Id - then the grand total with no GROUP BY University of Alberta 27

28 Materialized Views The CUBE operator is often used to precompute aggregations on all dimensions of a fact table and then save them as a materialized views to speed up future queries University of Alberta 28

29 ROLAP and MOLAP Relational OLAP: ROLAP OLAP data is stored in a relational database as previously described. Data cube is a way to think about a fact table. Multidimensional OLAP: MOLAP Vendor provides an OLAP server that implements a fact table as a data cube using some multi-dimensional (non-relational) implementation. provide proprietary, perhaps visual, languages that allow unsophisticated users to make queries that involve pivots, drilling down, or rolling up University of Alberta 29

30 Implementation Issues OLAP applications are characterized by a very large amount of data that is relatively static, with infrequent updates Thus, various aggregations can be precomputed and stored in the database Star joins, join indices, and bitmap indices can be used to improve efficiency Since updates are infrequent, the inefficiencies associated with updates are minimized University of Alberta 30