OLAP Systems and Multidimensional Expressions

Transcription

1 OLAP Systems and Multidimensional Expressions Krzysztof Dembczyński Institute of Computing Science Laboratory of Intelligent Decision Support Systems Politechnika Poznańska (Poznań University of Technology) Intelligent Decision Support Systems Master studies, first semester Academic year 2011/12 (summer course)

2 OLAP Servers SQL3 MDX Summary

3 OLAP Servers SQL3 MDX Summary

4 OLAP servers provide an effective solution for accessing and processing large volumes of high dimensional data.

5 Multidimensional Reports OLAP systems provide tools for multidimensional reporting.

6 Multidimensional Cube The proper data model for multidimensional reporting is the multidimensional one. More dimensions are possible.

7 Operators in Multidimensional Data Model: Roll up summarize data along a dimension hierarchy. Drill down go from higher level summary to lower level summary or detailed data. Slice and dice corresponds to selection and projection. Pivot reorient cube. Raking, Time functions, etc..

8 Lattice of Cuboids Different degrees of summarizations are presented as a lattice of cuboids. Example for the dimensions: time, product, location, supplier. Using this structure, one can easily show roll up and drill down operations.

9 For an n-dimensional data cube, the total number of cuboids that can be generated is: T = i=1,...,n (L i + 1), where L i is the number of levels associated with dimension i (excluding the virtual top level all since generalizing to all is equivalent to the removal of a dimension). For example, if the cube has 10 dimensions and each dimension has 4 levels, the total number of cuboids that can be generated will be: l = 5 10 = 9,

10 OLAP Servers: Relational OLAP (ROLAP), Multidimensional OLAP (MOLAP), Hybrid OLAP (HOLAP).

11 ROLAP servers use a relational or post-relational database management system to store and manage warehouse data. Optimization techniques: Denormalization, Materialized views, Partitioning, Joins, Indexes, Query processing.

12 Advantages of ROLAP Servers: Scalable with respect to the number of dimensions, Scalable with respect to the size of data, Sparsity is not a problem (fact tables contain only facts), Mature and well-developed technology. Disadvantage of ROLAP Servers: Worse performance than MOLAP, Additional data structures and optimization techniques used to improve the performance.

13 MOLAP Servers uses array-based multidimensional storage engines. Optimization techniques: Two-level storage representation: dense cubes are identified and stored as array structures, sparse cubes employ compression techniques, Materialized cubes.

14 Advantages of MOLAP Servers: Multidimensional views are directly mapped to data cube array structures efficient access to data, Can easily store subaggregates. Disadvantages of MOLAP Servers: Scalability problem in the case of larger number of dimensions, Not tailored for sparse data, Young technology, There are no existing standards.

15 Example Logical model consists of four dimensions: customer, product, location, and day; in the case of customers, products, locations and days, the data cube will contain cells! Huge number of cells is empty: a customer is not able to buy all the products in all the locations....

16 HOLAP Servers The hybrid approach combines ROLAP and MOLAP technology, benefiting from the greater scalability of ROLAP and the faster computation of MOLAP.

17 There are two main approaches for querying data warehouses: SQL3 (ROLAP) MDX (MOLAP)

18 OLAP Servers SQL3 MDX Summary

19 Star Schema

20 SQL Queries SQL group by SELECT Name, AVG(Grade) FROM Students_grades G, Student S WHERE G.Student = S.ID GROUP BY Name; Name AVG(Grade) Inmon 4.8 Kimball 4.7 Gates 4.0 Todman 4.5

21 SQL group by SELECT Academic_year, Name, AVG(Grade) FROM Students_grades G, Academic_year A, Professor P WHERE G.Professor = P.ID and G.Academic_year = A.ID GROUP BY Academic_year, Name; Academic_year Name AVG(Grade) 2001/2 Stefanowski /3 Stefanowski /4 Stefanowski /2 Słowiński /3 Słowiński /4 Słowiński /4 Dembczyński 4.8

22 SQL3 extends SQL by: GROUP BY ROLLUP, GROUP BY CUBE.

23 GROUP BY CUBE SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY CUBE (Time, Product, Location, Supplier);

24 GROUP BY CUBE SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY Time, Product, Location, Supplier UNION ALL SELECT Time, Product, Location, *, SUM(Gain) FROM Sales GROUP BY Time, Product, Location UNION ALL SELECT Time, Product, *, Location, SUM(Gain) FROM Sales GROUP BY Time, Product, Location UNION ALL... UNION ALL SELECT *, *, *, *, SUM(Gain) FROM Sales;

25 GROUP BY CUBE SELECT Academic_year, Name, AVG(Grade) FROM Students_grades GROUP BY CUBE(Academic_year, Name); Academic_year Name AVG(Grade) 2001/2 Stefanowski /2 Słowiński /3 Stefanowski /3 Słowiński /4 Stefanowski /4 Słowiński /4 Dembczyński /2 NULL /3 NULL /4 NULL 3.8 NULL Stefanowski 3.9 NULL Słowiński 3.6 NULL Dembczyński 4.8 NULL NULL 3.95

26 GROUP BY ROLLUP SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY ROLLUP (Time, Product, Location, Supplier);

27 GROUP BY ROLLUP SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY Time, Product, Location, Supplier UNION ALL SELECT Time, Product, Location, *, SUM(Gain) FROM Sales GROUP BY Time, Product, Location UNION ALL SELECT Time, Product, *, *, SUM(Gain) FROM Sales GROUP BY Time, Product UNION ALL SELECT Time, *, *, *, SUM(Gain) FROM Sales GROUP BY Time UNION ALL SELECT *, *, *, *, SUM(Gain) FROM Sales;

28 GROUP BY ROLLUP SELECT Academic_year, Name, AVG(Grade) FROM Students_grades G GROUP BY ROLLUP(Academic_year, Name); Academic_year Name AVG(Grade) 2001/2 Stefanowski /2 Słowiński /3 Stefanowski /3 Słowiński /4 Stefanowski /4 Słowiński /4 Dembczyński /2 NULL /3 NULL /4 NULL 3.8 NULL NULL 3.95

29 OLAP Servers SQL3 MDX Summary

30 Multidimensional Expressions For OLAP queries, MDX is an alternative to SQL3.

31 Multidimensional Expressions For OLAP queries, MDX is an alternative to SQL3. MDX SELECT [Time].[1997],[Time].[1998] ON COLUMNS, [Measures].[Sales],[Measures].[Cost] ON ROWS FROM Warehouse WHERE ([Store].[All Stores].[USA])

32 Multidimensional Expressions (MDX) An MDX query must contain the following information: The number of axes on which the result is presented. The set of members or tuples to include on each axis of the MDX query. The name of the cube that sets the context of the MDX query. The set of members or tuples to include on the slicer axis.

33 Multidimensional Expressions (MDX) The syntax of a basic MDX query that includes the use of the SELECT, FROM, and WHERE clauses: [WITH <SELECT WITH clause> [, <SELECT WITH clause>...]] SELECT [* (<SELECT query axis clause> [, <SELECT query axis clause>...])] FROM <SELECT subcube clause> [<SELECT slicer axis clause>] [<SELECT cell property list clause>]

34 Multidimensional Expressions (MDX) Main concepts of MDX: Dimension, Hierarchy, Level, Member, Measure, Tuple, Set, Axis.

35 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[All CARS].[Chevy], [CARS].[All CARS].[Ford]} ON ROWS, {[DATE].[All DATE].[March], [DATE].[All DATE].[April]} ON COLUMNS FROM MDDBCARS;

36 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[All CARS].[Chevy], [CARS].[All CARS].[Ford]} ON ROWS, {[DATE].[All DATE].[March], [DATE].[All DATE].[April]} ON COLUMNS FROM MDDBCARS; March April Chevy $ $ Ford $ $

37 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]} ON ROWS, {[DATE].[ALL DATE].[MARCH], [DATE].[ALL DATE].[APRIL]} ON COLUMNS FROM MDDBCARS WHERE ([MEASURES].[SALES_N])

38 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]} ON ROWS, {[DATE].[ALL DATE].[MARCH], [DATE].[ALL DATE].[APRIL]} ON COLUMNS FROM MDDBCARS WHERE ([MEASURES].[SALES_N]) Sales_N March April Chevy Ford

39 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]} ON COLUMNS, {[DATE].[ALL DATE].[JANUARY]:[DATE].[ALL DATE].[APRIL]} ON ROWS FROM MDDBCARS

40 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]} ON COLUMNS, {[DATE].[ALL DATE].[JANUARY]:[DATE].[ALL DATE].[APRIL]} ON ROWS FROM MDDBCARS Chevy Ford January $ $ February $ $ March $ $ April $ $

41 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]} ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS

42 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]} ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS Chevy Ford 1998 $ $ $ $ $ $ $ $

43 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[FORD].CHILDREN} ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS

44 Multidimensional Expressions (MDX): MDX SELECT {[CARS].[ALL CARS].[FORD].CHILDREN} ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS Ford Mustang Ford Taurus $ $ $ $ $ $ $ $

45 Multidimensional Expressions (MDX): MDX SELECT {([CARS].[ALL CARS].[CHEVY], [MEASURES].[SALES_SUM]), ([CARS].[ALL CARS].[CHEVY], [MEASURES].[SALES_N]), ([CARS].[ALL CARS].[FORD], [MEASURES].[SALES_SUM]), ([CARS].[ALL CARS].[FORD], [MEASURES].[SALES_N]) } ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS

46 Multidimensional Expressions (MDX): MDX SELECT {([CARS].[ALL CARS].[CHEVY], [MEASURES].[SALES_SUM]), ([CARS].[ALL CARS].[CHEVY], [MEASURES].[SALES_N]), ([CARS].[ALL CARS].[FORD], [MEASURES].[SALES_SUM]), ([CARS].[ALL CARS].[FORD], [MEASURES].[SALES_N]) } ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS Chevy Ford Sales_Sum Sales_N Sales_Sum Sales_N 1998 $ $ $ $ $ $ $ $

47 Multidimensional Expressions (MDX): MDX SELECT {CROSSJOIN({[CARS].[ALL CARS].[CHEVY], [CARS].[ALL CARS].[FORD]}, {[MEASURES].[SALES_SUM], [MEASURES].[SALES_N]}) } ON COLUMNS, {[DATE].MEMBERS} ON ROWS FROM MDDBCARS Chevy Ford Sales_Sum Sales_N Sales_Sum Sales_N 1998 $ $ $ $ $ $ $ $

48 Multidimensional Expressions (MDX): MDX SELECT NON EMPTY [Store Type].[Store Type].MEMBERS ON COLUMNS, FILTER([Store].[Store City].MEMBERS, (Measures.[Profit], [Time].[1997]) > ) ON ROWS FROM [Sales] WHERE (Measures.[Profit], [Time].[Year].[1997])

49 Multidimensional Expressions (MDX): MDX SELECT NON EMPTY [Store Type].[Store Type].MEMBERS ON COLUMNS, FILTER([Store].[Store City].MEMBERS, (Measures.[Profit], [Time].[1997]) > ) ON ROWS FROM [Sales] WHERE (Measures.[Profit], [Time].[Year].[1997]) Profit Normal 24 Hours Toronto $ $ Vancouver $ $ New York $ $ Chicago $ $

50 Multidimensional Expressions (MDX): MDX SELECT Measures.MEMBERS ON COLUMNS, ORDER({[Store].[Store City].MEMBERS}, Measures.[Sales Count], DESC) ON ROWS FROM [Sales]

51 Multidimensional Expressions (MDX): MDX SELECT Measures.MEMBERS ON COLUMNS, ORDER({[Store].[Store City].MEMBERS}, Measures.[Sales Count], DESC) ON ROWS FROM [Sales] Profit Sales Count... New York $ Chicago $ Toronto $ Vancouver $

52 Multidimensional Expressions (MDX): MDX WITH MEMBER [Time].[Year Difference] AS [Time].[2nd half] - [Time].[1st half] SELECT { [Account].[Income], [Account].[Expenses] } ON COLUMNS, { [Time].[1st half], [Time].[2nd half], [Time].[Year Difference] } ON ROWS FROM [Financials]

53 Multidimensional Expressions (MDX): MDX WITH MEMBER [Time].[Year Difference] AS [Time].[2nd half] - [Time].[1st half] SELECT { [Account].[Income], [Account].[Expenses] } ON COLUMNS, { [Time].[1st half], [Time].[2nd half], [Time].[Year Difference] } ON ROWS FROM [Financials] Income Expenses 1st Half st Half Year Difference

54 Multidimensional Expressions (MDX): MDX WITH MEMBER [Account].[Net Income] AS ([Account].[Income] - [Account].[Expenses]) / [Account].[Income] SELECT { [Account].[Income], [Account].[Expenses], [Account].[Net Income] } ON COLUMNS, { [Time].[1st half], [Time].[2nd half] } ON ROWS FROM [Financials]

55 Multidimensional Expressions (MDX): MDX WITH MEMBER [Account].[Net Income] AS ([Account].[Income] - [Account].[Expenses]) / [Account].[Income] SELECT { [Account].[Income], [Account].[Expenses], [Account].[Net Income] } ON COLUMNS, { [Time].[1st half], [Time].[2nd half] } ON ROWS FROM [Financials] Income Expenses Net Income 1st Half st Half

56 Multidimensional Expressions (MDX): MDX WITH MEMBER [Time].[Year Difference] AS [Time].[2nd half] - [Time].[1st half], SOLVE_ORDER = 1 MEMBER [Account].[Net Income] AS ([Account].[Income] - [Account].[Expenses]) / [Account].[Income], SOLVE_ORDER = 2 SELECT { [Account].[Income], [Account].[Expenses], [Account].[Net Income] } ON COLUMNS, { [Time].[1st half], [Time].[2nd half], [Time].[Year Difference] } ON ROWS FROM [Financials]

57 Multidimensional Expressions (MDX): Income Expenses Net Income 1st Half st Half Year Difference

58 Multidimensional Expressions (MDX): MDX WITH MEMBER [Time].[Year Difference] AS [Time].[2nd half] - [Time].[1st half], SOLVE_ORDER = 2 MEMBER [Account].[Net Income] AS ([Account].[Income] - [Account].[Expenses]) / [Account].[Income], SOLVE_ORDER = 1 SELECT { [Account].[Income], [Account].[Expenses], [Account].[Net Income] } ON COLUMNS, { [Time].[1st half], [Time].[2nd half], [Time].[Year Difference] } ON ROWS FROM [Financials]

59 Multidimensional Expressions (MDX): Income Expenses Net Income 1st Half st Half Year Difference

60 Multidimensional Expressions (MDX): MDX WITH SET [Quarter1] AS GENERATE([Time].[Year].MEMBERS, {[Time].CURRENTMEMBER.FIRSTCHILD}) SELECT [Quarter1] ON COLUMNS, [Store].[Store Name].MEMBERS ON ROWS FROM [Sales] WHERE (Measures.[Profit])

61 Multidimensional Expressions (MDX): MDX WITH SET [Quarter1] AS GENERATE([Time].[Year].MEMBERS, {[Time].CURRENTMEMBER.FIRSTCHILD}) SELECT [Quarter1] ON COLUMNS, [Store].[Store Name].MEMBERS ON ROWS FROM [Sales] WHERE (Measures.[Profit]) 1998Q1 1999Q1... Saturn $ $ Media Markt $ $ Avans $ $

62 OLAP Servers SQL3 MDX Summary

63 Summary Three types of OLAP servers: ROLAP, MOLAP, and HOLAP. Several approaches for querying data warehouses. ROLAP servers: SQL and SQL3. MOLAP servers: MDX.

64 Bibliography J. Han, M. Kamber, Data Mining: Concepts and Techniques, Morgan-Kaufmann 2000 SAS OLAP Server MDX Guide Third Edition, 2006 MSDN: SQL Server Developer Center, 2008