Summary. The Dimensional Fact Model. Goals and benefits Basic and advanced constructs. Logical design with the DFM Best practices for design

Transcription

1 Boosting the Data Warehouse Life-Cycle Through Conceptual Design Stefano Rizzi DISI University of Bologna Summary Methodological frameworks Prescriptive design Agile design The Dimensional Fact Model Goals and benefits Basic and advanced constructs Logical design with the DFM Best practices for design Knowledge documentation with the DFM Testing with the DFM Functional and usability tests A case study 2

2 Methodological framework Why? Designing a data warehouse is a long and complex project that does not always meet the needs of business users It is often perceived as too rigid and IT-centric and has become more complex with big data It requires an accurate planning aimed at devising satisfactory answers to organizational and architectural questions The risk of getting an unsatisfactory result in data warehousing projects is particularly high because of high user expectations Risks related to project management Risks associated with technology Risks related to data and design Risks related to the organization Methodologies are created by closely studying similar experiences and minimizing the risks for failure by founding new approaches on a constructive analysis of the mistakes made previously 4

3 A prescriptive view Bottom-up approach DWs are incrementally built and several data marts are iteratively created. Each data mart is based on a set of facts that are linked to a specific department and that can be interesting for a user group Leads to concrete results in a short time Does not require huge investments Enables designers to investigate one area at a time Gives managers a quick feedback about the actual benefits of the system being built Keeps the interest for the project constantly high May determine a partial vision of the business domain 5 Data mart design db administrator Source Analysis and Integration Front-end Programming Requirement Analysis Physical Design distinguishing between a phase of conceptual design and one of logical design Testing brings advantages to both designers and end-users end-user Conceptual Design ETL Design Data Volume and Workload designer Logical Design 6

4 Methodological scenarios Supply-driven approach data marts are designed based on a close operational data source analysis user requirements show designers which groups of data, relevant for decision-making processes, should be selected and how to define data group structures based on the multidimensional model Demand-driven approach it begins with the definition of information requirements of data mart users the problem of how to map those requirements into existing data sources is addressed at a later stage, when ETL procedures are implemented 7 Supply-driven approach glossary REQUIREMENT ANALYSIS interviews reconciled schema CONCEPTUAL DESIGN WORKLOAD AND DATA VOLUME data source schemata SOURCE ANALYSIS AND INTEGRATION fact schema logical model LOGICAL DESIGN workload data volume DBMS reconciled schema logical schema ETL DESIGN PHYSICAL DESIGN ETL schema physical schema 8

5 Supply-driven approach Pros an initial conceptual schema for data marts can be automatically derived from the reconciled layer that is, it strictly depends on data source structures ETL design is extremely streamlined because every single information piece stored in a data mart is directly associated with one or more source attributes the resulting data marts are quite stable in time, because they are rooted in source schemata that change less frequently than the requirements expressed by end users 9 Demand-driven approach glossary REQUIREMENT ANALYSIS interviews CONCEPTUAL DESIGN WORKLOAD AND DATA VOLUME data source schemata SOURCE ANALYSIS AND INTEGRATION fact schema logical model LOGICAL DESIGN workload data volume DBMS reconciled schema logical schema ETL DESIGN PHYSICAL DESIGN ETL schema physical schema 10

6 Demand-driven approach Pros users wishes play a leading role Cons designers are required to have strong leadership and meeting facilitation qualities to properly grab and integrate the different points of view designers make great efforts in the data-staging design phase facts, measures, and hierarchies are drawn directly from the specifications provided by users, and only at a later stage can designers check for the information required to be actually available in source databases this may undermine customers confidence in designers and in the advantage gained by data marts on the whole 11 An agile view Agile methodologies can be applied to data warehouse projects only partially Highlight on incremental delivery of valuable components for users......but several DW components are hardly perceived as valuable by users Strong project segmentation based on user stories: high-level functional requirements that can be implemented in a few days......but founding design on detailed functionalities required by users does not allow the multidimensional structure of data to be correctly determined However, some founding principles of agile methodologies can be reused Incrementality and risk-based iteration Prototyping User involvement Light and formal documentation Component reuse Automated schema generation 12

7 4WD (Four-Wheel-Drive) A methodology arising from the application of agile principles to real data warehouse projects: Fact-based iteration cycles Build a conceptual schema before implementation User involvement in testing activities Possibility of changing fact and data mart priorities 13 4WD (Four-Wheel-Drive) 14

8 Conceptual modeling Which formalism? While it is now universally recognized that a data mart is based on a multidimensional view of data, there is still no agreement on how to implement its conceptual design Use of the Entity-Relationship model is quite widespread throughout companies as a conceptual tool for standard documentation and design of relational databases, but it cannot be used to model DWs Designers often base their data marts design on the logical level that is, they directly define star schemata. But a star schema is nothing but a denormalized relational schema; it contains only the definition of a set of relations and integrity constraints! 16

9 The Dimensional Fact Model The DFM is a graphical model devised to: 1. lend effective support to conceptual design of data marts it is implementation-independent it is expressive it is non-ambiguous it is formally sound (based on FD theory) it can be automatically translated into a logical schema 2. be easily understood by both designers and end-users 3. provide clear and expressive design documentation It has been successfully experimented over the last 20 years in both the academic and industrial worlds The conceptual representation generated by the DFM consists of a set of fact schemata that model facts, measures, dimensions, and hierarchies 17 The Dimensional Fact Model Advantages: 1. it gives designers and end-users a platform-independent, nonambiguous, comprehensive picture of the DW content 2. it is 100% independent of the OLAP multidimensional engine chosen for deployment and of the target logical model 3. it enables effective communication between designers and endusers with the goal of more accurately formalizing requirement specifications 4. it decreases the overall complexity of design by breaking it into two distinct but inter-related phases 5. it streamlines the DW life-cycle by enabling logical design to be automated based on widely recognized best practices 18

10 The Dimensional Fact Model Advantages: 6. it provides clear and expressive design documentation, readable by both ICT and business people, which improves the overall system maintainability 7. it effectively integrates user-oriented business glossaries 8. it enables early testing of requirements based on the core workload expressed by users, thus reducing the probability of errors and misunderstanding 9. it encourages self-service BI by helping business users understand the information content of their DW 10. it enables computation of metrics to effectively assess the quality of design 19 DFM: basic concepts A fact is a concept relevant to decision-making processes. It typically models a set of events taking place within a company (e.g., sales, shipments, purchases,...). A fact has dynamic properties and evolves in some way over time A measure is a numerical property of a fact and describes a quantitative fact aspect that is relevant to analysis (e.g., every sale is quantified by its receipts) A dimension is a fact property with a finite domain and describes an analysis coordinate of the fact. Typical dimensions for the sales fact are products, stores, and dates fact dimension A fact expresses a many-to-many relationship between its dimensions measure 20

11 DFM: basic concepts Dimensional attributes are the dimensions and other possible attributes, always with discrete values, that describe them (e.g., a product is described by its type, by the category to which it belongs, by its brand, and by the department in which it is sold) A hierarchy is a directed tree whose nodes are dimensional attributes and whose arcs model many-to-one associations between dimensional attribute pairs. It includes a dimension, positioned at the tree s root, and all of the dimensional attributes that describe it dimensional attribute hierarchy 21 DFM vs. ERM 22

12 DFM vs. star schema DateId Date Week Month Quarter Year ProductID Product Type Category Brand MarketingGroup Department StoreID DateID ProductID NumberOfUnits TotalReceipts UnitPrice StoreID Store City State SalesManager SalesDistrict 23 DFM: advanced concepts A descriptive attribute stores additional information about a dimensional attribute but is not used for aggregation Some arcs in a fact schema can be optional descriptive attribute optional arc 24

13 DFM: advanced concepts A shared hierarchy is a shorthand to denote that a part of a hierarchy is replicated two or more times in a fact schema shared hierarchy 25 DFM: advanced concepts In a convergence, two dimensional attributes are connected by two or more distinct directed paths, and each of them still represents a functional dependency convergence 26

14 DFM: advanced concepts A cross-dimensional attribute is one whose value is defined by the combination of two or more dimensional attributes cross-dimensional attribute 27 DFM: advanced concepts A multiple arc models a many-to-many association between two dimensional attributes multiple arc 28

15 DFM: advanced concepts An incomplete hierarchy is a hierarchy where, for some instances, one ore more aggregation levels are missing (because they are unknown or undefined) incomplete hierarchy 29 DFM: advanced concepts In recursive hierarchies the father-child relationships between aggregation levels are consistent, but instances can have different lengths recursive hierarchy 30

16 DFM: advanced concepts Additivity shows how measures can be aggregated additivity matrix 31 The DFM in action Budget of the United States Government for Fiscal Year

17 The DFM in action University degrees 33 The DFM in action Retail orders & invoices 34

18 The DFM in action Activity analysis for a consulting company 35 Logical design with the DFM

19 Logical design It aims at determining a logical schema for the data mart starting from a conceptual schema Translation of conceptual schemata Optimization (view materialization, fragmentation) The principles it is based on are different from those used in operational databases data redundancy denormalization of relations conceptual schema workload data volume other constraints Logical design logical schema 37 Stars & Snowflakes WeekID Week Month ProductID Product Type Category Supplier StoreID WeekID ProductID Quantity Receipts StoreID Store City Country SalesManager WeekID Week Month ProductID Product Supplier TypeID TypeID Type Category StoreID WeekID ProductID Quantity Receipts StoreID Store CityID SalesManager CityID City Country 38

20 Translating conceptual schemata The basic rule for translating a fact schema into a star schema is: Create a fact table including all measures; for each hierarchy, create a dimension table including all attributes Besides this obvious rule, specific solutions must be taken for different constructs Some examples Shared hierarchies If a hierarchy is shared, there are two choices: I. Introduce redundancy by duplicating common attributes II. Snowflake on the first common attribute SaleFT StoreID ProductID DateID NumberOfUnits UnitPrice TotalReceipts StoreDT StoreID Store CityID... ProductDT ProductId Product Supplier CityID... CityDT CityID City State Country 40

21 Cross-dimensional attributes A cross-dimensional attribute b defines a many-to-many association between two or more dimensional attributes a 1..., a m Translating it to the star schema requires to create a bridge table including b and having a 1..., a m as the key SaleFT StoreID WeekID ProductID Quantity Receipts StoreDT StoreID Store State Country ProductDT ProductID Product Type Category Supplier Brand VatBT StoreState ProductCategory Vat 41 Multiple arcs When a hierarchy includes a many-to-many association, the best solution is to create an additional table (bridge table) to model the multiple arc: The key of the bridge table is made of the two attributes connected by the multiple arc A weight attribute allows to give different relevance to tuples Both weighted and impact queries are supported SaleFT BookID DateID Number Receipts BookDT BookID Book Genre AuthorDT AuthorID Author BT BookID AuthorID Weight 42

22 Degenerate dimensions This term refers to a hierarchy including only one attribute If the attribute is not too long, its values can be directly included in the FT Alternatively, we may use one DT to model several degenerate dimensions (junk dimension) Since there is no functional dependency between attributes, this solution is feasible only if the number of distinct values for the attributes involved is small OrderLineFT OrderID ProductID McsID Quantity Amount OrderDT OrderID Order Customer City McsDT McsID ShipmentMode ReturnCode LineStatus 43 Recursive hierarchies In recursive hierarchies, the number of aggregation levels vary from instance to instance A self-referencing foreign key can be used to model the recursion ActivityFT EmployeeID ProjectID DateID ActivityTypeID HoursWorked EmployeeDT EmployeeID Employee Role ManagerID A more powerful solution is to flatten out a hierarchy and set out all the links induced by it in a navigation table ActivityFT EmployeeID ProjectID DateID ActivityTypeID HoursWorked EmployeeDT EmployeeID Employee Role Navigation Table ParentID ChildID Level Leaf 44

23 Knowledge documentation with the DFM Data warehouse view 46

24 Data warehouse view 47 Data mart view 48

25 Data mart view 49 Data mart view 50

26 Fact view 51 Fact view 52

27 Testing with the DFM A winning combination The earlier an error is detected in the software life-cycle, the cheapest correcting that error is Multidimensional schemata are typically the first design artifact that is created and can be tested, so they have a primary role in ensuring early discovery of errors All DW design methodologies entail a multidimensional modeling phase, that is carried out either at the conceptual level or at the logical level A conceptual representation is by far more expressive and can more easily be understood by non-expert users Adopting a methodology that entails a conceptual design phase has a positive impact on the effectiveness of testing, which in turn brings several advantages in terms of design quality and accuracy and in terms of maintainability and reuse 54

28 What vs. how in testing multidimensional schema functional ETL physical schema front-end usability performance stress recovery security maintainability 55 Functional test of DFM schemata It aims at verifying that the multidimensional schemata produced for the data mart effectively supports user requirements The workload test verifies that the workload preliminarily expressed by users during requirement analysis is actually supported by the multidimensional schema Check, for each workload query, that the required measures have been included in the fact schema and that the required aggregation level can be expressed as a valid grouping set on the fact schema Should the workload be too large to be comprehensively tested, tests should be made on a sample of queries only 56

29 Functional test of DFM schemata It aims at verifying that the multidimensional schemata produced for the data mart effectively supports user requirements The hierarchy test verifies that the functional dependencies represented by hierarchies in the multidimensional schema are actually valid on source data Detect two types of errors: if a hierarchy includes a functional dependency that is contradicted by source data, then either (1) a modeling error has been done and the functional dependency should be removed from the multidimensional schema, or (2) source data are faulty, which should be taken care of by ETL Even when no such errors are detected, this test can lead designers to discover denormalization issues they were not aware of in source data, which has a significant impact on ETL design 57 Functional test of DFM schemata It aims at verifying that the multidimensional schemata produced for the data mart effectively supports user requirements The conformity test is aimed at assessing how well conformed hierarchies have been designed by evaluating the sparseness of the bus matrix If the bus matrix is very sparse, the designer probably failed to recognize the semantic and structural similarities between apparently different hierarchies (creating a conformed dimension to be shared by multiple facts would be preferable maybe) If the bus matrix is very dense, the designer probably failed to recognize the semantic and structural similarities between apparently different facts (it could be worth merging these facts into a single fact including the union of their measures) Dimension TICKET RET. ORDER STORE MOVEM. accounting type admin. block reason box type constraint reason counter currency currency type discount range discount reason entity environment exchange flag completed order flag item block reason 58

30 Functional test of DFM schemata It aims at verifying that the multidimensional schemata produced for the data mart effectively supports user requirements The similarity factor aids designers in evaluating whether the facts in a single data mart have been designed minimally or redundantly, and is computed for each ordered pair of facts Depending on the values of the similarity factors for two facts, the designer can infer that these facts are mainly overlapping, or that one of them is mostly included in the other by aggregation The similarity factor is particularly useful when assessing an existing data warehouse to quickly identify possibly redundant facts When conceptual design is query-driven, several quite similar facts may be erroneously designed, which is easily captured by the similarity factor RETAIL TICKETS ORDERS STORE MOVEMENTS RETAIL TICKETS ORDERS STORE MOVEMENTS Usability test of DFM schemata The size metrics counts the total number of attributes in each hierarchy/fact/data mart The complexity metrics counts the number of advanced DFM constructs (e.g., multiple arcs, recursive hierarchies, cross-dimensional attributes) Higher values suggest lower understandability Hierarchies with size above 25 and complexity above 3 may be difficult to understand for users, while hierarchies with size above 50 and complexity above 3 will most probably create problems with usability and maintainability 60

31 Usability test of DFM schemata The roll-up factor is specifically oriented to hierarchies and aims at evaluating their OLAP navigability in terms of the roll-up paths they express The roll-up factor of a hierarchy depends on its width and depth: a high value denotes a deep hierarchy, while a low value denotes a wide hierarchy The roll-up factor makes the designer aware that some hierarchies carry low roll-up expressiveness (i.e., they are characterized by short roll-up paths), so that he can check with the user if some rollup relationships have been forgotten during requirement analysis and conceptual design Missing existing roll-up relationships is one of the most common mistakes in multidimensional modeling, and leads to a proliferation of degenerate dimensions 61 A case study

32 Italian Shoes A leading brand in the international market that produces shoes, sells them through its own sales point network, and distributes them to third-party stores worldwide Development team: 1 project manager 1 application domain expert (requirement analysis and relations with business users) 2 designers (one for ETL, one for the front-end) 2 programmers (implementation) 1 Oracle expert (physical tuning) 2 testers Duration: 9 months 63 Testing before the DFM At designers and programmers full discretion (on ETL and front-end) Each team members tests her own work Only documentation: user-acceptance tests Precise functional analysis missing no clear specification of how to move from micro-analysis to logical design difficulties in documenting business rules for instance: 1. how is this KPI exactly defined? 2. on which group-by s is it valid? 3. for which business areas is it valid? 64

33 Testing after the DFM multidimensional schema functional ETL physical schema front-end usability performance stress recovery security maintainability 65 Conceptual schema Three data flows: Orders, Retail Tickets, Store Movements

34 Workload test - 1 Goal: check that the conceptual schema meets the preliminary workload all attributes and measures are present......at the right detail Retail Tickets Orders Store Movements % unsolved user queries 0% 0% 70% Retail Tickets and Orders were already partially implemented, so 0% was expected Store Movements was still under development (the Warehouse hierarchy was missing...) 67 Workload test - 2 Goal: share, explain, and discuss the three fact schemata with users 24 degenerate dimensions added 7 dimensions deleted 7 attributes added 32 attributes deleted 10 parent-child relationships in hierarchies changed 68

35 Similarity test Retail Tickets Orders Store Movements Retail Tickets Orders Store Movements a store movement can in principle be viewed as a particular type of order, but most user queries are related to either movements or orders After 2 months: Retail Tickets Orders Store Movements Retail Tickets Orders Store Movements Usability test Presence of 9 cross-dimensional attributes in the conceptual schema after an iteration, thanks to a more careful analysis and conceptual design, all these constructs could be avoided Low roll-up factor (0.21) for the product hierarchy a discussion with users revealed that the company lacks a shared view of the product hierarchy at this stage designers agreed to leave that hierarchy untouched this issue was listed for perfective maintenance Large number of large dimension tables for Orders (#DTs = 21, #attributes = 410) creation of some junk tables to reduce the number of degenerate dimension tables some snowflaking to reduce the number of attributes per dimension table 70

36 Total effort breakdown ETL 20% 24% Front-end 7% 11% 18% 20% Requirement Analysis Multidimensional design Database Testing 71 Testing effort breakdown 5% 10% MD schema 32% ETL 53% Front-end Physical schema 72

37 Conclusions Conceptual modeling brings advantages to... business analysts & data scientists (better understanding/ exploration of data, self-service BI) designers (streamlining of design activities, early validation of requirements, effective testing) BI architects (comprehensive documentation, control over evolution) The DFM... can be coupled with both prescriptive and agile design methodologies is supported by a CASE tool 73 Conclusions The DFM is already used in several projects in public and private companies (e.g., FIAT, Italian Ministry of Justice, University of Bologna, Yoox Group, GEOX, Health Authorities, etc.) Today, in big data contexts where companies store large quantities of unstructured data in data lakes, the DFM has a key role in ensuring that this huge wealth of information can be effectively explored and analyzed 74

38 References M. Golfarelli, S. Rizzi. Data Warehouse Testing: A Prototype- Based Methodology. Information and Software Technology, 53(11): , 2011 M. Golfarelli, S. Rizzi, E. Turricchia. Modern Software Engineering Methodologies Meet Data Warehouse Design: 4WD. Proceedings 13th International Conference on Data Warehousing and Knowledge Discovery, Toulouse, France, pp , 2011 M. Golfarelli, S. Rizzi. Data Warehouse Design: Modern Principles and Methodologies. McGraw-Hill, 2009 S. Rizzi. Conceptual modeling solutions for the data warehouse. In Data Warehousing and Mining: Concepts, Methodologies, Tools, and Applications, J. Wang (Ed.), Information Science Reference, pp , Thank you for your attention Questions?