TDWI strives to provide course books that are contentrich and that serve as useful reference documents after a class has ended.

Transcription

1 Previews of TDWI course books offer an opportunity to see the quality of our material and help you to select the courses that best fit your needs. The previews cannot be printed. TDWI strives to provide course books that are contentrich and that serve as useful reference documents after a class has ended. This preview shows selected pages that are representative of the entire course book; pages are not consecutive. The page numbers shown at the bottom of each page indicate their actual position in the course book. All table-of-contents pages are included to illustrate all of the topics covered by the course. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

2

3 TDWI Dimensional Data Modeling Primer From Requirements to Business Analysis

4 The Data Warehousing Institute takes pride in the educational soundness and technical accuracy of all of our courses. Please give us your comments we d like to hear from you. Address your feedback to: info@tdwi.org Publication Date: February 2011 Copyright by The Data Warehousing Institute. All rights reserved. No part of this document may be reproduced in any form, or by any means, without written permission from The Data Warehousing Institute. ii TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

5 TABLE OF CONTENTS Module 1 Dimensional Modeling Concepts Module 2 Requirements Gathering for Dimensional Modeling Module 3 Logical Dimensional Modeling Module 4 From Logical Model to Star Schema Module 5 Dimensional Data and Business Analysis Appendix A Bibliography and References..... A-1 Exercises Exercise Instructions and Worksheets W-1 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. iii

6 COURSE OBJECTIVES To learn: Concepts of dimensional data modeling The relationship between business metrics and dimensional data Similarities and differences between relational and dimensional data models Requirements gathering techniques for business metrics and dimensional data How to build a logical dimensional model Understand different types of dimensions and where to use each How to translate a logical dimensional model to a star schema design Testing and validating the dimensional model How dimensional data is used to deliver business analytics and OLAP capabilities iv TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

7 Dimensional Modeling Concepts Module 1 Dimensional Modeling Concepts Topic Page Dimensional Modeling in Context 1-2 Dimensional Modeling Basics 1-10 Comparing E-R and Dimensional Models 1-18 Concepts Summary 1-32 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 1-1

8 Dimensional Modeling Concepts TDWI Dimensional Data Modeling Primer Dimensional Modeling in Context Business Intelligence Defined Business Intelligence Defined The goal is improved decision making. Concepts and methodologies not just technology. Facts and fact-based systems are necessary to implement. 1-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

9 Dimensional Modeling Concepts Dimensional Modeling in Context Business Intelligence Defined BUSINESS INTELLIGENCE Howard Dresner (Gartner Group) first defined business intelligence as shown on the facing page. As BI became a mainstream term, additional definitions have emerged. Among them is the following definition: BI is neither a product nor a system. It is an architecture and a collection of integrated operational as well as decision-support applications and databases that provide the business community easy access to business data. (Larissa T. Moss and Shaku Arte, Business Intelligence Roadmap, Pearson Education, 2003) David Loshin defines business intelligence as: The processes, technologies, and tools needed to turn data into information, information into knowledge, and knowledge into plans that drive profitable business actions. Business intelligence encompasses data warehousing, business analytic tools, and content knowledge management. (Business Intelligence: The Savvy Manager s Guide, Addison Wesley, 2003) Business intelligence provides the ability to transform data into usable, actionable information for business purposes. BI requires: Collections of quality data and metadata important to the business The application of analytic tools, techniques, and processes The knowledge and skills to use business analysis to identify/create business information The organizational skills and motivation to develop a BI program and apply the results back into the business The foundation that enables BI is the enterprise architecture business, data, and technical. A well-implemented data warehousing program provides much of that foundation. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 1-3

10 Dimensional Modeling Concepts TDWI Dimensional Data Modeling Primer Dimensional Modeling Basics Dimensional Model Defined 1-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

11 Dimensional Modeling Concepts Dimensional Modeling Basics Dimensional Modeling Defined A BUSINESS MODEL OF PROCESS AND ACTIVITY Chris Adamson s definition of a dimensional model makes it clear that this is a process of modeling much more than data. While data is important as the means to collect and store measures, the modeling process is business oriented because the data simply represents the quantitative properties of business processes and activities. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 1-9

12 Dimensional Modeling Concepts TDWI Dimensional Data Modeling Primer Comparing E-R and Dimensional Models A Quick Review of Entity-Relationship Modeling 1-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

13 Dimensional Modeling Concepts Comparing E-R and Dimensional Models A Quick Review of Entity-Relationship Modeling ENTITY- RELATIONSHIP MODELS This diagram illustrates a simple entity-relationship (ER) model. The primary components of an ER model are: Entity An entity is a subject about which the business has the need, will, and means to collect data a person, place, thing, concept, or event that is of business interest. Entities are represented as labeled boxes in the ER model. Examples in the diagram include EMPLOYEE, JOB, and DEPARTMENT. Attribute An attribute is a property or characteristic of an entity that can be collected as data. Attributes are listed inside the box of the entities that they describe. Job_shift_code and job_title, for example, are attributes of the entity JOB. Relationship A relationship is an important association between pairs of entities that is of business interest and may be collected as data. Examples of relationships in this diagram include EMPLOYEE PERFORMS JOB and DEPARTMENT OWNS JOB. Other important ER concepts include: Cardinality, which describes the number of occurrences of each entity type that may participate in an occurrence of a relationship. Cardinality options include zero or one, exactly one, one or more, and zero or more. In this diagram some of the relationship cardinalities are: a PERSONNEL ACTION changes exactly one JOB; a JOB is changed by one or more PERSONNEL ACTIONs; a JOB is held by zero or one EMPLOYEEs. Sometimes cardinality as defined above is divided with cardinality indicating the maximum number and optionality indicating the minimum number. Specialization (also called subtyping) creates a hierarchy or parent/child relationship between an ENTITY super-type and its subtypes. Specialization makes sense when the entity sub-types have unique attributes or participate in relationships not common to all subtypes. In this diagram, PERSONNEL ACTION is an entity super-type with three sub-types. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 1-19

14 Dimensional Modeling Concepts TDWI Dimensional Data Modeling Primer 1-34 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

15 Requirements Gathering for Dimensional Modeling Module 2 Requirements Gathering for Dimensional Modeling Topic Page Business Context for Data Modeling 2-2 Business Questions as Requirements Models 2-8 Fact/Qualifier Analysis 2-14 Requirements Gathering Summary 2-24 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 2-1

16 Requirements Gathering for Dimensional Modeling TDWI Dimensional Data Modeling Primer Business Context for Data Modeling Business Value 2-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

17 Requirements Gathering for Dimensional Modeling Business Context for Data Modeling Business Value IMPORTANCE OF BUSINESS CONTEXT BUSINESS VALUE PERSPECTIVES Business context is essential to delivering business value. The context determines the nature of BI program the people who will use the information, the kinds of information they need, the business processes to be affected, and the information services to be provided. Business context provides the means to align business intelligence results with business goals. As previously described, dimensional modeling is really a business modeling process modeling of business processes and activities. The big picture view of modeling context begins by understanding how we create business value. The facing page illustrates three perspectives of value creation: Business management views value as positive results of business activities that align with the tactics, strategies, goals, and drivers of the business. Data management views value as business outcomes that are enabled through useful data that enhances knowledge to inform business decisions and actions. Performance management views value as successful execution of continuously evolving strategy and planning where data and information are critical elements of a feedback system for continuous growth and improvement. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 2-3

18 Requirements Gathering for Dimensional Modeling TDWI Dimensional Data Modeling Primer Business Questions as Requirements Models A Framework for Business Questions business questions (BPM / Customer) business questions (CRM / Product) Activity Process Organization Enterprise Customer Management Disciplines CRM BPM SCM BAM etc. Subject Areas Product Supplier Workforce Cost Value etc. Duration etc. business questions (SCM / Supplier) business questions (BAM / Workforce) 2-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

19 Requirements Gathering for Dimensional Modeling Business Questions as Requirements Models A Framework for Business Questions LIST OF BUSINESS QUESTIONS A DIMENSIONAL VIEW OF METRICS INSIDE THE FRAMEWORK METRIC CLASS WORDS Requirements modeling begins with business questions the foundation for all subsequent dimensional modeling activities. The first context-level model is a robust and representative list of business questions that are within the scope of the project. It is not possible to develop a complete and exhaustive list of all questions that might be asked. The objective is to create a list of questions that is fertile enough to drive development of a highly adaptable data structure capable of answering questions that are not yet known. A major strength of the dimensional model is its ability to provide such a data structure. The diagram on the facing page illustrates a three-dimensional view that is useful in managing a comprehensive set of business metrics. The dimensions are: Levels of metrics enterprise, organization, process, and activity that were previously discussed. Subject areas of your business, typically found in a subject area model and the basis for the subject-oriented data warehouse. Subjects vary depending on the industry and the things that make your business unique. They include such things as customer, supplier, product, service, finance, etc. Performance management methods used by your business. These include disciplines such as business performance management (BPM), customer relationship management (CRM), supply chain management (SCM), etc. Each intersection of the three dimensions is an opportunity to find business questions and to manage business metrics. At the intersection of enterprise, product, and CRM, you might ask: What needs to be known about products at the executive level to make CRM effective? What are the KPIs for products that are important to CRM? How are product measures for CRM related to those for BPM? How are product and customer measures for CRM related? Similar to the class words of common data elements (date, code, etc.) metrics have class words (e.g., cost, value, duration, count) that describe the kind of quantity that they express. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 2-9

20 Requirements Gathering for Dimensional Modeling TDWI Dimensional Data Modeling Primer Fact/Qualifier Analysis From Business Requirements to Data Requirements Mapping of Business Questions FACTS: Discrete items of business information. QUALIFIERS: Criteria used to access, sort, group, summarize, and present information. associations among facts & qualifiers 2-14 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

21 Requirements Gathering for Dimensional Modeling Fact/Qualifier Analysis From Business Requirements to Data Requirements STRUCTURING REQUIREMENTS FOR DATA Business questions are the basis of requirements analysis, but they are not the final result. Full understanding of requirements is achieved through analysis of those questions. Requirements are much more than simply knowing which data is needed. To provide a foundation for logical design, you ll also need to know how it will be accessed, navigated, and used. Business questions begin the transition to more formal data models through a matrix modeling technique known as fact/qualifier analysis. Each business question is examined to determine the fact(s) to be contained in the answer and the qualifiers that give the fact(s) meaning. The terms fact and qualifier describe roles that a data item assumes in a specific instance of business usage. Facts are discrete items of business information. Qualifiers are criteria used to access, sort, group, summarize, and present information. A single data item may be a fact in one instance and a qualifier in another. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 2-15

22 Requirements Gathering for Dimensional Modeling TDWI Dimensional Data Modeling Primer 2-26 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

23 Logical Dimensional Modeling Module 3 Logical Dimensional Modeling Topic Page Modeling Meters and Measures 3-2 Modeling Dimensions 3-4 More about Meters and Measures 3-12 Model Verification 3-18 Logical Modeling Summary 3-20 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 3-1

24 Logical Dimensional Modeling TDWI Dimensional Data Modeling Primer Modeling Meters and Measures A Group of Related Business Measures 3-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

25 Logical Dimensional Modeling Modeling Meters and Measures A Group of Related Business Measures FINDING METERS The first step of developing a logical dimensional model is identifying meters and measures. A meter is a group of related measures that can collectively be given a business name. The set of facts in the fact/qualifier matrix are the source of measures. Group those facts that are measures of the same business performance concept EMPLOYEE SATISFACTION, CUSTOMER LOYALTY, PRODUCT PROFITABILITY, WORKFORCE PRODUCTIVITY, etc. Each business performance concept becomes a meter and the set of related facts the measures contained by the meter. The example domain has been limited to EMPLOYEE SATISFACTION; thus, all of the facts are related by that domain. Sometimes business questions are listed and fact/qualifier analysis performed for a broader or more complex domain. You may, for example, have a set of questions that yield facts about both CUSTOMER VALUE and PRODUCT PROFITABILITY a combination of facts that aren t readily combined in a single meter. A further examination of questions 6 and 7 reveals that these are not measures of EMPLOYEE SATISFACTION. Question 8 is already addressed by question 3. These are therefore excluded from the scope of this model and would be addressed by a separate model. This relatively simple example yields one EMPLOYEE SATISFACTION meter containing three measures: job_turnover_rate, avg_length_of_employment, and number_of_complaints. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 3-3

26 Logical Dimensional Modeling TDWI Dimensional Data Modeling Primer Modeling Dimensions Adding Dimensions from the Qualifiers 3-4 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

27 Logical Dimensional Modeling Modeling Dimensions Adding Dimensions from the Qualifiers DIMENSIONS IN THE LOGICAL MODEL Once dimension hierarchies are known, the logical dimensional model is extended by adding dimensions and associating them with the meter. For every measure contained in the meter, all associated qualifiers must be represented by a dimension. To add dimensions to the model: 1. Include each dimension hierarchy previously identified. Associate only the lowest level of each hierarchy with the meter as a one-tomany relationship. 2. Examine the remaining qualifiers to find those that may be dimension attributes instead of dimension levels. In this example, employee gender and employee age are attributes that describe employee. Thus employee becomes the dimension level, with age and gender modeled as attributes. The dimension level employee is associated with the meter as a one-to-many relationship. 3. All remaining qualifiers are mapped as flat (single-level, nonhierarchical) dimensions, and each is associated with the meter using a one-to-many relationship. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 3-5

28 Logical Dimensional Modeling TDWI Dimensional Data Modeling Primer More about Meters and Measures Granularity and the Meter 3-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

29 Logical Dimensional Modeling More about Meters and Measures Granularity and the Meter BUSINESS METRICS LEVEL OF DETAIL Granularity describes the level of detail contained in the meter. Every measure in a meter must be at the same level of detail. Declaring the grain of the meter is an important verification step and yields an essential piece of information for the next steps of dimensional modeling. The grain is determined by the set of dimensions associated with the meter. By reading the cardinality of meter-to-dimension relationships, it is clear that every instance of EMPLOYEE SATISFACTION contains measures for a unique combination of: One month One employee One job One location, and One trade TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 3-13

30 Logical Dimensional Modeling TDWI Dimensional Data Modeling Primer Model Verification Testing the Model 3-18 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

31 Logical Dimensional Modeling Model Verification Testing the Model ADEQUACY TESTING CONSISTENCY TESTING At a minimum, the dimensional model structure must provide answers to the questions that were posed by the business representatives. One of the tests, therefore, is to refer to the original set of questions and verify that the structure supports answering each of the questions. If conformed dimensions exist, the model dimensions should be compared against those dimensions. While the best practice is to adopt the conformed dimension, this is not always done. Even if the conformed dimensions are not used directly, there should be no violations of the definitions and structures prescribed by those dimensions. If conformed dimensions do not exist, then dimensions should be compared to similar ones that are used in existing data marts. Any inconsistencies in the definitions and structures will likely lead to differing conclusions. EXPANSION TESTING As stated earlier, not all the business questions can be known in advance. A major strength of the dimensional model is its ability to answer some questions that have not been predefined. In testing the model, review the dimension attributes and compare them to attributes available in the business data model (if it exists) and in the operational systems. If any are missing, determine whether they should be added. If the definitions differ, be sure to resolve the differences. Similarly, there may be additional measures related to the model meter that could be useful to the business. Explore these with the business representatives and augment the model accordingly. Often period-toperiod comparisons are made, and inclusion of the previous period (e.g., same month last year) data as a distinct data element enables retrieval of the data for comparison from a single row. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 3-19

32 Logical Dimensional Modeling TDWI Dimensional Data Modeling Primer 3-22 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

33 From Logical Model to Star Schema Module 4 From Logical Model to Star Schema Topic Page Star Schema Dimensions 4-2 Star Schema Fact Tables 4-8 Star Schema Design Challenges 4-16 Modeling Process Summary 4-30 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 4-1

34 From Logical Model to Star Schema TDWI Dimensional Data Modeling Primer Star Schema Dimensions Naming the Dimensions 4-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

35 From Logical Model to Star Schema Star Schema Dimensions Naming the Dimensions FROM LOGICAL TO PHYSICAL Logical dimensional models are typically implemented in either of two forms: Multi-dimensional databases, also called cubes, are used with multi-dimensional OLAP (MOLAP) tools. For these implementations, the MOLAP tool includes a cube generation capability. The logical dimensional model captures the information needed to describe a desired cube to the tool. Star schema databases are used with relational OLAP (ROLAP) tools. For these implementations, physical modeling activities design the star schema. NAMED DIMENSIONS With either implementation, each dimension needs to be named. Dimension names are key navigation components for business analysts and others using OLAP tools; thus, it is important that the names have business meaning. In flat (single-level, non-hierarchical) dimensions, it is common practice for the dimension to have the same name as the dimension level. For multi-level dimensions, a collective name that encompasses all dimension levels is established. In the EMPLOYEE SATISFACTION example: Employee is a flat dimension and retains the name employee. Location is a flat dimension that retains the name location. Labor union contains trade, and is a multi-level dimension with the collective name labor organization. Year contains month, and is a multi-level dimension collectively named month. Department contains job, and is a hierarchical dimension named employment organization. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 4-3

36 From Logical Model to Star Schema TDWI Dimensional Data Modeling Primer Star Schema Fact Tables Modeling the Fact Table 4-8 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

37 From Logical Model to Star Schema Star Schema Fact Tables Modeling the Fact Table PHYSICAL MODELING CONVENTIONS The fact table in a star schema begins as a translation of the meter from the logical model. Little or no structural change occurs. The most significant changes here are in modeling standards and conventions primarily the naming standards that are applied. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 4-9

38 From Logical Model to Star Schema TDWI Dimensional Data Modeling Primer Star Schema Design Challenges Slowly Changing Dimensions 4-16 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

39 From Logical Model to Star Schema Star Schema Design Challenges Slowly Changing Dimensions DESIGNING FOR DIMENSION VOLATILITY One of the challenges of OLAP analysis is analyzing data over time when the dimensions don t remain constant across the time period of interest. Changing data values are common in dimensional data a problem known as slowly changing dimensions. The terminology doesn t mean to imply that some dimensions change more quickly than others; rather, that dimension values change less frequently than the values of measures in a fact table. When the data values in a dimension change, three options are possible to implement the dimension. Using Ralph Kimball s widely accepted terminology, these options are: Type 1 dimensions that overwrite dimension rows without retaining a history of dimension changes. Type 2 dimensions that preserve dimension history by inserting new rows into the dimension table. Type 3 dimensions that keep versions of dimension values by creating new columns for changed data. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 4-17

40 From Logical Model to Star Schema TDWI Dimensional Data Modeling Primer 4-32 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

41 Dimensional Data and Business Analytics Module 5 Dimensional Data and Business Analytics Topic Page Delivering Business Value 5-2 Effective Dimensional Modeling 5-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 5-1

42 Dimensional Data and Business Analytics TDWI Dimensional Data Modeling Primer Delivering Business Value Data Enabled Business Analysis 5-2 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

43 Dimensional Data and Business Analytics Delivering Business Value Data Enabled Business Analysis MANY DATA MARTS A typical business intelligence environment encompasses many different data marts at different levels of detail, with different focus and purpose, yet with substantial overlap of dimensions and some overlap of measures. The example that we ve followed throughout the course employee satisfaction is but a single data model in a more complex environment. The employee satisfaction model is a summary, and it is likely that a data mart exists to contain more detailed, atomic level data. It is also likely that many other data marts exist with similar measures and similar dimensions. Consistency and conformity across data marts is essential to support cross-functional business analysis. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 5-3

44 Dimensional Data and Business Analytics TDWI Dimensional Data Modeling Primer Effective Dimensional Modeling Critical Success Factors 5-6 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.

45 Dimensional Data and Business Analytics Effective Dimensional Modeling Critical Success Factors KEYS TO SUCCESS The facing page lists several best practices and success factors for highimpact dimensional modeling. TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY. 5-7

46 Dimensional Data and Business Analytics TDWI Dimensional Data Modeling Primer 5-12 TDWI. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. DO NOT COPY.