Dimensional Fact Model

Transcription

1 Dimensional Fact Model Stuttgart, 26/11/2014 Stefano stefano.cazzella{at}gmail.com BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 1

2 Complexity in SE and IS development The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible. Edsger Dijkstra, Notes on Structured Programming BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 2

3 Project Layers Business User requirements Conceptual model Design Technical choices Logical model Build Tecnology Physical model BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 3

4 Civil Engineering Example Business Design Build What the client wants The technical blueprint The desired building BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 4

5 Model-driven engineering Model transformation Business centric No tecnical details PIM PSM Tecnical design System architecture Tecnical deliverables System realization Build Model transformation BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 5

6 Project Layers for Data Mart Business DFM Dimensional Fact Model Design Relational model Build DBMS specific DDL BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 6

7 Why Dimensional Fact Model? 1 Formal language à well-specified syntax and an unequivocally interpretation (semantic) based on a sound algebraic definition 2 Simple and effective graphical notation (representation) 3 Specifically defined to represent multi-dimensional models 4 Does not imply any technical/implementation choice BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 7

8 DFM Notation Compendium BI ACADEMY - Stuttgart, 26/11/ Stefano Cazzella 8

9 Data Mart building process Business user s needs Technical specifications Data Mart + = Requirements definition Implementation strategy Deployment Model transformation Model transformation Multidimensional data model (Dimensional Fact Model) Logical data model (Relational model: tables, columns, etc.) Phisical data model (DDL with indexes, partions, etc.) BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 9

10 Data Mart building process Business user s needs Technical specifications Data Mart + = Requirements definition Formalize user s needs in a Implementation conceptual (business-centric) model, then strategy Deployment Model transformation Model transformation transform it in a logical model integrating technical specification Multidimensional data model (Dimensional Fact Model) and transform it again in a physical model that realizes the business requirements Logical data model (Relational model: tables, columns, etc.) Phisical data model (DDL with indexes, partions, etc.) BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 10

11 Business - From requisite to DFM Context: weblog analytics - the analysis of the visits of several web sites belonging to different domains (eg. Google Analytics) Requisite: monitoring and analyzing the number of visits and their monthly and daily average duration for each page of the websites, or each domain, distributed by the geographic region of the IP of the visitors. + þ Domain definition þ Aggregation rules þ Optional dependencies BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 11

12 Design choice Reference ROLAP model: Star-schema (denormalized dimension table) Snow-flake (hierarchies implemented by tables in 3NF) Hierarchy implementation strategy (for every dimension) Use natural key (the dimension attribute à PK column) Use surrogate key (add a new column with no business meaning) Use slow-changing dimension (SCD) of type 2 Use implicit dimension (no dimension table, only a column in the fact table) Domain ß à Data type association Text à VARCHAR(250) ; Currency à NUMBER(9,2) ; etc. Standard naming conventions and abbreviations Table name prefix (D for Dimensions, F for Facts) ; Number à NBR ; etc. BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 12

13 Transform DFM in a Relational Model Surrogate key SCD-2 Start date End date Model transformation Technical design choices: Reference ROLAP model à star-schema Hierarchy Viewerà use surrogate key Hierarchy Page à SCD Type 2 Fact grain BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 13

14 Build choice Choice the DBMS SqlServer Oracle Hive / Hadoop Generate constraints? Generate unique keys / primary keys / integrity constraints (foreign keys) Add specific indexes Add clustered indexes / column-store indexes / bitmap indexes / etc. Define table partitions Organize fact tables in partitions (by hash, value, range, etc.) Distribute data over multiple volumes Define file groups / tablespaces for tables, partitions, indexes BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 14

15 Phisical model and DDL (1) Implementation choices & best practice: DBMS à SQL Server Fact F_VISITS partitioned by year Column-store index on day and duration 2 distinct file groups for tables and indexes Partition scheme and functions File groups Columnstore index BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 15

16 Phisical model and DDL (2) Implementation choices & best practice: DBMS à Oracle Fact F_VISITS partitioned by year Bitmap index on viewer dimension 2 distinct table spaces for tables and indexes Table spaces Table partitions Bitmap index BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 16

17 BI Modeler In order to apply a model-driven approach, BI Project teams need a software tool to: þ Manage (draw) all the models - DFM, relational, etc. þ Support (and drive) the model transformation process There was (are) no many tools able to do that so, in 2006 I started working on the development of BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 17

18 DEMO Create a DFM about SALES from scratch Transform a DFM in a relational data model Add physical properties to the relational model Define the fact schema and its measures Add some dimensions / hierarchies Define and associate domains to attributes and measures Define an implementation strategy for Hierarchies Associate Data type to domains Apply a naming convention Choose a DBMS Create partitions Create indexes Generate DDL BI ACADEMY Launch@Germany - Stuttgart, 26/11/ Stefano Cazzella 18