Decision Guidance. Data Vault in Data Warehousing

Transcription

1 Decision Guidance Data Vault in Data Warehousing

2 DATA VAULT IN DATA WAREHOUSING Today s business environment requires data models, which are resilient to change and enable the integration of multiple data sources. More and more organizations consider implementing Data Vault in their new data warehouses as a part of the modernization of their BI systems. There are good reasons for this modeling method, but there are arguments against its usage, as well. This short overview should help to decide, whether Data Vault Modeling is an appropriate approach or not for your specific data warehouse project. WHAT IS DATA VAULT MODELING? Data Vault Modeling is a database modeling method, especially designed for data warehouses with a high number of structure changes. The basic concept of Data Vault is to split information in a way that allows easy integration and historization of the data. Additionally, the model can be enhanced without migration of the existing tables. With these three types of tables Hubs, Links and Satellites comprehensive and extensible data models can be built. Data Vault Modeling is typically used for modeling the Core layer of a data warehouse or to build an Enterprise Data Warehouse (EDW) with many different source systems. BI users do not access the Data Vault tables directly, but run their queries and reports on dimensional data marts that are loaded from the Data Vault layer.

3 Data Warehouse Source Systems Staging Area Data Vault Marts BI Platform Raw Vault Business Vault Metadata USE CASES OF DATA VAULT n Agile DWH projects. Agile software projects usually contain short development cycles with fast changing requirements and frequent data model extensions. n Data Warehouses with multiple source systems. Reporting on information from different source systems is only possible if the data was integrated before. n Large DWH projects. Data Vault is especially suitable for Enterprise Data Warehouses or DWH systems with high complexity and data from different departments.

4 CHANCES n Integration of data from different source systems. The source data is integrated using common business keys, stored in Hubs. The required business attributes are stored in separate Satellites per source system. This makes it easier to combine the information for further reports. n Parallel loading of data from different source systems. There is no pre-defined load order, data can be loaded into Data Vault independently of each other. n Complete historization of all attributes. Versioning of all attributes in the Satellites allows the traceability of all changes in the past and the extraction of the data at a specific point in time. n Easy extensibility of data model. Additional entities or attributes that are used for new requirements are implemented as additional tables in Data Vault. Existing tables are usually not changed. This helps to avoid data migration. RISKS n Large number of tables. Many model extensions can create data models with a high number of tables (Hubs, Links und Satellites). Accordingly, the number of ETL processes increases, too. n Complex extraction from Data Vault. While loading Data Vault tables is very simple, extracting the information to load data marts can be more extensive. For good performance, auxiliary tables may be required. n Knowledge of Data Vault required. The basic principles of Data Vault must be known to the entire project team. The developers must understand the ETL patterns for the different objects. n Business knowledge required. To be able to successfully create models with Data Vault, it is important to understand the business contexts. Otherwise, the risk is high in Data Vault that only the source data will be copied and historized.

5 n Simple and uniform ETL patterns. Loading of Hubs, Links and Satellites takes place according uniform rules, which are always constructed in the same way. n Appropriate business keys required. The determination of appropriate business keys is one of the biggest challenges in Data Vault. Unsuitable keys complicate integration of different sources and increase the complexity of loading data marts. RECOMMENDATIONS n Data Vault training for project team. The basic principles of Data Vault are simple, but should be understood by all involved developers and other project members. This includes at least an overview of Data Vault for the whole team, eventually also a certification of individual developers (see course Certified Data Vault Data Modeler ). n Naming conventions and design patterns. The consistent usage of uniform rules simplifies development and maintenance of the DWH system. Because of the high number of tables and ETL processes in Data Vault, exceptions and special cases quickly lead to a complex and hard-to-maintain system after a short time. n Usage of Data Warehouse generators. The high number of tables and ETL processes in Data Vault makes it worthwhile to use DWH automation tools such as bigenius. This not only reduces the development effort, but also ensures consistent usage of the conventions and design patterns within the project.

6 ADDITIONAL RESOUCES Whitepaper - Comparison of Data Modeling Methods for a Core Data Warehouse Blog of Dan Linstedt The Hans Blog (Hans Hultgren) Blog Data Vault Modeling (Dirk Lerner) Dani Schnider s Blog on Data Vault Modeling

7 Trivadis is an independent and leading IT Consultancy and Services Company in Germany, Switzerland, Austria and Denmark that prioritizes the consulting skills of its staff and equips them with the methods, tools and products they need to master the challenges presented by their projects with the maximum effectiveness and efficiency. These tools and products are evolved from and developed for practical use appropriate for their applications and simple to operate. This is our watchword in developing products for our clients. Trivadis AG Sägereistrasse 29 CH-8152 Glattbrugg