Kent Graziano

Transcription

1 Agile Data Warehouse Modeling: Introduction to Data Vault Modeling Kent Graziano

2 Agenda Bio What is a Data Vault? Where does it fit in an DW/BI architecture? How to design a Data Vault model Being agile with DV

3 My Bio Senior Technical Evangelist, Snowflake Computing Oracle ACE Director (DW/BI) Certified Data Vault Master and DV 2.0 Practitioner Former Member: Boulder BI Brain Trust Data Architecture and Data Warehouse Specialist 30+ years in IT 25+ years of Oracle-related work 20+ years of data warehousing experience Co-Author of The Business of Data Vault Modeling The Data Model Resource Book (1st Edition) Blogger Past-President of ODTUG and Rocky Mountain Oracle User Group

4 Data Vault Definition The Data Vault is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3 rd normal form (3NF) and star schema. The design is flexible, scalable, consistent and adaptable to the needs of the enterprise. Architected specifically to meet the needs of today s enterprise data warehouses Dan Linstedt: Defining the Data Vault TDAN.com Article

5 What is Data Vault Trying to Solve? What are our other Enterprise Data Warehouse options? Third-Normal Form (3NF): Complex primary keys (PK s) with cascading snapshot dates Star Schema (Dimensional): Difficult to reengineer fact tables for granularity changes Difficult to get it right the first time Not adaptable to rapid business change NOT AGILE! (C) Kent Graziano

6 Data Vault Evolution The work on the Data Vault approach began in the early 1990s, and completed around In 2002, the industry thought leaders were asked to review the architecture. This is when I attend my first DV seminar in Denver and met Dan! In 2003, Dan began teaching the modeling techniques to the mass public. In 2014, Dan introduced DV 2.0! (C) Kent Graziano

7 Where does a Data Vault Fit? LearnDataVault.com

8 Where does Data Vault fit? Data Vault goes here

9 Data Vault: 3 Simple Structures LearnDataVault.com

10 Data Vault Core Architecture Hubs = Unique List of Business Keys Links = Unique List of Relationships across keys Satellites = Descriptive Data Satellites have one and only one parent table Satellites cannot be Parents to other tables Hubs cannot be child tables All have some standard required columns LearnDataVault.com

11 1. Hub = Business Keys Hubs = Unique Lists of Business Keys Business Keys are used to TRACK and IDENTIFY key information New: DV 2.0 uses MD5 Hash of the BK for the PK (C) Kent Graziano

12 2: Links = Associations Links = Transactions and Associations They are used to hook together multiple sets of information In DV 2.0 the BK attributes may migrate to the Links for faster query (C) Kent Graziano

13 3. Satellites = Descriptors Satellites provide context for the Hubs and the Links Tracks changes over time - Like SCD 2 In DV 2.0 use HASH_DIFF to detect changes (C) Kent Graziano

14 Data Vault Model Flexibility (Agility) Goes beyond standard 3NF Highly normalized Hubs and Links only hold keys and meta data Satellites split by rate of change and/or source Enables Agile data modeling Easy to add to model without having to change existing structures and load routines Relationships (links) can be dropped and created on-demand. No more reloading history because of a missed requirement Based on natural business keys Not system surrogate keys Allows for integrating data across functions and source systems more easily All data relationships are key driven. (C) Kent Graziano

15 Data Vault Extensibility Adding new components to the EDW has NEAR ZERO impact to: Existing Loading Processes Existing Data Model Existing Reporting & BI Functions Existing Source Systems Existing Star Schemas and Data Marts (C) LearnDataVault.com

16 Data Vault Productivity Standardized modeling rules Highly repeatable and learnable modeling technique Can standardize load routines Delta Driven process Re-startable, consistent loading patterns. Can standardize extract routines Rapid build of new or revised Data Marts Can be automated Can use a BI-meta layer to virtualize the reporting structures Example: OBIEE Business Model and Mapping tool Example: BOBJ Universe Business Layer Can put views on the DV structures as well Simulate ODS/3NF or Star Schemas (C) Kent Graziano

17 Data Vault Adaptability The Data Vault holds granular historical relationships. Holds all history for all time, allowing any source system feeds to be reconstructed ondemand Easy generation of Audit Trails for data lineage and compliance. Data Mining can discover new relationships between elements Patterns of change emerge from the historical pictures and linkages. (C) Kent Graziano

18 How to be Agile using DV Model iteratively Use Data Vault data modeling technique Create basic components, then add over time Virtualize the Access Layer Don t waste time building facts and dimensions up front ETL and testing takes too long Project objects using pattern-based DV model with database views (or BI meta layer) Users see real reports with real data Can always build out for performance in another iteration (C) Kent Graziano

19 Worlds Smallest Data Vault Hub Customer Hub_Cust_Seq_ID Hub_Cust_Num Hub_Cust_Load_DTS Hub_Cust_Rec_Src Satellite Customer Name Hub_Cust_Seq_ID Sat_Cust_Load_DTS Sat_Cust_Load_End_DTS Sat_Cust_Name Sat_Cust_Rec_Src The Data Vault doesn t have to be BIG. An Data Vault can be built incrementally. Reverse engineering one component of the existing models is not uncommon. Building one part of the Data Vault, then changing the marts to feed from that vault is a best practice. The smallest Enterprise Data Warehouse consists of two tables: One Hub, One Satellite LearnDataVault.com

20 Notably In 2008 Bill Inmon stated that the Data Vault is the optimal approach for modeling the EDW in the DW2.0 framework. (DW2.0) The number of Data Vault users in the US surpassed 500 in 2010 and grows rapidly (

21 Organizations using Data Vault WebMD Health Services Anthem Blue-Cross Blue Shield MD Anderson Cancer Center Denver Public Schools Independent Purchasing Cooperative (IPC, Miami) Owner of Subway Kaplan US Defense Department Colorado Springs Utilities State Court of Wyoming Federal Express US Dept. Of Agriculture

22 Summary Data Vault provides a data modeling technique that allows: Model Agility Enabling rapid changes and additions Productivity Enabling low complexity systems with high value output at a rapid pace Easy projections of dimensional models So? Agile Data Warehousing? (C) Kent Graziano

23 SHAMELESS PLUG: Available on Amazon.com Better-Data-Modeling- Introduction- Engineeringebook/dp/B018BREV1C/

24 Super Charge Your Data Warehouse Available on Amazon.com Soft Cover or Kindle Format Now also available in PDF at LearnDataVault.com Hint: Kent is the Technical Editor

25 New DV 2.0 Book Is Here! Available on Amazon: m/building-scalable- Data-Warehouse- Vault/dp/ /

26 Other Data Vault References On YouTube: On Facebook:

27 Contact Information Kent Graziano Snowflake Computing On Visit my blog at